Toward Debugging Deep Reinforcement Learning Programs with RLExplorer

Deep reinforcement learning (DRL) has shown success in diverse domains such as robotics, computer games, and recommendation systems. However, like any other software system, DRL-based software systems are susceptible to faults that pose unique challenges for debugging and diagnosing. These faults of...

Full description

Saved in:
Bibliographic Details
Main Authors: Bouchoucha, Rached, Yahmed, Ahmed Haj, Patil, Darshan, Rajendran, Janarthanan, Nikanjam, Amin, Chandar, Sarath, Khomh, Foutse
Format: Conference Proceeding
Language:English
Subjects:
Debugging
Deep reinforcement learning
DRL Bugs
Fault diagnosis
Manuals
Monitoring
Runtime
Software Engineering
Software maintenance
Software systems
Software tools
Training
Video games
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep reinforcement learning (DRL) has shown success in diverse domains such as robotics, computer games, and recommendation systems. However, like any other software system, DRL-based software systems are susceptible to faults that pose unique challenges for debugging and diagnosing. These faults often result in unexpected behavior without explicit failures and error messages, making debugging difficult and time-consuming. Therefore, automating the monitoring and diagnosis of DRL systems is crucial to alleviate the burden on developers. In this paper, we propose RLExplorer, the first fault diagnosis approach for DRL-based software systems. RLExplorer automatically monitors training traces and runs diagnosis routines based on properties of the DRL learning dynamics to detect the occurrence of DRL-specific faults. It then logs the results of these diagnoses as warnings that cover theoretical concepts, recommended practices, and potential solutions to the identified faults. We conducted two sets of evaluations to assess RLExplorer. Our first evaluation of faulty DRL samples from Stack Overflow revealed that our approach can effectively diagnose real faults in 83% of the cases. Our second evaluation of RLExplorer with 15 DRL experts/developers showed that (1) RLExplorer could identify 3.6 times more defects than manual debugging and (2) RLExplorer is easily integrated into DRL applications.
ISSN:2576-3148
DOI:10.1109/ICSME58944.2024.00019