Deep reinforcement learning for automated radiation adaptation in lung cancer

• Published in: Medical physics (Lancaster) 2017-12, Vol.44 (12), p.6690-6705
• Main Authors: Tseng, Huan‐Hsin, Luo, Yi, Cui, Sunan, Chien, Jen‐Tzung, Ten Haken, Randall K., Naqa, Issam El
• Format: Article
• Language: English
• Subjects: adaptive radiotherapy; deep learning; lung cancer; reinforcement learning
• ISSN: 0094-2405; 2473-4209
• DOI: 10.1002/mp.12625

Purpose To investigate deep reinforcement learning (DRL) based on historical treatment plans for developing automated radiation adaptation protocols for nonsmall cell lung cancer (NSCLC) patients that aim to maximize tumor local control at reduced rates of radiation pneumonitis grade 2 (RP2). Methods In a retrospective population of 114 NSCLC patients who received radiotherapy, a three‐component neural networks framework was developed for deep reinforcement learning (DRL) of dose fractionation adaptation. Large‐scale patient characteristics included clinical, genetic, and imaging radiomics features in addition to tumor and lung dosimetric variables. First, a generative adversarial network (GAN) was employed to learn patient population characteristics necessary for DRL training from a relatively limited sample size. Second, a radiotherapy artificial environment (RAE) was reconstructed by a deep neural network (DNN) utilizing both original and synthetic data (by GAN) to estimate the transition probabilities for adaptation of personalized radiotherapy patients’ treatment courses. Third, a deep Q‐network (DQN) was applied to the RAE for choosing the optimal dose in a response‐adapted treatment setting. This multicomponent reinforcement learning approach was benchmarked against real clinical decisions that were applied in an adaptive dose escalation clinical protocol. In which, 34 patients were treated based on avid PET signal in the tumor and constrained by a 17.2% normal tissue complication probability (NTCP) limit for RP2. The uncomplicated cure probability (P+) was used as a baseline reward function in the DRL. Results Taking our adaptive dose escalation protocol as a blueprint for the proposed DRL (GAN + RAE + DQN) architecture, we obtained an automated dose adaptation estimate for use at ∼2/3 of the way into the radiotherapy treatment course. By letting the DQN component freely control the estimated adaptive dose per fraction (ranging from 1–5 Gy), the DRL automatically favored dose escalation/de‐escalation between 1.5 and 3.8 Gy, a range similar to that used in the clinical protocol. The same DQN yielded two patterns of dose escalation for the 34 test patients, but with different reward variants. First, using the baseline P+ reward function, individual adaptive fraction doses of the DQN had similar tendencies to the clinical data with an RMSE = 0.76 Gy; but adaptations suggested by the DQN were generally lower in magnitude (less aggressive). Second, by a