Time-distanced gates in long short-term memory networks

• Published in: Medical image analysis 2020-10, Vol.65, p.101785-101785, Article 101785
• Main Authors: Gao, Riqiang, Tang, Yucheng, Xu, Kaiwen, Huo, Yuankai, Bao, Shunxing, Antic, Sanja L., Epstein, Emily S., Deppen, Steve, Paulson, Alexis B., Sandler, Kim L., Massion, Pierre P., Landman, Bennett A.
• Format: Article
• Language: English
• Subjects: Algorithms; Benchmarks; Computed tomography; Data acquisition; Distanced LSTM; Gates; Humans; Irregular sampling; Long short-term memory; Longitudinal; Lung cancer; Lung cancer diagnosis; Lung nodules; Malignancy; Medical imaging; Memory, Short-Term; Natural Language Processing; Neural Networks, Computer; Nodules; Patients; Protocol (computers); Sampling; Temporal Emphasis Model; Tomography, X-Ray Computed
• ISSN: 1361-8415; 1361-8423
• DOI: 10.1016/j.media.2020.101785

•Explainable Time-Distanced Gates in LSTM to build Distanced LSTM.•The time distance of every scan to the last scan is modeled explicitly.•The newer scan in serial data is emphasized by our model.•Simulated data and 3 CT datasets including cross-and external-validation.•The document, dataset and the code of simulation are publicly available. The Long Short-Term Memory (LSTM) network is widely used in modeling sequential observations in fields ranging from natural language processing to medical imaging. The LSTM has shown promise for interpreting computed tomography (CT) in lung screening protocols. Yet, traditional image-based LSTM models ignore interval differences, while recently proposed interval-modeled LSTM variants are limited in their ability to interpret temporal proximity. Meanwhile, clinical imaging acquisition may be irregularly sampled, and such sampling patterns may be commingled with clinical usages. In this paper, we propose the Distanced LSTM (DLSTM) by introducing time-distanced (i.e., time distance to the last scan) gates with a temporal emphasis model (TEM) targeting at lung cancer diagnosis (i.e., evaluating the malignancy of pulmonary nodules). Briefly, (1) the time distance of every scan to the last scan is modeled explicitly, (2) time-distanced input and forget gates in DLSTM are introduced across regular and irregular sampling sequences, and (3) the newer scan in serial data is emphasized by the TEM. The DLSTM algorithm is evaluated with both simulated data and real CT images (from 1794 National Lung Screening Trial (NLST) patients with longitudinal scans and 1420 clinical studied patients). Experimental results on simulated data indicate the DLSTM can capture families of temporal relationships that cannot be detected with traditional LSTM. Cross-validation on empirical CT datasets demonstrates that DLSTM achieves leading performance on both regularly and irregularly sampled data (e.g., improving LSTM from 0.6785 to 0.7085 on F1 score in NLST). In external-validation on irregularly acquired data, the benchmarks achieved 0.8350 (CNN feature) and 0.8380 (with LSTM) on AUC score, while the proposed DLSTM achieves 0.8905. In conclusion, the DLSTM approach is shown to be compatible with families of linear, quadratic, exponential, and log-exponential temporal models. The DLSTM can be readily extended with other temporal dependence interactions while hardly increasing overall model complexity. [Display omitted]