Development and Validation of an Artificial Intelligence System to Optimize Clinician Review of Patient Records

• Published in: JAMA network open 2021-07, Vol.4 (7), p.e2117391-e2117391
• Main Authors: Chi, Ethan Andrew, Chi, Gordon, Tsui, Cheuk To, Jiang, Yan, Jarr, Karolin, Kulkarni, Chiraag V, Zhang, Michael, Long, Jin, Ng, Andrew Y, Rajpurkar, Pranav, Sinha, Sidhartha R
• Format: Article
• Language: English
• Subjects: Academic Medical Centers; Accuracy; Adult; Artificial Intelligence; Electronic health records; Female; Gastroenterology; Health Informatics; Humans; Information Storage and Retrieval - methods; Job Satisfaction; Male; Medical Records; Medical referrals; Middle Aged; Online Only; Original Investigation; Physicians; Physicians - psychology; Prospective Studies; Referral and Consultation; Task Performance and Analysis; Time Factors; User-Centered Design; Workload - psychology
• ISSN: 2574-3805; 2574-3805
• DOI: 10.1001/jamanetworkopen.2021.17391

Physicians are required to work with rapidly growing amounts of medical data. Approximately 62% of time per patient is devoted to reviewing electronic health records (EHRs), with clinical data review being the most time-consuming portion. To determine whether an artificial intelligence (AI) system developed to organize and display new patient referral records would improve a clinician's ability to extract patient information compared with the current standard of care. In this prognostic study, an AI system was created to organize patient records and improve data retrieval. To evaluate the system on time and accuracy, a nonblinded, prospective study was conducted at a single academic medical center. Recruitment emails were sent to all physicians in the gastroenterology division, and 12 clinicians agreed to participate. Each of the clinicians participating in the study received 2 referral records: 1 AI-optimized patient record and 1 standard (non-AI-optimized) patient record. For each record, clinicians were asked 22 questions requiring them to search the assigned record for clinically relevant information. Clinicians reviewed records from June 1 to August 30, 2020. The time required to answer each question, along with accuracy, was measured for both records, with and without AI optimization. Participants were asked to assess overall satisfaction with the AI system, their preferred review method (AI-optimized vs standard), and other topics to assess clinical utility. Twelve gastroenterology physicians/fellows completed the study. Compared with standard (non-AI-optimized) patient record review, the AI system saved first-time physician users 18% of the time used to answer the clinical questions (10.5 [95% CI, 8.5-12.6] vs 12.8 [95% CI, 9.4-16.2] minutes; P = .02). There was no significant decrease in accuracy when physicians retrieved important patient information (83.7% [95% CI, 79.3%-88.2%] with the AI-optimized vs 86.0% [95% CI, 81.8%-90.2%] without the AI-optimized record; P = .81). Survey responses from physicians were generally positive across all questions. Eleven of 12 physicians (92%) preferred the AI-optimized record review to standard review. Despite a learning curve pointed out by respondents, 11 of 12 physicians believed that the technology would save them time to assess new patient records and were interested in using this technology in their clinic. In this prognostic study, an AI system helped physicians extract relevant patient information in