Training, testing and benchmarking medical AI models using Clinical AIBench

• Published in: BenchCouncil Transactions on Benchmarks, Standards and Evaluations Standards and Evaluations, 2022-03, Vol.2 (1), p.100037, Article 100037
• Main Authors: Huang, Yunyou, Miao, Xiuxia, Zhang, Ruchang, Ma, Li, Liu, Wenjing, Zhang, Fan, Guan, Xianglong, Liang, Xiaoshuang, Lu, Xiangjiang, Tang, Suqing, Zhang, Zhifei
• Format: Article
• Language: English
• Subjects: Alzheimer’s disease; Benchmark; Clinical setting; Configurable clinical setting; COVID-19; Dental
• ISSN: 2772-4859; 2772-4859
• DOI: 10.1016/j.tbench.2022.100037

AI technology has been used in many clinical research fields, but most AI technologies are difficult to land in real-world clinical settings. In most current clinical AI research settings, the diagnosis task is to identify different types of diseases among the given ones. However, the diagnosis in real-world settings needs dynamically developing inspection strategies based on the existing resources of medical institutions and identifying different kinds of diseases out of many possibilities. To promote the development of different clinical AI technologies and the implementation of clinical applications, we propose a benchmark named Clinical AIBench for developing, verifying, and evaluating clinical AI technologies in real-world clinical settings. Specifically, Clinical AIBench can be used for: (1) Model training and testing: Researchers can use the data to train and test their models. (2)Model evaluation: Researchers can use Clinical AIBench to objectively, fairly, and comparably evaluate various models of different researchers. (3) Clinical value evaluation: Researchers can use the clinical indicators provided by Clinical AIBench to evaluate the clinical value of models, which will be applied in real-world clinical settings. For convenience, Clinical AIBench provides three different levels of clinical settings: restricted clinical setting, which is named closed clinical setting, data island clinical setting, and real-world clinical setting, which is called open clinical setting. In addition, Clinical AIBench covers three diseases: Alzheimer’s disease, COVID-19, and dental. Clinical AIBench provides python APIs to researchers. The data and source code are publicly available from the project website https://www.benchcouncil.org/clinical_aibench/. •A highly configurable scenario-based clinical benchmark suite is proposed.•Clinical AIBench provides a multi-level clinical setting and involves multi-diseases.•Clinical AIBench provides clinical indicators to evaluate the cost and the damage.