Multi-Magnification Image Search in Digital Pathology

• Published in: IEEE journal of biomedical and health informatics 2022-09, Vol.26 (9), p.4611-4622
• Main Authors: Rasoolijaberi, Maral, Babaei, Morteza, Riasatian, Abtin, Hemati, Sobhan, Ashrafi, Parsa, Gonzalez, Ricardo, Tizhoosh, Hamid R.
• Format: Article
• Language: English
• Subjects: Annotations; Archives & records; Bioinformatics; Brain tumors; Breast; CBIR; Digital imaging; Digital pathology; Feature extraction; Genomes; Histopathology; image Search; Medical imaging; Medical personnel; Medical research; Microscopes; Microscopy; multi-magnification; Pathology; Representations; Search engines; Search methods; Task analysis; Tumors; Urinary tract; Workflow
• ISSN: 2168-2194; 2168-2208
• DOI: 10.1109/JBHI.2022.3181531

This paper investigates the effect of magnification on content-based image search in digital pathology archives and proposes to use multi-magnification image representation. Image search in large archives of digital pathology slides provides researchers and medical professionals with an opportunity to match records of current and past patients and learn from evidently diagnosed and treated cases. When working with microscopes, pathologists switch between different magnification levels while examining tissue specimens to find and evaluate various morphological features. Inspired by the conventional pathology workflow, we have investigated several magnification levels in digital pathology and their combinations to minimize the gap between AI-enabled image search methods and clinical settings. The proposed searching framework does not rely on any regional annotation and potentially applies to millions of unlabelled (raw) whole slide images. This paper suggests two approaches for combining magnification levels and compares their performance. The first approach obtains a single-vector deep feature representation for a digital slide, whereas the second approach works with a multi-vector deep feature representation. We report the search results of 20×, 10×, and 5× magnifications and their combinations on a subset of The Cancer Genome Atlas (TCGA) repository. The experiments verify that cell-level information at the highest magnification is essential for searching for diagnostic purposes. In contrast, low-magnification information may improve this assessment depending on the tumor type. Our multi-magnification approach achieved up to 11% F1-score improvement in searching among the urinary tract and brain tumor subtypes compared to the single-magnification image search.