A deep learning based classifier framework for automated nuclear atypia scoring of breast carcinoma

• Published in: Engineering applications of artificial intelligence 2023-04, Vol.120, p.105949, Article 105949
• Main Authors: Mathew, Tojo, Johnpaul, C.I., Ajith, B., Kini, Jyoti R., Rajan, Jeny
• Format: Article
• Language: English
• Subjects: Breast cancer; Cancer grading; Deep learning; Histopathology; Nuclear atypia scoring; Slide image
• ISSN: 0952-1976; 1873-6769
• DOI: 10.1016/j.engappai.2023.105949

Nuclear atypia scoring is an essential procedure in the grading of breast carcinoma. Manual procedure of nuclear atypia scoring is error-prone, and marked by pathologists’ disagreement and low reproducibility. Automated methods are actively attempted by researchers to solve the problems of manual scoring. In this work, we propose a novel deep learning-based framework for automated nuclear atypia scoring of breast cancer from histopathology slide images. The framework consists of three major phases namely preprocessing, deep learning, and postprocessing. The original three-class problem of atypia scoring at slide level is not suitable for direct application of deep learning algorithms. This is due to the large dimensions and structural complexity of slide images, compounded by the small sample size of the available dataset. Redesign of this problem into a six-class nuclei classification problem through a set of preprocessing steps to facilitate effective use of deep learning algorithms, and the flexibility of the proposed three-phase framework to use different algorithms in each phase are the novel aspects of the proposed work. We used the publicly available slide image dataset MITOS-ATYPIA that contains 600 slide images of high spatial dimension for the experiments. A five-fold cross validation with the train-test sample ratio 80:20 in each fold is used for the performance evaluation. The performance of the method based on this framework exceeds the state-of-the-art with the results 0.8766, 0.8760, and 0.8745 for the metrics precision, recall, and F1 score respectively. [Display omitted] •A novel deep learning based framework for nuclear atypia scoring of breast cancer•Nuclei segmentation followed by a classification approach is applied in the method•Three-class atypia scoring task is re-designed as a six-class nuclei classification•Framework is flexible to use new algorithms in various stages for future improvement