Prior Information Guided Regularized Deep Learning for Cell Nucleus Detection

• Published in: IEEE transactions on medical imaging 2019-09, Vol.38 (9), p.2047-2058
• Main Authors: Tofighi, Mohammad, Guo, Tiantong, Vanamala, Jairam K. P., Monga, Vishal
• Format: Article
• Language: English
• Subjects: Algorithms; Animals; Artificial neural networks; Biomedical imaging; Cell Nucleus - physiology; Colon - cytology; Colon - diagnostic imaging; Colon - pathology; Computer architecture; convolutional neural networks; Cytology; Databases, Factual; Deep Learning; Image detection; Image edge detection; Image Processing, Computer-Assisted - methods; Image quality; Image segmentation; Information processing; learnable shapes; Machine learning; Microprocessors; Morphology; Neural networks; Nuclei; Nuclei (cytology); Nucleus detection; Regularization; Shape; shape priors; Shape recognition; Swine; Traveling salesman problem
• ISSN: 0278-0062; 1558-254X
• DOI: 10.1109/TMI.2019.2895318

Cell nuclei detection is a challenging research topic because of limitations in cellular image quality and diversity of nuclear morphology, i.e., varying nuclei shapes, sizes, and overlaps between multiple cell nuclei. This has been a topic of enduring interest with promising recent success shown by deep learning methods. These methods train convolutional neural networks (CNNs) with a training set of input images and known, labeled nuclei locations. Many such methods are supplemented by spatial or morphological processing. Using a set of canonical cell nuclei shapes, prepared with the help of a domain expert, we develop a new approach that we call shape priors (SPs) with CNNs (SPs-CNN). We further extend the network to introduce an SP layer and then allowing it to become trainable (i.e., optimizable). We call this network as tunable SP-CNN (TSP-CNN). In summary, we present new network structures that can incorporate "expected behavior" of nucleus shapes via two components: learnable layers that perform the nucleus detection and a fixed processing part that guides the learning with prior information. Analytically, we formulate two new regularization terms that are targeted at: 1) learning the shapes and 2) reducing false positives while simultaneously encouraging detection inside the cell nucleus boundary. Experimental results on two challenging datasets reveal that the proposed SP-CNN and TSP-CNN can outperform the state-of-the-art alternatives.