Using feature maps to unpack the CNN ‘Black box’ theory with two medical datasets of different modality

• Published in: Intelligent systems with applications 2023-05, Vol.18, p.200233, Article 200233
• Main Authors: Azam, Sami, Montaha, Sidratul, Fahim, Kayes Uddin, Rafid, A.K.M. Rakibul Haque, Mukta, Md. Saddam Hossain, Jonkman, Mirjam
• Format: Article
• Language: English
• Subjects: ANOVA test; Black box; Convolutional neural network; Feature map analysis; Geometric feature; T-test
• ISSN: 2667-3053; 2667-3053
• DOI: 10.1016/j.iswa.2023.200233

•Insight of CNNs by analyzing the extent of diversity of layer by layer feature maps.•Analyzing the number of produced black images in the feature maps.•Statistically analyzing feature maps for various iterations of convolutional layers.•Analyzing feature maps resulted from a single input image.•Iteration by iteration feature maps analysis to find inter-class variance. Convolutional neural networks (CNNs) have been established for a comprehensive range of computer vision problems across several benchmarks. Visualization and analysis of feature maps generated by convolutional layers can be an effective approach to explore the hidden and complex characteristic of a CNN model. Convolutional layers provide diverse feature maps however, the extent of this diversity needs to be explored. This research attempts to provide five insights of the ‘Black box’ mechanism of CNNs, using skin cancer dermoscopy and lung scan computed tomography (CT) Scan datasets by statistically analyzing layer by layer (three convolutional layers) feature maps using 17 geometrical and 6 intensity-based features to determine the characteristics and level of diversity. Significance and difference of the feature maps layer by layer, black feature maps analysis, difference of the feature maps to each other and to the original image, variations among the feature maps when running the model multiple times and inter-class variation among the feature maps for different iteration are explored. Various statistical methods including T-test, analysis of variance (ANOVA), mean, median, mean squared error (MSE), peak signal to noise ratio (PSNR), structural similarity index (SSIM), root mean squared error (RMSE), dice similarity score (DSC), universal image quality index (UQI) and Spectral angle mapper (SAM) are employed. Experimental results show that for the skin cancer dermoscopy dataset, a large number of black feature maps are produced (20–60%) while the proportion of black feature maps for the CT Scan dataset is comparatively low (2–20%). This demonstrates that for different datasets, feature maps with diverse characteristics can be produced. The layer by layer differences between the feature maps is evaluated using T-tests and ANOVA for seventeen geometrical features and six intensity-based features. For both datasets across most of the geometrical features and across most of the intensity-based features a significant diversity can be observed. The difference of the feature maps to each other a