Towards Classification of Architectural Styles of Chinese Traditional Settlements Using Deep Learning: A Dataset, a New Framework, and Its Interpretability

• Published in: Remote sensing (Basel, Switzerland) Switzerland), 2022-10, Vol.14 (20), p.5250
• Main Authors: Han, Qing, Yin, Chao, Deng, Yunyuan, Liu, Peilin
• Format: Article
• Language: English
• Subjects: Algorithms; architectural style classification; Architecture; Artificial intelligence; Artificial neural networks; Automation; Chinese traditional settlements; Classification; Classifiers; CNN interpretability; convolutional neural networks; Cultural heritage; Culture; Datasets; Deep learning; heritage preservation; Literature reviews; Machine learning; Neural networks; Remote sensing; Success; Tourism; Transfer learning; Visualization
• ISSN: 2072-4292; 2072-4292
• DOI: 10.3390/rs14205250

The classification of architectural style for Chinese traditional settlements (CTSs) has become a crucial task for developing and preserving settlements. Traditionally, the classification of CTSs primarily relies on manual work, which is inefficient and time consuming. Inspired by the tremendous success of deep learning (DL), some recent studies attempted to apply DL networks such as convolution neural networks (CNNs) to achieve automated classification of the architecture styles. However, these studies suffer overfitting problems of the CNNs, leading to inferior classification performance. Moreover, most of the studies apply the CNNs as a black box providing limited interpretability. To address these limitations, a new DL classification framework is proposed in this study to overcome the overfitting problem by transfer learning and learning-based data augmentation technique (i.e., AutoAugment). Furthermore, we also employ class activation map (CAM) visualization technique to help understand how the CNN classifiers work to abstract patterns from the input. Specifically, due to a lack of architectural style datasets for the CTSs, a new annotated dataset is first established with six representative classes. Second, several representative CNNs are leveraged to benchmark the new dataset. Third, to address the overfitting problem of the CNNs, a new DL framework is proposed which combines transfer learning and AutoAugment to improve the classification performance. Extensive experiments are conducted on the new dataset to demonstrate the effectiveness of our framework. The proposed framework achieves much better performance than baselines, greatly mitigating the overfitting problem. Additionally, the CAM visualization technique is harnessed to explain what and how the CNN classifiers implicitly learn for recognizing a specified architectural style.