Autonomous victim detection system based on deep learning and multispectral imagery

• Published in: Machine learning: science and technology 2023-03, Vol.4 (1), p.15018
• Main Authors: Cruz Ulloa, Christyan, Garrido, Luis, del Cerro, Jaime, Barrientos, Antonio
• Format: Article
• Language: English
• Subjects: Artificial neural networks; artificial vision; Catastrophic events; convolutional neural networks; Deep learning; Inspection; multispectral image; multispectral index; Optical measuring instruments; Qualitative analysis; Search and rescue missions; search and rescue robotics; Weather
• ISSN: 2632-2153; 2632-2153
• DOI: 10.1088/2632-2153/acb6cf

Post-disaster environments resulting from catastrophic events, leave sequels such as victims trapped in debris, which are difficult to detect by rescuers in a first inspection. Technological advances in electronics and perception have allowed the development of versatile and powerful optical sensors capable of capturing light in spectrums that humans cannot. new deep learning techniques, such as convolutional neural networks (CNNs), has allowed the generation of network models capable of autonomously detecting specific image patterns according to previous training. This work introduces an autonomous victim detection system to be deployed by using search and rescue robots. The proposed system defines new indexes based on combining the multispectral bands (Blue, Green, Red, Nir, Red Edge) to obtain new multispectral images where relevant characteristics of victims and the environment are highlighted. CNNs have been used as a second phase for automatically detecting victims in these new multispectral images. A qualitative and quantitative analysis of new indexes proposed by the authors has been carried out to evaluate their efficiency in contrast to the state-of-the-art ones. A data set has been generated to train different CNN models based on the best obtained index to analyze their effectiveness in detecting victims. The results show an efficiency of 92% in automatically detecting victims when applying the best multispectral index to new data. This method has also been contrasted with others based on thermal and RGB imagery to detect victims, where it has been proven that it generates better results in situations of outdoor environments and different weather conditions.