Deep Learning Based SWIR Object Detection in Long-Range Surveillance Systems: An Automated Cross-Spectral Approach

• Published in: Sensors (Basel, Switzerland) Switzerland), 2022-03, Vol.22 (7), p.2562
• Main Authors: Pavlović, Miloš S, Milanović, Petar D, Stanković, Miloš S, Perić, Dragana B, Popadić, Ilija V, Perić, Miroslav V
• Format: Article
• Language: English
• Subjects: Algorithms; Annotations; Cameras; Color; cross-spectral data annotation; Datasets; Deep Learning; Imaging systems; Light; multi-sensor imaging system; object detection; Performance evaluation; Sensors; Spectra; Surveillance; Surveillance equipment; Surveillance systems; SWIR imaging; Thermal imaging
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22072562

SWIR imaging bears considerable advantages over visible-light (color) and thermal images in certain challenging propagation conditions. Thus, the SWIR imaging channel is frequently used in multi-spectral imaging systems (MSIS) for long-range surveillance in combination with color and thermal imaging to improve the probability of correct operation in various day, night and climate conditions. Integration of deep-learning (DL)-based real-time object detection in MSIS enables an increase in efficient utilization for complex long-range surveillance solutions such as border or critical assets control. Unfortunately, a lack of datasets for DL-based object detection models training for the SWIR channel limits their performance. To overcome this, by using the MSIS setting we propose a new cross-spectral automatic data annotation methodology for SWIR channel training dataset creation, in which the visible-light channel provides a source for detecting object types and bounding boxes which are then transformed to the SWIR channel. A mathematical image transformation that overcomes differences between the SWIR and color channel and their image distortion effects for various magnifications are explained in detail. With the proposed cross-spectral methodology, the goal of the paper is to improve object detection in SWIR images captured in challenging outdoor scenes. Experimental tests for two object types (cars and persons) using a state-of-the-art YOLOX model demonstrate that retraining with the proposed automatic cross-spectrally created SWIR image dataset significantly improves average detection precision. We achieved excellent improvements in detection performance in various variants of the YOLOX model (nano, tiny and x).