Illumination-aware image fusion for around-the-clock human detection in adverse environments from Unmanned Aerial Vehicle

• Published in: Expert systems with applications 2022-10, Vol.204, p.117413, Article 117413
• Main Authors: Golcarenarenji, Gelayol, Martinez-Alpiste, Ignacio, Wang, Qi, Alcaraz-Calero, Jose Maria
• Format: Article
• Language: English
• Subjects: Deep machine learning; Human detection; Image fusion; Image registration; UAV
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2022.117413

This study proposes a novel illumination-aware image fusion technique and a Convolutional Neural Network (CNN) called BlendNet to significantly enhance the robustness and real-time performance of small human objects detection from Unmanned Aerial Vehicles (UAVs) in harsh and adverse operation environments. The proposed solution is particular useful for mission-critical public safety applications such as search and rescue operations in rural areas. The operation environments of such missions are featured with poor illumination condition and complex background such as dense vegetation and undergrowth in diverse weather conditions, and the missions have to address the challenges of detecting humans from UAVs at high altitudes, with a moving platform and from various viewing angles. To overcome these challenges, the proposed solution register and fuse the images using Enhanced Correlation Coefficient (ECC) and arithmetic image addition with customised weights techniques. The result of this fusion is fuelled with our new BlendNet AI model achieving 95.01 % of accuracy with 42.2 Frames Per Second (FPS) on Titan X GPU with input size of 608 pixels. The effectiveness of the proposed fusion method has been evaluated and compared with other methods using the KAIST public dataset. The experimental results show competitive performance of BlendNet in terms of both visual quality as well as quantitative assessment of high detection accuracy at high speed. •Design and development of a new three-step customised fusion method.•Design and development of a new real-time CNN for small object detection.•Validation of the proposed methods based on real-world trials of Police Scotland.•Allowing flexible operation capabilities in different illumination conditions.