PnP-UGCSuperGlue: deep learning drone image matching algorithm for visual localization

• Published in: The Journal of supercomputing 2024, Vol.80 (12), p.17711-17740
• Main Authors: Guo, Ya, Yang, Fan, Si, Yazhong, Yang, Yipu, Zhang, Wei, Zhang, Xiaolong, Zhou, Yatong
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Coding; Compilers; Computer Science; Deep learning; Drone aircraft; Feature extraction; Feature maps; Graph matching; Hovering; Interpreters; Localization; Machine learning; Modules; Phase matching; Processor Architectures; Programming Languages; Real time; Satellite imagery; Unmanned aerial vehicles
• ISSN: 0920-8542; 1573-0484
• DOI: 10.1007/s11227-024-06128-3

In response to the significant positioning errors that arise in visual localization algorithms for unmanned aerial vehicles (UAVs) when relying on drone image matching in areas devoid of satellite signals, we propose a deep learning-based algorithm named PnP-UGCSuperGlue. This algorithm employs a convolutional neural network (CNN) that is enhanced with a graph encoding module. The resulting enriched features contain vital information that refines the feature map and improves the overall accuracy of the visual localization process. The PnP-UGCSuperGlue framework initiates with the semantic feature extraction from both the real-time drone image and the geo-referenced image. This extraction process is facilitated by a CNN-based feature extractor. In the subsequent phase, a graph encoding module is integrated to aggregate the extracted features. This integration significantly enhances the quality of the generated feature keypoints and descriptors. Following this, a graph matching network is applied to leverage the generated descriptors, thereby facilitating a more precise feature point matching and filtering process. Ultimately, the perspective-n-point (PnP) method is utilized to calculate the rotation matrix and translation vector. This calculation is based on the results of the feature matching phase, as well as the camera intrinsic parameters and distortion coefficients. The proposed algorithm’s efficacy is validated through experimental evaluation, which demonstrates a mean absolute error of 0.0005 during the drone’s hovering state and 0.0083 during movement. These values indicate a significant reduction of 0.0010 and 0.0028, respectively, compared to the USuperGlue network.