Progressive Data Augmentation Method for Remote Sensing Ship Image Classification Based on Imaging Simulation System and Neural Style Transfer

• Published in: IEEE journal of selected topics in applied earth observations and remote sensing 2021, Vol.14, p.9176-9186
• Main Authors: Xiao, Qi, Liu, Bo, Li, Zengyi, Ni, Wei, Yang, Zhen, Li, Ligang
• Format: Article
• Language: English
• Subjects: Blurring; Classification; Convolutional neural networks; Current data; Data augmentation; Data models; Domain gap; Feature extraction; Image classification; Imaging; Imaging techniques; Machine learning; Marine vehicles; Methods; Neural networks; neural style transfer (NST); Remote sensing; ship simulation samples; Ships; Simulation; Three dimensional models; Training; Transfer learning
• ISSN: 1939-1404; 2151-1535
• DOI: 10.1109/JSTARS.2021.3109600

Deep learning has shown great power in processing remote sensing data, especially for fine-grained remote sensing ship image classification. However, the lack of a large amount of effective training data greatly limits the performance of neural networks. Based on current data augmentation methods, images of ships on the sea generated for remote sensing have the problem of distortion, blurring, and poor diversity. To tackle this problem, we propose a novel progressive remote sensing ship image data augmentation method that combines ship simulation samples and a neural style transfer (NST) based network to generate a large amount of transferred remote sensing ship images. Our method consists of two stages. The first stage uses a visible light imaging simulation system to generate ship simulation samples through three-dimensional models of real images. This stage can significantly increase the diversity of the training dataset. For the second stage, to eliminate the domain gap between real ship images and ship simulation samples, a few real images and a newly designed NST-based network called Sim2RealNet are employed to realize style transfer from simulation samples to real images. The proposed method was applied to a variety of ship targets to verify its effectiveness compared to other data augmentation methods on remote sensing image classification tasks. The experimental results demonstrate the effectiveness of the proposed method.