AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

• Published in: Publications of the Astronomical Society of the Pacific 2024-07, Vol.136 (7), p.75001
• Main Authors: Liu, Xiaoying, Liu, Yingbo, Yang, Lei, Wu, Shichao, Jiang, Rong, Xiang, Yongyuan
• Format: Article
• Language: English
• Subjects: Astronomical methods; Astronomy; Astronomy data analysis; Astronomy software; Computational methods; Data compression; Data storage; Deep learning; GPU computing; Solar images
• ISSN: 0004-6280; 1538-3873
• DOI: 10.1088/1538-3873/ad5b8a

Effective data compression technology is essential for addressing data storage and transmission needs, especially given the escalating volume and complexity of data generated by contemporary astronomy. In this study, we propose utilizing deep learning-based lossless compression techniques to improve compression efficiency. We begin with a qualitative and quantitative analysis of the temporal and spatial redundancy in solar observation data. Based on this analysis, we introduce a novel deep learning-based framework called AstroDLLC for the lossless compression of astronomical solar images. AstroDLLC first segments high-resolution images into blocks to ensure that deep learning model training does not rely on high-computation power devices. It then addresses the non-normality of the partitioned data through simple reversible computational methods. Finally, it utilizes Bit-swap to train deep learning models that capture redundant features across multiple image frames, thereby enhancing compression efficiency. Comprehensive evaluations using data from the New Vacuum Solar Telescope reveal that AstroDLLC achieves a maximum compression ratio of 3.00 per image, surpassing Gzip, RICE, and other lossless technologies. The performance of AstroDLLC underscores its potential to address data compression challenges in astronomy.