Chunk2vec: A novel resemblance detection scheme based on Sentence‐BERT for post‐deduplication delta compression in network transmission

• Published in: IET communications 2024-01, Vol.18 (2), p.145-159
• Main Authors: Wang, Chunzhi, Wang, Keguan, Li, Min, Wei, Feifei, Xiong, Neal
• Format: Article
• Language: English
• Subjects: data deduplication; deep learning; delta compression; Natural Language Processing; network transmission; resemblance detection
• ISSN: 1751-8628; 1751-8636
• DOI: 10.1049/cmu2.12719

Delta compression, as a complementary technique for data deduplication, has gained widespread attention in network storage systems. It can eliminate redundant data between non‐duplicate but similar chunks that cannot be identified by data deduplication. The network transmission overhead between servers and clients can be greatly reduced by using data deduplication and delta compression techniques. Resemblance detection is a technique that identifies similar chunks for post‐deduplication delta compression in network storage systems. The existing resemblance detection approaches fail to detect similar chunks with arbitrary similarity by setting a similarity threshold, which can be suboptimal. In this paper, the authors propose Chunk2vec, a resemblance detection scheme for delta compression that utilizes deep learning techniques and Approximate Nearest Neighbour Search technique to detect similar chunks with any given similarity range. Chunk2vec uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks. The experimental results on five real‐world datasets indicate that Chunk2vec improves the accuracy of resemblance detection for delta compression and achieves higher compression ratio than the state‐of‐the‐art resemblance detection technique. The existing resemblance detection approaches fail to identify similar chunks with arbitrary similarity by setting a similarity threshold. In this work, the authors propose a novel resemblance detection scheme called Chunk2vec, which uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks and applies the Approximate Nearest Neighbour Search technique to find the chunk's fingerprint feature vector with any given similarity range. This novel approach can significantly improve the accuracy of resemblance detection for post‐deduplication delta compression and greatly reduces the network transmission overhead between servers and clients.