Ripple: Enabling Decentralized Data Deduplication at the Edge

With its advantages in ensuring low data retrieval latency and reducing backhaul network traffic, edge computing is becoming a backbone solution for many latency-sensitive applications. An increasingly large number of data is being generated at the edge, stretching the limited capacity of edge stora...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2025-01, Vol.36 (1), p.55-66
Main Authors: Luo, Ruikun, He, Qiang, Chen, Feifei, Wu, Song, Jin, Hai, Yang, Yun
Format: Article
Language:English
Subjects:
Cloud computing
data deduplication
data index
data redundancy
data retrieval latency
Distributed databases
Edge computing
Indexes
Low latency communication
Memory
Metadata
Redundancy
Servers
Video on demand
Web sites
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With its advantages in ensuring low data retrieval latency and reducing backhaul network traffic, edge computing is becoming a backbone solution for many latency-sensitive applications. An increasingly large number of data is being generated at the edge, stretching the limited capacity of edge storage systems. Improving resource utilization for edge storage systems has become a significant challenge in recent years. Existing solutions attempt to achieve this goal through data placement optimization, data partitioning, data sharing, etc. These approaches overlook the data redundancy in edge storage systems, which produces substantial storage resource wastage. This motivates the need for an approach for data deduplication at the edge. However, existing data deduplication methods rely on centralized control, which is not always feasible in practical edge computing environments. This article presents Ripple, the first approach that enables edge servers to deduplicate their data in a decentralized manner. At its core, it builds a data index for each edge server, enabling them to deduplicate data without central control. With Ripple, edge servers can 1) identify data duplicates; 2) remove redundant data without violating data retrieval latency constraints; and 3) ensure data availability after deduplication. The results of trace-driven experiments conducted in a testbed system demonstrate the usefulness of Ripple in practice. Compared with the state-of-the-art approach, Ripple improves the deduplication ratio by up to 16.79% and reduces data retrieval latency by an average of 60.42%.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2024.3493953