On Tuning Raft for IoT Workload in Apache IoTDB

Raft has been widely adopted as the consensus protocol in various distributed systems, owing to its straight-forward interpretation and implementation. However, directly applying Raft may not fully meet the extremely high throughput requirement in the Internet of Things (IoT) scenarios. The case stu...

Full description

Saved in:
Bibliographic Details
Main Authors: Jiang, Tian, Huang, Xiangdong, Song, Shaoxu, Wang, Chen, Wang, Jianmin
Format: Conference Proceeding
Language:English
Subjects:
consensus protocol
Data structures
Dispatching
distributed system
IoT
Memory management
Systematics
Throughput
Time series analysis
Wind turbines
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Raft has been widely adopted as the consensus protocol in various distributed systems, owing to its straight-forward interpretation and implementation. However, directly applying Raft may not fully meet the extremely high throughput requirement in the Internet of Things (IoT) scenarios. The case study on real IoT applications reveals unique features, such as high concurrency, fluctuating traffic, fixed-size requests, and compressible data. It explains the bottlenecks of the Raft leader in dispatching, persistence, and memory management, for IoT applications. To this end, we propose to explore the opportunities of tuning Raft for the particular IoT workload, including alternative data structures, various compression algorithms, memory recycling strategies, etc. This paper presents a systematic evaluation of Raft by tuning the aspects above, in an open-source time series database Apache IoTDB. The extensive experiments demonstrate improved system parallelism, reduced information redundancy, and increased resource utilization. The throughput improvement ranges from 10% by replacing the dispatching data structure to nearly 200% by pre-serialization. The overall throughput can reach 4x compared with the original Raft implementation.
ISSN:2375-026X
DOI:10.1109/ICDE60146.2024.00399