Realistic and Scalable Benchmarking Cloud File Systems: Practices and Lessons from AliCloud

The past decade has witnessed the rapid boom of cloud computing. Many public cloud infrastructures have been implemented and serve millions of tenants. Cloud file systems, which take charge of petabyte-scale data storage, play a crucial role in the performance of cloud infrastructures. Typical cloud...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2017-11, Vol.28 (11), p.3272-3285
Main Authors: Ren, Zujie, Shi, Weisong, Wan, Jian, Cao, Feng, Lin, Jiangbin
Format: Article
Language:English
Subjects:
Benchmark testing
Benchmarks
Big Data
Cloud computing
Cloud file systems
Data storage
Distributed databases
performance benchmarking
Performance evaluation
Performance tests
Servers
Throughput
Workload
workload generator
Workloads
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The past decade has witnessed the rapid boom of cloud computing. Many public cloud infrastructures have been implemented and serve millions of tenants. Cloud file systems, which take charge of petabyte-scale data storage, play a crucial role in the performance of cloud infrastructures. Typical cloud file systems, including GFS, HDFS and Ceph, have attracted notable research efforts for performance evaluation and optimization. However, due to the heterogeneity and complexity of I/O workload characteristics in cloud environments, it is still challenging to conduct an accurate and efficient performance evaluation. To address this problem, we collected a two-week I/O workload trace from a 2,500-node production cluster in AliCloud, which is one of the largest cloud providers in Asia. Using the AliCloud trace, we characterized the I/O workload and data distribution, and compared two cloud services in multiple perspectives, including the request arrival pattern, request size, data population and so on. A list of observations and implications were derived and applied to help design a cloud file system benchmarking suite, called Porcupine. Porcupine aims to deploy a scalable and efficient performance evaluation on cloud file systems using realistic I/O workloads. We conducted a group of validation experiments, which demonstrated that Porcupine can achieve high accuracy and scalability. This paper provides our experiences and lessons in generating I/O workloads and deploying performance tests on cloud file systems, which we believe will be insightful to the cloud computing community in general.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2017.2715327