MetaFlow: A Scalable Metadata Lookup Service for Distributed File Systems in Data Centers

In large-scale distributed file systems, efficient metadata operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its sig...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on big data 2018-06, Vol.4 (2), p.203-216
Main Authors: Peng Sun, Yonggang Wen, Ta, Duong Nguyen Binh, Haiyong Xie
Format: Article
Language:English
Subjects:
B-tree
Big data
Linux
Metadata
Metadata management
Peer-to-peer computing
Servers
software-defined networking
System performance
Throughput
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In large-scale distributed file systems, efficient metadata operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system throughput by up to 70 percent, and increase system latency by a factor of up to 8 compared to ideal scenarios. In this paper, we present MetaFlow, a scalable metadata lookup service utilizing software-defined networking (SDN) techniques to distribute lookup workload over network components. MetaFlow tackles the lookup bottleneck problem by leveraging B-tree, which is constructed over the physical topology, to manage flow tables for SDN-enabled switches. Therefore, metadata requests can be forwarded to appropriate servers using only switches. Extensive performance evaluations in both simulations and testbed showed that MetaFlow increases system throughput by a factor of up to 3.2, and reduce system latency by a factor of up to 5 compared to DHT-based systems. We also deployed MetaFlow in a distributed file system, and demonstrated significant performance improvement.
ISSN:2332-7790
2332-7790
2372-2096
DOI:10.1109/TBDATA.2016.2612241