Fatman： Building Reliable Archival Storage Based on Low-Cost Volunteer Resources

We present Fatman, an enterprise-scale archival storage based on volunteer contribution resources from underutilized web servers, usually deployed on thousands of nodes with spare storage capacity. Fatman is specifically designed for enhancing the utilization of existing storage resources and cuttin...

Full description

Saved in:
Bibliographic Details
Published in:Journal of computer science and technology 2015-03, Vol.30 (2), p.273-282
Main Author: 覃安 胡殿明 刘俊 杨文君 谭待
Format: Article
Language:English
Subjects:
Analysis
Archives & records
Artificial Intelligence
Bandwidths
Budgeting
Cloud computing
Computer Science
Construction
Cost engineering
Crashes
Cutting
Data management
Data Structures and Information Theory
Design
Disks
Failure
Information Systems Applications (incl.Internet)
Internet
Metadata
Network storage
Nodes
Regular Paper
Resource utilization
Servers
Software Engineering
Storage capacity
Studies
Systems design
Theory of Computation
Web服务器
低成本
存储容量
存储资源
故障恢复
档案存储
档案系统
资源利用率
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present Fatman, an enterprise-scale archival storage based on volunteer contribution resources from underutilized web servers, usually deployed on thousands of nodes with spare storage capacity. Fatman is specifically designed for enhancing the utilization of existing storage resources and cutting down the hardware purchase cost. Two major concerned issues of the system design are maximizing the resource utilization of volunteer nodes without violating service level objectives （SLOs） and minimizing the cost without reducing the availability of archival system. Fatman has been widely deployed on tens of thousands of server nodes across several datacenters, providing more than 100 PB storage capacity and serving dozens of internal mass-data applications. The system realizes an efficient storage quota consolidation by strong isolation and budget limitation, to maximally support resource contribution without any degradation on host-level SLOs. It novelly improves data reliability by applying disk failure prediction to minish failure recovery cost, named fault-aware data management, dramatically reduces the mean time to repair （MTTR） by 76.3% and decreases file crash ratio by 35% on real-life product workload.
ISSN:1000-9000
1860-4749
DOI:10.1007/s11390-015-1521-6