Automating the debugging of datacenter applications with ADDA

Debugging data-intensive distributed applications running in datacenters is complex and time-consuming because developers do not have practical ways of deterministically replaying failed executions. The reason why building such tools is hard is that non-determinism that may be tolerable on a single...

Full description

Saved in:
Bibliographic Details
Main Authors: Zamfir, Cristian, Altekar, Gautam, Stoica, Ion
Format: Conference Proceeding
Language:English
Subjects:
Availability
Computer bugs
data-center
Debugging
Distributed databases
Hardware
Protocols
record-replay
reliability
Servers
storage
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Debugging data-intensive distributed applications running in datacenters is complex and time-consuming because developers do not have practical ways of deterministically replaying failed executions. The reason why building such tools is hard is that non-determinism that may be tolerable on a single node is exacerbated in large clusters of interacting nodes, and datacenter applications produce terabytes of intermediate data exchanged by nodes, thus making full input recording infeasible. We present ADDA, a replay-debugging system for datacenters that has lower recording and storage overhead than existing systems. ADDA is based on two techniques: First, ADDA provides control plane determinism, leveraging our observation that many typical datacenter applications consist of a separate "control plane" and "data plane", and most bugs reside in the former. Second, ADDA does not record "data plane" inputs, instead it synthesizes them during replay, starting from the application's external inputs, which are typically persisted in append-only storage for reasons unrelated to debugging. We evaluate ADDA and show that it deterministically replays real-world failures in Hypertable and Memcached.
ISSN:1530-0889
2158-3927
DOI:10.1109/DSN.2013.6575303