Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication f...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on software engineering 2008-07, Vol.34 (4), p.452-470
Main Authors: Rutherford, M.J., Carzaniga, A., Wolf, A.L.
Format: Article
Language:English
Subjects:
Adequacy
Alliances
Computational modeling
Computer networks
Computer simulation
Computer Society
Criteria
Digital Object Identifier
Discrete event simulation
Domain Name System
Domain names
Engineers
Failure
Failure analysis
Foils
Mathematical models
Network servers
Quality control
Simulation
Software engineering
Specification
Specifications
Studies
System testing
Systems development
Test coverage of specifications
Timing
Topology
Web server
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication failure, and timing. Furthermore, they do not provide the engineer with a means to evaluate the relative effectiveness of different criteria, nor the relative effectiveness of adequate test suites satisfying a given criterion. This paper makes three contributions to the development and use of test adequacy criteria for distributed systems: (1) a testing method based on discrete-event simulations; (2) a fault-based analysis technique for evaluating test suites and adequacy criteria; and (3) a series of case studies that validate the method and technique. The testing method uses a discrete-event simulation as an operational specification of a system, in which the behavioral effects of distribution are explicitly represented. Adequacy criteria and test cases are then defined in terms of this simulation-based specification. The fault-based analysis involves mutation of the simulation-based specification to provide a foil against which test suites and the criteria that formed them can be evaluated. Three distributed systems were used to validate the method and technique, including DNS, the domain name system.
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2008.33