Experimental analysis of the root causes of performance evaluation results: a backfilling case study

The complexity of modern computer systems may enable minor variations in performance evaluation procedures to actually determine the outcome. Our case study concerns the comparison of two parallel job schedulers, using different workloads and metrics. It shows that metrics may be sensitive to differ...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2005-02, Vol.16 (2), p.175-182
Main Author: Feitelson, D.G.
Format: Article
Language:English
Subjects:
backfilling
Computer aided software engineering
Delay
Discrete event simulation
experimental verification
parallel job scheduling
Performance analysis
Performance evaluation
Processor scheduling
Runtime
sensitivity of results
simulation
Supercomputers
Time factors
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The complexity of modern computer systems may enable minor variations in performance evaluation procedures to actually determine the outcome. Our case study concerns the comparison of two parallel job schedulers, using different workloads and metrics. It shows that metrics may be sensitive to different job classes, and not measure the performance of the whole workload in an impartial manner. Workload models may implicitly assume that some workload attribute is unimportant and does not warrant modeling; this too can turn out to be wrong. As such effects are hard to predict, a careful experimental methodology is needed in order to find and verify them.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2005.18