Measuring the Performance of Automated Planning Systems

In this paper, we describe existing performance measures for automated planning algorithms, and discuss the limitations and biases inherent in those performance measures. We point out the importance of developing a performance measure that explicitly the restrictive assumptions on which a planning a...

Full description

Saved in:
Bibliographic Details
Main Authors: Nau, Dana, Ghallab, Malik
Format: Report
Language:English
Subjects:
Administration and Management
ALGORITHMS
ARTIFICIAL INTELLIGENCE
AUTOMATED PLANNING ALGORITHMS
Cybernetics
DYNAMIC RESPONSE
METRICS
Mfg & Industrial Eng & Control of Product Sys
Operations Research
PERFORMANCE MEASURE
PERFORMANCE(ENGINEERING)
PLANNING
RESTRICTIONS
SYSTEMS ENGINEERING
WORKSHOPS
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we describe existing performance measures for automated planning algorithms, and discuss the limitations and biases inherent in those performance measures. We point out the importance of developing a performance measure that explicitly the restrictive assumptions on which a planning algorithm depends, and we propose a composite performance measure based on three factors: (1) the scope of the planning algorithm: which set of restrictive assumption are needed and which can be lifted, (2) the control knowledge and tuning required for each planning domain, (3) the size of the problems that can be solve in a reasonable amount of time in each area of its scope (i.e., for each combination of relaxed assumptions it can handle). Presented at the Performance Metrics for Intelligent Systems Workshop, held in Gaithersburg, MD on 24-26 August 2004. Published in the Proceedings of the Performance Metrics for Intelligent Systems Workshop, August 2004. Sponsored in part by National Science Foundation Grant no. IIS0412812. Prepared in cooperation with Laboratory for Analysis and Architecture of Systems (LAAS), Toulouse, France.