Validation of trace-driven simulation models: regression analysis revisited

• Main Authors: Kleijnen, Jack P. C., Bettonvil, Bert, Van Groenendaal, Willem J. H.
• Format: Conference Proceeding
• Language: English
• Subjects: Analytical models; Computational modeling; Computer simulation; Costs; Inventory management; Mathematics of computing; Mathematics of computing > Probability and statistics; Monte Carlo methods; Power generation economics; Power system modeling; Regression analysis; System testing
• DOI: 10.1145/256562.256650

For the validation of trace-driven simulation models this paper recommends a simple statistical test that uses elementary regression analysis in a novel way. This test concerns a (joint) null-hypothesis: the outputs of the simulated and the real systems have the same means and the same variances. Technically, the differences between simulated and real outputs are regressed on their sums, and the resulting slope and intercept are tested to be zero. This paper further proves that it is wrong to use a naive test that regresses the simulation outputs on the real outcomes, and hypothesizes that the resulting regression line gives a 45 /spl deg/ line through the origin. The new and the old tests are investigated in Monte Carlo experiments with inventory systems. The conclusion is that the new test has the correct type I error probability, whereas the old test (falsely) rejects a valid simulation model substantially more often than the nominal alpha level. The power of the new test increases, as the simulation model deviates more from the real system.