Needs Analysis and Technology Evaluation: Evaluation Case Study of Alternative Software for Controller Planning Work–Part 2

• Published in: Journal of cognitive engineering and decision making 2015-06, Vol.9 (2), p.186-207
• Main Authors: Billman, Dorrit, Arsintescu, Lucia, Feary, Michael, Lee, Jessica, Tiwary, Rachna
• Format: Article
• Language: English
• ISSN: 1555-3434; 2169-5032
• DOI: 10.1177/1555343414567775

A NASA Mission Control group provided us the opportunity to conduct a needs analysis, produce software guided by the needs analysis, and evaluate the software prototype. This paper reports our discriminative evaluation of new prototype software for Attitude Determination and Control Officers, who plan and execute maneuvers of the International Space Station with their Russian counterparts. On a specific, pragmatic level, our evaluation provided evidence of large performance improvement. Relative to legacy software, the new software reduced time and errors by half in a laboratory experiment using valid tasks identified from the needs analysis. Our discriminative evaluation also isolated specific benefits attributable to specific improvements in alignment of technology to the work. Our discriminative evaluation isolated contributing factors affecting performance by using item sets designed to differentially engage software components. This approach for identifying specific factors contributing to performance improvement may be reusable in many situations, particularly where it is not feasible to develop and test many software versions, each differing in just a single factor of interest.