Evaluation of XR Applications: A Tertiary Review

• Published in: ACM computing surveys 2024-05, Vol.56 (5), p.1-35, Article 110
• Main Authors: Becker, Artur, Freitas, Carla M. Dal Sasso
• Format: Article
• Language: English
• Subjects: Augmented reality; Computer science; HCI design and evaluation methods; Human-centered computing; Literature reviews; Mixed / augmented reality; Mixed reality; Taxonomy; Virtual reality
• ISSN: 0360-0300; 1557-7341
• DOI: 10.1145/3626517

Extended reality (XR) applications—encompassing virtual reality, augmented reality, and mixed reality—are finding their way into multiple domains. Each area has different motivations for employing and different criteria for evaluating XR. Multiple surveys describe XR and its evaluation in particular fields. However, these surveys do not always agree on the definition of XR. This lack of consensus makes it hard to compare and use learnings from XR research across areas. Through a tertiary systematic literature review, we analyzed 81 surveys from several fields to provide a comprehensive summary of the state of XR research regarding the evaluation of XR applications. We seek to understand (i) how is XR defined? (ii) why is XR employed? (iii) how is XR evaluated? (iv) what are the main criticisms and future research paths outlined by the surveys? and (v) how good are the surveys? We present our findings describing XR research in 10 categories. Given our findings, we propose that future research should build upon a solid XR taxonomy and depart from effectiveness into efficiency research—to understand not only if but also how XR achieves the desired outcomes.