VORI: A framework for testing voice user interface interactability

• Published in: High-Confidence Computing 2022-09, Vol.2 (3), p.100069, Article 100069
• Main Authors: Alrumayh, Abrar S., Tan, Chiu C.
• Format: Article
• Language: English
• Subjects: Home voice assistant; Smart speaker; Usability; Voice user interface
• ISSN: 2667-2952; 2667-2952
• DOI: 10.1016/j.hcc.2022.100069

The ability of Voice User Interface (VUI) to understand how users will express their commands naturally and intuitively is an essential component of user experience, especially when the user is interacting with the VUI for the first time. Designing an automated method for testing the usability of VUI is a challenge for two reasons. First, there are many different ways for a user to express the same intention, e.g. “play some music”, ””put some music on”, etc., that is difficult to determine in advance. Second, many VUI apps today typically rely on the platform service provider (e.g. Amazon, Google, etc.) to perform many of the speech recognition and natural language processing tasks, and these services are provided as a blackbox. Consequently, it is difficult for the app developer to obtain information about errors and user feedback. In this paper, we propose a framework, VORI, to systematically evaluate the interactability of VUI, as well as a new metric for quantifying the interactability of a VUI. We use VORI to analyze 127 applications on Alexa by sending over 82,931 commands. Our analysis results highlight that 41.7% of apps only accept strict input that has to exactly match the developer’s predefined sample commands with an interactability score of 20% or less. This suggests developers should consider a better interactability strategy in the design of VUIs, and more research is needed to further explore the design space to improve the interactability.