Gesto: Mapping UI Events to Gestures and Voice Commands

• Published in: Proceedings of the ACM on human-computer interaction 2019-06, Vol.3 (EICS), p.1-22, Article 5
• Main Authors: Park, Chang Min, Ki, Taeyeon, Ben Ali, Ali J., Pawar, Nikhil Sunil, Dantu, Karthik, Ko, Steven Y., Ziarek, Lukasz
• Format: Article
• Language: English
• Subjects: Human computer interaction (HCI); Human-centered computing; Interactive systems and tools; Ubiquitous and mobile computing; Ubiquitous and mobile computing systems and tools; User interface management systems
• ISSN: 2573-0142; 2573-0142
• DOI: 10.1145/3300964

Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.