Detecting in-car VR Motion Sickness from Lower Face Action Units

• Main Authors: Li, Gang, Guha, Tanaya, Onuoha, Ogechi, Qiu, Zhanyan, Grant, Alana, Feng, Zejian, Zhang, Zirui, Pohlmann, Kathariana, McGill, Mark, Brewster, Stephen, Pollick, Frank
• Format: Conference Proceeding
• Language: English
• Subjects: Cameras; Correlation; Faces; Feature extraction; Headphones; Lower Face Action Units; Machine learning; Motion sickness; Physiology; Pupils; Visualization
• ISSN: 2473-0726
• DOI: 10.1109/ISMAR62088.2024.00118

This paper presents the first in-car VR motion sickness (VRMS) detection model based on lower face action units (LF-AUs). Initially developed in a simulated in-car environment with 78 participants, the model's generalizability was later tested in realworld driving conditions. Motion sickness was induced using visual linear motion in the VR headset and physical horizontal rotation via a rotating chair. We used a convolutional neural network (MobileNetV3) to automatically extract LF-AUs from images of the users' mouth region, captured by the VR headset's built-in camera. These LF-AUs were then used to train a Support Vector Regression (SVR) model to estimate motion sickness scores. We compared the SVR model's performance using LF-AUs, pupil diameters, and physiological features (individually and in combination) from the same VR headset. Results showed that both individual LF-AU (right dimple) and combined LF-AUs had significant Pearson correlations with self-reported motion sickness scores and achieved lower root mean squared error compared to pupil diameters. The best detection results were obtained by combining LF-AUs and pupil diameters, while physiological features alone did not yield significant results. The LF-AUs-based model demonstrated encouraging generalizability across different settings in the independent studies.