From multimodal features to behavioural inferences: A pipeline to model engagement in human-robot interactions

• Published in: PloS one 2023-11, Vol.18 (11), p.e0285749-e0285749
• Main Authors: Joshi, Soham, Malavalli, Arpitha, Rao, Shrisha
• Format: Article
• Language: English
• Subjects: Accuracy; Analysis; Attitudes; Behavior; Beliefs, opinions and attitudes; Datasets; Emotions; Employees; Evaluation; Human acts; Human behavior; Interpersonal relations; Modelling; Personality; Personality traits; Physiology; Pipe lines; Robotics; Robots; Sensors; Social aspects; Support vector machines
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0285749

Modelling the engaging behaviour of humans using multimodal data collected during human-robot interactions has attracted much research interest. Most methods that have been proposed previously predict engaging behaviour directly from multimodal features, and do not incorporate personality inferences or any theories of interpersonal behaviour in human-human interactions. This work investigates whether personality inferences and attributes from interpersonal theories of behaviour (like attitude and emotion) further augment the modelling of engaging behaviour. We present a novel pipeline to model engaging behaviour that incorporates the Big Five personality traits, the Interpersonal Circumplex (IPC), and the Triandis Theory of Interpersonal Behaviour (TIB). We extract first-person vision and physiological features from the MHHRI dataset and predict the Big Five personality traits using a Support Vector Machine. Subsequently, we empirically validate the advantage of incorporating personality in modelling engaging behaviour and present a novel method that effectively uses the IPC to obtain scores for a human's attitude and emotion from their Big Five traits. Finally, our results demonstrate that attitude and emotion are correlates of behaviour even in human-robot interactions, as suggested by the TIB for human-human interactions. Furthermore, incorporating the IPC and the Big Five traits helps generate behavioural inferences that supplement the engaging behaviour prediction, thus enriching the pipeline. Engagement modelling has a wide range of applications in domains like online learning platforms, assistive robotics, and intelligent conversational agents. Practitioners can also use this work in cognitive modelling and psychology to find more complex and subtle relations between humans' behaviour and personality traits, and discover new dynamics of the human psyche. The code will be made available at: