From multimodal features to behavioural inferences: A pipeline to model engagement in human-robot interactions

Modelling the engaging behaviour of humans using multimodal data collected during human-robot interactions has attracted much research interest. Most methods that have been proposed previously predict engaging behaviour directly from multimodal features, and do not incorporate personality inferences...

Full description

Saved in:
Bibliographic Details
Published in:PloS one 2023-11, Vol.18 (11), p.e0285749-e0285749
Main Authors: Joshi, Soham, Malavalli, Arpitha, Rao, Shrisha
Format: Article
Language:English
Subjects:
Accuracy
Analysis
Attitudes
Behavior
Beliefs, opinions and attitudes
Datasets
Emotions
Employees
Evaluation
Human acts
Human behavior
Interpersonal relations
Modelling
Personality
Personality traits
Physiology
Pipe lines
Robotics
Robots
Sensors
Social aspects
Support vector machines
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Modelling the engaging behaviour of humans using multimodal data collected during human-robot interactions has attracted much research interest. Most methods that have been proposed previously predict engaging behaviour directly from multimodal features, and do not incorporate personality inferences or any theories of interpersonal behaviour in human-human interactions. This work investigates whether personality inferences and attributes from interpersonal theories of behaviour (like attitude and emotion) further augment the modelling of engaging behaviour. We present a novel pipeline to model engaging behaviour that incorporates the Big Five personality traits, the Interpersonal Circumplex (IPC), and the Triandis Theory of Interpersonal Behaviour (TIB). We extract first-person vision and physiological features from the MHHRI dataset and predict the Big Five personality traits using a Support Vector Machine. Subsequently, we empirically validate the advantage of incorporating personality in modelling engaging behaviour and present a novel method that effectively uses the IPC to obtain scores for a human's attitude and emotion from their Big Five traits. Finally, our results demonstrate that attitude and emotion are correlates of behaviour even in human-robot interactions, as suggested by the TIB for human-human interactions. Furthermore, incorporating the IPC and the Big Five traits helps generate behavioural inferences that supplement the engaging behaviour prediction, thus enriching the pipeline. Engagement modelling has a wide range of applications in domains like online learning platforms, assistive robotics, and intelligent conversational agents. Practitioners can also use this work in cognitive modelling and psychology to find more complex and subtle relations between humans' behaviour and personality traits, and discover new dynamics of the human psyche. The code will be made available at:
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0285749