Towards Dialogue Modeling Beyond Text

In this paper, we model aspects of communication beyond the words that are said. Specifically, we aim to detect interruptions and active listening events, which are important elements in any dialogue. We build a dataset with fine-grained annotations for each category and train multimodal models that...

Full description

Saved in:
Bibliographic Details
Main Authors: Wu, Tongzi, Zhou, Yuhao, Ling, Wang, Yang, Hojin, Veloso, Joana, Sun, Lin, Huang, Ruixin, Guimaraes, Norberto, Sanner, Scott
Format: Conference Proceeding
Language:English
Subjects:
Annotations
Dialogue
Machine Learning
Multimodality
Neural networks
Oral communication
Signal processing
Speech recognition
Transformers
Video Processing
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we model aspects of communication beyond the words that are said. Specifically, we aim to detect interruptions and active listening events, which are important elements in any dialogue. We build a dataset with fine-grained annotations for each category and train multimodal models that take into account all channels in a digital conversation, that is, the video, the audio, and the text. Our experiments show that multimodality is a necessary component in modeling the complexity of the non-textual components of the conversation as different artifacts require different modalities to capture effectively.
ISSN:2379-190X
DOI:10.1109/ICASSP49357.2023.10095598