Loading…

Multimodal modeling of collaborative problem-solving facets in triads

Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feed...

Full description

Saved in:
Bibliographic Details
Published in:User modeling and user-adapted interaction 2021-09, Vol.31 (4), p.713-751
Main Authors: Stewart, Angela E. B., Keirn, Zachary, D’Mello, Sidney K.
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3
cites cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3
container_end_page 751
container_issue 4
container_start_page 713
container_title User modeling and user-adapted interaction
container_volume 31
creator Stewart, Angela E. B.
Keirn, Zachary
D’Mello, Sidney K.
description Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.
doi_str_mv 10.1007/s11257-021-09290-y
format article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2569280112</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2569280112</sourcerecordid><originalsourceid>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</originalsourceid><addsrcrecordid>eNp9kE1LAzEQhoMoWKt_wNOC5-gkabKbo5T6ARUveg7ZfJQt6aYmu4X-e1NX8OZlZhje953hQeiWwD0BqB8yIZTXGCjBIKkEfDxDM8JrhgmT5BzNynaBSSOaS3SV8xaKSdRyhlZvYxi6XbQ6VKW60PWbKvrKxBB0G5MeuoOr9im2we1wjuFwEnht3JCrrq-G1Gmbr9GF1yG7m98-R59Pq4_lC16_P78uH9fYMCIHzJgwBgyjwmnPa25BCwGNFUxz3nrCGXFat0BbCQva2po5cJaVSQvPvWVzdDflloe-RpcHtY1j6stJRbmQtIHCoajopDIp5pycV_vU7XQ6KgLqhEtNuFTBpX5wqWMxscmUi7jfuPQX_Y_rG8wxbsk</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2569280112</pqid></control><display><type>article</type><title>Multimodal modeling of collaborative problem-solving facets in triads</title><source>Business Source Ultimate</source><source>ABI/INFORM Global</source><source>Springer Nature</source><creator>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</creator><creatorcontrib>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</creatorcontrib><description>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</description><identifier>ISSN: 0924-1868</identifier><identifier>EISSN: 1573-1391</identifier><identifier>DOI: 10.1007/s11257-021-09290-y</identifier><language>eng</language><publisher>Dordrecht: Springer Netherlands</publisher><subject>Automatic speech recognition ; Classifiers ; Collaboration ; Computer Science ; Construction ; Context ; Coordination ; Linguistics ; Machine learning ; Management of Computing and Information Systems ; Model accuracy ; Modelling ; Multimedia Information Systems ; Negotiations ; Problem solving ; Recreation ; User Interfaces and Human Computer Interaction ; Videoconferencing ; Visual tasks</subject><ispartof>User modeling and user-adapted interaction, 2021-09, Vol.31 (4), p.713-751</ispartof><rights>The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature 2021</rights><rights>The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature 2021.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</citedby><cites>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</cites><orcidid>0000-0002-6004-9266</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.proquest.com/docview/2569280112/fulltextPDF?pq-origsite=primo$$EPDF$$P50$$Gproquest$$H</linktopdf><linktohtml>$$Uhttps://www.proquest.com/docview/2569280112?pq-origsite=primo$$EHTML$$P50$$Gproquest$$H</linktohtml><link.rule.ids>314,780,784,11688,27924,27925,36060,44363,74895</link.rule.ids></links><search><creatorcontrib>Stewart, Angela E. B.</creatorcontrib><creatorcontrib>Keirn, Zachary</creatorcontrib><creatorcontrib>D’Mello, Sidney K.</creatorcontrib><title>Multimodal modeling of collaborative problem-solving facets in triads</title><title>User modeling and user-adapted interaction</title><addtitle>User Model User-Adap Inter</addtitle><description>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</description><subject>Automatic speech recognition</subject><subject>Classifiers</subject><subject>Collaboration</subject><subject>Computer Science</subject><subject>Construction</subject><subject>Context</subject><subject>Coordination</subject><subject>Linguistics</subject><subject>Machine learning</subject><subject>Management of Computing and Information Systems</subject><subject>Model accuracy</subject><subject>Modelling</subject><subject>Multimedia Information Systems</subject><subject>Negotiations</subject><subject>Problem solving</subject><subject>Recreation</subject><subject>User Interfaces and Human Computer Interaction</subject><subject>Videoconferencing</subject><subject>Visual tasks</subject><issn>0924-1868</issn><issn>1573-1391</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>M0C</sourceid><recordid>eNp9kE1LAzEQhoMoWKt_wNOC5-gkabKbo5T6ARUveg7ZfJQt6aYmu4X-e1NX8OZlZhje953hQeiWwD0BqB8yIZTXGCjBIKkEfDxDM8JrhgmT5BzNynaBSSOaS3SV8xaKSdRyhlZvYxi6XbQ6VKW60PWbKvrKxBB0G5MeuoOr9im2we1wjuFwEnht3JCrrq-G1Gmbr9GF1yG7m98-R59Pq4_lC16_P78uH9fYMCIHzJgwBgyjwmnPa25BCwGNFUxz3nrCGXFat0BbCQva2po5cJaVSQvPvWVzdDflloe-RpcHtY1j6stJRbmQtIHCoajopDIp5pycV_vU7XQ6KgLqhEtNuFTBpX5wqWMxscmUi7jfuPQX_Y_rG8wxbsk</recordid><startdate>20210901</startdate><enddate>20210901</enddate><creator>Stewart, Angela E. B.</creator><creator>Keirn, Zachary</creator><creator>D’Mello, Sidney K.</creator><general>Springer Netherlands</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7SC</scope><scope>7WY</scope><scope>7WZ</scope><scope>7XB</scope><scope>87Z</scope><scope>88G</scope><scope>8AL</scope><scope>8AO</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>8FL</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BEZIV</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FRNLG</scope><scope>FYUFA</scope><scope>F~G</scope><scope>GHDGH</scope><scope>GNUQQ</scope><scope>HCIFZ</scope><scope>JQ2</scope><scope>K60</scope><scope>K6~</scope><scope>K7-</scope><scope>L.-</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>M0C</scope><scope>M0N</scope><scope>M2M</scope><scope>P5Z</scope><scope>P62</scope><scope>PQBIZ</scope><scope>PQBZA</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PSYQQ</scope><scope>Q9U</scope><orcidid>https://orcid.org/0000-0002-6004-9266</orcidid></search><sort><creationdate>20210901</creationdate><title>Multimodal modeling of collaborative problem-solving facets in triads</title><author>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Automatic speech recognition</topic><topic>Classifiers</topic><topic>Collaboration</topic><topic>Computer Science</topic><topic>Construction</topic><topic>Context</topic><topic>Coordination</topic><topic>Linguistics</topic><topic>Machine learning</topic><topic>Management of Computing and Information Systems</topic><topic>Model accuracy</topic><topic>Modelling</topic><topic>Multimedia Information Systems</topic><topic>Negotiations</topic><topic>Problem solving</topic><topic>Recreation</topic><topic>User Interfaces and Human Computer Interaction</topic><topic>Videoconferencing</topic><topic>Visual tasks</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Stewart, Angela E. B.</creatorcontrib><creatorcontrib>Keirn, Zachary</creatorcontrib><creatorcontrib>D’Mello, Sidney K.</creatorcontrib><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Computer and Information Systems Abstracts</collection><collection>ProQuest_ABI/INFORM Collection</collection><collection>ABI/INFORM Global (PDF only)</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection</collection><collection>Psychology Database (Alumni)</collection><collection>Computing Database (Alumni Edition)</collection><collection>ProQuest Pharma Collection</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>Advanced Technologies &amp; Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Business Premium Collection</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>Business Premium Collection (Alumni)</collection><collection>Health Research Premium Collection</collection><collection>ABI/INFORM Global (Corporate)</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Central Student</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Computer Science Collection</collection><collection>ProQuest Business Collection (Alumni Edition)</collection><collection>ProQuest Business Collection</collection><collection>Computer Science Database</collection><collection>ABI/INFORM Professional Advanced</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>ABI/INFORM Global</collection><collection>Computing Database</collection><collection>Psychology Database (ProQuest)</collection><collection>Advanced Technologies &amp; Aerospace Database</collection><collection>ProQuest Advanced Technologies &amp; Aerospace Collection</collection><collection>One Business (ProQuest)</collection><collection>ProQuest One Business (Alumni)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest One Psychology</collection><collection>ProQuest Central Basic</collection><jtitle>User modeling and user-adapted interaction</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Stewart, Angela E. B.</au><au>Keirn, Zachary</au><au>D’Mello, Sidney K.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Multimodal modeling of collaborative problem-solving facets in triads</atitle><jtitle>User modeling and user-adapted interaction</jtitle><stitle>User Model User-Adap Inter</stitle><date>2021-09-01</date><risdate>2021</risdate><volume>31</volume><issue>4</issue><spage>713</spage><epage>751</epage><pages>713-751</pages><issn>0924-1868</issn><eissn>1573-1391</eissn><abstract>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</abstract><cop>Dordrecht</cop><pub>Springer Netherlands</pub><doi>10.1007/s11257-021-09290-y</doi><tpages>39</tpages><orcidid>https://orcid.org/0000-0002-6004-9266</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 0924-1868
ispartof User modeling and user-adapted interaction, 2021-09, Vol.31 (4), p.713-751
issn 0924-1868
1573-1391
language eng
recordid cdi_proquest_journals_2569280112
source Business Source Ultimate; ABI/INFORM Global; Springer Nature
subjects Automatic speech recognition
Classifiers
Collaboration
Computer Science
Construction
Context
Coordination
Linguistics
Machine learning
Management of Computing and Information Systems
Model accuracy
Modelling
Multimedia Information Systems
Negotiations
Problem solving
Recreation
User Interfaces and Human Computer Interaction
Videoconferencing
Visual tasks
title Multimodal modeling of collaborative problem-solving facets in triads
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-06T05%3A32%3A52IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Multimodal%20modeling%20of%20collaborative%20problem-solving%20facets%20in%20triads&rft.jtitle=User%20modeling%20and%20user-adapted%20interaction&rft.au=Stewart,%20Angela%20E.%20B.&rft.date=2021-09-01&rft.volume=31&rft.issue=4&rft.spage=713&rft.epage=751&rft.pages=713-751&rft.issn=0924-1868&rft.eissn=1573-1391&rft_id=info:doi/10.1007/s11257-021-09290-y&rft_dat=%3Cproquest_cross%3E2569280112%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2569280112&rft_id=info:pmid/&rfr_iscdi=true