Loading…
Multimodal modeling of collaborative problem-solving facets in triads
Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feed...
Saved in:
Published in: | User modeling and user-adapted interaction 2021-09, Vol.31 (4), p.713-751 |
---|---|
Main Authors: | , , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that this one cites Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
cited_by | cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3 |
---|---|
cites | cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3 |
container_end_page | 751 |
container_issue | 4 |
container_start_page | 713 |
container_title | User modeling and user-adapted interaction |
container_volume | 31 |
creator | Stewart, Angela E. B. Keirn, Zachary D’Mello, Sidney K. |
description | Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes. |
doi_str_mv | 10.1007/s11257-021-09290-y |
format | article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2569280112</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2569280112</sourcerecordid><originalsourceid>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</originalsourceid><addsrcrecordid>eNp9kE1LAzEQhoMoWKt_wNOC5-gkabKbo5T6ARUveg7ZfJQt6aYmu4X-e1NX8OZlZhje953hQeiWwD0BqB8yIZTXGCjBIKkEfDxDM8JrhgmT5BzNynaBSSOaS3SV8xaKSdRyhlZvYxi6XbQ6VKW60PWbKvrKxBB0G5MeuoOr9im2we1wjuFwEnht3JCrrq-G1Gmbr9GF1yG7m98-R59Pq4_lC16_P78uH9fYMCIHzJgwBgyjwmnPa25BCwGNFUxz3nrCGXFat0BbCQva2po5cJaVSQvPvWVzdDflloe-RpcHtY1j6stJRbmQtIHCoajopDIp5pycV_vU7XQ6KgLqhEtNuFTBpX5wqWMxscmUi7jfuPQX_Y_rG8wxbsk</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2569280112</pqid></control><display><type>article</type><title>Multimodal modeling of collaborative problem-solving facets in triads</title><source>Business Source Ultimate</source><source>ABI/INFORM Global</source><source>Springer Nature</source><creator>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</creator><creatorcontrib>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</creatorcontrib><description>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</description><identifier>ISSN: 0924-1868</identifier><identifier>EISSN: 1573-1391</identifier><identifier>DOI: 10.1007/s11257-021-09290-y</identifier><language>eng</language><publisher>Dordrecht: Springer Netherlands</publisher><subject>Automatic speech recognition ; Classifiers ; Collaboration ; Computer Science ; Construction ; Context ; Coordination ; Linguistics ; Machine learning ; Management of Computing and Information Systems ; Model accuracy ; Modelling ; Multimedia Information Systems ; Negotiations ; Problem solving ; Recreation ; User Interfaces and Human Computer Interaction ; Videoconferencing ; Visual tasks</subject><ispartof>User modeling and user-adapted interaction, 2021-09, Vol.31 (4), p.713-751</ispartof><rights>The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature 2021</rights><rights>The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature 2021.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</citedby><cites>FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</cites><orcidid>0000-0002-6004-9266</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.proquest.com/docview/2569280112/fulltextPDF?pq-origsite=primo$$EPDF$$P50$$Gproquest$$H</linktopdf><linktohtml>$$Uhttps://www.proquest.com/docview/2569280112?pq-origsite=primo$$EHTML$$P50$$Gproquest$$H</linktohtml><link.rule.ids>314,780,784,11688,27924,27925,36060,44363,74895</link.rule.ids></links><search><creatorcontrib>Stewart, Angela E. B.</creatorcontrib><creatorcontrib>Keirn, Zachary</creatorcontrib><creatorcontrib>D’Mello, Sidney K.</creatorcontrib><title>Multimodal modeling of collaborative problem-solving facets in triads</title><title>User modeling and user-adapted interaction</title><addtitle>User Model User-Adap Inter</addtitle><description>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</description><subject>Automatic speech recognition</subject><subject>Classifiers</subject><subject>Collaboration</subject><subject>Computer Science</subject><subject>Construction</subject><subject>Context</subject><subject>Coordination</subject><subject>Linguistics</subject><subject>Machine learning</subject><subject>Management of Computing and Information Systems</subject><subject>Model accuracy</subject><subject>Modelling</subject><subject>Multimedia Information Systems</subject><subject>Negotiations</subject><subject>Problem solving</subject><subject>Recreation</subject><subject>User Interfaces and Human Computer Interaction</subject><subject>Videoconferencing</subject><subject>Visual tasks</subject><issn>0924-1868</issn><issn>1573-1391</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>M0C</sourceid><recordid>eNp9kE1LAzEQhoMoWKt_wNOC5-gkabKbo5T6ARUveg7ZfJQt6aYmu4X-e1NX8OZlZhje953hQeiWwD0BqB8yIZTXGCjBIKkEfDxDM8JrhgmT5BzNynaBSSOaS3SV8xaKSdRyhlZvYxi6XbQ6VKW60PWbKvrKxBB0G5MeuoOr9im2we1wjuFwEnht3JCrrq-G1Gmbr9GF1yG7m98-R59Pq4_lC16_P78uH9fYMCIHzJgwBgyjwmnPa25BCwGNFUxz3nrCGXFat0BbCQva2po5cJaVSQvPvWVzdDflloe-RpcHtY1j6stJRbmQtIHCoajopDIp5pycV_vU7XQ6KgLqhEtNuFTBpX5wqWMxscmUi7jfuPQX_Y_rG8wxbsk</recordid><startdate>20210901</startdate><enddate>20210901</enddate><creator>Stewart, Angela E. B.</creator><creator>Keirn, Zachary</creator><creator>D’Mello, Sidney K.</creator><general>Springer Netherlands</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7SC</scope><scope>7WY</scope><scope>7WZ</scope><scope>7XB</scope><scope>87Z</scope><scope>88G</scope><scope>8AL</scope><scope>8AO</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>8FL</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BEZIV</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FRNLG</scope><scope>FYUFA</scope><scope>F~G</scope><scope>GHDGH</scope><scope>GNUQQ</scope><scope>HCIFZ</scope><scope>JQ2</scope><scope>K60</scope><scope>K6~</scope><scope>K7-</scope><scope>L.-</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>M0C</scope><scope>M0N</scope><scope>M2M</scope><scope>P5Z</scope><scope>P62</scope><scope>PQBIZ</scope><scope>PQBZA</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PSYQQ</scope><scope>Q9U</scope><orcidid>https://orcid.org/0000-0002-6004-9266</orcidid></search><sort><creationdate>20210901</creationdate><title>Multimodal modeling of collaborative problem-solving facets in triads</title><author>Stewart, Angela E. B. ; Keirn, Zachary ; D’Mello, Sidney K.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Automatic speech recognition</topic><topic>Classifiers</topic><topic>Collaboration</topic><topic>Computer Science</topic><topic>Construction</topic><topic>Context</topic><topic>Coordination</topic><topic>Linguistics</topic><topic>Machine learning</topic><topic>Management of Computing and Information Systems</topic><topic>Model accuracy</topic><topic>Modelling</topic><topic>Multimedia Information Systems</topic><topic>Negotiations</topic><topic>Problem solving</topic><topic>Recreation</topic><topic>User Interfaces and Human Computer Interaction</topic><topic>Videoconferencing</topic><topic>Visual tasks</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Stewart, Angela E. B.</creatorcontrib><creatorcontrib>Keirn, Zachary</creatorcontrib><creatorcontrib>D’Mello, Sidney K.</creatorcontrib><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Computer and Information Systems Abstracts</collection><collection>ProQuest_ABI/INFORM Collection</collection><collection>ABI/INFORM Global (PDF only)</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection</collection><collection>Psychology Database (Alumni)</collection><collection>Computing Database (Alumni Edition)</collection><collection>ProQuest Pharma Collection</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>Advanced Technologies & Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Business Premium Collection</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>Business Premium Collection (Alumni)</collection><collection>Health Research Premium Collection</collection><collection>ABI/INFORM Global (Corporate)</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Central Student</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Computer Science Collection</collection><collection>ProQuest Business Collection (Alumni Edition)</collection><collection>ProQuest Business Collection</collection><collection>Computer Science Database</collection><collection>ABI/INFORM Professional Advanced</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>ABI/INFORM Global</collection><collection>Computing Database</collection><collection>Psychology Database (ProQuest)</collection><collection>Advanced Technologies & Aerospace Database</collection><collection>ProQuest Advanced Technologies & Aerospace Collection</collection><collection>One Business (ProQuest)</collection><collection>ProQuest One Business (Alumni)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest One Psychology</collection><collection>ProQuest Central Basic</collection><jtitle>User modeling and user-adapted interaction</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Stewart, Angela E. B.</au><au>Keirn, Zachary</au><au>D’Mello, Sidney K.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Multimodal modeling of collaborative problem-solving facets in triads</atitle><jtitle>User modeling and user-adapted interaction</jtitle><stitle>User Model User-Adap Inter</stitle><date>2021-09-01</date><risdate>2021</risdate><volume>31</volume><issue>4</issue><spage>713</spage><epage>751</epage><pages>713-751</pages><issn>0924-1868</issn><eissn>1573-1391</eissn><abstract>Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.</abstract><cop>Dordrecht</cop><pub>Springer Netherlands</pub><doi>10.1007/s11257-021-09290-y</doi><tpages>39</tpages><orcidid>https://orcid.org/0000-0002-6004-9266</orcidid></addata></record> |
fulltext | fulltext |
identifier | ISSN: 0924-1868 |
ispartof | User modeling and user-adapted interaction, 2021-09, Vol.31 (4), p.713-751 |
issn | 0924-1868 1573-1391 |
language | eng |
recordid | cdi_proquest_journals_2569280112 |
source | Business Source Ultimate; ABI/INFORM Global; Springer Nature |
subjects | Automatic speech recognition Classifiers Collaboration Computer Science Construction Context Coordination Linguistics Machine learning Management of Computing and Information Systems Model accuracy Modelling Multimedia Information Systems Negotiations Problem solving Recreation User Interfaces and Human Computer Interaction Videoconferencing Visual tasks |
title | Multimodal modeling of collaborative problem-solving facets in triads |
url | http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-06T05%3A32%3A52IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Multimodal%20modeling%20of%20collaborative%20problem-solving%20facets%20in%20triads&rft.jtitle=User%20modeling%20and%20user-adapted%20interaction&rft.au=Stewart,%20Angela%20E.%20B.&rft.date=2021-09-01&rft.volume=31&rft.issue=4&rft.spage=713&rft.epage=751&rft.pages=713-751&rft.issn=0924-1868&rft.eissn=1573-1391&rft_id=info:doi/10.1007/s11257-021-09290-y&rft_dat=%3Cproquest_cross%3E2569280112%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c319t-336cc0c326eaf575d0a6608d63a55bf1531eaab02b9042bd73e0ed32bda6f5fd3%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2569280112&rft_id=info:pmid/&rfr_iscdi=true |