Loading…

AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficien...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-11
Main Authors: Mou, Xinyi, Liang, Jingcong, Lin, Jiayu, Zhang, Xinnong, Liu, Xiawei, Yang, Shiyue, Ye, Rong, Chen, Lei, Kuang, Haoyu, Huang, Xuanjing, Wei, Zhongyu
Format: Article
Language:English
Subjects:
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Mou, Xinyi
Liang, Jingcong
Lin, Jiayu
Zhang, Xinnong
Liu, Xiawei
Yang, Shiyue
Ye, Rong
Chen, Lei
Kuang, Haoyu
Huang, Xuanjing
Wei, Zhongyu
description Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_3121367365</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3121367365</sourcerecordid><originalsourceid>FETCH-proquest_journals_31213673653</originalsourceid><addsrcrecordid>eNqNy0sKwjAUheEgCBbtHi44LrSJbcWZiqLgrE6lhHCbpsZE83D9FnEBjs7g_86EJJSxIluvKJ2R1Pshz3Na1bQsWUJuW4kmNGg8bmCHRvQP7u7KSGisUFzD2QTUWo1KINgOLtzIyCXC9-gh9M5G2X-d4yKoN0Ij0HCnrF-Qace1x_S3c7I8Hq77U_Z09hXRh3aw0ZkxtaygBatqVpXsP_UBEkJD6w</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3121367365</pqid></control><display><type>article</type><title>AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</title><source>ProQuest Publicly Available Content database</source><creator>Mou, Xinyi ; Liang, Jingcong ; Lin, Jiayu ; Zhang, Xinnong ; Liu, Xiawei ; Yang, Shiyue ; Ye, Rong ; Chen, Lei ; Kuang, Haoyu ; Huang, Xuanjing ; Wei, Zhongyu</creator><creatorcontrib>Mou, Xinyi ; Liang, Jingcong ; Lin, Jiayu ; Zhang, Xinnong ; Liu, Xiawei ; Yang, Shiyue ; Ye, Rong ; Chen, Lei ; Kuang, Haoyu ; Huang, Xuanjing ; Wei, Zhongyu</creatorcontrib><description>Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Autonomous navigation ; Benchmarks ; Complexity ; Large language models ; Reasoning</subject><ispartof>arXiv.org, 2024-11</ispartof><rights>2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/3121367365?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25753,37012,44590</link.rule.ids></links><search><creatorcontrib>Mou, Xinyi</creatorcontrib><creatorcontrib>Liang, Jingcong</creatorcontrib><creatorcontrib>Lin, Jiayu</creatorcontrib><creatorcontrib>Zhang, Xinnong</creatorcontrib><creatorcontrib>Liu, Xiawei</creatorcontrib><creatorcontrib>Yang, Shiyue</creatorcontrib><creatorcontrib>Ye, Rong</creatorcontrib><creatorcontrib>Chen, Lei</creatorcontrib><creatorcontrib>Kuang, Haoyu</creatorcontrib><creatorcontrib>Huang, Xuanjing</creatorcontrib><creatorcontrib>Wei, Zhongyu</creatorcontrib><title>AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</title><title>arXiv.org</title><description>Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.</description><subject>Autonomous navigation</subject><subject>Benchmarks</subject><subject>Complexity</subject><subject>Large language models</subject><subject>Reasoning</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNqNy0sKwjAUheEgCBbtHi44LrSJbcWZiqLgrE6lhHCbpsZE83D9FnEBjs7g_86EJJSxIluvKJ2R1Pshz3Na1bQsWUJuW4kmNGg8bmCHRvQP7u7KSGisUFzD2QTUWo1KINgOLtzIyCXC9-gh9M5G2X-d4yKoN0Ij0HCnrF-Qace1x_S3c7I8Hq77U_Z09hXRh3aw0ZkxtaygBatqVpXsP_UBEkJD6w</recordid><startdate>20241123</startdate><enddate>20241123</enddate><creator>Mou, Xinyi</creator><creator>Liang, Jingcong</creator><creator>Lin, Jiayu</creator><creator>Zhang, Xinnong</creator><creator>Liu, Xiawei</creator><creator>Yang, Shiyue</creator><creator>Ye, Rong</creator><creator>Chen, Lei</creator><creator>Kuang, Haoyu</creator><creator>Huang, Xuanjing</creator><creator>Wei, Zhongyu</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20241123</creationdate><title>AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</title><author>Mou, Xinyi ; Liang, Jingcong ; Lin, Jiayu ; Zhang, Xinnong ; Liu, Xiawei ; Yang, Shiyue ; Ye, Rong ; Chen, Lei ; Kuang, Haoyu ; Huang, Xuanjing ; Wei, Zhongyu</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_31213673653</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Autonomous navigation</topic><topic>Benchmarks</topic><topic>Complexity</topic><topic>Large language models</topic><topic>Reasoning</topic><toplevel>online_resources</toplevel><creatorcontrib>Mou, Xinyi</creatorcontrib><creatorcontrib>Liang, Jingcong</creatorcontrib><creatorcontrib>Lin, Jiayu</creatorcontrib><creatorcontrib>Zhang, Xinnong</creatorcontrib><creatorcontrib>Liu, Xiawei</creatorcontrib><creatorcontrib>Yang, Shiyue</creatorcontrib><creatorcontrib>Ye, Rong</creatorcontrib><creatorcontrib>Chen, Lei</creatorcontrib><creatorcontrib>Kuang, Haoyu</creatorcontrib><creatorcontrib>Huang, Xuanjing</creatorcontrib><creatorcontrib>Wei, Zhongyu</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>ProQuest Publicly Available Content database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Mou, Xinyi</au><au>Liang, Jingcong</au><au>Lin, Jiayu</au><au>Zhang, Xinnong</au><au>Liu, Xiawei</au><au>Yang, Shiyue</au><au>Ye, Rong</au><au>Chen, Lei</au><au>Kuang, Haoyu</au><au>Huang, Xuanjing</au><au>Wei, Zhongyu</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</atitle><jtitle>arXiv.org</jtitle><date>2024-11-23</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2024-11
issn 2331-8422
language eng
recordid cdi_proquest_journals_3121367365
source ProQuest Publicly Available Content database
subjects Autonomous navigation
Benchmarks
Complexity
Large language models
Reasoning
title AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T04%3A57%3A38IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=AgentSense:%20Benchmarking%20Social%20Intelligence%20of%20Language%20Agents%20through%20Interactive%20Scenarios&rft.jtitle=arXiv.org&rft.au=Mou,%20Xinyi&rft.date=2024-11-23&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E3121367365%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_31213673653%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=3121367365&rft_id=info:pmid/&rfr_iscdi=true