Loading…

UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios

Multiagent reinforcement learning methods, such as VDN, QMIX, and QTRAN, that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multiagent scenarios, the number of agents and the size of the action s...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2023-04, Vol.34 (4), p.2093-2104
Main Authors: Chai, Jiajun, Li, Weifan, Zhu, Yuanheng, Zhao, Dongbin, Ma, Zhe, Sun, Kewu, Ding, Jishiyu
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63
cites cdi_FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63
container_end_page 2104
container_issue 4
container_start_page 2093
container_title IEEE transaction on neural networks and learning systems
container_volume 34
creator Chai, Jiajun
Li, Weifan
Zhu, Yuanheng
Zhao, Dongbin
Ma, Zhe
Sun, Kewu
Ding, Jishiyu
description Multiagent reinforcement learning methods, such as VDN, QMIX, and QTRAN, that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multiagent scenarios, the number of agents and the size of the action set actually vary over time. We call these unshaped scenarios, and the methods mentioned above fail in performing satisfyingly. In this article, we propose a new method, called Unshaped Networks for Multiagent Systems (UNMAS), which adapts to the number and size changes in multiagent systems. We propose the self-weighting mixing network to factorize the joint action-value. Its adaption to the change in agent number is attributed to the nonlinear mapping from each-agent Q value to the joint action-value with individual weights. Besides, in order to address the change in an action set, each agent constructs an individual action-value network that is composed of two streams to evaluate the constant environment-oriented subset and the varying unit-oriented subset. We evaluate UNMAS on various StarCraft II micromanagement scenarios and compare the results with several state-of-the-art MARL algorithms. The superiority of UNMAS is demonstrated by its highest winning rates especially on the most difficult scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative behaviors, while other MARL algorithms fail. Animated demonstrations and source code are provided in https://sites.google.com/view/unmas .
doi_str_mv 10.1109/TNNLS.2021.3105869
format article
fullrecord <record><control><sourceid>proquest_ieee_</sourceid><recordid>TN_cdi_proquest_journals_2795807864</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9525046</ieee_id><sourcerecordid>2795807864</sourcerecordid><originalsourceid>FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63</originalsourceid><addsrcrecordid>eNpdkEtPAjEQgBujEYL8AU3MJl68gG13-_JmiK8ImAgk3pqyTLGEfdjumvjv3RXkYC_Tmflm0n4InRM8JASrm_l0Op4NKaZkGBPMJFdHqEsJpwMaS3l8uIv3DuqHsMHN4ZjxRJ2iTpwkHCc46aKXxXRyN7uNJvW2cmYNeRW9gctt4VPI2mwMxucuX0dNKVrk4cOUsIpGRVGCN5X7gmiWQm68K8IZOrFmG6C_jz20eLifj54G49fH59HdeJDGjFQDtqJcCU5FyuhSpQpbkExgSXBbETgmqSCKWAvNF-yqQQGEYJJYlWBjedxD17u9pS8-awiVzlxIYbs1ORR10JRxoSRjpEWv_qGbovZ58zpNhWISC8mThqI7KvVFCB6sLr3LjP_WBOvWtv61rVvbem-7Gbrcr66XGawOI39uG-BiBzgAOLQVowwnPP4By4KBLQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2795807864</pqid></control><display><type>article</type><title>UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios</title><source>IEEE Xplore (Online service)</source><creator>Chai, Jiajun ; Li, Weifan ; Zhu, Yuanheng ; Zhao, Dongbin ; Ma, Zhe ; Sun, Kewu ; Ding, Jishiyu</creator><creatorcontrib>Chai, Jiajun ; Li, Weifan ; Zhu, Yuanheng ; Zhao, Dongbin ; Ma, Zhe ; Sun, Kewu ; Ding, Jishiyu</creatorcontrib><description>Multiagent reinforcement learning methods, such as VDN, QMIX, and QTRAN, that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multiagent scenarios, the number of agents and the size of the action set actually vary over time. We call these unshaped scenarios, and the methods mentioned above fail in performing satisfyingly. In this article, we propose a new method, called Unshaped Networks for Multiagent Systems (UNMAS), which adapts to the number and size changes in multiagent systems. We propose the self-weighting mixing network to factorize the joint action-value. Its adaption to the change in agent number is attributed to the nonlinear mapping from each-agent Q value to the joint action-value with individual weights. Besides, in order to address the change in an action set, each agent constructs an individual action-value network that is composed of two streams to evaluate the constant environment-oriented subset and the varying unit-oriented subset. We evaluate UNMAS on various StarCraft II micromanagement scenarios and compare the results with several state-of-the-art MARL algorithms. The superiority of UNMAS is demonstrated by its highest winning rates especially on the most difficult scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative behaviors, while other MARL algorithms fail. Animated demonstrations and source code are provided in https://sites.google.com/view/unmas .</description><identifier>ISSN: 2162-237X</identifier><identifier>EISSN: 2162-2388</identifier><identifier>DOI: 10.1109/TNNLS.2021.3105869</identifier><identifier>PMID: 34460404</identifier><identifier>CODEN: ITNNAL</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Algorithms ; Centralized training with decentralized execution (CTDE) ; Learning systems ; Marl ; Multi-agent systems ; multiagent ; Multiagent systems ; Reinforcement ; Reinforcement learning ; Semantics ; Source code ; StarCraft II ; Sun ; Task analysis ; Training</subject><ispartof>IEEE transaction on neural networks and learning systems, 2023-04, Vol.34 (4), p.2093-2104</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63</citedby><cites>FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63</cites><orcidid>0000-0001-5384-423X ; 0000-0001-8218-9633 ; 0000-0002-7611-064X</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9525046$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,27924,27925,54796</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/34460404$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Chai, Jiajun</creatorcontrib><creatorcontrib>Li, Weifan</creatorcontrib><creatorcontrib>Zhu, Yuanheng</creatorcontrib><creatorcontrib>Zhao, Dongbin</creatorcontrib><creatorcontrib>Ma, Zhe</creatorcontrib><creatorcontrib>Sun, Kewu</creatorcontrib><creatorcontrib>Ding, Jishiyu</creatorcontrib><title>UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios</title><title>IEEE transaction on neural networks and learning systems</title><addtitle>TNNLS</addtitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><description>Multiagent reinforcement learning methods, such as VDN, QMIX, and QTRAN, that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multiagent scenarios, the number of agents and the size of the action set actually vary over time. We call these unshaped scenarios, and the methods mentioned above fail in performing satisfyingly. In this article, we propose a new method, called Unshaped Networks for Multiagent Systems (UNMAS), which adapts to the number and size changes in multiagent systems. We propose the self-weighting mixing network to factorize the joint action-value. Its adaption to the change in agent number is attributed to the nonlinear mapping from each-agent Q value to the joint action-value with individual weights. Besides, in order to address the change in an action set, each agent constructs an individual action-value network that is composed of two streams to evaluate the constant environment-oriented subset and the varying unit-oriented subset. We evaluate UNMAS on various StarCraft II micromanagement scenarios and compare the results with several state-of-the-art MARL algorithms. The superiority of UNMAS is demonstrated by its highest winning rates especially on the most difficult scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative behaviors, while other MARL algorithms fail. Animated demonstrations and source code are provided in https://sites.google.com/view/unmas .</description><subject>Algorithms</subject><subject>Centralized training with decentralized execution (CTDE)</subject><subject>Learning systems</subject><subject>Marl</subject><subject>Multi-agent systems</subject><subject>multiagent</subject><subject>Multiagent systems</subject><subject>Reinforcement</subject><subject>Reinforcement learning</subject><subject>Semantics</subject><subject>Source code</subject><subject>StarCraft II</subject><subject>Sun</subject><subject>Task analysis</subject><subject>Training</subject><issn>2162-237X</issn><issn>2162-2388</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNpdkEtPAjEQgBujEYL8AU3MJl68gG13-_JmiK8ImAgk3pqyTLGEfdjumvjv3RXkYC_Tmflm0n4InRM8JASrm_l0Op4NKaZkGBPMJFdHqEsJpwMaS3l8uIv3DuqHsMHN4ZjxRJ2iTpwkHCc46aKXxXRyN7uNJvW2cmYNeRW9gctt4VPI2mwMxucuX0dNKVrk4cOUsIpGRVGCN5X7gmiWQm68K8IZOrFmG6C_jz20eLifj54G49fH59HdeJDGjFQDtqJcCU5FyuhSpQpbkExgSXBbETgmqSCKWAvNF-yqQQGEYJJYlWBjedxD17u9pS8-awiVzlxIYbs1ORR10JRxoSRjpEWv_qGbovZ58zpNhWISC8mThqI7KvVFCB6sLr3LjP_WBOvWtv61rVvbem-7Gbrcr66XGawOI39uG-BiBzgAOLQVowwnPP4By4KBLQ</recordid><startdate>20230401</startdate><enddate>20230401</enddate><creator>Chai, Jiajun</creator><creator>Li, Weifan</creator><creator>Zhu, Yuanheng</creator><creator>Zhao, Dongbin</creator><creator>Ma, Zhe</creator><creator>Sun, Kewu</creator><creator>Ding, Jishiyu</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QP</scope><scope>7QQ</scope><scope>7QR</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>JG9</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0001-5384-423X</orcidid><orcidid>https://orcid.org/0000-0001-8218-9633</orcidid><orcidid>https://orcid.org/0000-0002-7611-064X</orcidid></search><sort><creationdate>20230401</creationdate><title>UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios</title><author>Chai, Jiajun ; Li, Weifan ; Zhu, Yuanheng ; Zhao, Dongbin ; Ma, Zhe ; Sun, Kewu ; Ding, Jishiyu</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Algorithms</topic><topic>Centralized training with decentralized execution (CTDE)</topic><topic>Learning systems</topic><topic>Marl</topic><topic>Multi-agent systems</topic><topic>multiagent</topic><topic>Multiagent systems</topic><topic>Reinforcement</topic><topic>Reinforcement learning</topic><topic>Semantics</topic><topic>Source code</topic><topic>StarCraft II</topic><topic>Sun</topic><topic>Task analysis</topic><topic>Training</topic><toplevel>online_resources</toplevel><creatorcontrib>Chai, Jiajun</creatorcontrib><creatorcontrib>Li, Weifan</creatorcontrib><creatorcontrib>Zhu, Yuanheng</creatorcontrib><creatorcontrib>Zhao, Dongbin</creatorcontrib><creatorcontrib>Ma, Zhe</creatorcontrib><creatorcontrib>Sun, Kewu</creatorcontrib><creatorcontrib>Ding, Jishiyu</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005–Present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998–Present</collection><collection>IEEE/IET Electronic Library</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Calcium &amp; Calcified Tissue Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Chemoreception Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical &amp; Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology &amp; Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transaction on neural networks and learning systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Chai, Jiajun</au><au>Li, Weifan</au><au>Zhu, Yuanheng</au><au>Zhao, Dongbin</au><au>Ma, Zhe</au><au>Sun, Kewu</au><au>Ding, Jishiyu</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios</atitle><jtitle>IEEE transaction on neural networks and learning systems</jtitle><stitle>TNNLS</stitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><date>2023-04-01</date><risdate>2023</risdate><volume>34</volume><issue>4</issue><spage>2093</spage><epage>2104</epage><pages>2093-2104</pages><issn>2162-237X</issn><eissn>2162-2388</eissn><coden>ITNNAL</coden><abstract>Multiagent reinforcement learning methods, such as VDN, QMIX, and QTRAN, that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multiagent scenarios, the number of agents and the size of the action set actually vary over time. We call these unshaped scenarios, and the methods mentioned above fail in performing satisfyingly. In this article, we propose a new method, called Unshaped Networks for Multiagent Systems (UNMAS), which adapts to the number and size changes in multiagent systems. We propose the self-weighting mixing network to factorize the joint action-value. Its adaption to the change in agent number is attributed to the nonlinear mapping from each-agent Q value to the joint action-value with individual weights. Besides, in order to address the change in an action set, each agent constructs an individual action-value network that is composed of two streams to evaluate the constant environment-oriented subset and the varying unit-oriented subset. We evaluate UNMAS on various StarCraft II micromanagement scenarios and compare the results with several state-of-the-art MARL algorithms. The superiority of UNMAS is demonstrated by its highest winning rates especially on the most difficult scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative behaviors, while other MARL algorithms fail. Animated demonstrations and source code are provided in https://sites.google.com/view/unmas .</abstract><cop>United States</cop><pub>IEEE</pub><pmid>34460404</pmid><doi>10.1109/TNNLS.2021.3105869</doi><tpages>12</tpages><orcidid>https://orcid.org/0000-0001-5384-423X</orcidid><orcidid>https://orcid.org/0000-0001-8218-9633</orcidid><orcidid>https://orcid.org/0000-0002-7611-064X</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 2162-237X
ispartof IEEE transaction on neural networks and learning systems, 2023-04, Vol.34 (4), p.2093-2104
issn 2162-237X
2162-2388
language eng
recordid cdi_proquest_journals_2795807864
source IEEE Xplore (Online service)
subjects Algorithms
Centralized training with decentralized execution (CTDE)
Learning systems
Marl
Multi-agent systems
multiagent
Multiagent systems
Reinforcement
Reinforcement learning
Semantics
Source code
StarCraft II
Sun
Task analysis
Training
title UNMAS: Multiagent Reinforcement Learning for Unshaped Cooperative Scenarios
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T00%3A21%3A21IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_ieee_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=UNMAS:%20Multiagent%20Reinforcement%20Learning%20for%20Unshaped%20Cooperative%20Scenarios&rft.jtitle=IEEE%20transaction%20on%20neural%20networks%20and%20learning%20systems&rft.au=Chai,%20Jiajun&rft.date=2023-04-01&rft.volume=34&rft.issue=4&rft.spage=2093&rft.epage=2104&rft.pages=2093-2104&rft.issn=2162-237X&rft.eissn=2162-2388&rft.coden=ITNNAL&rft_id=info:doi/10.1109/TNNLS.2021.3105869&rft_dat=%3Cproquest_ieee_%3E2795807864%3C/proquest_ieee_%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c351t-5d2697627c52b9c90fe85708107c527031c7191ffe216fd976ee77581f940af63%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2795807864&rft_id=info:pmid/34460404&rft_ieee_id=9525046&rfr_iscdi=true