Dealing with Non-Stationarity in MARL via Trust-Region Decomposition
Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL). One of the reasons is the policy changes of agents during the learning process. Some existing works have discussed various consequences caused by non-stationarity with several kinds of measurement indicato...
Đã lưu trong:
| Những tác giả chính: | , , , , |
|---|---|
| Định dạng: | Bài viết |
| Được phát hành: |
2021
|
| Những chủ đề: | |
| Truy cập trực tuyến: | https://arxiv.org/abs/2102.10616 http://arxiv.org/abs/2102.10616 |
| Các nhãn: |
Không có thẻ, Là người đầu tiên thẻ bản ghi này!
|
| _version_ | 1879173543367802880 |
|---|---|
| author | Li, Wenhao Wang, Xiangfeng Jin, Bo Sheng, Junjie Zha, Hongyuan |
| author_browse | Jin, Bo Li, Wenhao Sheng, Junjie Wang, Xiangfeng Zha, Hongyuan |
| author_facet | Li, Wenhao Wang, Xiangfeng Jin, Bo Sheng, Junjie Zha, Hongyuan |
| author_sort | Li, Wenhao |
| building | arXiv |
| collection | Acceso abierto |
| description | Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL). One of the reasons is the policy changes of agents during the learning process. Some existing works have discussed various consequences caused by non-stationarity with several kinds of measurement indicators. This makes the objectives or goals of existing algorithms are inevitably inconsistent and disparate. In this paper, we introduce a novel notion, the $δ$-measurement, to explicitly measure the non-stationarity of a policy sequence, which can be further proved to be bounded by the KL-divergence of consecutive joint policies. A straightforward but highly non-trivial way is to control the joint policies' divergence, which is difficult to estimate accurately by imposing the trust-region constraint on the joint policy. Although it has lower computational complexity to decompose the joint policy and impose trust-region constraints on the factorized policies, simple policy factorization like mean-field approximation will lead to more considerable policy divergence, which can be considered as the trust-region decomposition dilemma. We model the joint policy as a pairwise Markov random field and propose a trust-region decomposition network (TRD-Net) based on message passing to estimate the joint policy divergence more accurately. The Multi-Agent Mirror descent policy algorithm with Trust region decomposition, called MAMT, is established by adjusting the trust-region of the local policies adaptively in an end-to-end manner. MAMT can approximately constrain the consecutive joint policies' divergence to satisfy $δ$-stationarity and alleviate the non-stationarity problem. Our method can bring noticeable and stable performance improvement compared with baselines in cooperative tasks of different complexity. |
| first_indexed | 2026-10-15T23:58:54Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-2102.10616</identifier><datestamp>2022-02-11</datestamp>
<dc:title>Dealing with Non-Stationarity in MARL via Trust-Region Decomposition</dc:title>
<dc:creator>Li, Wenhao</dc:creator>
<dc:creator>Wang, Xiangfeng</dc:creator>
<dc:creator>Jin, Bo</dc:creator>
<dc:creator>Sheng, Junjie</dc:creator>
<dc:creator>Zha, Hongyuan</dc:creator>
<dc:subject>Machine Learning</dc:subject>
<dc:subject>Computer Science and Game Theory</dc:subject>
<dc:subject>Multiagent Systems</dc:subject>
<dc:description>Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL). One of the reasons is the policy changes of agents during the learning process. Some existing works have discussed various consequences caused by non-stationarity with several kinds of measurement indicators. This makes the objectives or goals of existing algorithms are inevitably inconsistent and disparate. In this paper, we introduce a novel notion, the $δ$-measurement, to explicitly measure the non-stationarity of a policy sequence, which can be further proved to be bounded by the KL-divergence of consecutive joint policies. A straightforward but highly non-trivial way is to control the joint policies' divergence, which is difficult to estimate accurately by imposing the trust-region constraint on the joint policy. Although it has lower computational complexity to decompose the joint policy and impose trust-region constraints on the factorized policies, simple policy factorization like mean-field approximation will lead to more considerable policy divergence, which can be considered as the trust-region decomposition dilemma. We model the joint policy as a pairwise Markov random field and propose a trust-region decomposition network (TRD-Net) based on message passing to estimate the joint policy divergence more accurately. The Multi-Agent Mirror descent policy algorithm with Trust region decomposition, called MAMT, is established by adjusting the trust-region of the local policies adaptively in an end-to-end manner. MAMT can approximately constrain the consecutive joint policies' divergence to satisfy $δ$-stationarity and alleviate the non-stationarity problem. Our method can bring noticeable and stable performance improvement compared with baselines in cooperative tasks of different complexity. </dc:description>
<dc:description>39 pages, 23 figures, ICLR 2022 Camera Ready</dc:description>
<dc:date>2021-02-21</dc:date>
<dc:date>2022-02-10</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/2102.10616</dc:identifier>
<dc:identifier>http://arxiv.org/abs/2102.10616</dc:identifier>
</oai_dc:dc> |
| id | arxiv-2102.10616 |
| institution | arXiv |
| last_indexed | 2026-10-15T23:58:54Z |
| publishDate | 2021 |
| publishDateRange | 2021 |
| publishDateSort | 2021 |
| record_format | oai_dc |
| spelling | arxiv-2102.106162022-02-11 Dealing with Non-Stationarity in MARL via Trust-Region Decomposition Li, Wenhao Wang, Xiangfeng Jin, Bo Sheng, Junjie Zha, Hongyuan Machine Learning Computer Science and Game Theory Multiagent Systems Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL). One of the reasons is the policy changes of agents during the learning process. Some existing works have discussed various consequences caused by non-stationarity with several kinds of measurement indicators. This makes the objectives or goals of existing algorithms are inevitably inconsistent and disparate. In this paper, we introduce a novel notion, the $δ$-measurement, to explicitly measure the non-stationarity of a policy sequence, which can be further proved to be bounded by the KL-divergence of consecutive joint policies. A straightforward but highly non-trivial way is to control the joint policies' divergence, which is difficult to estimate accurately by imposing the trust-region constraint on the joint policy. Although it has lower computational complexity to decompose the joint policy and impose trust-region constraints on the factorized policies, simple policy factorization like mean-field approximation will lead to more considerable policy divergence, which can be considered as the trust-region decomposition dilemma. We model the joint policy as a pairwise Markov random field and propose a trust-region decomposition network (TRD-Net) based on message passing to estimate the joint policy divergence more accurately. The Multi-Agent Mirror descent policy algorithm with Trust region decomposition, called MAMT, is established by adjusting the trust-region of the local policies adaptively in an end-to-end manner. MAMT can approximately constrain the consecutive joint policies' divergence to satisfy $δ$-stationarity and alleviate the non-stationarity problem. Our method can bring noticeable and stable performance improvement compared with baselines in cooperative tasks of different complexity. 39 pages, 23 figures, ICLR 2022 Camera Ready 2021-02-21 2022-02-10 text https://arxiv.org/abs/2102.10616 http://arxiv.org/abs/2102.10616 |
| spellingShingle | Machine Learning Computer Science and Game Theory Multiagent Systems Li, Wenhao Wang, Xiangfeng Jin, Bo Sheng, Junjie Zha, Hongyuan Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title | Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title_full | Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title_fullStr | Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title_full_unstemmed | Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title_short | Dealing with Non-Stationarity in MARL via Trust-Region Decomposition |
| title_sort | dealing with non stationarity in marl via trust region decomposition |
| topic | Machine Learning Computer Science and Game Theory Multiagent Systems |
| url | https://arxiv.org/abs/2102.10616 http://arxiv.org/abs/2102.10616 |
| work_keys_str_mv | AT liwenhao dealingwithnonstationarityinmarlviatrustregiondecomposition AT wangxiangfeng dealingwithnonstationarityinmarlviatrustregiondecomposition AT jinbo dealingwithnonstationarityinmarlviatrustregiondecomposition AT shengjunjie dealingwithnonstationarityinmarlviatrustregiondecomposition AT zhahongyuan dealingwithnonstationarityinmarlviatrustregiondecomposition |