多智能体分时跟踪学习.docVIP

多智能体分时跟踪学习# 傅波，陈鑫** （中南大学信息科学与工程学院，长沙 410083）  5 10 15 20 25 30 35 40 摘要：合作式多智能体强化学习在虽然许多领域取得了较大的应用，但是维数灾和通信问题 依然存在，且是当前的热点和难点。针对这两个问题，本文首先提出了合作式多智能体联合 状态独立动作 Q 学习方法(join-state single-action Q-learning, JSQ-learning)，降低合作学习中 联合动作空间至独立动作空间的维数，从而有效缓解了维数灾；并在此基础上引入了基于策 略估计的强化学习算法 (JSQ-learning based policy prediction，JSQL-PP)从而免去了合作多智 能体间的通信，以上两种算法的收敛性均得到了证明。其次，针对学习中联合状态独立动作 奖赏函数的实现进行了讨论，可采用逼近的方法获得；最后提出了合作式多智能体分时跟踪 学习框架(Time-Tacking Framework，TTF)，以保证合作学习的结果趋于较优，并将 JSQL-PP 算法引入该框架获得了合作式多智能体分时跟踪算法(JSQL-TT)。仿真结果与传统 Q 学习算 法比较表明了 JSQL-PP 算法以及所提分时框架的有效性和优越性。. 关键词：合作式多智能体系统；Q 学习；收敛性；分时跟踪；策略预测 中图分类号：TP181 Multi-agent Cooperative Learning with Time-Tacking Framework Fu Bo, Chen Xin (School of Information Science and Engineering, Central South University, ChangSha 410083) Abstract: Thought MAS has been applied in many fields, dimensions and the communications problem always are the big challenges to be faced and taken a breakthrough in the cooperative multi-agent system, especially in large-scale space. In this paper the multi-agent algorithms of join-state single-action Q-learning (JSQ-learning) and JSQ-learning based policy prediction (JSQL-PP) are proposed from original definition of reinforcement learning to reduce the dimensions of the large-scale space and learn without communications in MAS. And both convergences of two algorithms are verified. Then, a multi-agent cooperative learning framework is built to make every agent has chances to improve the cooperative strategy with time-tracking. The learning algorithm JSQL-PP is extended to an algorithm for learning a better policy in the proposed framework (JSQL-TT). Our new algorithm can be shown to get a better strategy rapidly in large-scale space in experiments of two agents dating in the grid word and three agents line up problem. And the comparison results of the Q-learning and JSQL-TT have shown the effectiveness and superiority of our time-tracking framework. Keywords: cooperative multi-agent system; Q-learning; con