结合记忆网络和动态折扣系数的深度Q网络

钟榛 闫启帅

摘 要：深度强化学习立足于解决环境交互问题，实现智能体的连续序列决策。传统强化学习算法基于马尔科夫决策过程，未来的状态仅与当前的状态有关，忽略了序列决策过程中记忆对当前决策的重要影响。此外，奖励的折扣系数为固定值，难以描述不同训练阶段当前奖励值与未来奖励期望对当前决策的动态影响。通过结合深度Q网络和循环记忆网络的神经网络模型，将序列决策的长期记忆加入决策过程，同时设置动态折扣系数，对不同训练阶段的深度Q网络模型赋予适当的折扣系数，从而加速了深度Q网络模型收敛并提高了其性能。

关键词：深度强化学习;记忆神经网络;深度Q网络;动态折扣系数

中图分类号：TN925;TP273 文献标识码：A 文章编号：1003-5168（2021）22-0034-04

Abstract： Deep reinforcement learning is based on solving the problem of interaction with the environment to realize the continuous sequence decision-making of agents. The traditional reinforcement learning algorithm is based on Markov decision-making process， and the future state is only related to the current state， while ignoring the important influence of memory on the current decision-making in the sequential decision-making process. In addition， the discount coefficient of reward is fixed， which is difficult to describe the dynamic impact of current reward value and future reward expectation on current decision-making in different training stages. By combining the neural network model of deep Q network and circular memory network， the long-term memory of sequential decision-making is added to the decision-making process， and the dynamic discount coefficient is set to give appropriate discount coefficient to the deep Q network model in different training stages， so as to accelerate the convergence of deep Q network model and improve its performance.

Keywords： deep reinforcement learning;memory neural network; deep Q network; dynamic discount factor

強化学习算法着眼于智能体的连续序列决策，为智能机器人、无人驾驶等理论提供了强有力的理论支持。在智能体与环境交互过程中，强化学习算法实现了环境状态转移时的行为决策。因为环境状态通过人工设计的状态指标进行描述，所以强化学习算法的性能非常依赖于环境状态设计的优劣。复杂交互场景的环境状态指标不仅难以人工描述设计，而且数量庞大，大大增加了工作量，导致难以大规模应用推广。为了破解这一难题，2013年，MNIH等人提出深度Q网络（Deep Q-Network，DQN）模型，利用卷积神经网络逼近状态-行为的值函数，实现图像到行为决策的端到端输入、输出[1]。DQN不仅在Atari的2 600款游戏中超越以往算法，还在49款游戏中超越了专业游戏测试员，成为强化学习的里程碑式研究成果[2]。……