Efficient data use in incremental actor-critic algorithms
详细信息 查看全文
- 作者：Yuhu Cheng chengyuhu@163.comAuthor Vitae ; Huanting Feng fhtcumt@163.comAuthor Vitae ; Xuesong Wang ; wangxuesongcumt@163.comAuthor Vitae
- 关键词：Actor&ndash ; critic ; Reinforcement learning ; Incremental least-squares temporal difference ; Recursive least-squares temporal difference ; Policy evaluation ; Function approximation
- 刊名：Neurocomputing
- 出版年：2013
- 出版时间：20 September, 2013
- 年：2013
- 卷：116
- 期：Complete
- 页码：346-354
- 全文大小：793 K
文摘
Actor-critic (AC) reinforcement learning methods are on-line approximations to policy iterations and have wide application in solving large-scale Markov decision and high-dimensional learning control problems. In order to overcome data inefficiency of incremental AC algorithms based on temporal difference (AC-TD), two new incremental AC algorithms (i.e., AC-RLSTD and AC-iLSTD) are proposed by applying a recursive least-squares TD (RLSTD(¦Ë)) algorithm and an incremental least-squares TD (iLSTD(¦Ë)) algorithm to the Critic evaluation, which can make more efficient use of data than TD. The Critic estimates a value-function using the RLSTD(¦Ë) or iLSTD(¦Ë) algorithm and the Actor updates the policy based on a regular gradient obtained by the TD error. The improvement in learning evaluation efficiency of the Critic will contribute to the improvement in policy learning performance of the Actor. Simulation results on the learning control of an inverted pendulum and a mountain-car problem illustrate the effectiveness of the two proposed AC algorithms in comparison to the AC-TD algorithm. In addition the AC-iLSTD, using a greedy selection mechanism, can perform much better than the AC-iLSTD using a random selection mechanism. In the simulation, the effect of different parameter settings of the eligibility trace on the learning performance of AC algorithms is analyzed. Furthermore, it is found that different initial values of the variance matrix in the AC-RLSTD algorithm should be chosen appropriately to obtain better performance for different learning problems.