Reinforcement learning solution for HJB equation arising in constrained optimal control problem

详细信息    查看全文

• 作者：Biao Luo^a ; ^{biao.luo@hotmail.com" class="auth_mail" title="E-mail the corresponding author} ; Huai-Ning Wu^b ; ^{whn@buaa.edu.cn" class="auth_mail" title="E-mail the corresponding author} ; Tingwen Huang^c ; ^{tingwen.huang@qatar.tamu.edu" class="auth_mail" title="E-mail the corresponding author} ; Derong Liu^d ; ^{derong@ustb.edu.cn" class="auth_mail" title="E-mail the corresponding author}
• 关键词：Constrained optimal control ; Data-based ; Off-policy reinforcement learning ; Hamilton&ndash ; Jacobi&ndash ; Bellman equation ; The method of weighted residuals
• 刊名：Neural Networks
• 出版年：2015
• 出版时间：November 2015
• 年：2015
• 卷：71
• 期：Complete
• 页码：150-158
• 全文大小：530 K

文摘

The constrained optimal control problem depends on the solution of the complicated Hamilton–Jacobi–Bellman equation (HJBE). In this paper, a data-based off-policy reinforcement learning (RL) method is proposed, which learns the solution of the HJBE and the optimal control policy from real system data. One important feature of the off-policy RL is that its policy evaluation can be realized with data generated by other behavior policies, not necessarily the target policy, which solves the insufficient exploration problem. The convergence of the off-policy RL is proved by demonstrating its equivalence to the successive approximation approach. Its implementation procedure is based on the actor–critic neural networks structure, where the function approximation is conducted with linearly independent basis functions. Subsequently, the convergence of the implementation procedure with function approximation is also proved. Finally, its effectiveness is verified through computer simulations.