电动汽车充电控制的深度增强学习优化方法
|
摘要
随着用电信息采集系统的广泛推广,数据驱动的机器学习方法在电力系统优化运行领域的应用已引起广泛关注。该文基于电网在线运行状态数据采集,采用竞争深度Q网络(dueling deep Q network,DDQN)结构的深度增强学习方法开展电动汽车充电控制优化。首先选取观测状态与执行动作,定义状态动作估值函数,其次针对动作和状态维度上的绝对数值相差过大的问题,采用DDQN的Q函数,引入ε-greedy策略、记忆存储单元以及批量梯度下降法进行神经网络的分层学习,然后基于DDQN训练后的神经网络,开展电动汽车充电控制的深度增强学习优化。最后,结合IEEE33节点扩展算例说明所提电动汽车充电控制优化方法在满足各类用户出行的充电需求条件下,实现合理消纳可再生能源发电。
With the wide application of data acquisition system, the data-driven machine learning methods play an important role on optimization decision in power system. In this paper, the deep reinforcement learning(DRL) method with dueling deep Q network(DDQN) structure was designed to optimize the charging control of electric vehicles based on operation data. Firstly, the observation state and control action were selected, and the state action estimation function was defined. Secondly, considering the large difference between action and state dimensions, the Q function of DDQN was adopted, and the ε-greedy strategy, memory storage unit and batch gradient descent method were introduced to hierarchical learning of neural networks. Then based on the neural network after DDQN training, the DRL optimization of charging control for electric vehicles was carried out. Finally, combined with the IEEE33 node expansion example, the charging control optimization method proposed in this paper can realize the effective consumption of renewable energy under the condition of satisfying the charging demand of various users.
With the wide application of data acquisition system, the data-driven machine learning methods play an important role on optimization decision in power system. In this paper, the deep reinforcement learning(DRL) method with dueling deep Q network(DDQN) structure was designed to optimize the charging control of electric vehicles based on operation data. Firstly, the observation state and control action were selected, and the state action estimation function was defined. Secondly, considering the large difference between action and state dimensions, the Q function of DDQN was adopted, and the ε-greedy strategy, memory storage unit and batch gradient descent method were introduced to hierarchical learning of neural networks. Then based on the neural network after DDQN training, the DRL optimization of charging control for electric vehicles was carried out. Finally, combined with the IEEE33 node expansion example, the charging control optimization method proposed in this paper can realize the effective consumption of renewable energy under the condition of satisfying the charging demand of various users.
引文
[1]高红均,刘俊勇,沈晓东,等.主动配电网最优潮流研究及其应用实例[J].中国电机工程学报,2017,37(6):1634-1645.Gao Hongjun,Liu Junyong,Shen Xiaodong,et al.Optimal power flow research in active distribution network and its application examples[J].Proceedings of the CSEE,2017,37(6):1634-1645(in Chinese).
[2]姚高瑞,雷雪姣,刘西林,等.促进风电消纳的供需互动合约机制与多市场购电策略[J].电力系统自动化,2019,43(7):113-124.Yao Gaorui,Lei Xuejiao,Liu Xilin,et al.Supply-demand interaction contract mechanism and multi-market electricity purchasing strategy for improving wind power accommodation[J].Automation of Electric Power Systems,2019,43(7):113-124(in Chinese).
[3]王锡凡,肖云鹏,王秀丽.新形势下电力系统供需互动问题研究及分析[J].中国电机工程学报,2014,34(29):5018-5028.Wang Xifan,Xiao Yunpeng,Wang Xiuli.Study and analysis on supply-demand interaction of power systems under new circumstances[J].Proceedings of the CSEE,2014,34(29):5018-5028(in Chinese).
[4]胡泽春,宋永华,徐智威,等.电动汽车接入电网的影响与利用[J].中国电机工程学报,2012,32(4):1-10.Hu Zechun,Song Yonghua,Xu Zhiwei,et al.Impacts and utilization of electric vehicles integration into power systems[J].Proceedings of the CSEE,2012,32(4):1-10(in Chinese).
[5]杨铮,彭思成,廖清芬,等.面向综合能源楼宇的电动汽车辅助服务方案[J].电网技术,2017,41(9):2831-2843.Yang Zheng,Peng Sicheng,Liao Qingfen,et al.Ancillary services provided by electric vehicles for building integrated energy system[J].Power System Technology,2017,41(9):2831-2843(in Chinese).
[6]杨楠,董邦天,黄禹,等.考虑不确定性和多主体博弈的增量配电网源网荷协同规划方法[J].中国电机工程学报,2019,39(9):2689-2701.Yang Nan,Dong Bangtian,Huang Yu,et al.Incremental distribution network source-load collaborative planning method considering uncertainty and multi-agent game[J].Proceedings of the CSEE,2019,39(9):2689-2701(in Chinese).
[7]高红均,刘俊勇,魏震波,等.主动配电网分层鲁棒规划模型及其求解方法[J].中国电机工程学报,2017,37(5):1389-1401.Gao Hongjun,Liu Junyong,Wei Zhenbo,et al.A bi-level robust planning model of active distribution network and its solution method[J].Proceedings of the CSEE,2017,37(5):1389-1401(in Chinese).
[8]李逐云,雷霞,邱少引,等.考虑“源-网-荷”三方利益的主动配电网协调规划[J].电网技术,2017,41(2):378-387.Li Zhuyun,Lei Xia,Qiu Shaoyin,et al.Coordinated planning of active distribution network considering“source-grid-load”benefits[J].Power System Technology,2017,41(2):378-387(in Chinese).
[9]吴巨爱,薛禹胜,谢东亮,等.电动汽车参与运行备用的能力评估及其仿真分析[J].电力系统自动化,2018,42(13):101-107.Wu Juai,Xue Yusheng,Xie Dongliang,et al.Evaluation and simulation analysis of reserve capability for electric vehicles[J].Automation of Electric Power Systems,2018,42(13):101-107(in Chinese).
[10]王继业,季知祥,史梦洁,等.智能配用电大数据需求分析与应用研究[J].中国电机工程学报,2015,35(8):1829-1836.Wang Jiye,Ji Zhixiang,Shi Mengjie,et al.Scenario analysis and application research on big data in smart power distribution and consumption systems[J].Proceedings of the CSEE,2015,35(8):1829-1836(in Chinese).
[11]Sanseverino E R,Silvestre M L D,Mineo L,et al.Amulti-agent system reinforcement learning based optimal power flow for islanded microgrids[C]//IEEE,International Conference on Environment and Electrical Engineering.IEEE,2016:1-6.
[12]Xiao S Z,Li Q,Tao Y U,et al.Consensus transfer q-learning for decentralized generation command dispatch based on virtual generation tribe[J].IEEE Transactions on Smart Grid,2016(99):1-1.
[13]Mnih V,Kavukcuoglu K,Silver D,et al.Playing Atari with deep reinforcement learning[J].Computer Science,2013.
[14]Mnih V,Kavukcuoglu K,Silver D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529.
[15]Wang Z,Schaul T,Hessel M,et al.Dueling network architectures for deep reinforcement learning[J].2015:1995-2003.
[16]Van Hasselt H,Guez A,Silver D.Deep reinforcement learning with double q-learning[J].arXiv:1509.06461,2015.
[17]刘威,张东霞,王新迎,等.基于深度强化学习的电网紧急控制策略研究[J].中国电机工程学报,2018,38(1):109-119.Liu Wei,Zhang Dongxia,Wang Xinying,et al.A decision making strategy for generating unit tripping under emergency circumstances based on deep reinforcement learning[J].Proceedings of the CSEE,2018,38(1):109-119(in Chinese).
[2]姚高瑞,雷雪姣,刘西林,等.促进风电消纳的供需互动合约机制与多市场购电策略[J].电力系统自动化,2019,43(7):113-124.Yao Gaorui,Lei Xuejiao,Liu Xilin,et al.Supply-demand interaction contract mechanism and multi-market electricity purchasing strategy for improving wind power accommodation[J].Automation of Electric Power Systems,2019,43(7):113-124(in Chinese).
[3]王锡凡,肖云鹏,王秀丽.新形势下电力系统供需互动问题研究及分析[J].中国电机工程学报,2014,34(29):5018-5028.Wang Xifan,Xiao Yunpeng,Wang Xiuli.Study and analysis on supply-demand interaction of power systems under new circumstances[J].Proceedings of the CSEE,2014,34(29):5018-5028(in Chinese).
[4]胡泽春,宋永华,徐智威,等.电动汽车接入电网的影响与利用[J].中国电机工程学报,2012,32(4):1-10.Hu Zechun,Song Yonghua,Xu Zhiwei,et al.Impacts and utilization of electric vehicles integration into power systems[J].Proceedings of the CSEE,2012,32(4):1-10(in Chinese).
[5]杨铮,彭思成,廖清芬,等.面向综合能源楼宇的电动汽车辅助服务方案[J].电网技术,2017,41(9):2831-2843.Yang Zheng,Peng Sicheng,Liao Qingfen,et al.Ancillary services provided by electric vehicles for building integrated energy system[J].Power System Technology,2017,41(9):2831-2843(in Chinese).
[6]杨楠,董邦天,黄禹,等.考虑不确定性和多主体博弈的增量配电网源网荷协同规划方法[J].中国电机工程学报,2019,39(9):2689-2701.Yang Nan,Dong Bangtian,Huang Yu,et al.Incremental distribution network source-load collaborative planning method considering uncertainty and multi-agent game[J].Proceedings of the CSEE,2019,39(9):2689-2701(in Chinese).
[7]高红均,刘俊勇,魏震波,等.主动配电网分层鲁棒规划模型及其求解方法[J].中国电机工程学报,2017,37(5):1389-1401.Gao Hongjun,Liu Junyong,Wei Zhenbo,et al.A bi-level robust planning model of active distribution network and its solution method[J].Proceedings of the CSEE,2017,37(5):1389-1401(in Chinese).
[8]李逐云,雷霞,邱少引,等.考虑“源-网-荷”三方利益的主动配电网协调规划[J].电网技术,2017,41(2):378-387.Li Zhuyun,Lei Xia,Qiu Shaoyin,et al.Coordinated planning of active distribution network considering“source-grid-load”benefits[J].Power System Technology,2017,41(2):378-387(in Chinese).
[9]吴巨爱,薛禹胜,谢东亮,等.电动汽车参与运行备用的能力评估及其仿真分析[J].电力系统自动化,2018,42(13):101-107.Wu Juai,Xue Yusheng,Xie Dongliang,et al.Evaluation and simulation analysis of reserve capability for electric vehicles[J].Automation of Electric Power Systems,2018,42(13):101-107(in Chinese).
[10]王继业,季知祥,史梦洁,等.智能配用电大数据需求分析与应用研究[J].中国电机工程学报,2015,35(8):1829-1836.Wang Jiye,Ji Zhixiang,Shi Mengjie,et al.Scenario analysis and application research on big data in smart power distribution and consumption systems[J].Proceedings of the CSEE,2015,35(8):1829-1836(in Chinese).
[11]Sanseverino E R,Silvestre M L D,Mineo L,et al.Amulti-agent system reinforcement learning based optimal power flow for islanded microgrids[C]//IEEE,International Conference on Environment and Electrical Engineering.IEEE,2016:1-6.
[12]Xiao S Z,Li Q,Tao Y U,et al.Consensus transfer q-learning for decentralized generation command dispatch based on virtual generation tribe[J].IEEE Transactions on Smart Grid,2016(99):1-1.
[13]Mnih V,Kavukcuoglu K,Silver D,et al.Playing Atari with deep reinforcement learning[J].Computer Science,2013.
[14]Mnih V,Kavukcuoglu K,Silver D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529.
[15]Wang Z,Schaul T,Hessel M,et al.Dueling network architectures for deep reinforcement learning[J].2015:1995-2003.
[16]Van Hasselt H,Guez A,Silver D.Deep reinforcement learning with double q-learning[J].arXiv:1509.06461,2015.
[17]刘威,张东霞,王新迎,等.基于深度强化学习的电网紧急控制策略研究[J].中国电机工程学报,2018,38(1):109-119.Liu Wei,Zhang Dongxia,Wang Xinying,et al.A decision making strategy for generating unit tripping under emergency circumstances based on deep reinforcement learning[J].Proceedings of the CSEE,2018,38(1):109-119(in Chinese).