基于探索感知思维深度强化学习的自动发电控制
|
摘要
减少碳排放的重要途径之一是大规模引入新能源。然而,随着大规模风、光、电动汽车等新能源和分布式能源接入复杂互联电网,给电网带来严重的随机扰动问题。该文从自动发电控制角度,探索了一种动作探索感知思维的深度强化学习算法,即DDQN-AD。通过将神经网络的预测机制作为强化学习的动作选择机制,同时引入具有动作探索感知思维的AD策略,将区域控制误差与碳排放作为综合奖励函数,来获取强随机环境下的最优控制策略,进而解决分布式能源大规模接入电网所带来的随机扰动问题。对改进的IEEE标准两区域LFC模型,以及多区域多能生态系统模型进行仿真,结果显示DDQN-AD与已有的多种智能算法相比,具有更优的动态性能和在线学习能力,能够获得区域最优控制,减少碳排放。
One of the important ways to reduce carbon emissions is to introduce new energy on a large scale. However,with the access of large-scale wind, light, electric vehicles and other new energy sources and distributed energy to the complex grid, it will bring serious random disturbance problems to the grid. From the perspective of automatic generation control, a deep reinforcement learning algorithm with action exploration awareness was explored, namely DDQN-AD. By using the prediction mechanism of deep neural network as the action selection mechanism of reinforcement learning, and introducing the AD strategy with exploration-awareness, the regional control error and carbon emission were used as comprehensive reward functions to obtain the optimal strategies in a variety of random environments. In order to solve the problem of random disturbance caused by the large-scale access of new energy and distributed energy to the power grid, the simulation of the improved IEEE standard two-area load frequency control model and the multi-region pluripotent ecosystem model was carried out. From the result, it shows that DDQN-AD has greater online learning ability and better dynamic performance compared with many existing intelligent algorithms, and it can obtain regional optimal control and reduce carbon emissions.
One of the important ways to reduce carbon emissions is to introduce new energy on a large scale. However,with the access of large-scale wind, light, electric vehicles and other new energy sources and distributed energy to the complex grid, it will bring serious random disturbance problems to the grid. From the perspective of automatic generation control, a deep reinforcement learning algorithm with action exploration awareness was explored, namely DDQN-AD. By using the prediction mechanism of deep neural network as the action selection mechanism of reinforcement learning, and introducing the AD strategy with exploration-awareness, the regional control error and carbon emission were used as comprehensive reward functions to obtain the optimal strategies in a variety of random environments. In order to solve the problem of random disturbance caused by the large-scale access of new energy and distributed energy to the power grid, the simulation of the improved IEEE standard two-area load frequency control model and the multi-region pluripotent ecosystem model was carried out. From the result, it shows that DDQN-AD has greater online learning ability and better dynamic performance compared with many existing intelligent algorithms, and it can obtain regional optimal control and reduce carbon emissions.
引文
[1]周孝信,陈树勇,鲁宗相,等.能源转型中我国新一代电力系统的技术特征[J].中国电机工程学报,2018,38(7):1893-1904.Zhou Xiaoxin,Chen Shuyong,Lu Zongxiang,et al.Technology features of the new generation power system in China[J].Proceedings of the CSEE,2018,38(7):1893-1904(in Chinese).
[2]李岚淼,李龙国,李乃稳.城市雾霾成因及危害研究进展[J].环境工程,2017,35(12):92-97,104.Li Lanmiao,Li Longguo,Li Naiwen.Studies review on cause and damage of urban haze[J].Environmental Engineering,2017,35(12):92-97,104(in Chinese).
[3]Yin Linfei,Yu Tao,Zhou Lü,et al.Artificial emotional reinforcement learning for automatic generation control of large-scale interconnected power grids[J].IET Generation,Transmission&Distribution,2017,11(9):2305-2313.
[4]Daneshfar F,Bevrani H.Load-frequency control:a GA-based multi-agent reinforcement learning[J].IETGeneration,Transmission&Distribution,2010,4(1):13-26.
[5]Polaj?er B,Petrun M,Ritonja J.Adaptation of load-frequency-control target values based on the covariances between area-control errors[J].IEEETransactions on Power Systems,2018,33(6):5865-5874.
[6]张孝顺,李清,余涛,等.基于协同一致性迁移Q学习算法的虚拟发电部落AGC功率动态分配[J].中国电机工程学报,2017,37(5):1455-1466.Zhang Xiaoshun,Li Qing,Yu Tao,et al.Collaborative consensus transfer Q-learning based dynamic generation dispatch of automatic generation control with virtual generation tribe[J].Proceedings of the CSEE,2017,37(5):1455-1466(in Chinese).
[7]Yu T,Wang H Z,Zhou B,et al.Multi-agent correlated equilibrium Q(?)learning for coordinated smart generation control of interconnected power grids[J].IEEETransactions on Power Systems,2015,30(4):1669-1679.
[8]李滨,吴思缘,阳育德,等.大型风电场的控制性能评价标准及储能控制策略研究[J].中国电机工程学报,2017,37(16):4691-4698.LiBin,Wu Siyuan,Yang Yude,et al.A research on the control performance standard and energy storage control strategy for large scale wind farms[J].Proceedings of the CSEE,2017,37(16):4691-4698(in Chinese).
[9]Zhang Yuan,Rahbari-Asr N,Chow M Y.A robust distributed system incremental cost estimation algorithm for smart grid economic dispatch with Communications information losses[J].Journal of Network and Computer Applications,2016(59):315-324.
[10]余涛,袁野.基于平均报酬模型全过程R(?)学习的互联电网CPS最优控制[J].电力系统自动化,2010,34(21):27-33.Yu Tao,Yuan Ye.An average reward model based whole process R(?)-learning for optimal CPS control[J].Automation of Electric Power Systems,2010,34(21):27-33(in Chinese).
[11]李海,张宁,康重庆,等.可再生能源消纳影响因素的贡献度分析方法[J].中国电机工程学报,2019,39(4):1009-1017.Li Hai,Zhang Ning,Kang Chongqing,et al.Analytics of contribution degree for renewable energy accommodation factors[J].Proceedings of the CSEE,2019,39(4):1009-1017(in Chinese).
[12]孙舶皓,汤涌,仲悟之,等.基于分布式模型预测控制的包含大规模风电集群互联系统超前频率控制策略[J].中国电机工程学报,2017,37(21):6291-6302.Sun Bohao,Tang Yong,Zhong Wuzhi,et al.Multi-area interconnected power system advanced frequency control strategy considering large scale wind power cluster integration based on DMPC[J].Proceedings of the CSEE,2017,37(21):6291-6302(in Chinese).
[13]Padhy B P,Tyagi B.Artificial neural network based multi area automatic generation control scheme for a competitive electricity market environment[C].International Conference on Power Systems.Kharagpur,India:IEEE,2009,pp,1-6.
[14]孟祥萍,薛昌飞,张化光.多区域互联电力系统的PI滑模负荷频率控制[J].中国电机工程学报,2001,21(3):6-11.Meng Xiangping,Xue Changfei,Zhang Huaguang.PIsliding mode load frequency control of multi-area power systems[J].Proceedings of the CSEE,2001,21(3):6-11(in Chinese).
[15]Moghadam R,Modares H.Resilient adaptive optimal control of distributed multi-agent systems using reinforcement learning[J].IET Control Theory&Applications,2018,12(16):2165-2174.
[16]Yin Linfei,Yu Tao,Zhang Xiaoshun,et al.Relaxed deep learning for real-time economic generation dispatch and control with unified time scale[J].Energy,2018(149):11-23.
[17]Xi Lei,Yu Tao,Yang Bo,et al.A wolf pack hunting strategy based virtual tribes control for automatic generation control of smart grid[J].Applied Energy,2016(178):198-211.
[18]张孝顺,余涛,唐捷.基于CEQ(?)多智能体协同学习的互联电网性能标准控制指令动态分配优化算法[J].电工技术学报,2016,31(8):125-133.Zhang Xiaoshun,Yu Tao,Tang Jie.Dynamic optimal allocation algorithm for control performance standard order of interconnected power grids using synergetic learning of multi-agent CEQ(?)[J].Transactions of China Electrotechnical Society,2016,31(8):125-133(in Chinese).
[19]席磊,柳浪,黄悦华,等.基于虚拟狼群策略的分层分布式自动发电控制[J].电力系统自动化,2018,42(16):65-72.Xi Lei,Liu Lang,Huang Yuehua,et al.Hierarchical and distributed control method for automatic generation control based on virtual wolf pack strategy[J].Automation of Electric Power System,2018,42(16):65-72(in Chinese).
[20]余涛,梁海华,周斌.基于R(λ)学习的孤岛微电网智能发电控制[J].电力系统保护与控制,2012,40(13):7-13.Yu Tao,Liang Haihua,Zhou Bin.Smart power generation control for microgrids islanded operation based on R(λ)learning[J].Power System Protection and Control,2012,40(13):7-13(in Chinese).
[21]包涛,张孝顺,余涛,等.反映实时供需互动的Stackelberg博弈模型及其强化学习求解[J].中国电机工程学报,2018,38(10):2947-2955.Bao Tao,Zhang Xiaoshun,Yu Tao,et al.A stackelberg game model of real-time supply-demand interaction and the solving method via reinforcement learning[J].Proceedings of the CSEE,2018,38(10):2947-2955(in Chinese).
[22]Xi Lei,Yu Tao,Yang Bo,et al.A novel multi-agent decentralized win or learn fast policy hill-climbing with eligibility trace algorithm for smart generation control of interconnected complex power grids[J].Energy Conversion and Management,2015(103):82-93.
[23]Xi Lei,Chen Jianfeng,Huang Yuehua,et al.Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel[J].Energy,2018(153):977-987.
[24]Matthysen R,Huybrechs D.Function approximation on arbitrary domains using Fourier extension frames[J].SIAM Journal on Numerical Analysis,2018,56(3):1360-1385.
[25]Do T T,Cheung N M.Embedding based on function approximation for large scale image search[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2018,40(3):626-638.
[26]Elfwing S,Uchibe E,Doya K.Sigmoid-weighted linear units for neural network function approximation in reinforcement learning[J].Neural Networks,2018(107):3-11.
[27]祁琛.神经网络中近似乘法器的设计[J].电子技术,2018(3):45-49,44.Qi Shen.The selection of plaintiff of environmental public interest litigation in China[J].Electronic Technology,2018(3):45-49,44(in Chinese).
[28]席磊,陈建峰,黄悦华,等.基于具有动作自寻优能力的深度强化学习的智能发电控制[J].中国科学:信息科学,2018,48(10):1430-1449.Xi Lei,Chen Jianfeng,Huang Yuehua,et al.Intelligent power generation control based on deep reinforcement learning with action self-optimization ability[J].Chinese Science:Information Science,2018,48(10):1430-1449(in Chinese).
[29]Mnih V,Kavukcuoglu K,Silver D,et al.Playing atari with deep reinforcement learning[J].arXiv:Learning,2013:1-9.
[30]Mnih V,Kavukcuoglu K,Silver D,et al.Human-level control through deep reinforcement learning[J].Nature,2015(518):529-533.
[31]Van Hasselt H,Guez A,Silver D.Deep reinforcement learning with double Q-learning[C]//National Conference on Artificial Intelligence,2016:2094-2100.
[32]Banerjee B,Kraemer L.Reinforcement learning with action discovery[C].Proceedings of the Adaptive and Learning Agents.Toronto,Canada.2010:30-37.
[33]Yu Tao,Zhou Bin,Chan K W,et al.Stochastic optimal relaxed automatic generation control in Non-Markov environment based on multi-step Q(?)learning[J].IEEETransactions on Power Systems,2011,26(3):1272-1282.
[34]张刚,张峰,张利,等.考虑碳排放交易的日前调度双阶段鲁棒优化模型[J].中国电机工程学报,2018,38(18):5490-5499.Zhang Gang,Zhang Feng,Zhang Li,et al.Two-stage robust optimization model of day-ahead scheduling considering carbon emissions trading[J].Proceedings of the CSEE,2018,38(18):5490-5499(in Chinese).
[35]Group W.Hydraulic turbine and turbine control models for system dynamic studies[J].IEEE Transactions on Power Systems,1992,7(1):167-179.
[2]李岚淼,李龙国,李乃稳.城市雾霾成因及危害研究进展[J].环境工程,2017,35(12):92-97,104.Li Lanmiao,Li Longguo,Li Naiwen.Studies review on cause and damage of urban haze[J].Environmental Engineering,2017,35(12):92-97,104(in Chinese).
[3]Yin Linfei,Yu Tao,Zhou Lü,et al.Artificial emotional reinforcement learning for automatic generation control of large-scale interconnected power grids[J].IET Generation,Transmission&Distribution,2017,11(9):2305-2313.
[4]Daneshfar F,Bevrani H.Load-frequency control:a GA-based multi-agent reinforcement learning[J].IETGeneration,Transmission&Distribution,2010,4(1):13-26.
[5]Polaj?er B,Petrun M,Ritonja J.Adaptation of load-frequency-control target values based on the covariances between area-control errors[J].IEEETransactions on Power Systems,2018,33(6):5865-5874.
[6]张孝顺,李清,余涛,等.基于协同一致性迁移Q学习算法的虚拟发电部落AGC功率动态分配[J].中国电机工程学报,2017,37(5):1455-1466.Zhang Xiaoshun,Li Qing,Yu Tao,et al.Collaborative consensus transfer Q-learning based dynamic generation dispatch of automatic generation control with virtual generation tribe[J].Proceedings of the CSEE,2017,37(5):1455-1466(in Chinese).
[7]Yu T,Wang H Z,Zhou B,et al.Multi-agent correlated equilibrium Q(?)learning for coordinated smart generation control of interconnected power grids[J].IEEETransactions on Power Systems,2015,30(4):1669-1679.
[8]李滨,吴思缘,阳育德,等.大型风电场的控制性能评价标准及储能控制策略研究[J].中国电机工程学报,2017,37(16):4691-4698.LiBin,Wu Siyuan,Yang Yude,et al.A research on the control performance standard and energy storage control strategy for large scale wind farms[J].Proceedings of the CSEE,2017,37(16):4691-4698(in Chinese).
[9]Zhang Yuan,Rahbari-Asr N,Chow M Y.A robust distributed system incremental cost estimation algorithm for smart grid economic dispatch with Communications information losses[J].Journal of Network and Computer Applications,2016(59):315-324.
[10]余涛,袁野.基于平均报酬模型全过程R(?)学习的互联电网CPS最优控制[J].电力系统自动化,2010,34(21):27-33.Yu Tao,Yuan Ye.An average reward model based whole process R(?)-learning for optimal CPS control[J].Automation of Electric Power Systems,2010,34(21):27-33(in Chinese).
[11]李海,张宁,康重庆,等.可再生能源消纳影响因素的贡献度分析方法[J].中国电机工程学报,2019,39(4):1009-1017.Li Hai,Zhang Ning,Kang Chongqing,et al.Analytics of contribution degree for renewable energy accommodation factors[J].Proceedings of the CSEE,2019,39(4):1009-1017(in Chinese).
[12]孙舶皓,汤涌,仲悟之,等.基于分布式模型预测控制的包含大规模风电集群互联系统超前频率控制策略[J].中国电机工程学报,2017,37(21):6291-6302.Sun Bohao,Tang Yong,Zhong Wuzhi,et al.Multi-area interconnected power system advanced frequency control strategy considering large scale wind power cluster integration based on DMPC[J].Proceedings of the CSEE,2017,37(21):6291-6302(in Chinese).
[13]Padhy B P,Tyagi B.Artificial neural network based multi area automatic generation control scheme for a competitive electricity market environment[C].International Conference on Power Systems.Kharagpur,India:IEEE,2009,pp,1-6.
[14]孟祥萍,薛昌飞,张化光.多区域互联电力系统的PI滑模负荷频率控制[J].中国电机工程学报,2001,21(3):6-11.Meng Xiangping,Xue Changfei,Zhang Huaguang.PIsliding mode load frequency control of multi-area power systems[J].Proceedings of the CSEE,2001,21(3):6-11(in Chinese).
[15]Moghadam R,Modares H.Resilient adaptive optimal control of distributed multi-agent systems using reinforcement learning[J].IET Control Theory&Applications,2018,12(16):2165-2174.
[16]Yin Linfei,Yu Tao,Zhang Xiaoshun,et al.Relaxed deep learning for real-time economic generation dispatch and control with unified time scale[J].Energy,2018(149):11-23.
[17]Xi Lei,Yu Tao,Yang Bo,et al.A wolf pack hunting strategy based virtual tribes control for automatic generation control of smart grid[J].Applied Energy,2016(178):198-211.
[18]张孝顺,余涛,唐捷.基于CEQ(?)多智能体协同学习的互联电网性能标准控制指令动态分配优化算法[J].电工技术学报,2016,31(8):125-133.Zhang Xiaoshun,Yu Tao,Tang Jie.Dynamic optimal allocation algorithm for control performance standard order of interconnected power grids using synergetic learning of multi-agent CEQ(?)[J].Transactions of China Electrotechnical Society,2016,31(8):125-133(in Chinese).
[19]席磊,柳浪,黄悦华,等.基于虚拟狼群策略的分层分布式自动发电控制[J].电力系统自动化,2018,42(16):65-72.Xi Lei,Liu Lang,Huang Yuehua,et al.Hierarchical and distributed control method for automatic generation control based on virtual wolf pack strategy[J].Automation of Electric Power System,2018,42(16):65-72(in Chinese).
[20]余涛,梁海华,周斌.基于R(λ)学习的孤岛微电网智能发电控制[J].电力系统保护与控制,2012,40(13):7-13.Yu Tao,Liang Haihua,Zhou Bin.Smart power generation control for microgrids islanded operation based on R(λ)learning[J].Power System Protection and Control,2012,40(13):7-13(in Chinese).
[21]包涛,张孝顺,余涛,等.反映实时供需互动的Stackelberg博弈模型及其强化学习求解[J].中国电机工程学报,2018,38(10):2947-2955.Bao Tao,Zhang Xiaoshun,Yu Tao,et al.A stackelberg game model of real-time supply-demand interaction and the solving method via reinforcement learning[J].Proceedings of the CSEE,2018,38(10):2947-2955(in Chinese).
[22]Xi Lei,Yu Tao,Yang Bo,et al.A novel multi-agent decentralized win or learn fast policy hill-climbing with eligibility trace algorithm for smart generation control of interconnected complex power grids[J].Energy Conversion and Management,2015(103):82-93.
[23]Xi Lei,Chen Jianfeng,Huang Yuehua,et al.Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel[J].Energy,2018(153):977-987.
[24]Matthysen R,Huybrechs D.Function approximation on arbitrary domains using Fourier extension frames[J].SIAM Journal on Numerical Analysis,2018,56(3):1360-1385.
[25]Do T T,Cheung N M.Embedding based on function approximation for large scale image search[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2018,40(3):626-638.
[26]Elfwing S,Uchibe E,Doya K.Sigmoid-weighted linear units for neural network function approximation in reinforcement learning[J].Neural Networks,2018(107):3-11.
[27]祁琛.神经网络中近似乘法器的设计[J].电子技术,2018(3):45-49,44.Qi Shen.The selection of plaintiff of environmental public interest litigation in China[J].Electronic Technology,2018(3):45-49,44(in Chinese).
[28]席磊,陈建峰,黄悦华,等.基于具有动作自寻优能力的深度强化学习的智能发电控制[J].中国科学:信息科学,2018,48(10):1430-1449.Xi Lei,Chen Jianfeng,Huang Yuehua,et al.Intelligent power generation control based on deep reinforcement learning with action self-optimization ability[J].Chinese Science:Information Science,2018,48(10):1430-1449(in Chinese).
[29]Mnih V,Kavukcuoglu K,Silver D,et al.Playing atari with deep reinforcement learning[J].arXiv:Learning,2013:1-9.
[30]Mnih V,Kavukcuoglu K,Silver D,et al.Human-level control through deep reinforcement learning[J].Nature,2015(518):529-533.
[31]Van Hasselt H,Guez A,Silver D.Deep reinforcement learning with double Q-learning[C]//National Conference on Artificial Intelligence,2016:2094-2100.
[32]Banerjee B,Kraemer L.Reinforcement learning with action discovery[C].Proceedings of the Adaptive and Learning Agents.Toronto,Canada.2010:30-37.
[33]Yu Tao,Zhou Bin,Chan K W,et al.Stochastic optimal relaxed automatic generation control in Non-Markov environment based on multi-step Q(?)learning[J].IEEETransactions on Power Systems,2011,26(3):1272-1282.
[34]张刚,张峰,张利,等.考虑碳排放交易的日前调度双阶段鲁棒优化模型[J].中国电机工程学报,2018,38(18):5490-5499.Zhang Gang,Zhang Feng,Zhang Li,et al.Two-stage robust optimization model of day-ahead scheduling considering carbon emissions trading[J].Proceedings of the CSEE,2018,38(18):5490-5499(in Chinese).
[35]Group W.Hydraulic turbine and turbine control models for system dynamic studies[J].IEEE Transactions on Power Systems,1992,7(1):167-179.