基于纳什均衡迁移学习的碳–能复合流自律优化
|
摘要
提出了一种全新的纳什均衡迁移学习算法,并应用于求解大规模电力系统分散式碳–能复合流自律优化.首次引入碳排放责任分摊机制,避免了碳排放责任的重复计算.将大规模电网分解成若干小型区域电网,每个小型区域电网被定义为一个智能体,通过纳什博弈实现分散式自律优化.智能体利用记忆矩阵对寻优知识进行存储,并通过多个个体与环境的反复交互实现记忆更新;采用状态–动作链对记忆矩阵进行降维,有效避免了"维数灾难".此外,基于相似度的迁移学习可以对历史任务知识进行高效提炼,提高了新任务寻优效率.IEEE 57和300节点系统仿真表明:所提算法非常适合求解大规模电网的碳–能复合流自律优化,在保证纳什均衡解质量的同时,明显加快寻优速度.
This paper proposes a novel Nash equilibrium inspired transfer learning(NETL) for decentralized selforganizing optimal carbon-energy combined-flow of large-scale power systems. A shared responsibility of carbon emission is firstly considered, such that a double counting of carbon emission can be eliminated. Moreover, the whole power system is partitioned to the multiple subsystems, in which each subsystem is treated as an agent. The Nash game is introduced to satisfy the self-organizing optimal operation of each agent. Every agent stores its knowledge by the memory matrix, and a group of individuals is employed by agents to update their memories by interactions with the environment. The associated state-action chains are adopted to handle the curse of dimension. Transfer learning mechanism can refine the knowledge of the prior tasks efficiently thus dramatically accelerating the new tasks. The simulation on IEEE 57-bus system and IEEE300-bus system verify that NETL is particularly geared to handle the self-organizing optimal carbon-energy combinedflow of large-scale power systems, which can ensure the quality of the Nash equilibrium solution as well as significantly accelerate the searching speed.
This paper proposes a novel Nash equilibrium inspired transfer learning(NETL) for decentralized selforganizing optimal carbon-energy combined-flow of large-scale power systems. A shared responsibility of carbon emission is firstly considered, such that a double counting of carbon emission can be eliminated. Moreover, the whole power system is partitioned to the multiple subsystems, in which each subsystem is treated as an agent. The Nash game is introduced to satisfy the self-organizing optimal operation of each agent. Every agent stores its knowledge by the memory matrix, and a group of individuals is employed by agents to update their memories by interactions with the environment. The associated state-action chains are adopted to handle the curse of dimension. Transfer learning mechanism can refine the knowledge of the prior tasks efficiently thus dramatically accelerating the new tasks. The simulation on IEEE 57-bus system and IEEE300-bus system verify that NETL is particularly geared to handle the self-organizing optimal carbon-energy combinedflow of large-scale power systems, which can ensure the quality of the Nash equilibrium solution as well as significantly accelerate the searching speed.
引文
[1]YANG B,YU T,SHU H C,et al.Robust sliding-mode control of wind energy conversion systems for optimal power extraction via nonlinear perturbation observers[J].Applied Energy,2018,210:711–723.
[2]YANG B,ZHANG X S,YU T,et al.Grouped grey wolf optimizer for maximum power point tracking of doubly-fed induction generator based wind turbine[J].Energy Conversion and Management,2017,133:427–443.
[3]YANG B,HU Y L,HUANG H Y,et al.Perturbation estimation based robust state feedback control for grid connected DFIG wind energy conversion system[J].International Journal of Hydrogen Energy,2017,42(33):20994–21005.
[4]LU Z G,FENG T,LI X P.Low-carbon emission economic power dispatch using the multi-objective bacterial colony chemotaxis optimization algorithm considering carbon capture power plant[J].International Journal of Electrical Power&Energy Systems,2013,53(1):106–112.
[5]BOOTHANDFORD M E,ABANADES J C,ANTHONY E J,et al.Carbon capture and storage update[J].Energy&Environmental Science,2014,7(1):130–189.
[6]ZHANG Xiaoshun,YU Tao.Stochastic optimal generation command dispatch of interconnected power grids based on improved multi-objective technique for order preference similar to an ideal solution-Q algorithm[J].Control Theory&Applications,2015,32(4):497–503.(张孝顺,余涛.互联电网自动发电控制功率分配的改进逼近于理想解的排序—Q多目标优化算法[J].控制理论与应用,2015,32(4):497–503.)
[7]ZHOU Tianrui,KANG Chongqing,XU Qianyao,et al.Preliminary theoretical investigation on power system carbon emission flow[J].Automation of Electric Power Systems,2012,36(7):38–43.(周天睿,康重庆,徐乾耀,等.电力系统碳排放流分析理论初探[J].电力系统自动化,2012,36(7):38–43.)
[8]ZHOU Tianrui,KANG Chongqing,XU qianyao,et al.Preliminary investigation on a method for carbon emission flow calculation of power system[J].Automation of Electric Power Systems,2012,36(11):44–49.(周天睿,康重庆,徐乾耀,等.电力系统碳排放流的计算方法初探[J].电力系统自动化,2012,36(11):44–49.)
[9]LI Baowei,HU Zechun,SONG Yonghua,et al.Principle and model for assessment on carbon emission intensity cased by electricity at consumer side[J].Power System Technology,2012,36(8):6–11.(李保卫,胡泽春,宋永华,等.用户侧电力碳排放强度的评估原则与模型[J].电网技术,2012,36(8):6–11.)
[10]ZHANG X,YU T,YANG B.Approximate ideal multi-objective solution Q(λ)learning for optimal carbon-energy combined-flow in multi-energy power systems[J].Energy Conversion&Management,2015,106:543–556.
[11]AMIN S M.Smart grid security,privacy,and resilient architectures:Opportunities and challenges[C]//Power and Energy Society General Meeting.San Diego:IEEE,2012,6:1–2.
[12]GIRI J,PARASHAR M,TREHERN J,et al.The situation room:control center analytics for enhanced situational awareness[J].IEEE Power&Energy Magazine,2012,10(5):24–39
[13]LENZEN M,MURRAY J,SACK F,et al.Shared producer and consumer responsibility–theory and practice[J].Ecological Economics,2007,61(1):27–42.
[14]KIM B H,BALDICK R.Coarse-grained distributed optimal power flow[J].IEEE Transactions on Power Systems,1997,12(2):932–939.
[15]COHENG.Auxiliary problem principle and decomposition of optimization problems[J].Journal of Optimization Theory&Applications,1980,32(3):277–305.
[16]LU¨Wei,SUI Ruirui,FENG Enmin.Predictor-corrector improvement of Newton method[J].Control Theory&Applications,2015,32(12):1620–1626.(吕巍,隋瑞瑞,冯恩民.改进的牛顿预测–校正格式[J].控制理论与应用,2015,32(12):1620–1626.)
[17]BOYD S,PARIKH N,CHU E,et al.Distributed optimization and statistical learning via the alternating direction method of multipliers[J].Foundations&Trends?Rin Machine Learning,2011,3(1):1–122.
[18]ERDELJAN A,CAPKO D,VUKMIROVIC S,et al.Distributed PSO algorithm for data model partitioning in power distribution systems[J].Journal of Applied Research&Technology,2014,16(5):947–957.
[19]LIANG C H,CHUNG C Y,WONG K P,et al.Parallel optimal reactive power flow based on cooperative co-evolutionary differential evolution and power system decomposition[J].IEEE Transactions on Power Systems,2007,22(1):249–257.
[20]YU T,LIU J,CHAN K W,et al.Distributed multi-step Q(λ)learning for optimal power flow of large-scale power grids[J].International Journal of Electrical Power&Energy Systems,2012,42(1):614–620.
[21]HU J,WELLMAN M P.Nash Q-learning for general-sum stochastic games[J].Journal of Machine Learning Research,2003,4(4):1039–1069.
[22]GREENWALD A,HALL K.Correlated-Q learning[C]//Proceedings of the 20th International Conference on Machine Learning.Washington,USA:AAAI Press,2003,8:242–249.
[23]HU Y,GAO Y,AN B.Accelerating multiagent reinforcement learning by equilibrium transfer[J].IEEE Transactions on Cybernetics,2015,45(7):1289–1302.
[24]TAYLOR M E,STONE P.Transfer learning for reinforcement learning domains:A survey[J].Journal of Machine Learning Research,2009,10(10):1633–1685.
[25]BLANCA G,MANFRED L.A consistent input-output formulation of shared producer and consumer responsibility[J].Economic Systems Research,2005,17(4):365–391.
[26]NASH J.Non-cooperative games[J].Annals of Mathematics.1951,54(3):286–295.
[27]WATKINS C J C H,DAYAN P.Technical note:Q-learning[J].Machine Learning,1992,8(3/4):279–292.
[28]GAMBARDELLA L M,DORIGO M.Ant-Q:A Reinforcement learning approach to the traveling salesman problem[C]//Proceedings of the 20th International Conference on Machine Learning.Tahoe City,California,USA:Morgan Kaufmann Publishers,1995,170(3):252–260.
[29]GOMES E R,KOWALCZYK R.Dynamic analysis of multiagent Qlearning withε-greedy exploration[C]//International Conference on Machine Learning.New York:ACM,2009,6:369–376.
[30]FERNANDO F,VELOSO M.Probabilistic policy reuse in a reinforcement learning agent[C]//International Joint Conference on Autonomous Agents and Multiagent Systems.New York:ACM,2006,5:720–727.
[31]JOINES J A,HOUCK C R.On the use of non-stationary penalty functions to solve nonlinear constrained optimization problems with GA’s[C]//Evolutionary Computation,IEEE World Congress on Computational Intelligence,USA:IEEE,1994,6:579–584.
[32]LIU Yefeng,PAN Quanke,CHAI Tianyou.Application of improved genetic algorithm to magnetic materials group furnace optimization problem[J].Control Theory&Applications,2014,31(9):1221–1231.(刘业峰,潘全科,柴天佑.改进遗传算法在磁性材料组炉优化问题中的应用[J].控制理论与应用,2014,31(9):1221–1231.)
[33]ZHANG G,LI N,JIN W,et al.Novel quantum genetic algorithm and its applications[J].Acta Electronica Sinica,2006,1(1):31–36.
[34]KUANG Fangjun,XU Weihong,JIN Zhong.Artificial bee colony algorithm based on self-adaptive tent chaos search[J].Control Theory&Applications,2014,31(11):1502–1509.(匡芳君,徐蔚鸿,金忠.自适应Tent混沌搜索的人工蜂群算法[J].控制理论与应用,2014,31(11):1502–1509.)
[35]HE S,WU Q H,SAUNDERS J R.Group search optimizer:an optimization algorithm inspired by animal searching behavior[J].IEEE Transactions on Evolutionary Computation,2009,13(5):973–990.
[36]MIRJALILI S.The ant lion optimizer[J].Advances in Engineering Software,2015,83(6):80–98.
[37]YANG B,YU T,ZHANG X S,et al.Interactive teaching-learning optimizer for parameter tuning of VSC-HVDC systems with offshore wind farm integration[J].IET Generation,Transmission&Distribution,2017,DOI:10.1049/iet-gtd.2016.1768.
[38]ATASHPAZ-GARGARI E,LUCAS C.Imperialist competitive algorithm:An algorithm for optimization inspired by imperialistic competition[C]//IEEE Congress on Evolutionary Computation.Singapore:IEEE,2007,21(1):4661–4667.
[39]LAN Yipeng,LIU Yufei.Magnetic levitation linear motor H-infinity robust controller design and optimization of ant colony algorithm[J].Control Theory&Applications,2015,32(4):527–532.(蓝益鹏,刘宇菲.磁悬浮直线电动机H∞鲁棒控制器及其蚁群算法优化设计[J].控制理论与应用,2015,32(4):527–532.)
[40]FANG K T,MA C,WINKER P,et al.Uniform design:theory and application[J].Technometrics,2000,42(3):237–248.
[2]YANG B,ZHANG X S,YU T,et al.Grouped grey wolf optimizer for maximum power point tracking of doubly-fed induction generator based wind turbine[J].Energy Conversion and Management,2017,133:427–443.
[3]YANG B,HU Y L,HUANG H Y,et al.Perturbation estimation based robust state feedback control for grid connected DFIG wind energy conversion system[J].International Journal of Hydrogen Energy,2017,42(33):20994–21005.
[4]LU Z G,FENG T,LI X P.Low-carbon emission economic power dispatch using the multi-objective bacterial colony chemotaxis optimization algorithm considering carbon capture power plant[J].International Journal of Electrical Power&Energy Systems,2013,53(1):106–112.
[5]BOOTHANDFORD M E,ABANADES J C,ANTHONY E J,et al.Carbon capture and storage update[J].Energy&Environmental Science,2014,7(1):130–189.
[6]ZHANG Xiaoshun,YU Tao.Stochastic optimal generation command dispatch of interconnected power grids based on improved multi-objective technique for order preference similar to an ideal solution-Q algorithm[J].Control Theory&Applications,2015,32(4):497–503.(张孝顺,余涛.互联电网自动发电控制功率分配的改进逼近于理想解的排序—Q多目标优化算法[J].控制理论与应用,2015,32(4):497–503.)
[7]ZHOU Tianrui,KANG Chongqing,XU Qianyao,et al.Preliminary theoretical investigation on power system carbon emission flow[J].Automation of Electric Power Systems,2012,36(7):38–43.(周天睿,康重庆,徐乾耀,等.电力系统碳排放流分析理论初探[J].电力系统自动化,2012,36(7):38–43.)
[8]ZHOU Tianrui,KANG Chongqing,XU qianyao,et al.Preliminary investigation on a method for carbon emission flow calculation of power system[J].Automation of Electric Power Systems,2012,36(11):44–49.(周天睿,康重庆,徐乾耀,等.电力系统碳排放流的计算方法初探[J].电力系统自动化,2012,36(11):44–49.)
[9]LI Baowei,HU Zechun,SONG Yonghua,et al.Principle and model for assessment on carbon emission intensity cased by electricity at consumer side[J].Power System Technology,2012,36(8):6–11.(李保卫,胡泽春,宋永华,等.用户侧电力碳排放强度的评估原则与模型[J].电网技术,2012,36(8):6–11.)
[10]ZHANG X,YU T,YANG B.Approximate ideal multi-objective solution Q(λ)learning for optimal carbon-energy combined-flow in multi-energy power systems[J].Energy Conversion&Management,2015,106:543–556.
[11]AMIN S M.Smart grid security,privacy,and resilient architectures:Opportunities and challenges[C]//Power and Energy Society General Meeting.San Diego:IEEE,2012,6:1–2.
[12]GIRI J,PARASHAR M,TREHERN J,et al.The situation room:control center analytics for enhanced situational awareness[J].IEEE Power&Energy Magazine,2012,10(5):24–39
[13]LENZEN M,MURRAY J,SACK F,et al.Shared producer and consumer responsibility–theory and practice[J].Ecological Economics,2007,61(1):27–42.
[14]KIM B H,BALDICK R.Coarse-grained distributed optimal power flow[J].IEEE Transactions on Power Systems,1997,12(2):932–939.
[15]COHENG.Auxiliary problem principle and decomposition of optimization problems[J].Journal of Optimization Theory&Applications,1980,32(3):277–305.
[16]LU¨Wei,SUI Ruirui,FENG Enmin.Predictor-corrector improvement of Newton method[J].Control Theory&Applications,2015,32(12):1620–1626.(吕巍,隋瑞瑞,冯恩民.改进的牛顿预测–校正格式[J].控制理论与应用,2015,32(12):1620–1626.)
[17]BOYD S,PARIKH N,CHU E,et al.Distributed optimization and statistical learning via the alternating direction method of multipliers[J].Foundations&Trends?Rin Machine Learning,2011,3(1):1–122.
[18]ERDELJAN A,CAPKO D,VUKMIROVIC S,et al.Distributed PSO algorithm for data model partitioning in power distribution systems[J].Journal of Applied Research&Technology,2014,16(5):947–957.
[19]LIANG C H,CHUNG C Y,WONG K P,et al.Parallel optimal reactive power flow based on cooperative co-evolutionary differential evolution and power system decomposition[J].IEEE Transactions on Power Systems,2007,22(1):249–257.
[20]YU T,LIU J,CHAN K W,et al.Distributed multi-step Q(λ)learning for optimal power flow of large-scale power grids[J].International Journal of Electrical Power&Energy Systems,2012,42(1):614–620.
[21]HU J,WELLMAN M P.Nash Q-learning for general-sum stochastic games[J].Journal of Machine Learning Research,2003,4(4):1039–1069.
[22]GREENWALD A,HALL K.Correlated-Q learning[C]//Proceedings of the 20th International Conference on Machine Learning.Washington,USA:AAAI Press,2003,8:242–249.
[23]HU Y,GAO Y,AN B.Accelerating multiagent reinforcement learning by equilibrium transfer[J].IEEE Transactions on Cybernetics,2015,45(7):1289–1302.
[24]TAYLOR M E,STONE P.Transfer learning for reinforcement learning domains:A survey[J].Journal of Machine Learning Research,2009,10(10):1633–1685.
[25]BLANCA G,MANFRED L.A consistent input-output formulation of shared producer and consumer responsibility[J].Economic Systems Research,2005,17(4):365–391.
[26]NASH J.Non-cooperative games[J].Annals of Mathematics.1951,54(3):286–295.
[27]WATKINS C J C H,DAYAN P.Technical note:Q-learning[J].Machine Learning,1992,8(3/4):279–292.
[28]GAMBARDELLA L M,DORIGO M.Ant-Q:A Reinforcement learning approach to the traveling salesman problem[C]//Proceedings of the 20th International Conference on Machine Learning.Tahoe City,California,USA:Morgan Kaufmann Publishers,1995,170(3):252–260.
[29]GOMES E R,KOWALCZYK R.Dynamic analysis of multiagent Qlearning withε-greedy exploration[C]//International Conference on Machine Learning.New York:ACM,2009,6:369–376.
[30]FERNANDO F,VELOSO M.Probabilistic policy reuse in a reinforcement learning agent[C]//International Joint Conference on Autonomous Agents and Multiagent Systems.New York:ACM,2006,5:720–727.
[31]JOINES J A,HOUCK C R.On the use of non-stationary penalty functions to solve nonlinear constrained optimization problems with GA’s[C]//Evolutionary Computation,IEEE World Congress on Computational Intelligence,USA:IEEE,1994,6:579–584.
[32]LIU Yefeng,PAN Quanke,CHAI Tianyou.Application of improved genetic algorithm to magnetic materials group furnace optimization problem[J].Control Theory&Applications,2014,31(9):1221–1231.(刘业峰,潘全科,柴天佑.改进遗传算法在磁性材料组炉优化问题中的应用[J].控制理论与应用,2014,31(9):1221–1231.)
[33]ZHANG G,LI N,JIN W,et al.Novel quantum genetic algorithm and its applications[J].Acta Electronica Sinica,2006,1(1):31–36.
[34]KUANG Fangjun,XU Weihong,JIN Zhong.Artificial bee colony algorithm based on self-adaptive tent chaos search[J].Control Theory&Applications,2014,31(11):1502–1509.(匡芳君,徐蔚鸿,金忠.自适应Tent混沌搜索的人工蜂群算法[J].控制理论与应用,2014,31(11):1502–1509.)
[35]HE S,WU Q H,SAUNDERS J R.Group search optimizer:an optimization algorithm inspired by animal searching behavior[J].IEEE Transactions on Evolutionary Computation,2009,13(5):973–990.
[36]MIRJALILI S.The ant lion optimizer[J].Advances in Engineering Software,2015,83(6):80–98.
[37]YANG B,YU T,ZHANG X S,et al.Interactive teaching-learning optimizer for parameter tuning of VSC-HVDC systems with offshore wind farm integration[J].IET Generation,Transmission&Distribution,2017,DOI:10.1049/iet-gtd.2016.1768.
[38]ATASHPAZ-GARGARI E,LUCAS C.Imperialist competitive algorithm:An algorithm for optimization inspired by imperialistic competition[C]//IEEE Congress on Evolutionary Computation.Singapore:IEEE,2007,21(1):4661–4667.
[39]LAN Yipeng,LIU Yufei.Magnetic levitation linear motor H-infinity robust controller design and optimization of ant colony algorithm[J].Control Theory&Applications,2015,32(4):527–532.(蓝益鹏,刘宇菲.磁悬浮直线电动机H∞鲁棒控制器及其蚁群算法优化设计[J].控制理论与应用,2015,32(4):527–532.)
[40]FANG K T,MA C,WINKER P,et al.Uniform design:theory and application[J].Technometrics,2000,42(3):237–248.