基于多主体博弈与强化学习的并网型综合能源微网协调调度
|
摘要
针对传统集中式优化调度方法难以全面反映综合能源微网内不同智能体的利益诉求,以及人工智能技术在综合能源调度方面的应用亟待进一步挖掘等问题,提出了基于多主体博弈与强化学习的并网型综合能源微网协调调度模型和方法。首先,针对并网型综合能源微网中横向电气热冷各子系统及纵向源网荷储等各环节的不同投资与运营主体,开展了多智能体划分;其次,针对可再生能源服务商、微网系统能源服务商、电动汽车用户等智能体,分别构建了各自的决策模型,并建立了以多智能体间利益均衡为目标的联合博弈决策模型;再次,针对多主体博弈这一高维决策难题,引入人工智能求解方法,提出了基于Nash博弈和强化学习算法的综合能源微网协调调度方法;最后,通过实例验证了所提模型和方法的有效性与实用性。
Considering that the traditional centralized optimized scheduling methods cannot comprehensively reflect the interests of different agents in the integrated energy microgrid and the application of artificial intelligence techniques in integrated energy scheduling is deepened,the coordinated scheduling model and method for grid-connected integrated energy microgrid based on multi-agent game and reinforcement learning are proposed in this paper.Firstly,the multiple investment and operation agents are divided from the perspectives of electrical/heating/cooling subsystems and source/grid/load/storage links,respectively.Secondly,the decision-making models for renewable energy provider,microgrid energy provider and electric vehicle owner are constructed,and the joint game based decision-making model is established with the objective of balancing the multi-agent interests.Thirdly,the artificial intelligence techniques are introduced to solve the highly-dimensional multi-agent game decision-making problem,and the coordinated scheduling method for integrated energy microgrid based on Nash game and Q learning algorithm is proposed.Finally,the effectiveness and applicability of the model and method proposed are verified by case study.
Considering that the traditional centralized optimized scheduling methods cannot comprehensively reflect the interests of different agents in the integrated energy microgrid and the application of artificial intelligence techniques in integrated energy scheduling is deepened,the coordinated scheduling model and method for grid-connected integrated energy microgrid based on multi-agent game and reinforcement learning are proposed in this paper.Firstly,the multiple investment and operation agents are divided from the perspectives of electrical/heating/cooling subsystems and source/grid/load/storage links,respectively.Secondly,the decision-making models for renewable energy provider,microgrid energy provider and electric vehicle owner are constructed,and the joint game based decision-making model is established with the objective of balancing the multi-agent interests.Thirdly,the artificial intelligence techniques are introduced to solve the highly-dimensional multi-agent game decision-making problem,and the coordinated scheduling method for integrated energy microgrid based on Nash game and Q learning algorithm is proposed.Finally,the effectiveness and applicability of the model and method proposed are verified by case study.
引文
[1]艾芊,郝然.多能互补、集成优化能源系统关键技术及挑战[J].电力系统自动化,2018,42(4):2-10.DOI:10.7500/AEPS20170927008.AI Qian,HAO Ran.Key technologies and challenges for multienergy complementarity and optimization of integrated energy system[J].Automation of Electric Power Systems,2018,42(4):2-10.DOI:10.7500/AEPS20170927008.
[2]刘涤尘,马恒瑞,王波,等.含冷热电联供及储能的区域综合能源系统运行优化[J].电力系统自动化,2018,42(4):113-120.DOI:10.7500/AEPS20170512002.LIU Dichen,MA Hengrui,WANG Bo,et al.Operation optimization of regional integrated energy system with CCHPand energy storage system[J].Automation of Electric Power Systems,2018,42(4):113-120.DOI:10.7500/AEPS20170512002.
[3]杨永标,于建成,李奕杰,等.含光伏和蓄能的冷热电联供系统调峰调蓄优化调度[J].电力系统自动化,2017,41(6):6-12.DOI:10.7500/AEPS20160610002.YANG Yongbiao,YU Jiancheng,LI Yijie,et al.Optimal load leveling dispatch of CCHP incorporating photovoltaic and storage[J].Automation of Electric Power Systems,2017,41(6):6-12.DOI:10.7500/AEPS20160610002.
[4]刘洪,陈星屹,李吉峰,等.基于改进CPSO算法的区域电热综合能源系统经济调度[J].电力自动化设备,2017,37(6):193-200.LIU Hong,CHEN Xingyi,LI Jifeng,et al.Economic dispatch based on improved CPSO algorithm for regional power-heat integrated energy system[J].Electric Power Automation Equipment,2017,37(6):193-200.
[5]郝然,艾芊,朱宇超.基于多智能体一致性的能源互联网协同优化控制[J].电力系统自动化,2017,41(15):10-17.DOI:10.7500/AEPS20170209003.HAO Ran,AI Qian,ZHU Yuchao.Cooperative optimal control of energy Internet based on multi-agent consistency[J].Automation of Electric Power Systems,2017,41(15):10-17.DOI:10.7500/AEPS20170209003.
[6]林凯骏,吴俊勇,郝亮亮,等.基于非合作博弈的冷热电联供微能源网运行策略优化[J].电力系统自动化,2018,42(6):25-32.DOI:10.7500/AEPS20170714006.LIN Kaijun,WU Junyong,HAO Liangliang,et al.Optimization of operation strategy for micro-energy grid with CCHP systems based on non-cooperative game[J].Automation of Electric Power Systems,2018,42(6):25-32.DOI:10.7500/AEPS20170714006.
[7]曾君,王侨侨,刘俊峰,等.一种基于势博弈的微电网分布式运行优化算法[J].中国电机工程学报,2017,37(24):7195-7204.ZENG Jun,WANG Qiaoqiao,LIU Junfeng,et al.An operation optimization algorithm of microgrid based on potential game[J].Proceedings of the CSEE,2017,37(24):7195-7204.
[8]程乐峰,余涛,张孝顺,等.信息-物理-社会融合的智慧能源调度机器人及其知识自动化:框架、技术与挑战[J].中国电机工程学报,2018,38(1):25-40.CHENG Lefeng,YU Tao,ZHANG Xiaoshun,et al.Cyberphysical-social systems based smart energy robotic dispatcher and its knowledge automation:framework,techniques and challenges[J].Proceedings of the CSEE,2018,38(1):25-40.
[9]徐彪,尹项根,汪旸,等.基于模糊时间Petri网的电网故障诊断方法[J].电力系统自动化,2018,42(2):70-76.DOI:10.7500/AEPS20170320004.XU Biao,YIN Xianggen,WANG Yang,et al.Fault diagnosis method of power system based on fuzzy time Petri net[J].Automation of Electric Power Systems,2018,42(2):70-76.DOI:10.7500/AEPS20170320004.
[10]孔祥玉,郑锋,鄂志君,等.基于深度信念网络的短期负荷预测方法[J].电力系统自动化,2018,42(5):133-139.DOI:10.7500/AEPS20170826002.KONG Xiangyu,ZHENG Feng,E Zhijun,et al.Short-term load forecasting based on deep belief network[J].Automation of Electric Power Systems,2018,42(5):133-139.DOI:10.7500/AEPS20170826002.
[11]徐龙博,王伟,张滔,等.基于神经网络平均影响值的超短期风电功率预测[J].电力系统自动化,2017,41(21):40-45.DOI:10.7500/AEPS20170321005.XU Longbo,WANG Wei,ZHANG Tao,et al.Ultra-shortterm wind power prediction based on neural network and mean impact valve[J].Automation of Electric Power Systems,2017,41(21):40-45.DOI:10.7500/AEPS20170321005.
[12]LI Fudong,WU Min,HE Yong,et al.Optimal control in microgrid using multi-agent reinforcement learning[J].ISATransactions,2012,51(6):743-751.
[13]LI Ding,JAYAWEERA S K.Machine-learning aided optimal customer decisions for an interactive smart grid[J].IEEESystems Journal,2015,9(4):1529-1540.
[14]KOFINAS P,DOUNIS I A,VOUROS G A.Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids[J].Applied Energy,2018,219:53-67.
[15]王成山,洪博文,郭力,等.冷热电联供微网优化调度通用建模方法[J].中国电机工程学报,2013,33(31):26-33.WANG Chengshan,HONG Bowen,GUO Li,et al.A general modeling method for optimal dispatch of combined cooling,heating and power microgrid[J].Proceedings of the CSEE,2013,33(31):26-33.
[16]赵波.微电网优化配置关键技术及应用[M].北京:科学出版社,2015.ZHAO Bo.Key technology and application of microgrid optimal configuration[M].Beijing:Science Press,2015.
[17]罗运虎,邢丽冬,王勤,等.峰谷分时电价用户响应模型参数的最小二乘估计[J].华东电力,2009,37(1):67-69.LUO Yunhu,XING Lidong,WANG Qin,et al.Least-squares estimation of parameters of customer response models for peak and valley time-of-use electricity price[J].East China Electric Power,2009,37(1):67-69.
[18]罗敏,赵伟,林国营,等.基于电网峰谷分时电价联动的电动汽车有序充电电价研究[J].电器与能效管理技术,2015(24):78-82.LUO Min,ZHAO Wei,LIN Guoying,et al.Research on coordinated charging price of electric vehicle based on users’price sensitivity[J].Electrical&Energy Management Technology,2015(24):78-82.
[19]余涛,张水平.在策略SARSA算法在互联电网CPS最优控制中的应用[J].电力系统保护与控制,2013,41(1):211-216.YU Tao,ZHANG Shuiping.Optimal CPS control for interconnected power systems based on SARSA on-policy learning algorithm[J].Power System Protection and Control,2013,41(1):211-216.
[20]刘国静,韩学山,王尚,等.基于强化学习方法的风储合作决策[J].电网技术,2016,40(9):2729-2736.LIU Guojing,HAN Xueshan,WANG Shang,et al.Optimal decision-making in the cooperation of wind power and energy storage based on reinforcement learning algorithm[J].Power System Technology,2016,40(9):2729-2736.
[21]HU J L,WELLMAN M P.Nash Q-learning for general-sum stochastic games[J].Journal of Machine Learning Research,2004,4(6):1039-1069.
[22]罗艳红,梁佳丽,杨东升,等.计及可靠性的电-气-热能量枢纽配置与运行优化[J].电力系统自动化,2018,42(4):47-54.DOI:10.7500/AEPS20170831002.LUO Yanhong,LIANG Jiali,YANG Dongsheng,et al.Configuration and operation optimization of electric-gas-heat energy hub considering reliability[J].Automation of Electric Power Systems,2018,42(4):47-54.DOI:10.7500/AEPS20170831002.
[23]李正茂,张峰,梁军,等.含电热联合系统的微电网运行优化[J].中国电机工程学报,2015,35(14):3569-3576.LI Zhengmao,ZHANG Feng,LIANG Jun,et al.Optimization on microgrid with combined heat and power system[J].Proceedings of the CSEE,2015,35(14):3569-3576.
[24]TSIKALAKIS A G,HATZIARGYRION N D.Centralized control for optimizing microgrid operation[J].IEEETransactions on Energy Conversion,2008,23(1):241-248.
[25]Specifications of 2016Nissan LEAF[EB/OL].[2018-05-01].http://www.nissanusa.com/electric-cars/leaf/versions-specs/version.sv.html.
[2]刘涤尘,马恒瑞,王波,等.含冷热电联供及储能的区域综合能源系统运行优化[J].电力系统自动化,2018,42(4):113-120.DOI:10.7500/AEPS20170512002.LIU Dichen,MA Hengrui,WANG Bo,et al.Operation optimization of regional integrated energy system with CCHPand energy storage system[J].Automation of Electric Power Systems,2018,42(4):113-120.DOI:10.7500/AEPS20170512002.
[3]杨永标,于建成,李奕杰,等.含光伏和蓄能的冷热电联供系统调峰调蓄优化调度[J].电力系统自动化,2017,41(6):6-12.DOI:10.7500/AEPS20160610002.YANG Yongbiao,YU Jiancheng,LI Yijie,et al.Optimal load leveling dispatch of CCHP incorporating photovoltaic and storage[J].Automation of Electric Power Systems,2017,41(6):6-12.DOI:10.7500/AEPS20160610002.
[4]刘洪,陈星屹,李吉峰,等.基于改进CPSO算法的区域电热综合能源系统经济调度[J].电力自动化设备,2017,37(6):193-200.LIU Hong,CHEN Xingyi,LI Jifeng,et al.Economic dispatch based on improved CPSO algorithm for regional power-heat integrated energy system[J].Electric Power Automation Equipment,2017,37(6):193-200.
[5]郝然,艾芊,朱宇超.基于多智能体一致性的能源互联网协同优化控制[J].电力系统自动化,2017,41(15):10-17.DOI:10.7500/AEPS20170209003.HAO Ran,AI Qian,ZHU Yuchao.Cooperative optimal control of energy Internet based on multi-agent consistency[J].Automation of Electric Power Systems,2017,41(15):10-17.DOI:10.7500/AEPS20170209003.
[6]林凯骏,吴俊勇,郝亮亮,等.基于非合作博弈的冷热电联供微能源网运行策略优化[J].电力系统自动化,2018,42(6):25-32.DOI:10.7500/AEPS20170714006.LIN Kaijun,WU Junyong,HAO Liangliang,et al.Optimization of operation strategy for micro-energy grid with CCHP systems based on non-cooperative game[J].Automation of Electric Power Systems,2018,42(6):25-32.DOI:10.7500/AEPS20170714006.
[7]曾君,王侨侨,刘俊峰,等.一种基于势博弈的微电网分布式运行优化算法[J].中国电机工程学报,2017,37(24):7195-7204.ZENG Jun,WANG Qiaoqiao,LIU Junfeng,et al.An operation optimization algorithm of microgrid based on potential game[J].Proceedings of the CSEE,2017,37(24):7195-7204.
[8]程乐峰,余涛,张孝顺,等.信息-物理-社会融合的智慧能源调度机器人及其知识自动化:框架、技术与挑战[J].中国电机工程学报,2018,38(1):25-40.CHENG Lefeng,YU Tao,ZHANG Xiaoshun,et al.Cyberphysical-social systems based smart energy robotic dispatcher and its knowledge automation:framework,techniques and challenges[J].Proceedings of the CSEE,2018,38(1):25-40.
[9]徐彪,尹项根,汪旸,等.基于模糊时间Petri网的电网故障诊断方法[J].电力系统自动化,2018,42(2):70-76.DOI:10.7500/AEPS20170320004.XU Biao,YIN Xianggen,WANG Yang,et al.Fault diagnosis method of power system based on fuzzy time Petri net[J].Automation of Electric Power Systems,2018,42(2):70-76.DOI:10.7500/AEPS20170320004.
[10]孔祥玉,郑锋,鄂志君,等.基于深度信念网络的短期负荷预测方法[J].电力系统自动化,2018,42(5):133-139.DOI:10.7500/AEPS20170826002.KONG Xiangyu,ZHENG Feng,E Zhijun,et al.Short-term load forecasting based on deep belief network[J].Automation of Electric Power Systems,2018,42(5):133-139.DOI:10.7500/AEPS20170826002.
[11]徐龙博,王伟,张滔,等.基于神经网络平均影响值的超短期风电功率预测[J].电力系统自动化,2017,41(21):40-45.DOI:10.7500/AEPS20170321005.XU Longbo,WANG Wei,ZHANG Tao,et al.Ultra-shortterm wind power prediction based on neural network and mean impact valve[J].Automation of Electric Power Systems,2017,41(21):40-45.DOI:10.7500/AEPS20170321005.
[12]LI Fudong,WU Min,HE Yong,et al.Optimal control in microgrid using multi-agent reinforcement learning[J].ISATransactions,2012,51(6):743-751.
[13]LI Ding,JAYAWEERA S K.Machine-learning aided optimal customer decisions for an interactive smart grid[J].IEEESystems Journal,2015,9(4):1529-1540.
[14]KOFINAS P,DOUNIS I A,VOUROS G A.Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids[J].Applied Energy,2018,219:53-67.
[15]王成山,洪博文,郭力,等.冷热电联供微网优化调度通用建模方法[J].中国电机工程学报,2013,33(31):26-33.WANG Chengshan,HONG Bowen,GUO Li,et al.A general modeling method for optimal dispatch of combined cooling,heating and power microgrid[J].Proceedings of the CSEE,2013,33(31):26-33.
[16]赵波.微电网优化配置关键技术及应用[M].北京:科学出版社,2015.ZHAO Bo.Key technology and application of microgrid optimal configuration[M].Beijing:Science Press,2015.
[17]罗运虎,邢丽冬,王勤,等.峰谷分时电价用户响应模型参数的最小二乘估计[J].华东电力,2009,37(1):67-69.LUO Yunhu,XING Lidong,WANG Qin,et al.Least-squares estimation of parameters of customer response models for peak and valley time-of-use electricity price[J].East China Electric Power,2009,37(1):67-69.
[18]罗敏,赵伟,林国营,等.基于电网峰谷分时电价联动的电动汽车有序充电电价研究[J].电器与能效管理技术,2015(24):78-82.LUO Min,ZHAO Wei,LIN Guoying,et al.Research on coordinated charging price of electric vehicle based on users’price sensitivity[J].Electrical&Energy Management Technology,2015(24):78-82.
[19]余涛,张水平.在策略SARSA算法在互联电网CPS最优控制中的应用[J].电力系统保护与控制,2013,41(1):211-216.YU Tao,ZHANG Shuiping.Optimal CPS control for interconnected power systems based on SARSA on-policy learning algorithm[J].Power System Protection and Control,2013,41(1):211-216.
[20]刘国静,韩学山,王尚,等.基于强化学习方法的风储合作决策[J].电网技术,2016,40(9):2729-2736.LIU Guojing,HAN Xueshan,WANG Shang,et al.Optimal decision-making in the cooperation of wind power and energy storage based on reinforcement learning algorithm[J].Power System Technology,2016,40(9):2729-2736.
[21]HU J L,WELLMAN M P.Nash Q-learning for general-sum stochastic games[J].Journal of Machine Learning Research,2004,4(6):1039-1069.
[22]罗艳红,梁佳丽,杨东升,等.计及可靠性的电-气-热能量枢纽配置与运行优化[J].电力系统自动化,2018,42(4):47-54.DOI:10.7500/AEPS20170831002.LUO Yanhong,LIANG Jiali,YANG Dongsheng,et al.Configuration and operation optimization of electric-gas-heat energy hub considering reliability[J].Automation of Electric Power Systems,2018,42(4):47-54.DOI:10.7500/AEPS20170831002.
[23]李正茂,张峰,梁军,等.含电热联合系统的微电网运行优化[J].中国电机工程学报,2015,35(14):3569-3576.LI Zhengmao,ZHANG Feng,LIANG Jun,et al.Optimization on microgrid with combined heat and power system[J].Proceedings of the CSEE,2015,35(14):3569-3576.
[24]TSIKALAKIS A G,HATZIARGYRION N D.Centralized control for optimizing microgrid operation[J].IEEETransactions on Energy Conversion,2008,23(1):241-248.
[25]Specifications of 2016Nissan LEAF[EB/OL].[2018-05-01].http://www.nissanusa.com/electric-cars/leaf/versions-specs/version.sv.html.