数据流挖掘技术及应用研究
|
摘要
数据流技术已成为当今数据处理领域的研究热点,数据流挖掘是数据流技术研究的重要分支。数据流挖掘的核心思想是设计数据集单遍扫描算法,在有限的内存中存储少量代表数据流特征的概要数据,即时、快速地处理概要数据以得到响应结果,实现数据流实时、在线分析。对数据流进行聚类、分类具有广泛的应用价值。本文建立了数据流挖掘模型,完成了数据流概要的构建并实现了相应的增量更新算法,将聚类、分类算法应用于数据流这一新的数据模型上,对传统数据挖掘技术进行改进,实现了数据的实时处理和在线分析。在此基础上提出数据流挖掘技术在电力系统负荷分析和电能质量扰动在线识别领域中的应用框架,并给出了具体的实现方法,实验结果验证了方法的有效性。
Data stream technique is becoming hot in the field of data processing. Mining data streams is one of the important branches in data stream researches. The focus on mining data stream is to design one-pass scan algorithm over data set, and maintain a few synopsis data in memory. The synopsis data could express the main characteristic of data stream, and it is far smaller than the size of whole data stream.The synopsis data is used for the real-time and online analysis of data streams. The clustering and classification of data streams are widely used in many fields. This paper presents a reasonable model of mining data streams. The synopsis data structure and its incremental algorithm are designed. Traditional technique of data mining is improved to realize the real-time and online processing. Frameworks of the application in power load analysis and online power quality disturbance classification are proposed on the basis of mining data streams, and methods related to them are given. Experiment results show the effectiveness of the presented methods.
Data stream technique is becoming hot in the field of data processing. Mining data streams is one of the important branches in data stream researches. The focus on mining data stream is to design one-pass scan algorithm over data set, and maintain a few synopsis data in memory. The synopsis data could express the main characteristic of data stream, and it is far smaller than the size of whole data stream.The synopsis data is used for the real-time and online analysis of data streams. The clustering and classification of data streams are widely used in many fields. This paper presents a reasonable model of mining data streams. The synopsis data structure and its incremental algorithm are designed. Traditional technique of data mining is improved to realize the real-time and online processing. Frameworks of the application in power load analysis and online power quality disturbance classification are proposed on the basis of mining data streams, and methods related to them are given. Experiment results show the effectiveness of the presented methods.
引文
[1] 于戈,王国仁.实时数据流管理技术.计算机应用,2004,24(10):11~13,26
[2] Babcock B,Babu S,Datar M,Motwani R,Widom J.Models and issues in data stream systems.In:Popa L,eds.Proceedings of the 21st ACM Symposium on Principles of Databases Systems(PODS).Madison,Wisconsin,USA:ACM Press,2002.1~16
[3] Avnur R,Hellerstein J.Eddies:Continuously adaptive query processing.In:Chen W,Naughton JF,Bernstein PA,eds.Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data . Callas : ACM Press ,2000.261~272
[4] Hellerstein J,Franklin M,Chandrasekaran S,Deshpande A,Hildrum K,Madden S,Raman V,Shah MA.Adaptive query processing:Technology in evolution.IEEE Data Engineering Bulletin,2000,23(2):7~18
[5] Carney D,Cetinternel U,Cherniack M,Convey C,Lee S,Seidman G,Stonebraker M , Tatbul N , Zdonid S . Monitoring streams―a new class of DBMS applications.Technical Report,CS-02-01,Providence:Department of Computer Science,Brown University,2002
[6] Chen J,Dewitt D,Tian F,et al.NiagaraCQ:a scalable continuous query system for internet databases.In:Franklin MJ,eds.Proceedings of ACM SIGMOD Conference.Dallas,USA:ACM Press,2000.379~390
[7] Madden S,Franklin MJ.Fjording the stream:an architecture for queries over streaming sensor data.In:Proceedings of the 18th ICDE Conference.San Jose,USA:2002
[8] S Chandrasekaran,et al.Streaming queries over streaming data.In:Proceedings of the 28th International Conference on Very Large Data Bases.2002.203~214
[9] Guha S,Mishra N,Motwani R,O’Callaghan L.Clustering data streams.In:Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science(FOCS).2000.359~366
[10] O’Callaghan L,Mishia N,Meyerson A,et al.Streaming-data algorithms for high-quality clustering.In:Proceedings of IEEE International Conference on Data Engineering.San Jose,California,USA:2002.685~704
[11] Charu C.Aggarwal,Jiawei Han,Jianyong Wang.A framework for clustering evolving data streams.In:Proceedings of the 29th VLDB Conference.Berlin,Germany:2003.81~92
[12] Barbara D.Requirements for clustering data streams.ACM SIGKDD Explorations Newsletter,2003,3(2):23~27
[13] Cohen L,Avrahami G,Last M.Incremental info-fuzzy algorithm for real time data mining of non-stationary data streams . In : Proceedings of VLDB Conference.Berlin,Germany:2003.493~507
[14] P.Domingos,G.Hulten.Mining high-speed data streams.In:Proceedings of the 6th International Conference on Knowledge Discovery and Data Mining(KDD).2000.71~80
[15] P.Domingos,G.Hulten,L.Spencer.Mining time-changing data streams.In:Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining(KDD).2001.97~106
[16] R.Jin,G.Agrawal.Efficient decision tree construction on streaming data.In:Proceedings of ACM SIGKDD.2003
[17] R.John,G.Sara,C.Evans.A coverage based ensemble algorithm(CBEA)for streaming data.In:16th IEEE International Conference on Tools with Artificial Intelligence.2004.106~112
[18] 杨颖,韩忠明,杨磊.数据流的核心技术与应用发展研究综述.计算机应用研究,2005,22(11):4~7
[19] 姚丽娟,李庆忠.传感器采样数据流查询技术.计算机工程与应用,2004,40(4):208~210
[20] 姚丽娟.数据流查询处理及在传感器采样系统中的应用:[硕士学位论文].山东:山东大学,2003
[21] 刘佳.基于流数据技术的 Internet 信息监测系统的研究:[硕士学位文].秦皇岛:燕山大学,2004
[22] 郑斌祥,席裕.面向电力系统的数据挖掘研究.电气自动化,2000,22(6):4~7
[23] 于之虹,郭志忠.数据挖掘与电力系统.电网技术,2001,25(8):58~62
[24] 廖志伟,孙雅明.数据挖掘技术及其在电力系统中的应用.电力系统自动化,2001,25(11):62~66
[25] 金澈清,钱卫宁,周傲英.流数据分析与管理综述.软件学报,2004,15(8):1172~1181
[26] 刘景春,王永利.数据流处理技术.佳木斯大学学报(自然科学版),2004,22(4):556~560
[27] 闫朝升.数据流联机分析处理技术的研究:[硕士学位论文].黑龙江:黑龙江大学,2004
[28] 刘景春.数据流管理系统关键技术研究.佳木斯大学学报(自然科学版),2005,23(4):505~510
[29] 王永利.数据流概要与数据流分析若干关键问题研究:[博士学位论文].南京:东南大学,2006
[30] 吴成涛.数据流管理系统中概要数据结构算法的研究和实现:[硕士学位论文].南京:东南大学,2006
[31] Y Zhu,D Shasha.StatStream:Statistical monitoring of thousands of data streams in real time.In:Proceedings of the 28th VLDB Conference.Hong Kong,China:2002.358~369
[32] Bulut A,Singh K.SWAT:Hierarchical stream summarization in large networks.In:IEEE International Conference on Data Endineering.2003.563~575
[33] 杨颖,陈秋莲,杨磊.基于小波变换的数据流快速索引的研究.计算机应用,2006,26(11):2657~2660
[34] 富春岩,葛茂松,刘艳玲.小波方法在数据流中的应用.佳木斯大学学报(自然科学版),2006,24(3):343~345
[35] Jiawei Han,Micheline Kamber.数据挖掘概念与技术,(范明,孟小峰,等).北京:机械工业出版社,2001年
[36] 方金城.分类挖掘算法综述.沈阳工程学院学报(自然科学版),2006,2(1):73~76
[37] 姜园,张朝阳,仇佩亮,周东方.用于数据挖掘的聚类算法.电子与信息学报,2005,27(4):655~662
[38] 蒋盛益,李庆华,李新.数据流挖掘算法研究综述.计算机工程与设计,2005,26(5):1130~1169
[39] 倪志伟,黄玲,李锋刚,忻凌.数据流管理与挖掘研究.合肥工业大学学报(自然科学版),2005,28(9):1157~1162
[40] 宋卫东,毛宇光,张玲东.数据流挖掘技术研究.微机发展,2005,15(8):139~141,144
[41] 郭兴凯.数据流挖掘技术的研究:[硕士学位论文].黑龙江:黑龙江大学,2005
[42] 朱蔚恒,印鉴,谢益煌.基于数据流的任意形状聚类算法.软件学报,2006,17(3):379~387
[43] 刘赏,黄亚楼,倪维健.流数据聚类模型变化检测策略.计算机工程与应用,2006,42(5):15~18
[44] 赵法信,刘俊岭.数据流聚类算法研究.通化师范学院学报,2006,27(2):29~32
[45] 马瑞民,王小龙.NBCC:一种数据流上变化的挖掘算法.计算机工程与应用,2006,42(7):166~168
[46] 黄树成,刘丽.数据流上挖掘决策树的关键技术研究.常州工学院学报,2005,18(5):19~25
[47] 孙超利.基于决策树的数据流挖掘算法的研究.太原科技大学学报,2006,27(4):268~272
[48] Figueiredo V,Rodrigues F,Vale Z,Gouveia J B.An electric energy consumer characterization framework based on data mining techniques . IEEE TRANSACTIONS ON POWER SYSTEMS,2005,20(2):596~602
[49] Chang R F,Lu C N.Load profiling and its applications in power market.In:Power Engineering Society General Meeting.Toronto:2003.974~978
[50] 段铷,张彩庆,刘爱芳.模糊聚类在电力用户分类中的应用.电力需求侧管理,2005,7(5):18~20
[51] 鞠平,陈谦,熊传平,黄丽,方朝雄,陈峰,赵红嘎,戴琦,鄢安河,付红军.基于日负荷曲线的负荷分类和综合建模.电力系统自动化,2006,30(16):6~9
[52] 黄梅,贺仁睦,杨少兵,张佰春.东北电网负荷模型的分类与应用.电力系统自动化,2005,29(4):85~87
[53] 黄梅,贺仁睦,杨少兵.模糊聚类在负荷实测建模中的应用.电网技术,2006,30(14):49~52
[54] 李培强,李欣然,陈辉华,唐外文.基于模糊聚类的电力负荷特性的分类与综合.中国电机工程学报,2005,25(24):73~78
[55] 李培强,李欣然,唐外文,刘艳阳.模糊 C 均值聚类在电力负荷建模中的应用研究.湖南大学学报(自然科学版),2006,33(3):41~45
[56] 李培强,李欣然,陈凤,唐外文,朱军飞.模糊聚类在统计综合法负荷建模中的应用.电力自动化设备,2003,23(5):43~45,74
[57] 李黎.基于聚类分析的负荷数据分类方法的研究:[硕士学位论文].北京:华北电力大学,2004
[58] 朱树人,徐增敏,夏卓群,彭妮.模糊聚类在用电管理中的应用.长沙电力学院学报(自然科学版),2005,20(1):21~24
[59] 周明,李庚银,倪以信.电力市场下电力需求侧管理实施机制初探.电网技术,2005,29(5):6~11
[60] 王明俊.市场环境下的负荷管理和需求侧管理.电网技术,2005,29(5):1~5
[61] 吴杰康,任震,黄雯莹,黄福全.在全面开放的电力市场中用户用电管理及其策略.电网技术,2001,25(8):50~53,57
[62] 黄志伟,严正,陈允平,倪以信.电力市场环境下配电系统综合自动化探讨.电力自动化设备,2004,24(5):79~83
[63] 陈祝赢,朱炯,刘康生.宁波城区夏季电力负荷分析及在有序用电管理中的应用.华东电力,2005,33(11):43~45
[64] 郭联哲,谭忠富,李晓军.基于用户响应下的分时电价优化设计模型与方法.电网技术,2006,30(5):24~28
[65] 张智光.城网供用电的经济性与现行电价政策的合理性分析.电网技术,2001,25(6):48~51
[66] 陈德裕,温克昌.对我国电价改革的几点建议.电网技术,2000,24(4):1~6
[67] 马金明,秦雷鸣,朱振青.供电公司分时售电电价研究.电力系统自动化,2002,26(5):9~12,59
[68] 李扬,王治华,卢毅,李军红,张长沪.峰谷分时电价的实施及大工业用户的响应.电力系统自动化,2001,25(8):45~48.
[69] 林海学.现代电能质量的基本问题.电网技术,2001,25(10):5~12
[70] 肖湘宁,尹忠东.徐永海现代电能质量问题综述.电气时代,2004,(11):48~52
[71] 张晓慧,郑维平,洪丹鹰.电能质量扰动检测与分类方法的应用策略.东北电力技术,2005,26(5):5~8
[72] 徐袭,石敏.一种基于粗糙集与小波变换的电能质量分类方法.电力自动化设备,2005,25(11):15~18
[73] 赵凤展,杨仁刚.基于时域、小波变换和 FFT 的电能质量扰动识别.继电器,2006,34(8):50~55
[2] Babcock B,Babu S,Datar M,Motwani R,Widom J.Models and issues in data stream systems.In:Popa L,eds.Proceedings of the 21st ACM Symposium on Principles of Databases Systems(PODS).Madison,Wisconsin,USA:ACM Press,2002.1~16
[3] Avnur R,Hellerstein J.Eddies:Continuously adaptive query processing.In:Chen W,Naughton JF,Bernstein PA,eds.Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data . Callas : ACM Press ,2000.261~272
[4] Hellerstein J,Franklin M,Chandrasekaran S,Deshpande A,Hildrum K,Madden S,Raman V,Shah MA.Adaptive query processing:Technology in evolution.IEEE Data Engineering Bulletin,2000,23(2):7~18
[5] Carney D,Cetinternel U,Cherniack M,Convey C,Lee S,Seidman G,Stonebraker M , Tatbul N , Zdonid S . Monitoring streams―a new class of DBMS applications.Technical Report,CS-02-01,Providence:Department of Computer Science,Brown University,2002
[6] Chen J,Dewitt D,Tian F,et al.NiagaraCQ:a scalable continuous query system for internet databases.In:Franklin MJ,eds.Proceedings of ACM SIGMOD Conference.Dallas,USA:ACM Press,2000.379~390
[7] Madden S,Franklin MJ.Fjording the stream:an architecture for queries over streaming sensor data.In:Proceedings of the 18th ICDE Conference.San Jose,USA:2002
[8] S Chandrasekaran,et al.Streaming queries over streaming data.In:Proceedings of the 28th International Conference on Very Large Data Bases.2002.203~214
[9] Guha S,Mishra N,Motwani R,O’Callaghan L.Clustering data streams.In:Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science(FOCS).2000.359~366
[10] O’Callaghan L,Mishia N,Meyerson A,et al.Streaming-data algorithms for high-quality clustering.In:Proceedings of IEEE International Conference on Data Engineering.San Jose,California,USA:2002.685~704
[11] Charu C.Aggarwal,Jiawei Han,Jianyong Wang.A framework for clustering evolving data streams.In:Proceedings of the 29th VLDB Conference.Berlin,Germany:2003.81~92
[12] Barbara D.Requirements for clustering data streams.ACM SIGKDD Explorations Newsletter,2003,3(2):23~27
[13] Cohen L,Avrahami G,Last M.Incremental info-fuzzy algorithm for real time data mining of non-stationary data streams . In : Proceedings of VLDB Conference.Berlin,Germany:2003.493~507
[14] P.Domingos,G.Hulten.Mining high-speed data streams.In:Proceedings of the 6th International Conference on Knowledge Discovery and Data Mining(KDD).2000.71~80
[15] P.Domingos,G.Hulten,L.Spencer.Mining time-changing data streams.In:Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining(KDD).2001.97~106
[16] R.Jin,G.Agrawal.Efficient decision tree construction on streaming data.In:Proceedings of ACM SIGKDD.2003
[17] R.John,G.Sara,C.Evans.A coverage based ensemble algorithm(CBEA)for streaming data.In:16th IEEE International Conference on Tools with Artificial Intelligence.2004.106~112
[18] 杨颖,韩忠明,杨磊.数据流的核心技术与应用发展研究综述.计算机应用研究,2005,22(11):4~7
[19] 姚丽娟,李庆忠.传感器采样数据流查询技术.计算机工程与应用,2004,40(4):208~210
[20] 姚丽娟.数据流查询处理及在传感器采样系统中的应用:[硕士学位论文].山东:山东大学,2003
[21] 刘佳.基于流数据技术的 Internet 信息监测系统的研究:[硕士学位文].秦皇岛:燕山大学,2004
[22] 郑斌祥,席裕.面向电力系统的数据挖掘研究.电气自动化,2000,22(6):4~7
[23] 于之虹,郭志忠.数据挖掘与电力系统.电网技术,2001,25(8):58~62
[24] 廖志伟,孙雅明.数据挖掘技术及其在电力系统中的应用.电力系统自动化,2001,25(11):62~66
[25] 金澈清,钱卫宁,周傲英.流数据分析与管理综述.软件学报,2004,15(8):1172~1181
[26] 刘景春,王永利.数据流处理技术.佳木斯大学学报(自然科学版),2004,22(4):556~560
[27] 闫朝升.数据流联机分析处理技术的研究:[硕士学位论文].黑龙江:黑龙江大学,2004
[28] 刘景春.数据流管理系统关键技术研究.佳木斯大学学报(自然科学版),2005,23(4):505~510
[29] 王永利.数据流概要与数据流分析若干关键问题研究:[博士学位论文].南京:东南大学,2006
[30] 吴成涛.数据流管理系统中概要数据结构算法的研究和实现:[硕士学位论文].南京:东南大学,2006
[31] Y Zhu,D Shasha.StatStream:Statistical monitoring of thousands of data streams in real time.In:Proceedings of the 28th VLDB Conference.Hong Kong,China:2002.358~369
[32] Bulut A,Singh K.SWAT:Hierarchical stream summarization in large networks.In:IEEE International Conference on Data Endineering.2003.563~575
[33] 杨颖,陈秋莲,杨磊.基于小波变换的数据流快速索引的研究.计算机应用,2006,26(11):2657~2660
[34] 富春岩,葛茂松,刘艳玲.小波方法在数据流中的应用.佳木斯大学学报(自然科学版),2006,24(3):343~345
[35] Jiawei Han,Micheline Kamber.数据挖掘概念与技术,(范明,孟小峰,等).北京:机械工业出版社,2001年
[36] 方金城.分类挖掘算法综述.沈阳工程学院学报(自然科学版),2006,2(1):73~76
[37] 姜园,张朝阳,仇佩亮,周东方.用于数据挖掘的聚类算法.电子与信息学报,2005,27(4):655~662
[38] 蒋盛益,李庆华,李新.数据流挖掘算法研究综述.计算机工程与设计,2005,26(5):1130~1169
[39] 倪志伟,黄玲,李锋刚,忻凌.数据流管理与挖掘研究.合肥工业大学学报(自然科学版),2005,28(9):1157~1162
[40] 宋卫东,毛宇光,张玲东.数据流挖掘技术研究.微机发展,2005,15(8):139~141,144
[41] 郭兴凯.数据流挖掘技术的研究:[硕士学位论文].黑龙江:黑龙江大学,2005
[42] 朱蔚恒,印鉴,谢益煌.基于数据流的任意形状聚类算法.软件学报,2006,17(3):379~387
[43] 刘赏,黄亚楼,倪维健.流数据聚类模型变化检测策略.计算机工程与应用,2006,42(5):15~18
[44] 赵法信,刘俊岭.数据流聚类算法研究.通化师范学院学报,2006,27(2):29~32
[45] 马瑞民,王小龙.NBCC:一种数据流上变化的挖掘算法.计算机工程与应用,2006,42(7):166~168
[46] 黄树成,刘丽.数据流上挖掘决策树的关键技术研究.常州工学院学报,2005,18(5):19~25
[47] 孙超利.基于决策树的数据流挖掘算法的研究.太原科技大学学报,2006,27(4):268~272
[48] Figueiredo V,Rodrigues F,Vale Z,Gouveia J B.An electric energy consumer characterization framework based on data mining techniques . IEEE TRANSACTIONS ON POWER SYSTEMS,2005,20(2):596~602
[49] Chang R F,Lu C N.Load profiling and its applications in power market.In:Power Engineering Society General Meeting.Toronto:2003.974~978
[50] 段铷,张彩庆,刘爱芳.模糊聚类在电力用户分类中的应用.电力需求侧管理,2005,7(5):18~20
[51] 鞠平,陈谦,熊传平,黄丽,方朝雄,陈峰,赵红嘎,戴琦,鄢安河,付红军.基于日负荷曲线的负荷分类和综合建模.电力系统自动化,2006,30(16):6~9
[52] 黄梅,贺仁睦,杨少兵,张佰春.东北电网负荷模型的分类与应用.电力系统自动化,2005,29(4):85~87
[53] 黄梅,贺仁睦,杨少兵.模糊聚类在负荷实测建模中的应用.电网技术,2006,30(14):49~52
[54] 李培强,李欣然,陈辉华,唐外文.基于模糊聚类的电力负荷特性的分类与综合.中国电机工程学报,2005,25(24):73~78
[55] 李培强,李欣然,唐外文,刘艳阳.模糊 C 均值聚类在电力负荷建模中的应用研究.湖南大学学报(自然科学版),2006,33(3):41~45
[56] 李培强,李欣然,陈凤,唐外文,朱军飞.模糊聚类在统计综合法负荷建模中的应用.电力自动化设备,2003,23(5):43~45,74
[57] 李黎.基于聚类分析的负荷数据分类方法的研究:[硕士学位论文].北京:华北电力大学,2004
[58] 朱树人,徐增敏,夏卓群,彭妮.模糊聚类在用电管理中的应用.长沙电力学院学报(自然科学版),2005,20(1):21~24
[59] 周明,李庚银,倪以信.电力市场下电力需求侧管理实施机制初探.电网技术,2005,29(5):6~11
[60] 王明俊.市场环境下的负荷管理和需求侧管理.电网技术,2005,29(5):1~5
[61] 吴杰康,任震,黄雯莹,黄福全.在全面开放的电力市场中用户用电管理及其策略.电网技术,2001,25(8):50~53,57
[62] 黄志伟,严正,陈允平,倪以信.电力市场环境下配电系统综合自动化探讨.电力自动化设备,2004,24(5):79~83
[63] 陈祝赢,朱炯,刘康生.宁波城区夏季电力负荷分析及在有序用电管理中的应用.华东电力,2005,33(11):43~45
[64] 郭联哲,谭忠富,李晓军.基于用户响应下的分时电价优化设计模型与方法.电网技术,2006,30(5):24~28
[65] 张智光.城网供用电的经济性与现行电价政策的合理性分析.电网技术,2001,25(6):48~51
[66] 陈德裕,温克昌.对我国电价改革的几点建议.电网技术,2000,24(4):1~6
[67] 马金明,秦雷鸣,朱振青.供电公司分时售电电价研究.电力系统自动化,2002,26(5):9~12,59
[68] 李扬,王治华,卢毅,李军红,张长沪.峰谷分时电价的实施及大工业用户的响应.电力系统自动化,2001,25(8):45~48.
[69] 林海学.现代电能质量的基本问题.电网技术,2001,25(10):5~12
[70] 肖湘宁,尹忠东.徐永海现代电能质量问题综述.电气时代,2004,(11):48~52
[71] 张晓慧,郑维平,洪丹鹰.电能质量扰动检测与分类方法的应用策略.东北电力技术,2005,26(5):5~8
[72] 徐袭,石敏.一种基于粗糙集与小波变换的电能质量分类方法.电力自动化设备,2005,25(11):15~18
[73] 赵凤展,杨仁刚.基于时域、小波变换和 FFT 的电能质量扰动识别.继电器,2006,34(8):50~55