云计算环境下基于MapReduce的并行化排列熵算法
|
摘要
针对工业监测数据在特征提取环节需要处理的数据集越来越大、实效性要求越来越高的问题,设计了一种在云计算平台MaxCompute环境下并行化的排列熵(Permutation Entropy,PE)算法。采用MaxCompute表存储海量的监测数据,基于MaxCompute扩展MapReduce模型设计了并行化排列熵算法,用于海量监测历史数据的批量排列熵特征提取。通过在单机和云计算平台环境下测试,算法具有良好的可扩展性,并可以适应大规模数据集,算法性能与数据量成线性关系。
Aiming at the problem that the data sets of industrial monitoring data need to be processed in the feature extraction process are becoming larger and larger,and the requirement of practicality is getting higher and higher,a parallel permutation entropy algorithm based on cloud computing platform MaxCompute is designed.MaxCompute table is used to store massive monitoring data.Based on MaxCompute,MapReduce modelis extended,a parallel permutation entropy algorithm is designed to extract batch permutation entropy features of massive monitoring historical data.By testing on a standalone computer and cloud computing platform,the algorithm has good scalability and can adapt to largescale data sets.The performance of the algorithm is linear with the amount of data.
Aiming at the problem that the data sets of industrial monitoring data need to be processed in the feature extraction process are becoming larger and larger,and the requirement of practicality is getting higher and higher,a parallel permutation entropy algorithm based on cloud computing platform MaxCompute is designed.MaxCompute table is used to store massive monitoring data.Based on MaxCompute,MapReduce modelis extended,a parallel permutation entropy algorithm is designed to extract batch permutation entropy features of massive monitoring historical data.By testing on a standalone computer and cloud computing platform,the algorithm has good scalability and can adapt to largescale data sets.The performance of the algorithm is linear with the amount of data.
引文
[1]周国亮,宋亚奇,王桂兰,等.状态监测大数据存储及聚类划分研究[J].电工技术学报,2013,28(s2):337-344.ZHOU Guoliang,SONG Yaqi,WANG Guilan,et al.Research of conditin monitoring big data storage and clustering[J].Transactions of China Electrotechnical Society,2013,28(s2):337-344.
[2]郝成元,吴绍洪,李双成.排列熵应用于气候复杂性度量[J].地理研究,2007,26(1):46-52.HAO Chengyuan,WU Shaohong,LI Shuangcheng.Measurement of climate complexity using permutatin entropy[J].Geographical Research,2007,26(1):46-52.
[3]郑近德,程军圣,杨宇.基于LCD和排列熵的滚动轴承故障诊断[J].振动、测试与诊断,2014,34(5):802-806.
[4]姚文坡,刘铁兵,戴加飞,等.脑电信号的多尺度排列熵分析[J].物理学报,2014,63(7):078704.YAO Wenpo,LIU Tiebing,DAI Jiafei,et al.Multiscale permutation entropy analysis of electroencephalogram[J].Acta Phys.Sin.,2014,63(7):078704.
[5]孙克辉,谈国强,盛利元.基于排列熵算法的混沌伪随机序列复杂性分析[J].计算机工程与应用,2008,44(3):47-49.SUN Kehui,TAN Guoqiang,SHENG Liyuan.Analysis of chaotic pseudo-random sequence complexity based on permutation entropy[J].Computer Engineering and Applications,2008,44(3):47-49.
[6]冯辅周,饶国强,司爱威,等.排列熵算法的应用与发展[J].装甲兵工程学院学报,2012,26(2):34-38.FENG Fuzhou,RAO Guoqiang,SI Aiwei,et al.Application and development of permutation entropy algorithm[J].Journal of Academy of Armored Force Engineering,2012,26(2):34-38.
[7]FRANK B,POMPE B,SCHNEIDER U,et al.Permutation entropy improves fetal behavioural state classification based on heart rate analysis from biomagnetic recordings in near term fetuses[J].Medical&Biological Engineering&Computing,2006,44(3):179-187.
[8]冯辅周,饶国强,司爱威,等.排列熵算法研究及其在振动信号突变检测中的应用[J].振动工程学报,2012,25(2):221-224.FENG Fuzhou,RAO Guoqiang,SI Aiwei,et al.Research and application of the arithmetic of PE in testing the sudden change of vibration signal[J].Journal of Vibration Engineering,2012,25(2):221-224.
[9]吴凯峰,刘万涛,李彦虎,等.基于云计算的电力大数据分析技术与应用[J].中国电力,2015,48(2):111-116,127.WU Kaifeng,LIU Wantao,LI Yanhu,et al.Cloud-computing based power big data analysis technology and its application[J].Electric Power,2015,48(2):111-116,127.
[10]衡星辰,周力.分布式技术在电力大数据高性能处理中的应用[J].电力信息与通信技术,2013,11(9):40-43.HENG Xingchen,ZHOU Li.Application of distributed t e c h n o l o g i e s i n t h e p o w e r b i g d a t a h i g h-p e r f o r m a n c e processing[J].Electric Power Information and Communication Technology,2013,11(9):40-43.
[11]裘德新.电力设备状态监测数据的特征压缩与提取方法研究[D].南昌:华东交通大学,2017.
[12]李心蕊.MaxCompute SQL的应用与发展研究[J].信息技术,2017(5):144-146.LI Xinrui.Application and development of MaxCompute SQL[J].Information Technology,2017(5):144-146.
[13]DEAN J,GHEMAWAT S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113.
[14]齐小刚,胡秋秋,刘立芳.基于MapReduce的并行异常检测算法[J/OL].智能系统学报,2019(2):1-7.[2018-12-18].http://kns.cnki.net/kcms/detail/23.1538.tp.20181214.1001.002.html.
[15]冯祥,张媛媛.基于Hadoop的MapReduce运行流程研究[J].福建电脑,2018,34(12):118,173.
[16]王万良,胡禹.基于MapReduce的CTK加权聚类改进算法[J].微电子学与计算机,2018,35(12):105-109,114.WANG Wanliang,HU Yu.Improved CTK weighting clustering algorithm based on MapReduce[J].Microelectronics&Computer,2018,35(12):105-109,114.
[17]张翌娜,马晓君,张建伟,等.基于排列熵算法的弧形闸门振动状态监测[J].水电能源科学,2018,36(7):163-166.ZHANG Yina,MA Xiaojun,ZHANG Jianwei,et al.Vibration state monitoring of radial gate based on permutation entropy method[J].Water Resources and Power,2018,36(7):163-166.
[18]李从善,刘天琪,李兴源,等.基于排列熵算法的电力系统故障信号分析[J].电子科技大学学报,2015,44(2):233-238.LI Congshan,LIU Tianqi,LI Xingyuan,et al.Power system fault signal analysis based on permutation entropy algorithm[J].Journal of University of Electronic Science and Technology of China,2015,44(2):233-238.
[19]李刚,焦亚菲,刘福炎,等.联合采用熵权和灰色系统理论的电力大数据质量综合评估[J].电力建设,2016,37(12):24-31.LI Gang,JIAO Yafei,LIU Fuyan,et al.Comprehensive evaluation of big data quality in power systems with entropy weight and grey system theory[J].Electric Power Construction,2016,37(12):24-31.
[2]郝成元,吴绍洪,李双成.排列熵应用于气候复杂性度量[J].地理研究,2007,26(1):46-52.HAO Chengyuan,WU Shaohong,LI Shuangcheng.Measurement of climate complexity using permutatin entropy[J].Geographical Research,2007,26(1):46-52.
[3]郑近德,程军圣,杨宇.基于LCD和排列熵的滚动轴承故障诊断[J].振动、测试与诊断,2014,34(5):802-806.
[4]姚文坡,刘铁兵,戴加飞,等.脑电信号的多尺度排列熵分析[J].物理学报,2014,63(7):078704.YAO Wenpo,LIU Tiebing,DAI Jiafei,et al.Multiscale permutation entropy analysis of electroencephalogram[J].Acta Phys.Sin.,2014,63(7):078704.
[5]孙克辉,谈国强,盛利元.基于排列熵算法的混沌伪随机序列复杂性分析[J].计算机工程与应用,2008,44(3):47-49.SUN Kehui,TAN Guoqiang,SHENG Liyuan.Analysis of chaotic pseudo-random sequence complexity based on permutation entropy[J].Computer Engineering and Applications,2008,44(3):47-49.
[6]冯辅周,饶国强,司爱威,等.排列熵算法的应用与发展[J].装甲兵工程学院学报,2012,26(2):34-38.FENG Fuzhou,RAO Guoqiang,SI Aiwei,et al.Application and development of permutation entropy algorithm[J].Journal of Academy of Armored Force Engineering,2012,26(2):34-38.
[7]FRANK B,POMPE B,SCHNEIDER U,et al.Permutation entropy improves fetal behavioural state classification based on heart rate analysis from biomagnetic recordings in near term fetuses[J].Medical&Biological Engineering&Computing,2006,44(3):179-187.
[8]冯辅周,饶国强,司爱威,等.排列熵算法研究及其在振动信号突变检测中的应用[J].振动工程学报,2012,25(2):221-224.FENG Fuzhou,RAO Guoqiang,SI Aiwei,et al.Research and application of the arithmetic of PE in testing the sudden change of vibration signal[J].Journal of Vibration Engineering,2012,25(2):221-224.
[9]吴凯峰,刘万涛,李彦虎,等.基于云计算的电力大数据分析技术与应用[J].中国电力,2015,48(2):111-116,127.WU Kaifeng,LIU Wantao,LI Yanhu,et al.Cloud-computing based power big data analysis technology and its application[J].Electric Power,2015,48(2):111-116,127.
[10]衡星辰,周力.分布式技术在电力大数据高性能处理中的应用[J].电力信息与通信技术,2013,11(9):40-43.HENG Xingchen,ZHOU Li.Application of distributed t e c h n o l o g i e s i n t h e p o w e r b i g d a t a h i g h-p e r f o r m a n c e processing[J].Electric Power Information and Communication Technology,2013,11(9):40-43.
[11]裘德新.电力设备状态监测数据的特征压缩与提取方法研究[D].南昌:华东交通大学,2017.
[12]李心蕊.MaxCompute SQL的应用与发展研究[J].信息技术,2017(5):144-146.LI Xinrui.Application and development of MaxCompute SQL[J].Information Technology,2017(5):144-146.
[13]DEAN J,GHEMAWAT S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113.
[14]齐小刚,胡秋秋,刘立芳.基于MapReduce的并行异常检测算法[J/OL].智能系统学报,2019(2):1-7.[2018-12-18].http://kns.cnki.net/kcms/detail/23.1538.tp.20181214.1001.002.html.
[15]冯祥,张媛媛.基于Hadoop的MapReduce运行流程研究[J].福建电脑,2018,34(12):118,173.
[16]王万良,胡禹.基于MapReduce的CTK加权聚类改进算法[J].微电子学与计算机,2018,35(12):105-109,114.WANG Wanliang,HU Yu.Improved CTK weighting clustering algorithm based on MapReduce[J].Microelectronics&Computer,2018,35(12):105-109,114.
[17]张翌娜,马晓君,张建伟,等.基于排列熵算法的弧形闸门振动状态监测[J].水电能源科学,2018,36(7):163-166.ZHANG Yina,MA Xiaojun,ZHANG Jianwei,et al.Vibration state monitoring of radial gate based on permutation entropy method[J].Water Resources and Power,2018,36(7):163-166.
[18]李从善,刘天琪,李兴源,等.基于排列熵算法的电力系统故障信号分析[J].电子科技大学学报,2015,44(2):233-238.LI Congshan,LIU Tianqi,LI Xingyuan,et al.Power system fault signal analysis based on permutation entropy algorithm[J].Journal of University of Electronic Science and Technology of China,2015,44(2):233-238.
[19]李刚,焦亚菲,刘福炎,等.联合采用熵权和灰色系统理论的电力大数据质量综合评估[J].电力建设,2016,37(12):24-31.LI Gang,JIAO Yafei,LIU Fuyan,et al.Comprehensive evaluation of big data quality in power systems with entropy weight and grey system theory[J].Electric Power Construction,2016,37(12):24-31.