SCADA系统中的压缩算法的研究
|
摘要
历史数据库是SCADA系统的一个重要组成部分,数据库中长期积累的数据是SCADA系统的宝贵资源,为管理人员分析和处理故障提供了第一手资料,同趋势分析、事件处理、报表打印等服务紧密结合。随着铁路跨越式的的改革与发展和铁路的第六次提速的顺利进行,电铁SCADA系统朝着大系统和网络化控制系统方向发展。这种新型的PSCADA分布式系统,调度中心所需采集的信息量越来越大,要确保这些海量的数据能够实时存储,并尽可能地节约存储成本,提高存储效率,必须对数据进行有效的压缩处理。本文在充分考虑电铁SCADA系统实时历史数据库和历史数据的特点的前提下,对历史数据的压缩策略与压缩算法进行了研究,研究结果概述如下：
1)研究了SCADA系统中实时历史数据库系统。首先分析了实时历史数据库的系统特点,其次详细地说明了数据库中数据分布和数据读写流程,最后根据历史数据库中数据的特点,阐明了数据压缩的可行性和必要性。
2)研究了无损压缩技术。首先介绍了数据压缩的概念,阐述了数据压缩的原理、分类和应用。其次对无损压缩算法中包括基于统计模型的压缩算法,LZ系列基于字典模型的压缩算法和其它算法等算法的原理及实现通过实例进行全面讲解,最后使用C++实现了PPM编码、LZW编码、BWT编码的编程,对这三种编码算法进行性能比较。
3)研究了有损压缩技术。对在实时历史数据压缩中应用最为普遍的旋转门算法进行了研究,通过仿真测试说明在旋转门算法中是影响压缩质量与性能的唯一设定值。一旦压缩偏移量参数设置与实际数据特性不符,将严重限制了SDT算法的性能。因此研究了一种改进的旋转门算法,可以自适应的调整压缩偏移量,并对改进的旋转门算法进行仿真测试。
4)研究了数据压缩技术在实时历史数据库中的实现,通过动态链接库实现数据压缩和解压模块,研究数据压缩模块的总体框架,针对时间戳、开关量、整型量和模拟量分别采用了不同的数据压缩方法。
As an important component of SCADA system, historical database saves important data in SCADA system day and night, so it provides first-hand information for management analysis of processing failures. In addition, historical database plays a great part in trend analysis, event processing, report printing and other services. With the reform and development of the railway and smooth progress of The sixth speed, the PSCADA system develop to the direction of large systems and networked control systems. The information which collected by the dispatch center in the new PSCADA distributed system is growing. The data must be effectively compression to ensure the vast amounts of data to real-time storage, to save storage costs as much as possible and improve storage efficiency. This paper researches the historical data compression strategy and the compression algorithm with the premise of fully considering of the characteristics of real-time history of the PSCADA system database and historical data. The findings are summarized as follows:
1) The real-time history database system in SCADA system. Firstly, the characteristics of the real-time historical database are analyzed. Secondly, we describe the data distribution in the database and data reading and writing process. Finally, according to the characteristics of data in the historical database, we clarify the feasibility and necessity of data compression.
2) Lossless compression technique is researched. Firstly, we introduce the concept principle, classification and application of data compression. Secondly, we explain fully the principle and implementation of the lossless compression algorithms include compression algorithm based on the statistical model, LZ series based on the dictionary model and other algorithms through examples. Finally,we implement three coding algorithms including the PPM encoding, the LZW encoding, the BWT encoding by C++.
3) Lossy compression techniques are researched. The SDT algorithm which is most common in the real-time historical data compression is researched. We can see that the compression parameters E affect the compression quality and performance in the SDT algorithm through the simulation. Once E is does not match with the actual data characteristics, There will be severely limitation in the performance of the SDT algorithm. So we study an improved SDT algorithm which can adaptively adjust the E, and do the simulations.
4) The application of the data compression technology in real-time history database is researched in this paper. The data compression and decompression module are achieved through the dynamic link library. The overall framework of the data compression module is reaearched. Different signals.such as:timestamp, switch type, integer type and analog type,have different recording methods.
1)研究了SCADA系统中实时历史数据库系统。首先分析了实时历史数据库的系统特点,其次详细地说明了数据库中数据分布和数据读写流程,最后根据历史数据库中数据的特点,阐明了数据压缩的可行性和必要性。
2)研究了无损压缩技术。首先介绍了数据压缩的概念,阐述了数据压缩的原理、分类和应用。其次对无损压缩算法中包括基于统计模型的压缩算法,LZ系列基于字典模型的压缩算法和其它算法等算法的原理及实现通过实例进行全面讲解,最后使用C++实现了PPM编码、LZW编码、BWT编码的编程,对这三种编码算法进行性能比较。
3)研究了有损压缩技术。对在实时历史数据压缩中应用最为普遍的旋转门算法进行了研究,通过仿真测试说明在旋转门算法中是影响压缩质量与性能的唯一设定值。一旦压缩偏移量参数设置与实际数据特性不符,将严重限制了SDT算法的性能。因此研究了一种改进的旋转门算法,可以自适应的调整压缩偏移量,并对改进的旋转门算法进行仿真测试。
4)研究了数据压缩技术在实时历史数据库中的实现,通过动态链接库实现数据压缩和解压模块,研究数据压缩模块的总体框架,针对时间戳、开关量、整型量和模拟量分别采用了不同的数据压缩方法。
As an important component of SCADA system, historical database saves important data in SCADA system day and night, so it provides first-hand information for management analysis of processing failures. In addition, historical database plays a great part in trend analysis, event processing, report printing and other services. With the reform and development of the railway and smooth progress of The sixth speed, the PSCADA system develop to the direction of large systems and networked control systems. The information which collected by the dispatch center in the new PSCADA distributed system is growing. The data must be effectively compression to ensure the vast amounts of data to real-time storage, to save storage costs as much as possible and improve storage efficiency. This paper researches the historical data compression strategy and the compression algorithm with the premise of fully considering of the characteristics of real-time history of the PSCADA system database and historical data. The findings are summarized as follows:
1) The real-time history database system in SCADA system. Firstly, the characteristics of the real-time historical database are analyzed. Secondly, we describe the data distribution in the database and data reading and writing process. Finally, according to the characteristics of data in the historical database, we clarify the feasibility and necessity of data compression.
2) Lossless compression technique is researched. Firstly, we introduce the concept principle, classification and application of data compression. Secondly, we explain fully the principle and implementation of the lossless compression algorithms include compression algorithm based on the statistical model, LZ series based on the dictionary model and other algorithms through examples. Finally,we implement three coding algorithms including the PPM encoding, the LZW encoding, the BWT encoding by C++.
3) Lossy compression techniques are researched. The SDT algorithm which is most common in the real-time historical data compression is researched. We can see that the compression parameters E affect the compression quality and performance in the SDT algorithm through the simulation. Once E is does not match with the actual data characteristics, There will be severely limitation in the performance of the SDT algorithm. So we study an improved SDT algorithm which can adaptively adjust the E, and do the simulations.
4) The application of the data compression technology in real-time history database is researched in this paper. The data compression and decompression module are achieved through the dynamic link library. The overall framework of the data compression module is reaearched. Different signals.such as:timestamp, switch type, integer type and analog type,have different recording methods.
引文
[1]张志檀.实时数据库原理及应用[M].北京：中国石化出版社,2001.
[2]钱清泉.电气化铁道微机监控技术[M].北京：中国铁道出版社,2000.
[3]钟山,彭俊斌.浅析工业监控组态软件的应用及发展[J].江苏船舶,2002,(5)：22-25.
[4]傅钦翠.新一代调度自动化系统[J].华东交通大学学报,2003,20(5)：87-90.
[5]赖前程.电铁实时监控系统组播通信安全性研究[D].四川：西南交通大学,2007.
[6]Mark Nelson著,贾起东译.数据压缩技术原理与范例[M].北京：科学出版社,1995.
[7]于翔.数据压缩技术分析[J].青海大学学报(自然科学版),2002,20(5)：52-54.
[8]D.A.Huffman, A method for the construction of minimum redundancy codes, proceedings of the I.R.E,1952,12.
[9]吴国清,陈虹.一种科学数据无损压缩算法[J].计算机工程与应用,2006,05：172-175.
[10]Dmitry Shkarin, PPM:one step to practicality, http://www.freebsdchina.prg/.
[11]Ziv.J, LEMPEL.A, A universal algorithm for sequential data compression[J]. IEEE Transactions on Information Theory 23(1977):337-343.
[12]Ziv.J, LEMPEL.A, Compression of indibidual sequences via variable-rate coding[J]. IEEE Transactions on Information Theory 24(1978):530-536.
[13]Mark Nelson, LZW Data compression, http://www.datacompression.info/.
[14]Mark Nelson, Data compression with the Burrows-Wheeler Transform, http://www.datacompression.info/.
[15]袁占亭,张秋余,冯涛等.数据通信中文本文件无损压缩算法的实现[J].计算机工程与应用,2001,9：44-46.
[16]王咏刚,笨笨数据压缩教程,http://www.contextfree.net.
[17]李飞鹏,秦前清,李德仁.海量遥感影像数据库实时压缩系统的设计与实现[J].计算机工程与应用,2003,26：9-11.
[18]白慧慧,丁晓明,张韬.一种基于上下文的树形加权概率统计算法[J].电声技术,2004,2.
[19]徐长斌,吕子平.动态交通路况信息的高效存取方案[J].公路交通科技：37-38.
[20]赵志军,穆海军.基于小波神经网络的心电数据压缩方法研究[J].自动化技术与应用[J],2007,26(7)：1-3.
[21]傅慧灵,地图数字化中矢量数据压缩算法研究,硕士学位论文,太原理工大学,2005年5月.
[22]彭春华,林中达.PI实时数据库及其在电厂SIS系统中的应用[LJ].工业控制计算机, 2003,16(6)：28-33.
[23]王同庆.实时系统中的历史数据库[J].自动化仪表,1996,17(12)：32-34.
[24]J Stankovic, S Son, J Hansson.Misconceptions about real-time databases [J]. IEEE Computer,1999:32(6):29-36.
[25]K Ramamritham.Real-time databases [J].International Journal of Distributed and Parallel Databases.1993,1:199-226.
[26]熊道辉.基于Qt的SCADA系统数据服务器软件设计[D].四川：西南交通大学,2011.
[27]高宁波,金宏,王宏安.历史数据实时压缩方法研究[J].计算机工程与应用,2004,28：167-171.
[28]王同庆.实时数据库原理及压缩技术分析[J].华北电力技术,2004,6：17-19.
[29]潘海涛,何洁月.流程工业实时数据压缩技术的研究与实现[J].计算机工程,2003,2：275-278.
[30]黄文君,徐慧.数据压缩技术在实时数据库中的应用研究[J].仪器仪表学报,2006,27(6)：911-914.
[31]文水英.实时数据库中历史数据压缩算法的研究[D].长沙：中南大学,2008.
[32]吴家安.数据压缩技术及应用.北京：科学出版社,2009.
[33]姜磊,黄广君.自适应的无损数据压缩算法[J].计算机工程,2008,34(1)：101-104.
[34]Cleary J G, Teahan W J.Unbounded length Contexts for PPM[J].Thev Computer Journal.1993,36(5):32-36.
[35]徐慧.实时数据库中数据压缩算法的研究[D].杭州：浙江大学,2006.
[36]Hiuk Jae Shim, Byeungwoo Jeon.DH-LZW:lossless data hiding method in LZW compression[C].PCM 2004, LNCS 3333, 739-746, Springer-Verlag Berlin Heidelberg, 2004.
[37]赵利强,于涛,王建林.基于SQL数据库的过程数据压缩方法[J].计算机工程,2008,34(14)：58-59.
[38]张伟,盛剑国,黄文君.实时数据在线压缩与解压缩方法：中国,200310108294.1[P].2005-5-4.
[39]汤同奎,王豪,邵惠鹤.过程数据压缩技术综述[J].计算机与应用化学,2000,17(3)：193-197.
[40]赵众,顾幸生,蒋慰孙.基于正交小波变换的化工生产过程数据分析.华东理工大学学报,1998,24(2)：192-197,234.
[41]Hale J C, Sellars H L. Historieal data recording for process [J].Comput Chem Eng Prog, 1981,77(1):38-43.
[42]邹嵘,何星,李根蔚.分段线性趋势数据压缩算法在电力调度系统中的应用[J].测 控技术,2003,22(4)：24-25.
[43]王正洪.改进的SDT算法及其在过程数据压缩中的应用[J].煤矿自动化,2000,6：9-11.
[44]Mah.R.S.H, Tamhane.A.C, Tung.S.H.etc.Process trending with Piecewise linear smoothing[J].Computers Chem.Engng.1995,19(2):129-137.
[45]Gray R M. Vector quantization [J]. IEEE ASAP Magazine, Apr.1984:4-29.
[46]Franzen A.Compression of process data with a wavelet method[J].Steel Research.1998, 69(1):28-31.
[47]梁淑宝,张培林,曹建军.过程数据压缩在故障诊断中的应用[J].微计算机信息,2010,26(9)：24-25.
[48]Bristol E H. Swinging Door Trending:Adaptive Trend Recording[J]. ISA National Conference Proceedings,1990:749-753.
[49]张宝军,平玲娣,潘雪增等.高速网络环境下数据的压缩存储[J].浙江大学学报(工学版),2009,43(9)：1585-1591.
[50]胡寿松.自动控制原理[M].北京：科学出版社,2001.
[51]周钊.基于实时数据库的历史数据处理技术研究[D].北京：东北电力大学,2008.
[52]王超,张东来,张斌等.电力系统周期性数据大比率压缩算法[J].电力系统自动化,2009,33(24)：34-37.
[2]钱清泉.电气化铁道微机监控技术[M].北京：中国铁道出版社,2000.
[3]钟山,彭俊斌.浅析工业监控组态软件的应用及发展[J].江苏船舶,2002,(5)：22-25.
[4]傅钦翠.新一代调度自动化系统[J].华东交通大学学报,2003,20(5)：87-90.
[5]赖前程.电铁实时监控系统组播通信安全性研究[D].四川：西南交通大学,2007.
[6]Mark Nelson著,贾起东译.数据压缩技术原理与范例[M].北京：科学出版社,1995.
[7]于翔.数据压缩技术分析[J].青海大学学报(自然科学版),2002,20(5)：52-54.
[8]D.A.Huffman, A method for the construction of minimum redundancy codes, proceedings of the I.R.E,1952,12.
[9]吴国清,陈虹.一种科学数据无损压缩算法[J].计算机工程与应用,2006,05：172-175.
[10]Dmitry Shkarin, PPM:one step to practicality, http://www.freebsdchina.prg/.
[11]Ziv.J, LEMPEL.A, A universal algorithm for sequential data compression[J]. IEEE Transactions on Information Theory 23(1977):337-343.
[12]Ziv.J, LEMPEL.A, Compression of indibidual sequences via variable-rate coding[J]. IEEE Transactions on Information Theory 24(1978):530-536.
[13]Mark Nelson, LZW Data compression, http://www.datacompression.info/.
[14]Mark Nelson, Data compression with the Burrows-Wheeler Transform, http://www.datacompression.info/.
[15]袁占亭,张秋余,冯涛等.数据通信中文本文件无损压缩算法的实现[J].计算机工程与应用,2001,9：44-46.
[16]王咏刚,笨笨数据压缩教程,http://www.contextfree.net.
[17]李飞鹏,秦前清,李德仁.海量遥感影像数据库实时压缩系统的设计与实现[J].计算机工程与应用,2003,26：9-11.
[18]白慧慧,丁晓明,张韬.一种基于上下文的树形加权概率统计算法[J].电声技术,2004,2.
[19]徐长斌,吕子平.动态交通路况信息的高效存取方案[J].公路交通科技：37-38.
[20]赵志军,穆海军.基于小波神经网络的心电数据压缩方法研究[J].自动化技术与应用[J],2007,26(7)：1-3.
[21]傅慧灵,地图数字化中矢量数据压缩算法研究,硕士学位论文,太原理工大学,2005年5月.
[22]彭春华,林中达.PI实时数据库及其在电厂SIS系统中的应用[LJ].工业控制计算机, 2003,16(6)：28-33.
[23]王同庆.实时系统中的历史数据库[J].自动化仪表,1996,17(12)：32-34.
[24]J Stankovic, S Son, J Hansson.Misconceptions about real-time databases [J]. IEEE Computer,1999:32(6):29-36.
[25]K Ramamritham.Real-time databases [J].International Journal of Distributed and Parallel Databases.1993,1:199-226.
[26]熊道辉.基于Qt的SCADA系统数据服务器软件设计[D].四川：西南交通大学,2011.
[27]高宁波,金宏,王宏安.历史数据实时压缩方法研究[J].计算机工程与应用,2004,28：167-171.
[28]王同庆.实时数据库原理及压缩技术分析[J].华北电力技术,2004,6：17-19.
[29]潘海涛,何洁月.流程工业实时数据压缩技术的研究与实现[J].计算机工程,2003,2：275-278.
[30]黄文君,徐慧.数据压缩技术在实时数据库中的应用研究[J].仪器仪表学报,2006,27(6)：911-914.
[31]文水英.实时数据库中历史数据压缩算法的研究[D].长沙：中南大学,2008.
[32]吴家安.数据压缩技术及应用.北京：科学出版社,2009.
[33]姜磊,黄广君.自适应的无损数据压缩算法[J].计算机工程,2008,34(1)：101-104.
[34]Cleary J G, Teahan W J.Unbounded length Contexts for PPM[J].Thev Computer Journal.1993,36(5):32-36.
[35]徐慧.实时数据库中数据压缩算法的研究[D].杭州：浙江大学,2006.
[36]Hiuk Jae Shim, Byeungwoo Jeon.DH-LZW:lossless data hiding method in LZW compression[C].PCM 2004, LNCS 3333, 739-746, Springer-Verlag Berlin Heidelberg, 2004.
[37]赵利强,于涛,王建林.基于SQL数据库的过程数据压缩方法[J].计算机工程,2008,34(14)：58-59.
[38]张伟,盛剑国,黄文君.实时数据在线压缩与解压缩方法：中国,200310108294.1[P].2005-5-4.
[39]汤同奎,王豪,邵惠鹤.过程数据压缩技术综述[J].计算机与应用化学,2000,17(3)：193-197.
[40]赵众,顾幸生,蒋慰孙.基于正交小波变换的化工生产过程数据分析.华东理工大学学报,1998,24(2)：192-197,234.
[41]Hale J C, Sellars H L. Historieal data recording for process [J].Comput Chem Eng Prog, 1981,77(1):38-43.
[42]邹嵘,何星,李根蔚.分段线性趋势数据压缩算法在电力调度系统中的应用[J].测 控技术,2003,22(4)：24-25.
[43]王正洪.改进的SDT算法及其在过程数据压缩中的应用[J].煤矿自动化,2000,6：9-11.
[44]Mah.R.S.H, Tamhane.A.C, Tung.S.H.etc.Process trending with Piecewise linear smoothing[J].Computers Chem.Engng.1995,19(2):129-137.
[45]Gray R M. Vector quantization [J]. IEEE ASAP Magazine, Apr.1984:4-29.
[46]Franzen A.Compression of process data with a wavelet method[J].Steel Research.1998, 69(1):28-31.
[47]梁淑宝,张培林,曹建军.过程数据压缩在故障诊断中的应用[J].微计算机信息,2010,26(9)：24-25.
[48]Bristol E H. Swinging Door Trending:Adaptive Trend Recording[J]. ISA National Conference Proceedings,1990:749-753.
[49]张宝军,平玲娣,潘雪增等.高速网络环境下数据的压缩存储[J].浙江大学学报(工学版),2009,43(9)：1585-1591.
[50]胡寿松.自动控制原理[M].北京：科学出版社,2001.
[51]周钊.基于实时数据库的历史数据处理技术研究[D].北京：东北电力大学,2008.
[52]王超,张东来,张斌等.电力系统周期性数据大比率压缩算法[J].电力系统自动化,2009,33(24)：34-37.