基于虚拟化的水务分布式大数据存储平台设计
|
摘要
针对自然界与人类社会二元水循环产生的海量水务观测数据,现有水务数据管理系统存在存储负荷大,数据库扩展困难,查询速度慢的问题,无法满足存储与分析的需要。针对问题,首先,结合虚拟化技术、Hadoop基础架构,设计分布式大数据存储平台的基本架构;其次,依据现有水务大数据情况及实际业务数据库表,实现分布式大数据存储平台的设计;最后,完成从集中式平台到分布式平台的数据迁移代码实现,并进行数据迁移实验测试。实验结果验证了分布式大数据存储平台设计方案的可行性与有效性,可为大规模行业数据的存储与处理提供一种理想的分布式解决方案。
In view of the massive water observation data generated by the dual water cycle of nature and human society,the existing water data management system has the problems of large storage load,difficult database expansion and slow query speed,which cannot meet the needs of storage and analysis.To solve the problems,firstly,the basic architecture of distributed big data storage platform is designed by combining the popular virtualization technology and hadoop infrastructure.Secondly,the design of distributed big data storage platform is realized according to the existing big data of water utilities and the actual business database table.Finally,the data migration code from the centralized platform to the distributed platform is completed,and the data migration experiment is carried out.The experimental results verify the feasibility and effectiveness of the design scheme of the distributed big data storage platform,which can provide an ideal distributed solution for the storage and processing of large-scale industrial data.
In view of the massive water observation data generated by the dual water cycle of nature and human society,the existing water data management system has the problems of large storage load,difficult database expansion and slow query speed,which cannot meet the needs of storage and analysis.To solve the problems,firstly,the basic architecture of distributed big data storage platform is designed by combining the popular virtualization technology and hadoop infrastructure.Secondly,the design of distributed big data storage platform is realized according to the existing big data of water utilities and the actual business database table.Finally,the data migration code from the centralized platform to the distributed platform is completed,and the data migration experiment is carried out.The experimental results verify the feasibility and effectiveness of the design scheme of the distributed big data storage platform,which can provide an ideal distributed solution for the storage and processing of large-scale industrial data.
引文
[1]史建兵.南京市高淳区智慧水务信息化系统建设初探[J].中国水利,2014(9):5253.
[2]杨明祥,蒋云钟,田雨,等.智慧水务建设需求探析[J].清华大学学报(自然科学版),2014(1):133-136.
[3]胡传廉.基于新技术条件的城市“智慧水网”发展规划初探[J].中国水利,2011(11):39-41.
[4]FAGIANI M,SQUARTINI S,GABRIELLI L,et al.Computational intelligence in Smart Water and gas grids:An up-to-date overview[C]//International Joint Conference on Neural Networks.Beijing:IEEE,2014:921-926.
[5]LIU J,LI S L,DAI M,et al. Research of earthquake big data storage based on Hbase[J].Journal of Geodesy&Geodynamics,2015,35(5):890-893.
[6]范建永,龙明,熊伟.基于HBase的矢量空间数据分布式存储研究[J].地理与地理信息科学,2012,2(5):39-42.
[7]姚林,张永库. NoSQL的分布式存储与扩展解决方法[J].计算机工程,2012,38(6):40-42.
[8]胡文波,徐造林.分布式存储方案的设计与研究[J].计算机技术与发展,2010,20(4):65-68.
[9]李彭军,陈光杰,郭文明.基于HDFS的区域医学影像分布式存储架构设计[J].南方医科大学学报,2011,31(3):495-498.
[10]吴润泽,蔡永涛,陈文伟,等.面向多源异构数据源的实际范围索引树索引方法[J].电力系统自动化,2016,40(11):121-125.
[2]杨明祥,蒋云钟,田雨,等.智慧水务建设需求探析[J].清华大学学报(自然科学版),2014(1):133-136.
[3]胡传廉.基于新技术条件的城市“智慧水网”发展规划初探[J].中国水利,2011(11):39-41.
[4]FAGIANI M,SQUARTINI S,GABRIELLI L,et al.Computational intelligence in Smart Water and gas grids:An up-to-date overview[C]//International Joint Conference on Neural Networks.Beijing:IEEE,2014:921-926.
[5]LIU J,LI S L,DAI M,et al. Research of earthquake big data storage based on Hbase[J].Journal of Geodesy&Geodynamics,2015,35(5):890-893.
[6]范建永,龙明,熊伟.基于HBase的矢量空间数据分布式存储研究[J].地理与地理信息科学,2012,2(5):39-42.
[7]姚林,张永库. NoSQL的分布式存储与扩展解决方法[J].计算机工程,2012,38(6):40-42.
[8]胡文波,徐造林.分布式存储方案的设计与研究[J].计算机技术与发展,2010,20(4):65-68.
[9]李彭军,陈光杰,郭文明.基于HDFS的区域医学影像分布式存储架构设计[J].南方医科大学学报,2011,31(3):495-498.
[10]吴润泽,蔡永涛,陈文伟,等.面向多源异构数据源的实际范围索引树索引方法[J].电力系统自动化,2016,40(11):121-125.