应用分布式存储技术优化省级CIMISS数据服务能力
|
摘要
随着气象数据量的不断增长,进一步提升CIMISS数据管理和服务能力的需求变得日益迫切。为解决存储系统动态扩展能力不足、并行计算与吞吐效率低下等限制CIMISS继续发展的问题,采用分布式文件系统和NAS技术替代GPFS建设共享文件系统,实现非结构化气象数据的存储功能;采用分布式数据库替代Oracle RAC建设关系数据库管理系统,实现结构化气象数据的存储功能和非结构化气象数据的索引功能。实践证明,该方案能够有效地改善CIMISS的数据存储能力、并发响应能力,适应未来气象业务对数据存储和应用的需求。
As the meteorological data continuously grows,the needs to further improve the data management and service capacity of CIMISS(China Integrated Meteorological Information Sharing System)has become increasingly urgent.The storage system of CIMISS has some problems,such as insufficient dynamic expansion ability,poor parallel computing ability and low data throughput.To solve these problems that restrict the development of CIMISS,agroup of distributed file systems and a distributed database system are established. Combining with NAS(Network Attached Storage)technology,the distributed file system replaces GPFS(General Parallel File System)and sets up shared file systems to store the unstructured meteorological data.The distributed database system replaces Oracle RAC and sets up an RDBMS(Relational Database Management System)to store the structured meteorological data and index meteorological data files.The practice has proved that the proposed scheme can effectively improve the data storage capacity and concurrent response ability of CIMISS to meet the needs of meteorological operations for data storage and application in the future.
As the meteorological data continuously grows,the needs to further improve the data management and service capacity of CIMISS(China Integrated Meteorological Information Sharing System)has become increasingly urgent.The storage system of CIMISS has some problems,such as insufficient dynamic expansion ability,poor parallel computing ability and low data throughput.To solve these problems that restrict the development of CIMISS,agroup of distributed file systems and a distributed database system are established. Combining with NAS(Network Attached Storage)technology,the distributed file system replaces GPFS(General Parallel File System)and sets up shared file systems to store the unstructured meteorological data.The distributed database system replaces Oracle RAC and sets up an RDBMS(Relational Database Management System)to store the structured meteorological data and index meteorological data files.The practice has proved that the proposed scheme can effectively improve the data storage capacity and concurrent response ability of CIMISS to meet the needs of meteorological operations for data storage and application in the future.
引文
[1]熊安元,赵芳,王颖,等.全国综合气象信息共享系统的设计与实现[J].应用气象学报,2015,26(4):500-512.
[2]赵芳,熊安元,张小缨,等.全国综合气象信息共享平台架构设计技术特征[J].应用气象学报,2017,28(6):750-757.
[3]王旻燕,邓莉,赵芳,等.CIMISS中气象卫星数据存储和服务模型[J].安徽农业科学,2012,40(8):4785-4789.
[4]马渝勇,徐晓莉,宋智,等.省级气象信息共享系统的设计与实现[J].应用气象学报,2011,22(4):505-512.
[5]沈文海,赵芳,高华云,等.国家级气象资料存储检索系统的建立[J].应用气象学报,2004,15(6):727-736.
[6]钱建梅,孙安来,徐喆,等.风云气象卫星数据存档与服务系统[J].应用气象学报,2012,23(3):369-376.
[7]李集明,沈文海,王国复.气象信息共享平台及其关键技术研究[J].应用气象学报,2006,17(5):621-628.
[8]季永华,孙超,刘一鸣,等.CIMISS中气象观测资料处理入库效率优化方法[J].气象科技,2017,45(2):29-34.
[9]姜召凤.Oracle RAC数据库缓存优化方法研究[D].大连:大连海事大学,2009.
[10]周晓丹.基于Oracle RAC平台的海量数据DML处理性能[D].厦门:厦门大学,2007.
[11]周晓丹,冯少荣,薛永生.Oracle 10g RAC核心技术研究与分析[J].计算机工程,2007,33(7):53-55.
[12]周江,王伟平,孟丹,等.面向大数据分析的分布式文件系统关键技术[J].计算机研究与发展,2014,51(2):382-394.
[13]赵铁柱.分布式文件系统性能建模及应用研究[D].广州:华南理工大学,2011.
[14]董乾豪.通用可扩展的分布式文件系统性能测试框架研究与实现[D].南京:南京大学,2016.
[15]康俊锋.云计算环境下高分辨率遥感影像存储与高效管理技术研究[D].杭州:浙江大学,2011.
[16]姚杰.分布式存储系统文件级连续数据保护技术研究[D].武汉:华中科技大学,2009.
[17]王瑾,刘洋,曾坚永.等.基于多源数据的分布式实时数据库迁移技术[J].华电技术,2017,39(6):10-13,36,77.
[2]赵芳,熊安元,张小缨,等.全国综合气象信息共享平台架构设计技术特征[J].应用气象学报,2017,28(6):750-757.
[3]王旻燕,邓莉,赵芳,等.CIMISS中气象卫星数据存储和服务模型[J].安徽农业科学,2012,40(8):4785-4789.
[4]马渝勇,徐晓莉,宋智,等.省级气象信息共享系统的设计与实现[J].应用气象学报,2011,22(4):505-512.
[5]沈文海,赵芳,高华云,等.国家级气象资料存储检索系统的建立[J].应用气象学报,2004,15(6):727-736.
[6]钱建梅,孙安来,徐喆,等.风云气象卫星数据存档与服务系统[J].应用气象学报,2012,23(3):369-376.
[7]李集明,沈文海,王国复.气象信息共享平台及其关键技术研究[J].应用气象学报,2006,17(5):621-628.
[8]季永华,孙超,刘一鸣,等.CIMISS中气象观测资料处理入库效率优化方法[J].气象科技,2017,45(2):29-34.
[9]姜召凤.Oracle RAC数据库缓存优化方法研究[D].大连:大连海事大学,2009.
[10]周晓丹.基于Oracle RAC平台的海量数据DML处理性能[D].厦门:厦门大学,2007.
[11]周晓丹,冯少荣,薛永生.Oracle 10g RAC核心技术研究与分析[J].计算机工程,2007,33(7):53-55.
[12]周江,王伟平,孟丹,等.面向大数据分析的分布式文件系统关键技术[J].计算机研究与发展,2014,51(2):382-394.
[13]赵铁柱.分布式文件系统性能建模及应用研究[D].广州:华南理工大学,2011.
[14]董乾豪.通用可扩展的分布式文件系统性能测试框架研究与实现[D].南京:南京大学,2016.
[15]康俊锋.云计算环境下高分辨率遥感影像存储与高效管理技术研究[D].杭州:浙江大学,2011.
[16]姚杰.分布式存储系统文件级连续数据保护技术研究[D].武汉:华中科技大学,2009.
[17]王瑾,刘洋,曾坚永.等.基于多源数据的分布式实时数据库迁移技术[J].华电技术,2017,39(6):10-13,36,77.