一次基于系统性能分析的存储设备升级
|
摘要
综合气象信息共享系统(CIMISS)收集、处理、存储、共享海量的气象数据,随着设备运行多年,接入数据越来越多,系统在性能上存在瓶颈。通过分析系统业务需求和存储磁盘的IO,得出run文件系统IOPS值较大,数据吞吐量大,这部分对存储设备的性能要求较高,而ORACLE数据库和data文件系统是大数据容量存储部分,同时也对随机IO有较高的要求。基于业务分析同时兼顾性价比,配置新的存储设备,采用2个Active-Active工作模式的32 GB高速缓存控制器,配置SSD固态硬盘和高速SAS磁盘满足高性能IO需求,也有容量需求较大的4 TB SATA盘,并且能灵活扩容。实施中利用FAST VP功能实现分层数据管理,通过一段时间的稳定运行,run文件系统数据最"热",在SSD和SAS上分别占比为50. 63%和34.22%,而ORACLE数据库和data文件系统中大部分数据访问率较低,因此,这些"冷数据"会自动迁移到SA-TA盘上,分别占比64.7%和74.31%。
China integrated meteorological information service system(CIMISS) collects,processes,stores and shares massive meteorological data.With the equipment running for many years,more and more data are accessed,and the system has a bottlenecks in performance.Through the analysis of the system business requirements and IO of storage disk,it is concluded that the run file system has larger IOPS and high data throughput.The ORACLE database and the data file system need large data capacity,and also have high requirements for the random IO.Configuring new storage based on business analysis and cost performance,it uses 2 Active-Active mode controller of32 GB cache,with SSD solid disk and high speed SAS disk for IO performance requirements,and 4 TB SATA disk with capacity requirements,which can be expanded flexibly.In the implementation using FAST VP function to realize hierarchical data management,through a period of steady running,the run file system has the most"hot"data,accounting for 50.63% and 34.22% respectively on SSD and SAS.The access rate of most data in ORACLE database and data file system is low,so these"cold data"will automatically be migrated to SATA disk,accounting for 64.7% and 74.31% respectively.
China integrated meteorological information service system(CIMISS) collects,processes,stores and shares massive meteorological data.With the equipment running for many years,more and more data are accessed,and the system has a bottlenecks in performance.Through the analysis of the system business requirements and IO of storage disk,it is concluded that the run file system has larger IOPS and high data throughput.The ORACLE database and the data file system need large data capacity,and also have high requirements for the random IO.Configuring new storage based on business analysis and cost performance,it uses 2 Active-Active mode controller of32 GB cache,with SSD solid disk and high speed SAS disk for IO performance requirements,and 4 TB SATA disk with capacity requirements,which can be expanded flexibly.In the implementation using FAST VP function to realize hierarchical data management,through a period of steady running,the run file system has the most"hot"data,accounting for 50.63% and 34.22% respectively on SSD and SAS.The access rate of most data in ORACLE database and data file system is low,so these"cold data"will automatically be migrated to SATA disk,accounting for 64.7% and 74.31% respectively.
引文
[1]GANTZ J,REINSEL D.Extracting value from chaos[R].[s.l.]:IDC IVIEW,2011.
[2]GHEMAWAT S,GOBIOFF H,LEUNG S T.The Google file system[J].ACM SIGOPS Operating Systems Review,2003,37(5):29-43.
[3]SHVACHKO K,KUANG H,RADIA S,et al.The Hadoop distributed file system[C]//IEEE 26th symposium on mass storage systems and technologies.Incline Village,NV,USA:IEEE,2010:1-10.
[4]LAKSHMAN A,MALIK P.Cassandra:a decentralized structured storage system[J].ACM SIGOPS Operating Systems Review,2010,44(2):35-40.
[5]BAKER J,BOND C,CORBETT J,et al.Megastore:providing scalable,highly available storage for interactive services[J]//5th biennial conference on innovative data system research.Asilomar,CA,USA:[s.n.],2011:223-234.
[6]ARNOLD J.OpenStack swift:using,administering,and developing for swift object storage[M].[s.l.]:O’Reilly Media,Inc.,2014.
[7]杨润芝,马强,李德泉,等.内存转发模型在CIMISS数据收发系统中的应用[J].应用气象学报,2012,23(3):377-384.
[8]王旻燕,邓莉,赵芳,等.CIMISS中气象卫星数据存储和服务模型[J].安徽农业科学,2012,40(8):4785-4789.
[9]李德泉,阮宇智,杨润芝,等.基于RBAC的气象多维数据权限管理模型的建立[J].应用气象学报,2012,23(5):614-623.
[10]季永华,孙超,刘一鸣,等.CIMISS中气象观测资料处理入库效率优化方法[J].气象科技,2017,45(1):30-35.
[11]杨菲菲,杨昕,马强.基于Rman和TSM的Oracle数据库备份方法研究[J].计算机技术与发展,2017,27(7):164-169.
[12]赵芳,熊安元,张小缨,等.全国综合气象信息共享平台架构设计技术特征[J].应用气象学报,2017,28(6):750-758.
[13]熊安元,赵芳,王颖,等.全国综合气象信息共享系统的设计与实现[J].应用气象学报,2015,26(4):500-512.
[14]中国气象局.气象行业标准《气象资料分类编码》标准(QX/T?102-2009)[S].北京:气象出版社,2009.
[15]李存琛.海量数据分布式存储技术的研究与应用[D].北京:北京邮电大学,2013.
[2]GHEMAWAT S,GOBIOFF H,LEUNG S T.The Google file system[J].ACM SIGOPS Operating Systems Review,2003,37(5):29-43.
[3]SHVACHKO K,KUANG H,RADIA S,et al.The Hadoop distributed file system[C]//IEEE 26th symposium on mass storage systems and technologies.Incline Village,NV,USA:IEEE,2010:1-10.
[4]LAKSHMAN A,MALIK P.Cassandra:a decentralized structured storage system[J].ACM SIGOPS Operating Systems Review,2010,44(2):35-40.
[5]BAKER J,BOND C,CORBETT J,et al.Megastore:providing scalable,highly available storage for interactive services[J]//5th biennial conference on innovative data system research.Asilomar,CA,USA:[s.n.],2011:223-234.
[6]ARNOLD J.OpenStack swift:using,administering,and developing for swift object storage[M].[s.l.]:O’Reilly Media,Inc.,2014.
[7]杨润芝,马强,李德泉,等.内存转发模型在CIMISS数据收发系统中的应用[J].应用气象学报,2012,23(3):377-384.
[8]王旻燕,邓莉,赵芳,等.CIMISS中气象卫星数据存储和服务模型[J].安徽农业科学,2012,40(8):4785-4789.
[9]李德泉,阮宇智,杨润芝,等.基于RBAC的气象多维数据权限管理模型的建立[J].应用气象学报,2012,23(5):614-623.
[10]季永华,孙超,刘一鸣,等.CIMISS中气象观测资料处理入库效率优化方法[J].气象科技,2017,45(1):30-35.
[11]杨菲菲,杨昕,马强.基于Rman和TSM的Oracle数据库备份方法研究[J].计算机技术与发展,2017,27(7):164-169.
[12]赵芳,熊安元,张小缨,等.全国综合气象信息共享平台架构设计技术特征[J].应用气象学报,2017,28(6):750-758.
[13]熊安元,赵芳,王颖,等.全国综合气象信息共享系统的设计与实现[J].应用气象学报,2015,26(4):500-512.
[14]中国气象局.气象行业标准《气象资料分类编码》标准(QX/T?102-2009)[S].北京:气象出版社,2009.
[15]李存琛.海量数据分布式存储技术的研究与应用[D].北京:北京邮电大学,2013.