PetroV分布式文件系统的设计与实现
|
摘要
为了实现可存储ZB级文件数量、单一文件TB级大小的行业基础数据的需求,以及利于实践"端到端"的、驱动油气重大发现的"地质智能"深度学习解决方案,本文设计、开发并验证了一种充分考虑油气勘探行业基础数据特点的分布式文件系统(PetroV Distributed File System,PetroV DFS)。结合全球地理网格剖分编码和地质信息编码,PetroV DFS建立了基于空间位置索引的分布式文件命名、分配与管理机制,可高效管理ZB级数量的文件并快速定位;基于ST-Based KIDA元数据建模下的本地空间数据库集成机制和空间索引、八叉树切分、空间键值对等三种分布式数据子块存储机制,可快速读写一个TB量级基础数据。PetroV DFS具有"地理位置相近、存储位置相近"的特点——同一地理区域的基础数据存储于同一数据中心的同一批机柜,在提升本地基础数据的访问和计算速度的基础上,能"就近"高效利用不同类型基础数据。以440GB叠前地震数据文件的分布式存储、全时窗频率振幅属性分布式计算为例,利用C++泛化编程技术实现的PetroV DFS可有效部署于当前普通计算机中,为后续深入实践不同类型深度学习解决方案提供新的、可行的大数据存储模式。
Because of the key software architecture requirements of middleware in the field of geological intelligence in big data,big computing,and big models,this paper designs and implements a new petroleum valuation distributed file system(PetroV DFS).PetroV DFS adopts a two-level data storage mechanism including local spatial database storage under ST-based KIDA metadata modeling and distributed data chunk file storage based on geographic meshing and geological information coding.The storage kinds of distributed chunk file include spatial index chunk for basin-level geological mapping(two-dimensional digital cores),octree branch chunk for three-dimensional volume data(seismic data,three-dimensional digital cores),and spatial key-value pairs for conventional,imaging log data storage.Following the geographical mesh subdivision algorithm,PetroV DFS supports the storage of different data types from the global geographic scale to single well or even smaller scales,and there is no upper allocation limits on the number of data chunk files.Especially,if there are no any constraints in the case of scale-out inexpensive servers from different regions(e.g.,different local data center),it is no upper limits on the amount of data stored.PetroV DFS ensures that"geographically close and storage location close",namely,data from the same geographic area are stored in the same rack of the same data center.For the segmentation of 440 Gprestack seismic data files and the calculation of the full-time frequency-amplitude attributes,the generic programming of PetroV DFS can be effectively deployed on the commodity computers,and gets almost the same calculation effects preserved ever by highperformance computers in addition to significantly improving the professional data management.
Because of the key software architecture requirements of middleware in the field of geological intelligence in big data,big computing,and big models,this paper designs and implements a new petroleum valuation distributed file system(PetroV DFS).PetroV DFS adopts a two-level data storage mechanism including local spatial database storage under ST-based KIDA metadata modeling and distributed data chunk file storage based on geographic meshing and geological information coding.The storage kinds of distributed chunk file include spatial index chunk for basin-level geological mapping(two-dimensional digital cores),octree branch chunk for three-dimensional volume data(seismic data,three-dimensional digital cores),and spatial key-value pairs for conventional,imaging log data storage.Following the geographical mesh subdivision algorithm,PetroV DFS supports the storage of different data types from the global geographic scale to single well or even smaller scales,and there is no upper allocation limits on the number of data chunk files.Especially,if there are no any constraints in the case of scale-out inexpensive servers from different regions(e.g.,different local data center),it is no upper limits on the amount of data stored.PetroV DFS ensures that"geographically close and storage location close",namely,data from the same geographic area are stored in the same rack of the same data center.For the segmentation of 440 Gprestack seismic data files and the calculation of the full-time frequency-amplitude attributes,the generic programming of PetroV DFS can be effectively deployed on the commodity computers,and gets almost the same calculation effects preserved ever by highperformance computers in addition to significantly improving the professional data management.
引文
[1] 盛秀杰,金之钧,彭成,等.PetroV软件架构设计中的一些思考与实现[J].石油地球物理勘探,2015,50(4):766-774.SHENG Xiujie,JIN Zhijun,PENG Cheng,et al.Some ideas in PetroV architecture design and development[J].Oil Geophysical Prospecting,2015,50(4):766-774.
[2] 盛秀杰,金之钧,彭成,等.PetroV分布式数据存储与分析框架设计[J].石油地球物理勘探,2017,52(4):875-883.SHENG Xiujie,JIN Zhijun,PENG Cheng,et al.The design of distributed data storage and analytics framework of PetroV[J].Oil Geophysical Prospecting,2017,52(4):875-883.
[3] 孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-164.MENG Xiaofeng,CI Xiang.Big data management:Concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-164.
[4] 赵长海,晏海华,王宏琳,等.面向地震数据处理的并行与分布式编程框架[J].石油地球物理勘探,2010,45(1):147-152.ZHAO Changhai,YAN Haihua,WANG Honglin,et al.Seismic data processing oriented parallel and distributed programming framework[J].Oil Geophysical Prospecting,2010,45(1):147-152.
[5] 王珊,萨师煊.数据库系统概论(第4版)[M].北京:高等教育出版社,2005.WANG Shan,SA Shixuan.Database System Concepts(4th)[M].Higher Education Press,Beijing,2005.
[6] McKusick K,Quinlan S.GFS:Evolution on fast-forward[J].Communication of the ACM,2010,53(3):42-49.
[7] Dean J and Ghemawat S.Map reduce:simplified data processing on large clusters[J].Communication of the ACM,2008,51(3):107-113.
[8] 王宏琳,罗国安.国产地震处理解释软件的发展[J].石油地球物理勘探,2013,48(2):325-331.WANG Honglin,LUO Guoan.Seismic data processing and interpretation software progress in China[J].Oil Geophysical Prospecting,2013,48(2):325-331.
[9] 熊翥.我国物探技术的进步及展望[J].石油地球物理勘探,2003,38(5):565-578.XIONG Zhu.Progress and prospect of geophysical prospecting technology in China[J].Oil Geophysical Prospecting,2003,38(5):565-578.
[10] Brewer E A.Towards robust distributed systems[J].Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing,New York,2000.
[11] 成景旺,毛宁波,吕晓春,等.非分裂完全匹配层边界存储时间域全波形反演[J].石油地球物理勘探,2018,53(4):754-764.CHENG Jingwang,MAO Ningbo,LYU Xiaochun,et al.Time-domain full waveform inversion with CFS-NPML boundary storage[J].Oil Geophysical Prospecting,2018,53(4):754-764.
[12] 邓世广,王淑艳,赵文津,等.基于OpenMP并行计算的匹配追踪时频分析方法[J].石油地球物理勘探,2018,53(3):454-461.DENG Shiguang,WANG Shuyan,ZHAO Wenjin,et al.A matching pursuit time-frequency analysis me-thod based on OpenMP parallel computing[J].Oil Geophysical Prospecting,2018,53(3):454-461.
[13] 亢永敢,赵改善,魏嘉,等.基于Hadoop的Kirchhoff叠前时间偏移并行算法[J].石油地球物理勘探,2015,50(6):1213-1218.KANG Yonggan,ZHAO Gaishan,WEI Jia,et al.Pa-rallel algorithms of Kirchhoff prestack time migration based on Hadoop[J].Oil Geophysical Prospecting,2015,50(6):1213-1218.
[14] 张丰麒,金之钧,盛秀杰,等.贝叶斯三参数低频软约束同步反演[J].石油地球物理勘探,2016,51(5):965-975.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.Bayesian prestack three-term inversion with soft Low-frequence constraint[J].Oil Geophysical Prospecting,2016,51(5):965-975.
[15] 张丰麒,金之钧,盛秀杰,等.基于低频软约束的叠前AVA稀疏层反演[J].石油地球物理勘探,2017,52(4):770-782.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.AVA sparse layer inversion with the soft-low frequency constraint[J].Oil Geophysical Prospecting,2017,52(4):770-782.
[16] 张丰麒,金之钧,盛秀杰,等.一种稳健的弹性阻抗反演方法[J].石油地球物理勘探,2017,52(2):294-303.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.A stable elastic impedance inversion approach[J].Oil Geophysical Prospecting,2017,52(2):294-303.
[17] 张丰麒,金之钧,盛秀杰,等.基于基追踪-BI_Zoeppritz方程广义线性脆性指数直接反演方法[J].地球物理学报,2017,60(10):3954-3968.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.A direct inversion for brittleness index based on GLI with basic-pursuit decomposition[J].Chinese Journal of Geophysics,2017,60(10):3954-3968.
[2] 盛秀杰,金之钧,彭成,等.PetroV分布式数据存储与分析框架设计[J].石油地球物理勘探,2017,52(4):875-883.SHENG Xiujie,JIN Zhijun,PENG Cheng,et al.The design of distributed data storage and analytics framework of PetroV[J].Oil Geophysical Prospecting,2017,52(4):875-883.
[3] 孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-164.MENG Xiaofeng,CI Xiang.Big data management:Concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-164.
[4] 赵长海,晏海华,王宏琳,等.面向地震数据处理的并行与分布式编程框架[J].石油地球物理勘探,2010,45(1):147-152.ZHAO Changhai,YAN Haihua,WANG Honglin,et al.Seismic data processing oriented parallel and distributed programming framework[J].Oil Geophysical Prospecting,2010,45(1):147-152.
[5] 王珊,萨师煊.数据库系统概论(第4版)[M].北京:高等教育出版社,2005.WANG Shan,SA Shixuan.Database System Concepts(4th)[M].Higher Education Press,Beijing,2005.
[6] McKusick K,Quinlan S.GFS:Evolution on fast-forward[J].Communication of the ACM,2010,53(3):42-49.
[7] Dean J and Ghemawat S.Map reduce:simplified data processing on large clusters[J].Communication of the ACM,2008,51(3):107-113.
[8] 王宏琳,罗国安.国产地震处理解释软件的发展[J].石油地球物理勘探,2013,48(2):325-331.WANG Honglin,LUO Guoan.Seismic data processing and interpretation software progress in China[J].Oil Geophysical Prospecting,2013,48(2):325-331.
[9] 熊翥.我国物探技术的进步及展望[J].石油地球物理勘探,2003,38(5):565-578.XIONG Zhu.Progress and prospect of geophysical prospecting technology in China[J].Oil Geophysical Prospecting,2003,38(5):565-578.
[10] Brewer E A.Towards robust distributed systems[J].Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing,New York,2000.
[11] 成景旺,毛宁波,吕晓春,等.非分裂完全匹配层边界存储时间域全波形反演[J].石油地球物理勘探,2018,53(4):754-764.CHENG Jingwang,MAO Ningbo,LYU Xiaochun,et al.Time-domain full waveform inversion with CFS-NPML boundary storage[J].Oil Geophysical Prospecting,2018,53(4):754-764.
[12] 邓世广,王淑艳,赵文津,等.基于OpenMP并行计算的匹配追踪时频分析方法[J].石油地球物理勘探,2018,53(3):454-461.DENG Shiguang,WANG Shuyan,ZHAO Wenjin,et al.A matching pursuit time-frequency analysis me-thod based on OpenMP parallel computing[J].Oil Geophysical Prospecting,2018,53(3):454-461.
[13] 亢永敢,赵改善,魏嘉,等.基于Hadoop的Kirchhoff叠前时间偏移并行算法[J].石油地球物理勘探,2015,50(6):1213-1218.KANG Yonggan,ZHAO Gaishan,WEI Jia,et al.Pa-rallel algorithms of Kirchhoff prestack time migration based on Hadoop[J].Oil Geophysical Prospecting,2015,50(6):1213-1218.
[14] 张丰麒,金之钧,盛秀杰,等.贝叶斯三参数低频软约束同步反演[J].石油地球物理勘探,2016,51(5):965-975.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.Bayesian prestack three-term inversion with soft Low-frequence constraint[J].Oil Geophysical Prospecting,2016,51(5):965-975.
[15] 张丰麒,金之钧,盛秀杰,等.基于低频软约束的叠前AVA稀疏层反演[J].石油地球物理勘探,2017,52(4):770-782.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.AVA sparse layer inversion with the soft-low frequency constraint[J].Oil Geophysical Prospecting,2017,52(4):770-782.
[16] 张丰麒,金之钧,盛秀杰,等.一种稳健的弹性阻抗反演方法[J].石油地球物理勘探,2017,52(2):294-303.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.A stable elastic impedance inversion approach[J].Oil Geophysical Prospecting,2017,52(2):294-303.
[17] 张丰麒,金之钧,盛秀杰,等.基于基追踪-BI_Zoeppritz方程广义线性脆性指数直接反演方法[J].地球物理学报,2017,60(10):3954-3968.ZHANG Fengqi,JIN Zhijun,SHENG Xiujie,et al.A direct inversion for brittleness index based on GLI with basic-pursuit decomposition[J].Chinese Journal of Geophysics,2017,60(10):3954-3968.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |