地震数据处理中的并行计算技术研究
摘要
地震数据处理一直是高性能计算的主要领域之一,本文以海量地震数据处理为研究对象,其主要工作和创新点包括三方面的内容:设计并实现了网络化的并行地震数据处理支撑框架;为框架设计了一个基于netCDF数据格式的并行I/O子系统,并对其关键算法进行了深入研究;将MPI+OpenMP模型引入到现有的并行地震数据处理中,实现了一个可以支持多种并行地震数据处理模型的支撑库。
首先,设计并实现了网络化的并行地震数据处理框架,框架为实验人员提供一个集成的实验环境,内容包括地震数据处理和数据显示分析。框架提供的流程构建器允许用户在客户端利用服务器上模块库中的模块来构建处理流程、提供交互式的模块参数设置界面,具有新建工程、模板保存等强大的编辑功能。平台同时为地震数据处理模块开发者提供一个集成的开发平台,利用该平台定义的具有较好容错能力的API接口,开发者不需要了解与平台相关的分布式计算相关知识,专注模块处理算法的开发。实践证明,借助框架可使油气勘探研究人员开发并行数据处理和显示模块的效率大幅度提高。
其次,在深入分析原有并行文件系统和netCDF地震数据格式特点的基础上,为并行地震数据处理系统设计了一个基于netCDF的高效、高可靠的专用并行文件系统,重点研究了文件系统的文件分片算法,数据备份/恢复技术,并在Linux集群上实现该系统原型。实验表明,文件系统可以使地震数据处理最常见的简单并行达到很高的并行效率,且文件越大这种优势越明显。
最后,将基于消息传递(MPI)的并行地震处理模型与共享存储(OpenMP)模型相结合,实现了一个适合于SMP集群并行地震数据支撑库,将涉及到消息传递的并行地震数据操作以直观的API的形式提供给开发人员,利用支撑库提供的API开发了一些测试模块,实验证明:支撑库可支持现有地震数据处理和显示模块的多种并行计算模型,并且能够获得较高的并行加速比和计算效率。
Seismic data processing is a main areas of high performance computing,Taking seismic data processing for its research object, The main content and innovative points of this paper can be outlined as the following aspects: We present a integrated software framework for parallel seismic data processing on pc cluster,design a parallel I/O subsystem based on netCDF for the framework, Combines the existing parallel seismic data processing patterns based on MPI with share memory programming paradigm(openMP), providing a paralle seismic data processing support library for the framework.
First,We present a integrated software framework for parallel seismic data processing on pc cluster. A modern and feature-rich graphical user interface (GUI) was developed for the system allowing edit and submission of parallel processing flows and interaction with running jobs. Multiple jobs can be executed on the pc cluster and controlled from the same GUI. The framework provides a module integrated development environment (MIDE) to pc cluster module developer. system monitor, database, memory management and the MPI functionalities are provided to developer as API, which can significantly simplify the code development for the system. A multi-user development and processing environment is available. The system can greatly reduce time for transplanting serial processing modules to pc cluster, and get a high speedup.
Second,After in-depth analysis the existing parall filesystem and netCDF data format,this paper design a efficient, reliable parallel I/O filesystem base on netCDF for the framework,the filesystem uses a new data scatter algorithm,Comparing to the existing parall filesystem,the filesystem improve greatly in fault-tolerant and data backup/recovery, The experiment demonstrates can thoroughly support Simple seismic data processing patterns and get a high parall speedup.
Last,This paper combines the existing parallel seismic data processing patterns based on MPI with share memory programming paradigm(OpenMP),providing a parallel seismic data processing support library suitable for SMP cluster, The experiment demonstrates that the platform can thoroughly support current seismic data processing patterns and get a high parall speedup.
首先,设计并实现了网络化的并行地震数据处理框架,框架为实验人员提供一个集成的实验环境,内容包括地震数据处理和数据显示分析。框架提供的流程构建器允许用户在客户端利用服务器上模块库中的模块来构建处理流程、提供交互式的模块参数设置界面,具有新建工程、模板保存等强大的编辑功能。平台同时为地震数据处理模块开发者提供一个集成的开发平台,利用该平台定义的具有较好容错能力的API接口,开发者不需要了解与平台相关的分布式计算相关知识,专注模块处理算法的开发。实践证明,借助框架可使油气勘探研究人员开发并行数据处理和显示模块的效率大幅度提高。
其次,在深入分析原有并行文件系统和netCDF地震数据格式特点的基础上,为并行地震数据处理系统设计了一个基于netCDF的高效、高可靠的专用并行文件系统,重点研究了文件系统的文件分片算法,数据备份/恢复技术,并在Linux集群上实现该系统原型。实验表明,文件系统可以使地震数据处理最常见的简单并行达到很高的并行效率,且文件越大这种优势越明显。
最后,将基于消息传递(MPI)的并行地震处理模型与共享存储(OpenMP)模型相结合,实现了一个适合于SMP集群并行地震数据支撑库,将涉及到消息传递的并行地震数据操作以直观的API的形式提供给开发人员,利用支撑库提供的API开发了一些测试模块,实验证明:支撑库可支持现有地震数据处理和显示模块的多种并行计算模型,并且能够获得较高的并行加速比和计算效率。
Seismic data processing is a main areas of high performance computing,Taking seismic data processing for its research object, The main content and innovative points of this paper can be outlined as the following aspects: We present a integrated software framework for parallel seismic data processing on pc cluster,design a parallel I/O subsystem based on netCDF for the framework, Combines the existing parallel seismic data processing patterns based on MPI with share memory programming paradigm(openMP), providing a paralle seismic data processing support library for the framework.
First,We present a integrated software framework for parallel seismic data processing on pc cluster. A modern and feature-rich graphical user interface (GUI) was developed for the system allowing edit and submission of parallel processing flows and interaction with running jobs. Multiple jobs can be executed on the pc cluster and controlled from the same GUI. The framework provides a module integrated development environment (MIDE) to pc cluster module developer. system monitor, database, memory management and the MPI functionalities are provided to developer as API, which can significantly simplify the code development for the system. A multi-user development and processing environment is available. The system can greatly reduce time for transplanting serial processing modules to pc cluster, and get a high speedup.
Second,After in-depth analysis the existing parall filesystem and netCDF data format,this paper design a efficient, reliable parallel I/O filesystem base on netCDF for the framework,the filesystem uses a new data scatter algorithm,Comparing to the existing parall filesystem,the filesystem improve greatly in fault-tolerant and data backup/recovery, The experiment demonstrates can thoroughly support Simple seismic data processing patterns and get a high parall speedup.
Last,This paper combines the existing parallel seismic data processing patterns based on MPI with share memory programming paradigm(OpenMP),providing a parallel seismic data processing support library suitable for SMP cluster, The experiment demonstrates that the platform can thoroughly support current seismic data processing patterns and get a high parall speedup.
引文
[1] (美)亨尼西,(美)帕特森,郑纬民等译,计算机体系结构:量化研究方法.第三版.北京:电子工业出版社,2004
[2] 陈国良.并行计算机体系结构.第一版.北京:高等教育出版社,2002
[3] Avery Ching, Kenin Coloma, Alok. Challenges for Parallel I/O in Grid Computing. In:Engineering the Grid: Status and Perspective, American: Scientific Publishers, 2006,625-632
[4] 杨红霞,赵改善. 21 世纪的地震数据处理系统,石油物探.2001,13(4):126-141
[5] Ching Avery, Liao Wei-keng. Evaluating I/O Characteristics and Methods for Storing Structured Scientific Data. In:Proceedings of the 20th International Parallel and Distributed Processing Symposium. IEEE,2006,1-15
[6] Kenin Coloma, Alok Choudhary, Avery Ching, et al. Power and Performance in I/O for Scientific Applications. In:Proceedings of the Next Generation Software Workshop, IEEE,2005,1-8
[7] 李冀,李晓明,陆桑璐.数据筛选技术在并行 I/O 中的应用.电子学报,2001,29(2): 249-252
[8] Carns P, Ligon W, Ross R, et al. PVFS: A Parallel Virtual File System for Linux Clusters.In:Proceedings of the 4th Annual Linux Showcase and Conference. Atlanta:GA,2000,317-327
[9] 张若洋,吕杨.PVFS 在 linux 集群上的应用研究.西南科技大学学报,2006,21(1): 10-17
[10] The parallel virtual file system 2 (PVFS2). http://www.pvfs.org/pvfs2/, 2007-12-20
[11] Seamons K E, Chen Y, Jones P. Server directed collective I/O in Panda. In: Proceedings of Supercomputing. California:ACM,1995,57-62
[12] Kotz D. Disk-Directed I/O for MIMD Multiprocessors. ACM Transactions on Computer Systems (TOCS).1997,15(1):41-74
[13] Wesley Perkins. Parallel geophysical processing. The Leading Edg, 1996, 15(12):1359-1361
[14] 王 宏 琳 , 高 绘 生 . 地 震 并 行 处 理 模 式 与 应 用 框 架 . 计 算 机 学 报 ,2001,24(2): 202-208
[15] 魏嘉.可扩展的地震数据并行处理.石油物探.1997,4:12-17
[16] 都志辉,高性能计算之并行编程技术-MPI 并行程序设计.北京:清华大学出版社,2001
[17] 陈文光,武永卫.MPI 与 OpenMP 并行程序设计.北京:清华大学出版社,2004
[18] 张云泉.高性能计算-体系结构,并行编程,性能评价与展望.http://www.rdcps. ac.cn/,2007-12-22
[19] 黄铠,徐志伟.可扩展并行计算-技术结构与编程.北京:机械工业出版社,2000
[20] Kurt Geihs. Middleware Challenges Ahead. computer IEEE,2001,34(6):24-31
[21] Doug Wille. Immersive environments enhance team collaboration. World Oil , 1999 , 220(5) :72-74
[22] Fred Aminzadeh. Future geophysical technology trends. The Leading Edge of Geophysics , 1996 ,15(6):729-735
[23] 李蕾,方明科,杜欣.计算机机群技术及其在Web领域中的应用. 现代计算机 . 2006,5: 60-62
[24] Dimitri Bevc. Internet based seismic processing: The future of geophysical computing. In:Annual Meeting Abstracts , SEG, 2000, 2119-2122
[25] William K Aylor. A business case for seismic data transmission by satellite. In:Annual Meeting Abstracts, SEG, 2000,187-235
[26] 杨晓云,庄建军,李淑英.集群技术在石油地震处理中的应用研究.高性能计算技术.2003,163:43-45
[27] 赵改善. 集群技术及其在石油工业中的应用. 石油物探.2001,40(3) :118-126
[28] Herman Chang. 3-D prestack Kirchhoff depth migration: From prototype to production in a massively parallel processorof geophysical computing. In: Annual Meeting Abstracts . SEG, 2000, 2119-2122
[29] 李家康. 大规模并行处理机三维叠前深度偏移.石油物探. 1998, 37 (2): 77-87
[30] Hengchang Dai. Parallel processing of Prestack Kirchhoff Time Migration on a PC Cluster. Computers and Geosciences.2005,31(1):891-899
[31] Chen Li. Parallel performance optimization of large-scale unstructured data visualization for the earth simulator. In Proceedings of the Fourth Eurographics Workshop on Parallel Graphics and Visualization. 2002, 133-140
[32] 陈 莉 . 大 规 模 数 据 场 的 并 行 可 视 化 . 浙 江 大 学 学 报 ( 理 学 版 ).2001,28(2): 222-226
[33] 何成万,何克清.基于角色的设计模式建模和实现方法.软件学报,2006,17(4): 658-669
[34] 曾 闽 山 , 侯 岩 松 . 海 量 地 震 数 据 网 格 化 算 法 分 析 与 研 究 . 石 油 天 然 气 学报,2006,28(2):72-75
[35] 曹凤海,易昌华,李秀山.地震勘探测量数据处理的全面解决方案,物探装备.2006,16(1):59-62
[36] 陈茂山,王云高.油气勘探软件用户界面设计方法探索与实践.勘探地球物理进展.2004,27(2):132-138
[37] 刘宏涛, 李义杰, 于长江.基于 B/S 模式三层结构地震信息管理系统的设计.东北地震研究.2006,22(1):75-80
[38] 李冀,陈晓林,陆桑璐.一个基于 NOW 的并行 I/O 系统.软件学报,2000,12(11): 1654-1659
[39] 赵欣,陈道蓄,谢立.一个基于 NOW 的跨平台并行文件系统的设计和实现.计算机科学.2000,27(2):21-25
[40] 李群,谢立,孙钟秀.并行文件系统的设计.计算机科学.1996,23(2):36-39
[41] Li Jiangwei, Liao Weikeng, Alok Choudhary, Parallel netCDF: A High Performance Scientific I/O Interface.ACM,2003
[42] Hadjidoukas P E, Papatheodorou T S. OpenMP extensions for master-slave message passing computing.Parallel Computing.2005,31,1155-1167
[43] Inho Park, Seon Wook Kim.Study of OpenMP applications on the InfiniBand based software distributedshared-memory system. Parallel Computing. 2005,31: 1099-1113
[44] Lu H, Hu Y C, Zwaenepoe W. OpenMP on networks of work-stations. Supercomputing'98 ,Orlando,1998
[45] Hu W, Shi W, Tang Z, etal. A lock-based cache coherence protocol for scope consistency. Journal of Computer Science and Technology.1998,13 (2):97-109
[46] Kengo Nakajima.Parallel iterative solvers for finite-element methods using an OpenMP/MPI hybrid programming model on the Earth Simulator.Parallel Computing.2005,31:1048-1065
[47] Lei Huang,Barbara Chapman, Zhenying Liu.Towards a more efficient implementation of OpenMP for clusters via translation to global arrays.Parallel Computing.2005,31:1114-1139
[48] 陈 勇 , 陈 国 良 , 李 春 生 .SMP 集 群 混 合 编 程 模 型 研 究 . 小 型 微 型 计 算 机 系统,2004,25(10):1763-1767
[49] 赵永华,迟学斌.基于 SMP 集群的 MPI+OpenMP 混合编程模型及有效实现.微电学与计算机.2005 ,22(10):7-11
[50] 吴少刚,章隆兵,蔡飞.一种适用于机群 OpenMP 系统的有效调度算法.计算机研究与发展.2004,41(7):1298-1305
[2] 陈国良.并行计算机体系结构.第一版.北京:高等教育出版社,2002
[3] Avery Ching, Kenin Coloma, Alok. Challenges for Parallel I/O in Grid Computing. In:Engineering the Grid: Status and Perspective, American: Scientific Publishers, 2006,625-632
[4] 杨红霞,赵改善. 21 世纪的地震数据处理系统,石油物探.2001,13(4):126-141
[5] Ching Avery, Liao Wei-keng. Evaluating I/O Characteristics and Methods for Storing Structured Scientific Data. In:Proceedings of the 20th International Parallel and Distributed Processing Symposium. IEEE,2006,1-15
[6] Kenin Coloma, Alok Choudhary, Avery Ching, et al. Power and Performance in I/O for Scientific Applications. In:Proceedings of the Next Generation Software Workshop, IEEE,2005,1-8
[7] 李冀,李晓明,陆桑璐.数据筛选技术在并行 I/O 中的应用.电子学报,2001,29(2): 249-252
[8] Carns P, Ligon W, Ross R, et al. PVFS: A Parallel Virtual File System for Linux Clusters.In:Proceedings of the 4th Annual Linux Showcase and Conference. Atlanta:GA,2000,317-327
[9] 张若洋,吕杨.PVFS 在 linux 集群上的应用研究.西南科技大学学报,2006,21(1): 10-17
[10] The parallel virtual file system 2 (PVFS2). http://www.pvfs.org/pvfs2/, 2007-12-20
[11] Seamons K E, Chen Y, Jones P. Server directed collective I/O in Panda. In: Proceedings of Supercomputing. California:ACM,1995,57-62
[12] Kotz D. Disk-Directed I/O for MIMD Multiprocessors. ACM Transactions on Computer Systems (TOCS).1997,15(1):41-74
[13] Wesley Perkins. Parallel geophysical processing. The Leading Edg, 1996, 15(12):1359-1361
[14] 王 宏 琳 , 高 绘 生 . 地 震 并 行 处 理 模 式 与 应 用 框 架 . 计 算 机 学 报 ,2001,24(2): 202-208
[15] 魏嘉.可扩展的地震数据并行处理.石油物探.1997,4:12-17
[16] 都志辉,高性能计算之并行编程技术-MPI 并行程序设计.北京:清华大学出版社,2001
[17] 陈文光,武永卫.MPI 与 OpenMP 并行程序设计.北京:清华大学出版社,2004
[18] 张云泉.高性能计算-体系结构,并行编程,性能评价与展望.http://www.rdcps. ac.cn/,2007-12-22
[19] 黄铠,徐志伟.可扩展并行计算-技术结构与编程.北京:机械工业出版社,2000
[20] Kurt Geihs. Middleware Challenges Ahead. computer IEEE,2001,34(6):24-31
[21] Doug Wille. Immersive environments enhance team collaboration. World Oil , 1999 , 220(5) :72-74
[22] Fred Aminzadeh. Future geophysical technology trends. The Leading Edge of Geophysics , 1996 ,15(6):729-735
[23] 李蕾,方明科,杜欣.计算机机群技术及其在Web领域中的应用. 现代计算机 . 2006,5: 60-62
[24] Dimitri Bevc. Internet based seismic processing: The future of geophysical computing. In:Annual Meeting Abstracts , SEG, 2000, 2119-2122
[25] William K Aylor. A business case for seismic data transmission by satellite. In:Annual Meeting Abstracts, SEG, 2000,187-235
[26] 杨晓云,庄建军,李淑英.集群技术在石油地震处理中的应用研究.高性能计算技术.2003,163:43-45
[27] 赵改善. 集群技术及其在石油工业中的应用. 石油物探.2001,40(3) :118-126
[28] Herman Chang. 3-D prestack Kirchhoff depth migration: From prototype to production in a massively parallel processorof geophysical computing. In: Annual Meeting Abstracts . SEG, 2000, 2119-2122
[29] 李家康. 大规模并行处理机三维叠前深度偏移.石油物探. 1998, 37 (2): 77-87
[30] Hengchang Dai. Parallel processing of Prestack Kirchhoff Time Migration on a PC Cluster. Computers and Geosciences.2005,31(1):891-899
[31] Chen Li. Parallel performance optimization of large-scale unstructured data visualization for the earth simulator. In Proceedings of the Fourth Eurographics Workshop on Parallel Graphics and Visualization. 2002, 133-140
[32] 陈 莉 . 大 规 模 数 据 场 的 并 行 可 视 化 . 浙 江 大 学 学 报 ( 理 学 版 ).2001,28(2): 222-226
[33] 何成万,何克清.基于角色的设计模式建模和实现方法.软件学报,2006,17(4): 658-669
[34] 曾 闽 山 , 侯 岩 松 . 海 量 地 震 数 据 网 格 化 算 法 分 析 与 研 究 . 石 油 天 然 气 学报,2006,28(2):72-75
[35] 曹凤海,易昌华,李秀山.地震勘探测量数据处理的全面解决方案,物探装备.2006,16(1):59-62
[36] 陈茂山,王云高.油气勘探软件用户界面设计方法探索与实践.勘探地球物理进展.2004,27(2):132-138
[37] 刘宏涛, 李义杰, 于长江.基于 B/S 模式三层结构地震信息管理系统的设计.东北地震研究.2006,22(1):75-80
[38] 李冀,陈晓林,陆桑璐.一个基于 NOW 的并行 I/O 系统.软件学报,2000,12(11): 1654-1659
[39] 赵欣,陈道蓄,谢立.一个基于 NOW 的跨平台并行文件系统的设计和实现.计算机科学.2000,27(2):21-25
[40] 李群,谢立,孙钟秀.并行文件系统的设计.计算机科学.1996,23(2):36-39
[41] Li Jiangwei, Liao Weikeng, Alok Choudhary, Parallel netCDF: A High Performance Scientific I/O Interface.ACM,2003
[42] Hadjidoukas P E, Papatheodorou T S. OpenMP extensions for master-slave message passing computing.Parallel Computing.2005,31,1155-1167
[43] Inho Park, Seon Wook Kim.Study of OpenMP applications on the InfiniBand based software distributedshared-memory system. Parallel Computing. 2005,31: 1099-1113
[44] Lu H, Hu Y C, Zwaenepoe W. OpenMP on networks of work-stations. Supercomputing'98 ,Orlando,1998
[45] Hu W, Shi W, Tang Z, etal. A lock-based cache coherence protocol for scope consistency. Journal of Computer Science and Technology.1998,13 (2):97-109
[46] Kengo Nakajima.Parallel iterative solvers for finite-element methods using an OpenMP/MPI hybrid programming model on the Earth Simulator.Parallel Computing.2005,31:1048-1065
[47] Lei Huang,Barbara Chapman, Zhenying Liu.Towards a more efficient implementation of OpenMP for clusters via translation to global arrays.Parallel Computing.2005,31:1114-1139
[48] 陈 勇 , 陈 国 良 , 李 春 生 .SMP 集 群 混 合 编 程 模 型 研 究 . 小 型 微 型 计 算 机 系统,2004,25(10):1763-1767
[49] 赵永华,迟学斌.基于 SMP 集群的 MPI+OpenMP 混合编程模型及有效实现.微电学与计算机.2005 ,22(10):7-11
[50] 吴少刚,章隆兵,蔡飞.一种适用于机群 OpenMP 系统的有效调度算法.计算机研究与发展.2004,41(7):1298-1305