HSWAP:适用于高性能计算环境的数值模拟工作流管理平台
|
摘要
针对高性能计算(HPC)环境中的"建模、计算、分析、优化"一体化应用构建的问题,设计了支持数值模拟软件封装和数值模拟工作流交互设计的数值模拟工作流管理平台——HSWAP。首先,基于对数值模拟活动的运行特征共性建模构建组件模型;然后,利用工作流表达数值模拟活动间的控制、数据依赖关系,建立形式化的数值模拟工作流模型,所形成的工作流模型可在平台中自动解析并适配高性能计算资源,从而实现批量关联数值模拟任务的自动生成与调度,为领域用户屏蔽高性能计算资源的使用细节。平台提供Web Portal服务,支持图形数值模拟程序的交互界面推送。目前该平台已在超算中心实际生产环境得到部署应用,可在2人月内完成包含10个以下数值模拟软件、20个以内计算任务节点的数值模拟工作流的集成。
Concerning the construction of integrated application of "modeling, computation, analysis, optimization" workflow under High Performance Computing(HPC) environment, HPC Simulation Workflow Application Platform(HSWAP) supporting numerical simulation software encapsulation and numerical simulation workflow interaction design was developed. Firstly, based on the modeling of runtime characteristics of numerical simulation activities, the component model was built. Then, the control and data dependency relationships between simulation activities were represented by the workflow, creating a formal numerical simulation workflow model. The formed workflow model was able to be automatically parsed in the platform to adapt to HPC resources. Therefore, HSWAP platform could be used for automatic generation and scheduling of a batch of related numerical simulation tasks, screening technical details of HPC resources from domain users. The platform provided Web Portal services, which supports the push of interactive interfaces of graphical numerical simulation programs. The platform is already deployed and applied at Supercomputing Center and with this platform, the integration of numerical simulation workflows with up to 10 numerical simulation softwares and 20 computing task nodes can be completed in 2 person-month.
Concerning the construction of integrated application of "modeling, computation, analysis, optimization" workflow under High Performance Computing(HPC) environment, HPC Simulation Workflow Application Platform(HSWAP) supporting numerical simulation software encapsulation and numerical simulation workflow interaction design was developed. Firstly, based on the modeling of runtime characteristics of numerical simulation activities, the component model was built. Then, the control and data dependency relationships between simulation activities were represented by the workflow, creating a formal numerical simulation workflow model. The formed workflow model was able to be automatically parsed in the platform to adapt to HPC resources. Therefore, HSWAP platform could be used for automatic generation and scheduling of a batch of related numerical simulation tasks, screening technical details of HPC resources from domain users. The platform provided Web Portal services, which supports the push of interactive interfaces of graphical numerical simulation programs. The platform is already deployed and applied at Supercomputing Center and with this platform, the integration of numerical simulation workflows with up to 10 numerical simulation softwares and 20 computing task nodes can be completed in 2 person-month.
引文
[1]曹荣强,王小宁,卢莎莎,等.基于Portlet的高性能计算应用集成组件[J].科研信息化技术与应用,2017,8(2):18-30.(CAOR Q,WANG X N,LU S S,et al.High performance computing application integration toolkits based on Portlet[J].e-Science Technology&Application,2017,8(2):18-30.)
[2]TOWNS J,COCKERILL T,DAHAN M,et al.XSEDE:accelerating scientific discovery[J].Computing in Science&Engineering,2014,16(5):62-74.
[3]迟学斌,肖海力,王小宁,等.国家重点研发计划助力国家高性能计算环境服务化建设迈上新台阶[J].科研信息化技术与应用,2016,7(4):84-88.(CHI X B,XIAO H L,WANG X N,et al.A further promoting toward building China national grid supported by the national key research and development program of China[J].e-Science Technology&Application,2016,7(4):84-88.)
[4]TAYLOR I J,DEELMAN E,GANNON D B,et al.Workflows for e-Science:Scientific Workflows for Grids[M].Berlin:Springer,2007:9-16.
[5]DEELMAN E,SINGH G,SU M,et al.Pegasus:a framework for mapping complex scientific workflows onto distributed systems[J].Scientific Programming,2005,13(3):219-237.
[6]BERMAN F.From Tera Grid to knowledge grid[J].Communications of the ACM,2001,44(11):27-28.
[7]FOSTER I,KESSELMAN C.Globus:a metacomputing infrastructure toolkit[J].The International Journal of Supercomputer Applications and High Performance Computing,1997,11(2):115-128.
[8]吴响,邓笋根,陆忠华.国内外科学工作流综述研究[J].科研信息化技术与应用,2014,5(5):86-95.(WU X,DENG S G,LUZ H.A review of the study on the scientific workflow[J].e-Science Technology&Application,2014,5(5):86-95.)
[9]ATKINSON M,GESING S,MONTAGNAT J,et al.Scientific workflows:past,present and future[J].Future Generation Computer Systems,2017,75:216-227.
[10]YU J,BUYYA R.A taxonomy of scientific workflow systems for grid computing[J].ACM SIGMOD Record,2005,34(3):44-49.
[11]DEELMAN E,PETERKA T,ALTINTAS I,et al.The future of scientific workflows[J].The International Journal of High Performance Computing Applications,2018,32(1):159-175.
[12]李于锋,莫则尧,肖永浩,等.超算环境科学工作流应用平台的引擎设计和资源调度[J].计算机应用研究,2019,36(7):1-7.(LI Y F,MO Z Y,XIAO Y H,et al.Engine design and resource scheduling of scientific workflow application platform in supercomputing[J].Application Research of Computers,2019,36(7):1-7.)
[13]LU S,PAI D,HUA J,et al.A task abstraction and mapping approach to the shimming problem in scientific workflows[C]//Proceedings of the 2009 IEEE International Conference on Services Computing.Piscataway,NJ:IEEE,2009:284-291.
[14]HUERTA E A,HAAS R,JHA S,et al.Supporting high-performance and high-throughput computing for experimental science[J].Computing and Software for Big Science,2019,3(1):5.
[15]JAIN A,ONG S P,CHEN W,et al.Fire Works:a dynamic workflow system designed for high-throughput applications[J].Concurrency and Computation:Practice and Experience,2015,27(17):5037-5059.
[2]TOWNS J,COCKERILL T,DAHAN M,et al.XSEDE:accelerating scientific discovery[J].Computing in Science&Engineering,2014,16(5):62-74.
[3]迟学斌,肖海力,王小宁,等.国家重点研发计划助力国家高性能计算环境服务化建设迈上新台阶[J].科研信息化技术与应用,2016,7(4):84-88.(CHI X B,XIAO H L,WANG X N,et al.A further promoting toward building China national grid supported by the national key research and development program of China[J].e-Science Technology&Application,2016,7(4):84-88.)
[4]TAYLOR I J,DEELMAN E,GANNON D B,et al.Workflows for e-Science:Scientific Workflows for Grids[M].Berlin:Springer,2007:9-16.
[5]DEELMAN E,SINGH G,SU M,et al.Pegasus:a framework for mapping complex scientific workflows onto distributed systems[J].Scientific Programming,2005,13(3):219-237.
[6]BERMAN F.From Tera Grid to knowledge grid[J].Communications of the ACM,2001,44(11):27-28.
[7]FOSTER I,KESSELMAN C.Globus:a metacomputing infrastructure toolkit[J].The International Journal of Supercomputer Applications and High Performance Computing,1997,11(2):115-128.
[8]吴响,邓笋根,陆忠华.国内外科学工作流综述研究[J].科研信息化技术与应用,2014,5(5):86-95.(WU X,DENG S G,LUZ H.A review of the study on the scientific workflow[J].e-Science Technology&Application,2014,5(5):86-95.)
[9]ATKINSON M,GESING S,MONTAGNAT J,et al.Scientific workflows:past,present and future[J].Future Generation Computer Systems,2017,75:216-227.
[10]YU J,BUYYA R.A taxonomy of scientific workflow systems for grid computing[J].ACM SIGMOD Record,2005,34(3):44-49.
[11]DEELMAN E,PETERKA T,ALTINTAS I,et al.The future of scientific workflows[J].The International Journal of High Performance Computing Applications,2018,32(1):159-175.
[12]李于锋,莫则尧,肖永浩,等.超算环境科学工作流应用平台的引擎设计和资源调度[J].计算机应用研究,2019,36(7):1-7.(LI Y F,MO Z Y,XIAO Y H,et al.Engine design and resource scheduling of scientific workflow application platform in supercomputing[J].Application Research of Computers,2019,36(7):1-7.)
[13]LU S,PAI D,HUA J,et al.A task abstraction and mapping approach to the shimming problem in scientific workflows[C]//Proceedings of the 2009 IEEE International Conference on Services Computing.Piscataway,NJ:IEEE,2009:284-291.
[14]HUERTA E A,HAAS R,JHA S,et al.Supporting high-performance and high-throughput computing for experimental science[J].Computing and Software for Big Science,2019,3(1):5.
[15]JAIN A,ONG S P,CHEN W,et al.Fire Works:a dynamic workflow system designed for high-throughput applications[J].Concurrency and Computation:Practice and Experience,2015,27(17):5037-5059.