基于网格计算的分布计算协同机制与技术
|
摘要
随着科学计算的日趋复杂,多单位协作日益频繁,大型科学与工程项目的组织与实施都面临着众多人员、大量异构计算资源、多信息系统之间的协作与交互。
本文首先从研究目前国内外流行的网格计算技术出发,对网格技术进行分析比较,重点对分布协同计算机制进行了讨论。同时,根据研究工作需要,深入研究了网格资源管理和网格作业调度相关技术及理论,分析总结了目前分布网格资源调度技术主要的应用领域及发展趋势。
其次,在对网格资源调度技术进行分析的基础上,对网格作业调度流程及其协同分配的难点、机制等问题进行深入的研究。
第三,着重对网格计算中资源协同调度机制进行深入的研究,根据该机制的资源协同分配需求,提出了一个以Globus Toolkit为平台的基于网格计算的分布协同计算模型。在基于经济模型的作业调度机制的竞价机制中,给出了一个较为完善的价格函数模型以及相应的作业预测完成时间模型,利用设计的价格函数模型来实现资源提供的主观能动性,确定资源的调配更加合理、快速的完成。同时,在分布式计算模型中还给出了一个基于备份的作业调度容错机制。
With the increasing complexity of scientific calculation, the organization and implementation of large scientific project are facing cooperation and interaction among the human resources, heterogeneous computational resource and Multidimensional Information System.
In this essay, we first researched on the grid computing technique which is popular in home and abroad. We analyzed and compared the grid computing technique and discussed Distributed Coherent computing technique. At the same time, according to the need of the research work, we studied the technical theory of Grid resource management and scheduling and summarized the current application area and development tendency of it.
Secondly, based on the research work of Grid resource scheduling, we made in-depth study on grid job scheduling and co-allocation in the grid.
Third, we emphasized the research work on resource cooperation scheduling in grid computing. According to the need of resource cooperation scheduling, we put forward a Distributed Coherent computing model based on Globus Toolkit software platform. After studying the grid job scheduling bidding mechanisms based on the economic mode, cost function model and related job finish time Prediction Model were introduced. Using cost function model to realize subjective initiative of recourse supply, which will make recourse scheduling rational and fast. And the distributed computing model also has the job scheduling Fault Tolerance.
In the last part, we point out the deficiencies of out proposal in this paper, and forecast the development tendency of grid job scheduling.
本文首先从研究目前国内外流行的网格计算技术出发,对网格技术进行分析比较,重点对分布协同计算机制进行了讨论。同时,根据研究工作需要,深入研究了网格资源管理和网格作业调度相关技术及理论,分析总结了目前分布网格资源调度技术主要的应用领域及发展趋势。
其次,在对网格资源调度技术进行分析的基础上,对网格作业调度流程及其协同分配的难点、机制等问题进行深入的研究。
第三,着重对网格计算中资源协同调度机制进行深入的研究,根据该机制的资源协同分配需求,提出了一个以Globus Toolkit为平台的基于网格计算的分布协同计算模型。在基于经济模型的作业调度机制的竞价机制中,给出了一个较为完善的价格函数模型以及相应的作业预测完成时间模型,利用设计的价格函数模型来实现资源提供的主观能动性,确定资源的调配更加合理、快速的完成。同时,在分布式计算模型中还给出了一个基于备份的作业调度容错机制。
With the increasing complexity of scientific calculation, the organization and implementation of large scientific project are facing cooperation and interaction among the human resources, heterogeneous computational resource and Multidimensional Information System.
In this essay, we first researched on the grid computing technique which is popular in home and abroad. We analyzed and compared the grid computing technique and discussed Distributed Coherent computing technique. At the same time, according to the need of the research work, we studied the technical theory of Grid resource management and scheduling and summarized the current application area and development tendency of it.
Secondly, based on the research work of Grid resource scheduling, we made in-depth study on grid job scheduling and co-allocation in the grid.
Third, we emphasized the research work on resource cooperation scheduling in grid computing. According to the need of resource cooperation scheduling, we put forward a Distributed Coherent computing model based on Globus Toolkit software platform. After studying the grid job scheduling bidding mechanisms based on the economic mode, cost function model and related job finish time Prediction Model were introduced. Using cost function model to realize subjective initiative of recourse supply, which will make recourse scheduling rational and fast. And the distributed computing model also has the job scheduling Fault Tolerance.
In the last part, we point out the deficiencies of out proposal in this paper, and forecast the development tendency of grid job scheduling.
引文
[1]都志辉,陈渝,刘鹏.网格计算.清华大学出版社.2002-8
[2]陈云芳,穆鸿.网格计算及其应用.江苏通信技术.2003
[3]王伟.网格中信息服务的设计与实现.CNKI系列数据库编辑出版.2006
[4]王犇.CGSP网格服务自动化测试技术研究与实现.西北工业大学硕士学位论文.2006-3
[5]柴晓路.Web服务架构与开放互操作技术.清华大学出版社,2002
[6]金海,李其胜,邹德清,陈汉华.信息服务网格中资源代理的实现策略.计算机应用与软件.2004
[7]李胜利,张琦,张浩,陈宝利.MPI-I0中Col lect ive I/0的优化.计算机工程与科学.2004
[8]黄斌.基于网格的高校综合信息系统设计.吉首大学学报.2003
[9]武秀川.计算网格的资源管理方法研究.计算机科学.2003
[10]Ian Foster,Carl Kesselman,Globus;A Metacomputing Infrastructure Toolkit,International Journal of Supercomputer Applications and High Performance Computing,11(2);115-128(1997).
[11]Andrew S.Grimshaw,William A.Wulf,and the Legion team,The Legion Vision of a Worldwide Virtual Computer,Communications of the ACM,40(1);39-45(1997).
[12]全球网格论坛GGF;http;//www.ggf.org/(Current March 17,2006).
[13]Miguel L.Bote-Lorenzo,YannisA.Dimitriadis,Eduardofiomez-Sanchez,Grid characteristics and uses;A grid definition,Grid Computing,Springer-Verlag,Berlin,291-298(2004).
[14]唐志敏,网格技术专题介绍,计算机研究与发展,39(8)(2002).
[15]高全泉.企业级分布式计算的基本需求及中间件技术.计算机科学.2001
[16]Distributed Computing(HPDC-10).August 2001.NW Washington,DC USA;IEEEComputer Society,2001
[17]B.Allcock,I.Foster,V.Nefedova,et al.High-performance remote access to climatesimulation data;a challenge problem for data grid technologies.In;Conference onHigh Performance Networking and Computing.Denver,Colorado.2001.NY USA;ACM Press,2001
[18]朱朝艳,郑耀,魏贵义.网格环境中高性能数据传输客户端的实现.江南大学学报.2005
[19]刘鹏展,刘亮,杨寿保.GridFTP传输性能分析.计算机工程与应用.2005
[20]Fran Berman,Rich Wolski,Silvia Figueira,Jennifer Schopf,Gary Shao,Application-Level Scheduling on Distributed Heterogeneous Networks,Proceedings of Supercomputing 96,Pittsburgh,PA,1996.
[21]James Frey,Todd Tannenbaum,Miron Livny,Ian T.Foster,Steven Tuecke,Condor-G;A Computation Management Agent for Multi-institutional Grids,Cluster Computing,5(3);237-246(2002).
[22]Rajkumar Buyya,David Abramson,Jonathan Giddy,Nimrod/G;An Architecture for a Resource Management and Scheduling System in a Global Computational Grid,Proceedings of Fourth International Conference/Exhibition on High Performance Computing in the Asia-Pacific Region,vol.1,2000,283-289.
[23]刘防修.网格计算技术的特点及关键技术.华东船舶工业学院学报.自然科学版,2003
[24]G.Allen,T.Dramlitsch,I.Foster.et al.Supporting Efficient Execution inHeterogeneous Distributed Computing Environments with Cactus and Globus.Conference on High Performance Networking and Computing.Denver,Colorado.2001.New York,NY,USA;ACM Press,2001
[25]F.Berman,A.Chien,K.Cooper,et al.The GRADS Project;Software Support forHigh-Level Grid Application Development.International Journal of High-Performance
[26]J.Leigh,G.Dawe,J.Talandis,et al.AGAVE;Access Grid Augmented VirtualEnvironment.In;Access Grid Retreat 2001 Series.Argonne National Laboratory.2001.USA;Illinois,2001.
[27]应宏.网格系统组成与体系结构分析.西南师范大学学报;自然科学版.2004
[28]彭定,傅秀芬,谢翠萍,高冉,侯文国.网格数据传输协议探讨.网络与通信.2004
[29]Adabala,S.Kapadia,Fortes,et al.Interfacing wide-area network computing andcluster software;Condor,DQS and PBS via PUNCH.In;Adabala eds.Proceedings theNinth International Symposium on High-Performance Distribute Computing.Pittsburgh,USA.2000.LosAlamitos,USA;IEEEComput.Soc,2000
[30]都志辉,陈渝,刘鹏,王小鸽,杜江,周念生.以服务为中心的网格体系结构OGSA.计算机科学,2003
[31]徐钊,刘志强.第三代Internet-网格及其关键技术研究[J].计算机与网络.2005
[32]祝延波.国内外网格研究浅析[J].青海科技.2005
[33]邓永红.网格及网格计算技术综述[J].有线电视技术.2005
[34]Serafeim Zanikolas and Rizos Sakellariou.A Taxonomy of Grid Monitoring Systems.Future Generation Computer Systems.21(1).January 2005
[35]Foster.Globus Toolkit Version 4;Software for Service-Oriented Systems.IFIPInternational Conference on Network and Parallel Computing,Springer-Verlag LNCS 3779.pp 2-13,2005
[36]章勤,鄢娟,金海等.昊宇网络计算平台体系结构研究.计算机研究与发展,2003
[37]鄢娟,金海,韩宗芬等.昊宇;基于LVS的开放式网络计算平台.计算机工程与应用,录用编号;23651
[38]Adabala,S.Kapadia,Fortes,et al.Interfacing wide-area network computing andcluster software;Condor,DQS and PBS via PUNCH.In;Adabala eds. Proceedings theNinth International Symposium on High-Performance Distribute Computing.Pittsburgh,USA.2000.LosAlamitos,USA;IEEE Comput.Soc,2000
[39]J.Frey,T.Tannenbaum,I.Foster,et al.Condor-G;AComputationManagement Agentfor Multi-lnstitutional Grids.In;10th International Symposium on High PerformanceDistributed Computing,2001.USA;IEEE Press.2001
[40]R.Buyya,J.Giddy,D.Abramson.A Case for Economy Grid Architecture forService-Oriented Grid Computing.In;10th IEEE International HeterogeneousComputing Workshop(HCW2001).San Francisco,California,USA.2001.NewportBeach,CA,USA;IEEE Computer Society.2001-2
[41]A.S.Grimshaw,A.Ferrari,F.C.Knabe,et al.Wide-Area Computing;Resource Sharingon a Large Scale.IEEE Computer.1999
[42]A.Grimshaw,W.Wulf,et al.The Legion Vision of aWorldwide Virtual Computer.Communications of the ACM.January 1997
[43]罗秉安,张立臣.网格及其典型研究项目简介.微型机与应用.2003
[44]A.Grimshaw,W.Wulf,et al.The Legion Vision of a Worldwide Virtual Computer.Communications of the ACM.January 1997.
[45]张秋余,袁占亭,萧海东.网格计算技术的应用.甘肃工业大学学报.2003
[46]A.S.Grimshaw,A.Ferrari,F.C.Knabe,et al.Wide-Area Computing;Resource Sharingon a Large Scale.IEEE Computer.1999
[47]房向明,杨寿保.网格计算系统的安全体系结构模型研究.2003
[48]J.Frey,T.Tannenbaum,I.Foster,et al.Condor-G;A Computation Management Agentfor Multi-Institutional Grids.In;10th International Symposium on High PerformanceDistributed Computing,2001.USA;IEEE Press,2001
[49]R.Buyya,J.Giddy,D.Abramson.A Case for Economy Grid Architecture forService-Oriented Grid Computing.In;10th IEEE International HeterogeneousComputing Workshop(HCW2001).San Francisco,California,USA.2001.NewportBeach,CA,USA;IEEE Computer Society,2001
[2]陈云芳,穆鸿.网格计算及其应用.江苏通信技术.2003
[3]王伟.网格中信息服务的设计与实现.CNKI系列数据库编辑出版.2006
[4]王犇.CGSP网格服务自动化测试技术研究与实现.西北工业大学硕士学位论文.2006-3
[5]柴晓路.Web服务架构与开放互操作技术.清华大学出版社,2002
[6]金海,李其胜,邹德清,陈汉华.信息服务网格中资源代理的实现策略.计算机应用与软件.2004
[7]李胜利,张琦,张浩,陈宝利.MPI-I0中Col lect ive I/0的优化.计算机工程与科学.2004
[8]黄斌.基于网格的高校综合信息系统设计.吉首大学学报.2003
[9]武秀川.计算网格的资源管理方法研究.计算机科学.2003
[10]Ian Foster,Carl Kesselman,Globus;A Metacomputing Infrastructure Toolkit,International Journal of Supercomputer Applications and High Performance Computing,11(2);115-128(1997).
[11]Andrew S.Grimshaw,William A.Wulf,and the Legion team,The Legion Vision of a Worldwide Virtual Computer,Communications of the ACM,40(1);39-45(1997).
[12]全球网格论坛GGF;http;//www.ggf.org/(Current March 17,2006).
[13]Miguel L.Bote-Lorenzo,YannisA.Dimitriadis,Eduardofiomez-Sanchez,Grid characteristics and uses;A grid definition,Grid Computing,Springer-Verlag,Berlin,291-298(2004).
[14]唐志敏,网格技术专题介绍,计算机研究与发展,39(8)(2002).
[15]高全泉.企业级分布式计算的基本需求及中间件技术.计算机科学.2001
[16]Distributed Computing(HPDC-10).August 2001.NW Washington,DC USA;IEEEComputer Society,2001
[17]B.Allcock,I.Foster,V.Nefedova,et al.High-performance remote access to climatesimulation data;a challenge problem for data grid technologies.In;Conference onHigh Performance Networking and Computing.Denver,Colorado.2001.NY USA;ACM Press,2001
[18]朱朝艳,郑耀,魏贵义.网格环境中高性能数据传输客户端的实现.江南大学学报.2005
[19]刘鹏展,刘亮,杨寿保.GridFTP传输性能分析.计算机工程与应用.2005
[20]Fran Berman,Rich Wolski,Silvia Figueira,Jennifer Schopf,Gary Shao,Application-Level Scheduling on Distributed Heterogeneous Networks,Proceedings of Supercomputing 96,Pittsburgh,PA,1996.
[21]James Frey,Todd Tannenbaum,Miron Livny,Ian T.Foster,Steven Tuecke,Condor-G;A Computation Management Agent for Multi-institutional Grids,Cluster Computing,5(3);237-246(2002).
[22]Rajkumar Buyya,David Abramson,Jonathan Giddy,Nimrod/G;An Architecture for a Resource Management and Scheduling System in a Global Computational Grid,Proceedings of Fourth International Conference/Exhibition on High Performance Computing in the Asia-Pacific Region,vol.1,2000,283-289.
[23]刘防修.网格计算技术的特点及关键技术.华东船舶工业学院学报.自然科学版,2003
[24]G.Allen,T.Dramlitsch,I.Foster.et al.Supporting Efficient Execution inHeterogeneous Distributed Computing Environments with Cactus and Globus.Conference on High Performance Networking and Computing.Denver,Colorado.2001.New York,NY,USA;ACM Press,2001
[25]F.Berman,A.Chien,K.Cooper,et al.The GRADS Project;Software Support forHigh-Level Grid Application Development.International Journal of High-Performance
[26]J.Leigh,G.Dawe,J.Talandis,et al.AGAVE;Access Grid Augmented VirtualEnvironment.In;Access Grid Retreat 2001 Series.Argonne National Laboratory.2001.USA;Illinois,2001.
[27]应宏.网格系统组成与体系结构分析.西南师范大学学报;自然科学版.2004
[28]彭定,傅秀芬,谢翠萍,高冉,侯文国.网格数据传输协议探讨.网络与通信.2004
[29]Adabala,S.Kapadia,Fortes,et al.Interfacing wide-area network computing andcluster software;Condor,DQS and PBS via PUNCH.In;Adabala eds.Proceedings theNinth International Symposium on High-Performance Distribute Computing.Pittsburgh,USA.2000.LosAlamitos,USA;IEEEComput.Soc,2000
[30]都志辉,陈渝,刘鹏,王小鸽,杜江,周念生.以服务为中心的网格体系结构OGSA.计算机科学,2003
[31]徐钊,刘志强.第三代Internet-网格及其关键技术研究[J].计算机与网络.2005
[32]祝延波.国内外网格研究浅析[J].青海科技.2005
[33]邓永红.网格及网格计算技术综述[J].有线电视技术.2005
[34]Serafeim Zanikolas and Rizos Sakellariou.A Taxonomy of Grid Monitoring Systems.Future Generation Computer Systems.21(1).January 2005
[35]Foster.Globus Toolkit Version 4;Software for Service-Oriented Systems.IFIPInternational Conference on Network and Parallel Computing,Springer-Verlag LNCS 3779.pp 2-13,2005
[36]章勤,鄢娟,金海等.昊宇网络计算平台体系结构研究.计算机研究与发展,2003
[37]鄢娟,金海,韩宗芬等.昊宇;基于LVS的开放式网络计算平台.计算机工程与应用,录用编号;23651
[38]Adabala,S.Kapadia,Fortes,et al.Interfacing wide-area network computing andcluster software;Condor,DQS and PBS via PUNCH.In;Adabala eds. Proceedings theNinth International Symposium on High-Performance Distribute Computing.Pittsburgh,USA.2000.LosAlamitos,USA;IEEE Comput.Soc,2000
[39]J.Frey,T.Tannenbaum,I.Foster,et al.Condor-G;AComputationManagement Agentfor Multi-lnstitutional Grids.In;10th International Symposium on High PerformanceDistributed Computing,2001.USA;IEEE Press.2001
[40]R.Buyya,J.Giddy,D.Abramson.A Case for Economy Grid Architecture forService-Oriented Grid Computing.In;10th IEEE International HeterogeneousComputing Workshop(HCW2001).San Francisco,California,USA.2001.NewportBeach,CA,USA;IEEE Computer Society.2001-2
[41]A.S.Grimshaw,A.Ferrari,F.C.Knabe,et al.Wide-Area Computing;Resource Sharingon a Large Scale.IEEE Computer.1999
[42]A.Grimshaw,W.Wulf,et al.The Legion Vision of aWorldwide Virtual Computer.Communications of the ACM.January 1997
[43]罗秉安,张立臣.网格及其典型研究项目简介.微型机与应用.2003
[44]A.Grimshaw,W.Wulf,et al.The Legion Vision of a Worldwide Virtual Computer.Communications of the ACM.January 1997.
[45]张秋余,袁占亭,萧海东.网格计算技术的应用.甘肃工业大学学报.2003
[46]A.S.Grimshaw,A.Ferrari,F.C.Knabe,et al.Wide-Area Computing;Resource Sharingon a Large Scale.IEEE Computer.1999
[47]房向明,杨寿保.网格计算系统的安全体系结构模型研究.2003
[48]J.Frey,T.Tannenbaum,I.Foster,et al.Condor-G;A Computation Management Agentfor Multi-Institutional Grids.In;10th International Symposium on High PerformanceDistributed Computing,2001.USA;IEEE Press,2001
[49]R.Buyya,J.Giddy,D.Abramson.A Case for Economy Grid Architecture forService-Oriented Grid Computing.In;10th IEEE International HeterogeneousComputing Workshop(HCW2001).San Francisco,California,USA.2001.NewportBeach,CA,USA;IEEE Computer Society,2001