基于网络监测的网格计算优化调度模型
|
摘要
网格计算技术的产生是各类大型应用对计算资源和计算能力的需求不断增长的结果。所以，网格计算的运行需要网格资源监控工具对其进行实时监控，进而对各类资源的进行全面、系统地调度。特别是，在多台可能不在同一地点的主机上运行网格计算程序时，将会在网络上形成大量的数据流通。因此，为了使网格计算能稳定、高效地进行，需要高速、稳定的网络连接。网络监测能及时显示网络连接的实时性能，也能根据历史记录预测网络的流通状况。网络监测组件再将这些监测信息发布给相关的网格中间件，让它们根据这些监测信息及时调整自身的行为，以提高网络利用率，让管理者恰当地对网格计算进行调度，最终提高网格计算的性能。
网格系统自身或管理者要想根据网络连接状况科学地调度、优化计算过程，这需要一定的调度、优化方案。这些方案可以做成网格组件，由其它中间件随时调用。
本文首先在第一章介绍了网格计算的定义、分类，网格系统的特点、功能，典型的五层沙漏网格体系结构模型，和结合了Web Services技术和网格技术的新的开放网格服务体系结构OGSA。并介绍了一些目前较有名的网格系统，如Globus。
在第二章简单介绍了网格应用的研究现状和发展趋势，
在第三章介绍了网格系统中的典型网络监测技术，包括其结构、网络监测中的网络运行性能参数和主要的网络监测工具，以及将监测结果发布给网格中间件的方法，最后介绍了欧洲数据网格EDG的网络监测实验床。
对于优化网格计算来说，网络的流通性能无疑是非常重要的。在第四章，在分析现有调度方案的基础上，提出了一个用于优化选择的网间价值函数，在此基础上，给出了一种新的基于网络的优化连通服务以及其架构。文中给出了由一般的网络运行状态参数复合成的更大颗粒的参数，如亲近函数(Closoness Function) C_(ij)(pl_(ij)，r_(ij)，th_(ij))和邻近函数(Proximity Function) P(SE_i)。此优化服务的用途广泛，文中给出的资源代理服务(Resource Brokers)和数据管理服务(Data Management)能充分利用网络的状态，以选择更优的路由或网格节点来优化网格计算性能。
The use of the technology of grid computing is the result that many large applications require more and more computing resources and higher and higher computing capability. The kinds of the resources that involved in grid computing are complicate, their amount is large, and their constructions are changeful. So that grid-monitoring tools must monitor the grid computing at every moment when grid computing is running. Especially, when a grid-computing program is running in several computers that probably lived in deferent localities, there will be generated large volume of data streams in the network. In this way, if grid computing wants to run stably and efficiently, there is a need for a high rate and stable network link. Network monitoring can display the characteristic of the network and forecast the currency status of the network according to the historic log file. And network-monitoring components publish this information to the grid middleware. The grid middleware will adjust its behavior according to t
his information and improve the use efficiency of the network, and the administrator can schedule the grid computing. So the performance of the grid computing can be improved.
If the grid system itself or the administrator wants to schedule and optimize the grid computing in a scientific way, there need some good scheduling and optimizing schemas. These schemas can be made as grid component that used by other middleware at any time.
In the first chapter of this article, we can read the introduce of definition and catalog of the grid computing, the characteristic and function of the grid system, the model of the classical five layers of hourglass grid architecture and OGSA, which is new and combined by the technology of Web Services and grid, and several famous grid systems, for example, Globus.
In the second chapter, these are introduced that the present research status of the grid application, its trend of improvement
In the third chapter, a classic network monitoring technology
in grid system is introduced, including its architecture, the network performance metric in the network monitoring, the main network monitoring tools and the way in which monitoring results publish to grid middleware. At last, the network monitoring testbed of European DataGrid is introduced.
In the fourth chapter, we consider that given the fundamental relevance of network performance for the optimization of the grid computing, the concept of a novel service called the Network-based Optimization Service is introduced and its architecture, based on the use of internal cost functions, is presented. We show how network metrics can be combined to form complex compound metrics like the Closeness C;.(/?/;.,r;.,thr} and the Proximity function/>(,$?.).
The Optimization Service can be used in a variety of different use cases; in this paper we show how Resource Brokers and Data Management can make use of network status for a considerable improvement of their decision-taking tasks.
网格系统自身或管理者要想根据网络连接状况科学地调度、优化计算过程，这需要一定的调度、优化方案。这些方案可以做成网格组件，由其它中间件随时调用。
本文首先在第一章介绍了网格计算的定义、分类，网格系统的特点、功能，典型的五层沙漏网格体系结构模型，和结合了Web Services技术和网格技术的新的开放网格服务体系结构OGSA。并介绍了一些目前较有名的网格系统，如Globus。
在第二章简单介绍了网格应用的研究现状和发展趋势，
在第三章介绍了网格系统中的典型网络监测技术，包括其结构、网络监测中的网络运行性能参数和主要的网络监测工具，以及将监测结果发布给网格中间件的方法，最后介绍了欧洲数据网格EDG的网络监测实验床。
对于优化网格计算来说，网络的流通性能无疑是非常重要的。在第四章，在分析现有调度方案的基础上，提出了一个用于优化选择的网间价值函数，在此基础上，给出了一种新的基于网络的优化连通服务以及其架构。文中给出了由一般的网络运行状态参数复合成的更大颗粒的参数，如亲近函数(Closoness Function) C_(ij)(pl_(ij)，r_(ij)，th_(ij))和邻近函数(Proximity Function) P(SE_i)。此优化服务的用途广泛，文中给出的资源代理服务(Resource Brokers)和数据管理服务(Data Management)能充分利用网络的状态，以选择更优的路由或网格节点来优化网格计算性能。
The use of the technology of grid computing is the result that many large applications require more and more computing resources and higher and higher computing capability. The kinds of the resources that involved in grid computing are complicate, their amount is large, and their constructions are changeful. So that grid-monitoring tools must monitor the grid computing at every moment when grid computing is running. Especially, when a grid-computing program is running in several computers that probably lived in deferent localities, there will be generated large volume of data streams in the network. In this way, if grid computing wants to run stably and efficiently, there is a need for a high rate and stable network link. Network monitoring can display the characteristic of the network and forecast the currency status of the network according to the historic log file. And network-monitoring components publish this information to the grid middleware. The grid middleware will adjust its behavior according to t
his information and improve the use efficiency of the network, and the administrator can schedule the grid computing. So the performance of the grid computing can be improved.
If the grid system itself or the administrator wants to schedule and optimize the grid computing in a scientific way, there need some good scheduling and optimizing schemas. These schemas can be made as grid component that used by other middleware at any time.
In the first chapter of this article, we can read the introduce of definition and catalog of the grid computing, the characteristic and function of the grid system, the model of the classical five layers of hourglass grid architecture and OGSA, which is new and combined by the technology of Web Services and grid, and several famous grid systems, for example, Globus.
In the second chapter, these are introduced that the present research status of the grid application, its trend of improvement
In the third chapter, a classic network monitoring technology
in grid system is introduced, including its architecture, the network performance metric in the network monitoring, the main network monitoring tools and the way in which monitoring results publish to grid middleware. At last, the network monitoring testbed of European DataGrid is introduced.
In the fourth chapter, we consider that given the fundamental relevance of network performance for the optimization of the grid computing, the concept of a novel service called the Network-based Optimization Service is introduced and its architecture, based on the use of internal cost functions, is presented. We show how network metrics can be combined to form complex compound metrics like the Closeness C;.(/?/;.,r;.,thr} and the Proximity function/>(,$?.).
The Optimization Service can be used in a variety of different use cases; in this paper we show how Resource Brokers and Data Management can make use of network status for a considerable improvement of their decision-taking tasks.
引文
[1] 都志辉，陈渝，刘鹏；网格计算；清华大学出版社，2002年11月份出版
[2] 黄昶，陆伟，吴朝晖；Grid技术研究现状及应用；计算机科学，2002VOL．29 NO：12
[3] Ian Foster, Carl Kesselman; The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration; Globus Project, 2002, http://www.globus.org/research/papers/ogsa.pdf
[4] Foster, I. "The anatomy of the grid" enabling scalable virtual organizations", CCGRID2001 (First IEEE/ACM International Symposium on Cluster Computing and the Grid)
[5] Foster I., The Grid Enabling Resource Sharing within Virtual Organizations, http://www-fp.mcs.anl.gov/~foster/Talks/WWWGridsMay2OO2. ppt,http://www-fp.mcs.anl.gov/~foster/Talks/GridTutorial.ppt
[6] OGSA, http://www.gridforum.org/ogsi-wg/drafts/ogsa_draft2.9_2002-O6-22.pdf, http://www.Globus.org/ogsa/
[7] Globus Toolkit 2. x, http://www.globus.org/toolkit/
[8] European DataGrid Project, http://www.eu-datagrid.org/
[9] European DataCrid Work Package 7 (EDG WPT); http://ccmail, in2p3, fr/archives/datagrid-wpT-1.html
[10] European DataGrid WP3, Information and Monitoring Services; http://hepunx. rl. ac. uk/edg/wp3/
[11] European DataGrid Testbed, http://ccwp7.in2p3.fr/
[12] GGF(Global Grid Forum); http://www.gridforum.org/
[13] RFC 2330, Framework for IP Performance Metrics
http://www.ietf. cnri. reston, va. us/rfc/rfc2330. txt?number=2330
[14] http://www.slac.stanford.edu/comp/net/wan-mon/tutorial. html#variable
[15] PingER-http://www-iepm.slac.stanford.edu/pinger/
[16] RTPL-http://www.phys.uu.nl/~wwwfi/rtpl/
[17] iperf; http://dast.nlanr.net/Projects/Iperf/
[18] RFC 1487, LDAP, A Light-Weight Directory Access Protocol http://www.ietf.cnri.reston.va.us/rfc/rfc1487.txt?number=1487
[19] Tierney, B.; Adyt, R.; Gunter, D.; Smith, W.; Taylor, V.; Wolski, R.; Swany, M.; A Grid Monitoring Architecture; the Grid Monitoring Architecture Working Group, Global Grid Forum (http://www-didc.lbl.gov/GGF-PERF/GMA-WG/)
[20] Definition of architecture, technical plan and evaluation criteria for scheduling, resource management, security and job description; Deliverable 1.2, DataGRID Project, Work package 1, http://serverll.infn.it/workload-grid/documents.htm
[21] Stockinger, H.; Samar, A.; Allcock, B.; Foster, I.; Holtman, K.; Tierney, B.; File and Object Replication in Data Grids, Proceedings of 10th IEEE Symposium on High Performance and Distributed Computing (HPDC-10), San Francisco, California, Aug 2001
[22] Czajkowski, K.; Fitzgerald, S.; Foster, I.; Kesselman, C.; Grid Information Services for Distributed Resource Sharing; Proceedings of the Tenth IEEE Interna-tional Symposium on High-Performance Distributed Computing (HPDC-10), IEEE Press, August 2001
[23] Almes, G.; Kalidindi, S.; Zekauskas, M.; A Round-trip Delay Metric for IPPM, RFC 2681
[24] Yajnik, M.; Moon, S.B.; Kurose, J.; Towsley, D.; Measurement and Modeling of the Temporal Dependence in Packet Loss; Proc. IEEE Infocom'99 (New York, NY, March 1999)
[25] Mathis, M.; Semke, J.; Mahdavi, J.; Ott, T.; The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Computer Communications Review, volume 27, number 3, July 1997
[26] Thain, D.; Livny, M.; Multiple Bypass: Interposition Agents for Distributed Com-puting, Journal of Cluster Computing, vol. 4, pages 39-47, 2001
[27] 中国网格信息中转站；http://www.gridhome.com/
[28] 清华大学计算机系网格研究组；http://hpclab.cs.tsinghua.edu.cn/
[29] 刘鹏；网格应用研究现状；http://www.gridhome.com/grid/paperppt/GridApplications.pdf
[30] 刘鹏；我国网格研究现状；http://www.gridhome.com/grid/paperppt/OurCountry.pdf
[31] 刘鹏：网格发展趋势；http://www.gridhome.com/grid/paperppt/GridTrend.pdf
[32] 刘鹏；网格概念的界定；http://www.gridhome.com/grid/paperppt/GridConcept.pdf
[33] M Swany, R. Wolski, Representing Dynamic Performance Information in Grid Environments with the Network Weather Service, 2nd IEEE International Symposium on Cluster Computing and the Grid (CCGrid2002), Berlin, Germany, May 2002.
[34] Bandwidth; http://www.cacr.caltech.edu/Uews/sc02/bandwid.pdf
[35] Sebastian Ho. Survey of Grid Meta-Scheduler.
http://student.bii.a-star.edu.sg/~sebastianh/Scheduler_Survey.pdf
[36] Condor-G. Unversity of Wisconsin, madison. http://www.cs.wisc.edu/condor
[37] R. Wolski, N. Spring, and J. Hayes. The network weather service: A distributed resource performance forecasting service for metacomputing. Future Generation Computer Systems, 1999.
[38] 郑然，李胜利，金海；网格资源管理与调度模型的研究；华中科技大学学报，Vol．29 No．12，2001
[39] 曹鸿强，肖侬，卢锡城，刘艳；一种基于市场机制的计算网格资源分配方法；计算机研究与发展，Vol．39，No．8，2002
[40] Rai jkumar Buyya编，郑纬民，石威，汪东升等译；高性能群集计算：结构与系统(第一卷)；电子工业出版社，2001年6月；P352-346
[2] 黄昶，陆伟，吴朝晖；Grid技术研究现状及应用；计算机科学，2002VOL．29 NO：12
[3] Ian Foster, Carl Kesselman; The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration; Globus Project, 2002, http://www.globus.org/research/papers/ogsa.pdf
[4] Foster, I. "The anatomy of the grid" enabling scalable virtual organizations", CCGRID2001 (First IEEE/ACM International Symposium on Cluster Computing and the Grid)
[5] Foster I., The Grid Enabling Resource Sharing within Virtual Organizations, http://www-fp.mcs.anl.gov/~foster/Talks/WWWGridsMay2OO2. ppt,http://www-fp.mcs.anl.gov/~foster/Talks/GridTutorial.ppt
[6] OGSA, http://www.gridforum.org/ogsi-wg/drafts/ogsa_draft2.9_2002-O6-22.pdf, http://www.Globus.org/ogsa/
[7] Globus Toolkit 2. x, http://www.globus.org/toolkit/
[8] European DataGrid Project, http://www.eu-datagrid.org/
[9] European DataCrid Work Package 7 (EDG WPT); http://ccmail, in2p3, fr/archives/datagrid-wpT-1.html
[10] European DataGrid WP3, Information and Monitoring Services; http://hepunx. rl. ac. uk/edg/wp3/
[11] European DataGrid Testbed, http://ccwp7.in2p3.fr/
[12] GGF(Global Grid Forum); http://www.gridforum.org/
[13] RFC 2330, Framework for IP Performance Metrics
http://www.ietf. cnri. reston, va. us/rfc/rfc2330. txt?number=2330
[14] http://www.slac.stanford.edu/comp/net/wan-mon/tutorial. html#variable
[15] PingER-http://www-iepm.slac.stanford.edu/pinger/
[16] RTPL-http://www.phys.uu.nl/~wwwfi/rtpl/
[17] iperf; http://dast.nlanr.net/Projects/Iperf/
[18] RFC 1487, LDAP, A Light-Weight Directory Access Protocol http://www.ietf.cnri.reston.va.us/rfc/rfc1487.txt?number=1487
[19] Tierney, B.; Adyt, R.; Gunter, D.; Smith, W.; Taylor, V.; Wolski, R.; Swany, M.; A Grid Monitoring Architecture; the Grid Monitoring Architecture Working Group, Global Grid Forum (http://www-didc.lbl.gov/GGF-PERF/GMA-WG/)
[20] Definition of architecture, technical plan and evaluation criteria for scheduling, resource management, security and job description; Deliverable 1.2, DataGRID Project, Work package 1, http://serverll.infn.it/workload-grid/documents.htm
[21] Stockinger, H.; Samar, A.; Allcock, B.; Foster, I.; Holtman, K.; Tierney, B.; File and Object Replication in Data Grids, Proceedings of 10th IEEE Symposium on High Performance and Distributed Computing (HPDC-10), San Francisco, California, Aug 2001
[22] Czajkowski, K.; Fitzgerald, S.; Foster, I.; Kesselman, C.; Grid Information Services for Distributed Resource Sharing; Proceedings of the Tenth IEEE Interna-tional Symposium on High-Performance Distributed Computing (HPDC-10), IEEE Press, August 2001
[23] Almes, G.; Kalidindi, S.; Zekauskas, M.; A Round-trip Delay Metric for IPPM, RFC 2681
[24] Yajnik, M.; Moon, S.B.; Kurose, J.; Towsley, D.; Measurement and Modeling of the Temporal Dependence in Packet Loss; Proc. IEEE Infocom'99 (New York, NY, March 1999)
[25] Mathis, M.; Semke, J.; Mahdavi, J.; Ott, T.; The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Computer Communications Review, volume 27, number 3, July 1997
[26] Thain, D.; Livny, M.; Multiple Bypass: Interposition Agents for Distributed Com-puting, Journal of Cluster Computing, vol. 4, pages 39-47, 2001
[27] 中国网格信息中转站；http://www.gridhome.com/
[28] 清华大学计算机系网格研究组；http://hpclab.cs.tsinghua.edu.cn/
[29] 刘鹏；网格应用研究现状；http://www.gridhome.com/grid/paperppt/GridApplications.pdf
[30] 刘鹏；我国网格研究现状；http://www.gridhome.com/grid/paperppt/OurCountry.pdf
[31] 刘鹏：网格发展趋势；http://www.gridhome.com/grid/paperppt/GridTrend.pdf
[32] 刘鹏；网格概念的界定；http://www.gridhome.com/grid/paperppt/GridConcept.pdf
[33] M Swany, R. Wolski, Representing Dynamic Performance Information in Grid Environments with the Network Weather Service, 2nd IEEE International Symposium on Cluster Computing and the Grid (CCGrid2002), Berlin, Germany, May 2002.
[34] Bandwidth; http://www.cacr.caltech.edu/Uews/sc02/bandwid.pdf
[35] Sebastian Ho. Survey of Grid Meta-Scheduler.
http://student.bii.a-star.edu.sg/~sebastianh/Scheduler_Survey.pdf
[36] Condor-G. Unversity of Wisconsin, madison. http://www.cs.wisc.edu/condor
[37] R. Wolski, N. Spring, and J. Hayes. The network weather service: A distributed resource performance forecasting service for metacomputing. Future Generation Computer Systems, 1999.
[38] 郑然，李胜利，金海；网格资源管理与调度模型的研究；华中科技大学学报，Vol．29 No．12，2001
[39] 曹鸿强，肖侬，卢锡城，刘艳；一种基于市场机制的计算网格资源分配方法；计算机研究与发展，Vol．39，No．8，2002
[40] Rai jkumar Buyya编，郑纬民，石威，汪东升等译；高性能群集计算：结构与系统(第一卷)；电子工业出版社，2001年6月；P352-346