校园数据网格关键技术研究与设计
摘要
数据网格是一种网格计算系统，主要用来处理数据——有约束的共享和管理大量的分布式数据。数据网格技术是研究的热点，主要集中在元数据管理和复制管理两个方面。校园网络环境中存在大量的信息孤岛，许多资源和信息不能得到有效的利用和共享。
本文通过对数据网格中元数据管理和复制管理技术的分析研究，设计了校园数据网格系统来解决校园网络环境中的问题。主要从以下几个方面进行了研究：
1．元数据管理的分析研究：本文分析了当前元数据管理模型的特点，并在此基础上提出了一种局部自治的、分布式的、三层结构的元数据管理模型。
2．副本的创建策略分析研究：本文分析了现有的副本创建策略，并提出了适用于校园数据网格系统的缓存加最佳用户副本创建策略。在局部自治域之间采用缓存副本创建策略，在局部自治域内采用最佳用户策略。
3．副本的定位与选择机制分析研究：本文分析了副本定位与选择的各种机制，选取了适用于校园数据网格系统的机制。副本的定位采用副本目录来实现，副本目录中记录逻辑文件到物理文件的映射信息，来完成副本定位。副本的选择采用简单、高效的IBL算法来实现。
4．副本的一致性管理分析研究：本文分析了现有的副本一致性策略，提出了适用于本文提出的元信息管理模型的副本一致性策略。
在研究数据网格关键技术的基础上，结合本文的研究成果设计了校园数据网格系统，并给出了系统初步的原型实现。
A data grid is a grid computing system that deals with data—the controlled sharing and management of large amounts of distributed data. Technology of Data Grid is a research hot. Metadata and replica management are crucial aspect. There are great deals of information isolated island, many resources and information can't get valid utilization and share in the campus network environment.
To investigate on technology of metadata and replica management in Data Grid, designs Campus Data Grid to resolve the problem within environment of the campus network. This paper carries on a research from several followings:
1. Metadata management: To analyze features of some related model of metadata management. On the basis of research a kind of local autonomy, distributed and three layers model of metadata management is suggested.
2. Replica creation strategy: To analyze some related works on the replica creation strategy. Caching plus best client replica creation strategy for Campus Data Grid environment is suggested. Between of local autonomy adopts caching strategy. Inside of local autonomy adopts best client strategy.
3. Replica location and selection mechanism: To analyze some related works on the replica location and selection mechanism. Mechanism for Campus Data Grid environment is selected. Replica catalog is used to implement replica location. Replica catalog storage mapping of logical file to physical file that implements the location. Replica selection adopts simple and efficient IBL algorithm.
4. Replica consistency management: To analyze some related replica consistency management strategy. Replica consistency management strategy for metadata model of this paper is suggested.
On basis of investigating on key technology of Data Grid, this paper designs Campus Data Grid system with production of research and realizes primary system model.
本文通过对数据网格中元数据管理和复制管理技术的分析研究，设计了校园数据网格系统来解决校园网络环境中的问题。主要从以下几个方面进行了研究：
1．元数据管理的分析研究：本文分析了当前元数据管理模型的特点，并在此基础上提出了一种局部自治的、分布式的、三层结构的元数据管理模型。
2．副本的创建策略分析研究：本文分析了现有的副本创建策略，并提出了适用于校园数据网格系统的缓存加最佳用户副本创建策略。在局部自治域之间采用缓存副本创建策略，在局部自治域内采用最佳用户策略。
3．副本的定位与选择机制分析研究：本文分析了副本定位与选择的各种机制，选取了适用于校园数据网格系统的机制。副本的定位采用副本目录来实现，副本目录中记录逻辑文件到物理文件的映射信息，来完成副本定位。副本的选择采用简单、高效的IBL算法来实现。
4．副本的一致性管理分析研究：本文分析了现有的副本一致性策略，提出了适用于本文提出的元信息管理模型的副本一致性策略。
在研究数据网格关键技术的基础上，结合本文的研究成果设计了校园数据网格系统，并给出了系统初步的原型实现。
A data grid is a grid computing system that deals with data—the controlled sharing and management of large amounts of distributed data. Technology of Data Grid is a research hot. Metadata and replica management are crucial aspect. There are great deals of information isolated island, many resources and information can't get valid utilization and share in the campus network environment.
To investigate on technology of metadata and replica management in Data Grid, designs Campus Data Grid to resolve the problem within environment of the campus network. This paper carries on a research from several followings:
1. Metadata management: To analyze features of some related model of metadata management. On the basis of research a kind of local autonomy, distributed and three layers model of metadata management is suggested.
2. Replica creation strategy: To analyze some related works on the replica creation strategy. Caching plus best client replica creation strategy for Campus Data Grid environment is suggested. Between of local autonomy adopts caching strategy. Inside of local autonomy adopts best client strategy.
3. Replica location and selection mechanism: To analyze some related works on the replica location and selection mechanism. Mechanism for Campus Data Grid environment is selected. Replica catalog is used to implement replica location. Replica catalog storage mapping of logical file to physical file that implements the location. Replica selection adopts simple and efficient IBL algorithm.
4. Replica consistency management: To analyze some related replica consistency management strategy. Replica consistency management strategy for metadata model of this paper is suggested.
On basis of investigating on key technology of Data Grid, this paper designs Campus Data Grid system with production of research and realizes primary system model.
引文
[1] I. Foster, C. Kesselman. The Grid: Blueprint for a Future ComputingInfrastructure. California: Morgan Kaufmann Publishers, 1999.
[2] I. Foster. The Grid: A New Infrastructure for 21st Century Science. PhysicsToday, 2002, 55(2),42-47.
[3] J. M. Schopf, B. Nitzberg. Grids: Top Ten Questions. Scientific Programming, special issue on Grid Computing, 2002, 10(2),103-111.
[4] I. Foster. What is the Grid? A Three Point Checklist, Argonne National Laboratory & University of Chicago,2002. http://www.mcs.csuhayward.edu/～tebo/Classes/6580/papers/WhatIsTheGrid.pdf
[5] 都志辉，陈渝，刘鹏．网格计算，北京，清华大学出版社，2002，9-12．
[6] I. Foster, C. Kesselman, S. Tuecke. The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International J. Supercomputer Applications, 2001,15(3).
[7] OGSA规范, http://www.gridforum.org/ogsi-wg/drafts/GS_Spec_draf103_2002-07-17.pdf.
[8] 都志辉，陈渝，刘鹏，王小鸽，杜江，周念生．以服务为中心的网格体系结构OGSA，计算机科学，2003．
[9] S. Tuecke, K. Czajkowski, I. Foster, J. Frey, S. Graham, C. Kesselman,etc. OpenGrid Services Infrastructure (OGSI) Version 1.0,2003. http://xml.coverpages.org/OGSI-SpecificationV110.pdf
[10] K. Czajkowski, Donald F Ferguson, I. Foster, J. Frey, S. Graham,etc. The WS-Resource Framework Version 1.0,2004. http://www.globus.org/wsrf/specs/ws-wsrf.pdf
[11] 肖侬，编织数据网格，计算机世界，2002．
[12] CERN in 2 minutes,http://public.web.cem.ch/Public/whatiscern.html
[13] The World Wide Web,http://public.web.cem.ch/Public/ACHIEVEMENTS/web.html
[14] http://www.globus.org/
[15] SRB, http://www.sdsc.edu/srb/index.php/Main_Page
[16] IVDGL, http://www.ivdgl.org/
[17] PPDG, http://www.ppdg.net/
[18] 区颖薇，网络环境下资源管理模式——元数据，图书馆学研究，2002．
[19] Ray S. Atarashi,J. Kishigami,S. Sugimoto. Metadata and newchallenges.In: Proceedings of the 2003 Symposium on Applications and theInternet Workshops,2003,395-398.
[20] 马珉，元数据——组织网上信息资源的基本格式，情报科学，2002，20(4)．
[21] 刘嘉，元数据：理念与应用，中国图书馆学报，2001，5．
[22] http://www.sdsc.edu/srb/index.php/SRB_User_Manual
[23] http://www.sdsc.edu/srb/index.php/MCAT
[24] S. Fitzgerald, I. Foster, C. Kesselman, G. Laszewski, W. Smith, S. Tuecke.A Directory Service for Configuring High-Performance DistributedComputations. Proc.6th IEEE on Symposium High-Performance DistributedComputing,1997, 365-375.
[25] 付伟，数据网格中元信息服务系统的设计与实现，国防科学技术大学，硕士论文，2003，38-41．
[26] S. Goel, R. Buyya. DATA REPLICATION STRATEGIES IN WIDE AREADISTRIBOTED SYSTEMS. Grid Computing and Distributed Systems (GRIDS)Laboratory Department of Computer Science and Software Engineering TheUniversity of Melboume, Australia,2006,http://jarrett.cs.mu.oz.au/～raj/papers/DataReplicationInDSChapter2006.pdf
[27] L. Guy, P. Kunszt, E. Laure,H. Stockinger, K. Stockinger. Replica Managementin Data Grids. CERN, European Organization for Nuclear Research CH-1211Geneva 23, Switzerland, July 1,2002.
[28] K. Ranganathan, I. Foster. Identifying Dynamic Replication Strategies for a High—Performance Data Grid. Proceedings of the Second International Workshopon Grid Computing,Denver, November, 2002.
[29] M. Carman, F. Zini, L. Serafini, K. Stockinger. Towards an Economy-Based Optimisation of File Access and Replication on a Data Grid. 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2002.
[30] K. Ranganathan,I. Foster. Design and Evaluation of Dynamic Replication Strategies for a High Performance Data Grid. International Conference on Computing in High Energy and Nuclear Physics, 2001.
[31]孙海燕,数据网格副本管理关键技术研究,国防科学技术大学,博士论文, 2005,57—58．2005, 57-58.
[32] Getting Started with the Globus Replica Catalog. http://www.globus.Org/toolkit/docs/2.4/datagrid/deliverables/replicaGettingStar ted.pdf
[33] S. Vazhkudai,S. Tuecke,I. Foster. Replica Selection in the Globus Data Grid. Proceedings of the First IEEE/ACM International Conference on Cluster Computing and the Grid (CCGRID 2001),IEEE Computer Society Press, May 2001.
[34] I. Foster, A. Iamnitchi, M. Ripeanu, A. Chervenak, E. Deelman, C. Kesselman, W. Hoschek, P. Kunszt, H. Stockinger, K. Stockinger. Giggle:A Framework for Constructing Scalable Replica Location Services. Proceeding of the 15th annual IEEE Supercomputing Conference,2002.
[35] R. Wolski,'Neil T. Spring, J. Hayes. The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. The Journal of Future Generation Computer Systems, 1999,757-768.
[36] B. Tierney, W. Johnston, B. Crowley, G. Hoo, C. Brooks,D. Gunter. The Netlogger methodology for high performance distributed systems performance analysis. Proceedings of the Seventh IEEE International Symposium on HPDC 7,1998,260-267.
[37] M.Faerman, A. Su, R. Wolski, F. Berman. Adaptive Performance Prediction for Distributed Data-Intensive Applications. Proceedings of the 1999 ACM/IEEE conference on Supercomputing, 1999.
[38] Yu Hu,Jennifer M Schopf. EBL for Replica Selection in Data-Intensive Grid Applications. Master's Thesis, Department of Computer Science, University of Chicago, 2003.
[39] A. Domenici, F. Donno, G. Pucciani, H. Stockinger, K. Stockinger. Replica consistency in a Data Grid. 9th International Workshop on Advanced Computing and Analysis Techniques in Physics Research,2003.
[40] Yuzhong Sun and Zhiwei Xu. Grid Replication Coherence Protocol. The 18th International Parallel and Distributed Processing Symposium, Santa Fe, USA, April 2004,232-239.
[41] OptorSim, http://edg-wp2.web.cern.ch/edg-wp2/optimization/optorsim.html
[42] http://www.ogsadai.org.uk/about/ogsa-dai
[43] OGSA-DAI Architecture http://www.ogsadai.org.Uk/documentation/ogsadai-wsrf-2.2/doc/background/arc hitecture.html
[44] Interacting with Data Service Resources. http://www.ogsadai.org.uk/documentation/ogsadai-wsrf-2.2/doc/background/../i nteraction/index.html
[45] B.Allcock,J.Besterl,J.Bresnahan,Ann L.Chervenak,I.Foster,C. Kesselman,S. Meder,V.Nefedova,D.Quesnel,S.Tuecke. Data Management and Transfer in High Performance Computational Grid Environments.http://www.globus.org/alliance/publications/papers/dataMgmt.pdf
[46] http://www.cogkit.org/
[47] http://www.globus.org/security/overview.html
[48] http://www.ietf.org/rfc/rfc2743.txt
[2] I. Foster. The Grid: A New Infrastructure for 21st Century Science. PhysicsToday, 2002, 55(2),42-47.
[3] J. M. Schopf, B. Nitzberg. Grids: Top Ten Questions. Scientific Programming, special issue on Grid Computing, 2002, 10(2),103-111.
[4] I. Foster. What is the Grid? A Three Point Checklist, Argonne National Laboratory & University of Chicago,2002. http://www.mcs.csuhayward.edu/～tebo/Classes/6580/papers/WhatIsTheGrid.pdf
[5] 都志辉，陈渝，刘鹏．网格计算，北京，清华大学出版社，2002，9-12．
[6] I. Foster, C. Kesselman, S. Tuecke. The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International J. Supercomputer Applications, 2001,15(3).
[7] OGSA规范, http://www.gridforum.org/ogsi-wg/drafts/GS_Spec_draf103_2002-07-17.pdf.
[8] 都志辉，陈渝，刘鹏，王小鸽，杜江，周念生．以服务为中心的网格体系结构OGSA，计算机科学，2003．
[9] S. Tuecke, K. Czajkowski, I. Foster, J. Frey, S. Graham, C. Kesselman,etc. OpenGrid Services Infrastructure (OGSI) Version 1.0,2003. http://xml.coverpages.org/OGSI-SpecificationV110.pdf
[10] K. Czajkowski, Donald F Ferguson, I. Foster, J. Frey, S. Graham,etc. The WS-Resource Framework Version 1.0,2004. http://www.globus.org/wsrf/specs/ws-wsrf.pdf
[11] 肖侬，编织数据网格，计算机世界，2002．
[12] CERN in 2 minutes,http://public.web.cem.ch/Public/whatiscern.html
[13] The World Wide Web,http://public.web.cem.ch/Public/ACHIEVEMENTS/web.html
[14] http://www.globus.org/
[15] SRB, http://www.sdsc.edu/srb/index.php/Main_Page
[16] IVDGL, http://www.ivdgl.org/
[17] PPDG, http://www.ppdg.net/
[18] 区颖薇，网络环境下资源管理模式——元数据，图书馆学研究，2002．
[19] Ray S. Atarashi,J. Kishigami,S. Sugimoto. Metadata and newchallenges.In: Proceedings of the 2003 Symposium on Applications and theInternet Workshops,2003,395-398.
[20] 马珉，元数据——组织网上信息资源的基本格式，情报科学，2002，20(4)．
[21] 刘嘉，元数据：理念与应用，中国图书馆学报，2001，5．
[22] http://www.sdsc.edu/srb/index.php/SRB_User_Manual
[23] http://www.sdsc.edu/srb/index.php/MCAT
[24] S. Fitzgerald, I. Foster, C. Kesselman, G. Laszewski, W. Smith, S. Tuecke.A Directory Service for Configuring High-Performance DistributedComputations. Proc.6th IEEE on Symposium High-Performance DistributedComputing,1997, 365-375.
[25] 付伟，数据网格中元信息服务系统的设计与实现，国防科学技术大学，硕士论文，2003，38-41．
[26] S. Goel, R. Buyya. DATA REPLICATION STRATEGIES IN WIDE AREADISTRIBOTED SYSTEMS. Grid Computing and Distributed Systems (GRIDS)Laboratory Department of Computer Science and Software Engineering TheUniversity of Melboume, Australia,2006,http://jarrett.cs.mu.oz.au/～raj/papers/DataReplicationInDSChapter2006.pdf
[27] L. Guy, P. Kunszt, E. Laure,H. Stockinger, K. Stockinger. Replica Managementin Data Grids. CERN, European Organization for Nuclear Research CH-1211Geneva 23, Switzerland, July 1,2002.
[28] K. Ranganathan, I. Foster. Identifying Dynamic Replication Strategies for a High—Performance Data Grid. Proceedings of the Second International Workshopon Grid Computing,Denver, November, 2002.
[29] M. Carman, F. Zini, L. Serafini, K. Stockinger. Towards an Economy-Based Optimisation of File Access and Replication on a Data Grid. 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2002.
[30] K. Ranganathan,I. Foster. Design and Evaluation of Dynamic Replication Strategies for a High Performance Data Grid. International Conference on Computing in High Energy and Nuclear Physics, 2001.
[31]孙海燕,数据网格副本管理关键技术研究,国防科学技术大学,博士论文, 2005,57—58．2005, 57-58.
[32] Getting Started with the Globus Replica Catalog. http://www.globus.Org/toolkit/docs/2.4/datagrid/deliverables/replicaGettingStar ted.pdf
[33] S. Vazhkudai,S. Tuecke,I. Foster. Replica Selection in the Globus Data Grid. Proceedings of the First IEEE/ACM International Conference on Cluster Computing and the Grid (CCGRID 2001),IEEE Computer Society Press, May 2001.
[34] I. Foster, A. Iamnitchi, M. Ripeanu, A. Chervenak, E. Deelman, C. Kesselman, W. Hoschek, P. Kunszt, H. Stockinger, K. Stockinger. Giggle:A Framework for Constructing Scalable Replica Location Services. Proceeding of the 15th annual IEEE Supercomputing Conference,2002.
[35] R. Wolski,'Neil T. Spring, J. Hayes. The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. The Journal of Future Generation Computer Systems, 1999,757-768.
[36] B. Tierney, W. Johnston, B. Crowley, G. Hoo, C. Brooks,D. Gunter. The Netlogger methodology for high performance distributed systems performance analysis. Proceedings of the Seventh IEEE International Symposium on HPDC 7,1998,260-267.
[37] M.Faerman, A. Su, R. Wolski, F. Berman. Adaptive Performance Prediction for Distributed Data-Intensive Applications. Proceedings of the 1999 ACM/IEEE conference on Supercomputing, 1999.
[38] Yu Hu,Jennifer M Schopf. EBL for Replica Selection in Data-Intensive Grid Applications. Master's Thesis, Department of Computer Science, University of Chicago, 2003.
[39] A. Domenici, F. Donno, G. Pucciani, H. Stockinger, K. Stockinger. Replica consistency in a Data Grid. 9th International Workshop on Advanced Computing and Analysis Techniques in Physics Research,2003.
[40] Yuzhong Sun and Zhiwei Xu. Grid Replication Coherence Protocol. The 18th International Parallel and Distributed Processing Symposium, Santa Fe, USA, April 2004,232-239.
[41] OptorSim, http://edg-wp2.web.cern.ch/edg-wp2/optimization/optorsim.html
[42] http://www.ogsadai.org.uk/about/ogsa-dai
[43] OGSA-DAI Architecture http://www.ogsadai.org.Uk/documentation/ogsadai-wsrf-2.2/doc/background/arc hitecture.html
[44] Interacting with Data Service Resources. http://www.ogsadai.org.uk/documentation/ogsadai-wsrf-2.2/doc/background/../i nteraction/index.html
[45] B.Allcock,J.Besterl,J.Bresnahan,Ann L.Chervenak,I.Foster,C. Kesselman,S. Meder,V.Nefedova,D.Quesnel,S.Tuecke. Data Management and Transfer in High Performance Computational Grid Environments.http://www.globus.org/alliance/publications/papers/dataMgmt.pdf
[46] http://www.cogkit.org/
[47] http://www.globus.org/security/overview.html
[48] http://www.ietf.org/rfc/rfc2743.txt