基于网格环境的资源存取的研究
摘要
近年来，随着网络化和智能化趋势的发展，人类的应用迅速朝着高性能、多样性、多功能发展，许多大规模的科学计算不仅仅需要一台高性能计算机，它还更需要由多种机器组成、多个系统合作、多个科学仪器设备相连的网络虚拟超级计算机。这些应用要求将地理上分布、异构的多种计算资源通过高速网络连接起来，共同完成计算任务。网格正是在这种背景下诞生的，其目标是将地理上分布、异构的高性能计算机、数据服务器，大型检索存储系统和可视化、虚拟实现系统等，通过高速互连网络连接并集成起来，共同完成一些缺乏有效研究办法的重大应用研究问题。通过利用网络上的大量计算资源、丰富的开发工具和友好的人机界面，以及各种不同性能和功能的信息系统，进行高性能的并行与分布式计算，解决核模拟、气象、航天、生物化学、神经和环境等重大科学应用领域的计算问题。
网格技术的产生主要有以下几个原因：
①计算资源的广域分布
②广域互连技术的发展
③广域范围内资源共享的需求
网格计算技术在产生的初期主要集中在高性能科学计算领域，而现在网格并不局限于此，出现了适用于不同应用领域的网格技术，如生物网格、信息网格等。网格最大的优点之一是有利于全国乃至全球范围内各种计算和数据资源的共享。这些资源包括多种文件系统、数据库资源、高性能计算机、贵重设备、软件等。
中国科学院计算技术研究所承担的国家863项目---面向网格的超级服务
四川大学硕士学位论文
器曙光4000，提出了织女星网格(vega Grid)研究，它继承了已有计算机的设
计方法和思想，即将网格看成是一台虚拟的、具有单一系统映像的超级计算机
系统，其实质就是利用网格技术实现计算资源和信息的共享。网格中的数据资
源各种各样，表示、存储的形式也各不同。一些数据可能以文件形式存储，一
些数据存储在数据库或数据仓库中，另一些存储在如同HPSS的Archive档案系
统中，还有一些数据是由多个分布存储系统中的数据组成的。方便而有效地访
问各种异构数据组成的数据集合是网格的一个主要功能和关键技术。如果为每
一种数据存储方式提供一种访问方法，用户是不可能接受的。因此，网格必须
抽象各种数据存储系统，形成一个抽象模型，为不同的救据存储系统提供统一
的数据访问接口。数据访问服务的作用就是将存储、检索数据集等高层用户的
请求映射为异构分布式存储环境中的底层存储访问操作，实现广域范围内对数
据有效的统一访问和管理。
本文就网格环境下资源存取的设计与实现问题进行了深入的研究，并在现
有的技术条件下提出并详细分析了作者的设计方案，介绍了具体实现工作，同
时就设计与实现工作中的关键技术问题进行了分析和讨论。
With the rapid development of science and technology, more and more large scale science computing need collaboration of HPC and multi-system, which give birth to the Grid. By high-speed network, the Grid aim at integrating all -resources in order to complete specified task, such as nuclear simulation, distributed computing, aviation.
The grid technology is a hotspot in HPC research. The architecture design is the basis for constructing grid system. Based on the existing grid technologies, the design of Vega Grid is proposed and make research on some important issues studied. The fundamental idea of Vega Grid is to build a virtual, single grid super computer.
One of the most strongpoint of Grid is sharing the resource in world wide, which include HPC, data resources, software. The data resources, which may be based on different operation system or different storage system, include all kinds of files, database resource and other data resource stored in distributed data warehouse. In order to cover the difference of data resources, grid must give the abstract model, which can provide a uniform interface for access data in grid.
The paper introduces the background of Vega Grid, and analyze the access of data resources in Vega Grid, then it designs the architecture for access of data resources,and explains the implementation and pivotal techniques based on the practices of author.
网格技术的产生主要有以下几个原因：
①计算资源的广域分布
②广域互连技术的发展
③广域范围内资源共享的需求
网格计算技术在产生的初期主要集中在高性能科学计算领域，而现在网格并不局限于此，出现了适用于不同应用领域的网格技术，如生物网格、信息网格等。网格最大的优点之一是有利于全国乃至全球范围内各种计算和数据资源的共享。这些资源包括多种文件系统、数据库资源、高性能计算机、贵重设备、软件等。
中国科学院计算技术研究所承担的国家863项目---面向网格的超级服务
四川大学硕士学位论文
器曙光4000，提出了织女星网格(vega Grid)研究，它继承了已有计算机的设
计方法和思想，即将网格看成是一台虚拟的、具有单一系统映像的超级计算机
系统，其实质就是利用网格技术实现计算资源和信息的共享。网格中的数据资
源各种各样，表示、存储的形式也各不同。一些数据可能以文件形式存储，一
些数据存储在数据库或数据仓库中，另一些存储在如同HPSS的Archive档案系
统中，还有一些数据是由多个分布存储系统中的数据组成的。方便而有效地访
问各种异构数据组成的数据集合是网格的一个主要功能和关键技术。如果为每
一种数据存储方式提供一种访问方法，用户是不可能接受的。因此，网格必须
抽象各种数据存储系统，形成一个抽象模型，为不同的救据存储系统提供统一
的数据访问接口。数据访问服务的作用就是将存储、检索数据集等高层用户的
请求映射为异构分布式存储环境中的底层存储访问操作，实现广域范围内对数
据有效的统一访问和管理。
本文就网格环境下资源存取的设计与实现问题进行了深入的研究，并在现
有的技术条件下提出并详细分析了作者的设计方案，介绍了具体实现工作，同
时就设计与实现工作中的关键技术问题进行了分析和讨论。
With the rapid development of science and technology, more and more large scale science computing need collaboration of HPC and multi-system, which give birth to the Grid. By high-speed network, the Grid aim at integrating all -resources in order to complete specified task, such as nuclear simulation, distributed computing, aviation.
The grid technology is a hotspot in HPC research. The architecture design is the basis for constructing grid system. Based on the existing grid technologies, the design of Vega Grid is proposed and make research on some important issues studied. The fundamental idea of Vega Grid is to build a virtual, single grid super computer.
One of the most strongpoint of Grid is sharing the resource in world wide, which include HPC, data resources, software. The data resources, which may be based on different operation system or different storage system, include all kinds of files, database resource and other data resource stored in distributed data warehouse. In order to cover the difference of data resources, grid must give the abstract model, which can provide a uniform interface for access data in grid.
The paper introduces the background of Vega Grid, and analyze the access of data resources in Vega Grid, then it designs the architecture for access of data resources,and explains the implementation and pivotal techniques based on the practices of author.
引文
1. The Anatomy of the Grid: Enabling Scalable Virtual Organizations. I. Foster, C. Kesselman, S. Tuecke, Intl. J. Supereomputer Applications, 2001.
2. The Globus Project: A Status Report. I. Foster, C. Kesselman, Proc. IPPS/SPDP '98 Heterogeneous Computing Workshop, pg. 4-18, 1998.
3. A National-Scale Authentication Infrastructure. R. Butler, D. Engert, I. Foster, C. Kesselman, S. Tuecke, J. Volmer, V. Welch. IEEE Computer, 33(12): 60-66, 2000.
4. A Security Architecture for Computational Grids. I. Foster, C. Kesselman, G. Tsudik, S. Tuecke, Proc. 5th ACM Conference on Computer and Communications Security Conference, pg. 83-92, 1998.
5. A Resource Management Architecture for Metacomputing Systems. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke. Proc. IPPS/SPDP '98 Workshop on Job Scheduling Strategies for Parallel Processing, 1998
6. A Directory Service for Configuring High-Performance Distributed Computations. S. Fitzgerald, I. Foster, C. Kesselman, G. von Laszewski, W. Smith, S. Tuecke. Proc. 6th IEEE Symp. on High-Performance Distributed Computing, pg. 365-375, 1997.
7. GASS: A Data Movement and Access Service for Wide Area Computing Systems. J. Bester, I. Foster, C. Kesselman, J. Tedesco, S. Tuecke. (Sixth Workshop on I/O in Parallel and Distributed Systems, May 5, 1999).
8. A Fault Detection Service for Wide Area Distributed Computations. P. Stelling, I. Foster, C. Kesselman, C. Lee, G. von Laszewski, Proc. 7th IEEE Symp. on High Performance Distributed Computing, 268-278, 1998.
9. Managing Multiple Communication Methods in High-Performance Networked Computing Systems. I. Foster, J. Geisler, C. Kesselman, S.
Tuecke. J. Parallel and Distributed Computing, 40: 35-48, 1997.
10. Application Experiences with the Globus Toolkit. S. Brunett, K. Czajkowski, S. Fitzgerald, I. Foster, A. Johnson, C. Kesselman, J. Leigh, S. Tuecke, 7th IEEE Symp. on High Performance Distributed Computing. July 1998.
11. L. Smarr and C. Catlett, Metacomputing, Communication of the ACM, 35(1992), pp. 44-52.
12. T. DeFanti, I. Foster, M. Papka, R. Stevents, and T. Kuhfuss, Overview of the I-WAY: Wide area visual supercomputing, International Journal of Supercomputer Applications, 10(1996), pp. 123-130.
13. C. E. Nugent, T. E. Vollman, and J. Ruml, An experimental comparison of techniques for the assignment of facilities to locations, Operations Research 16(1968) pp. 150-173.
14. J. P. Goux, S. Kulkami, J. Linderoth, and M. Yoder, An enabling framework for master-worker computing applications on the computational grid, available at URL: http://www. mcs. anl. gov/metaneos/papers/mw2.ps, June 2000.
15. Sculley, A. and Woods, W. B2B Exchanges: The Killer Application in the Business-to-Business Internet Revolution. ISI Publications, 2000.
16. Barry, J., Aparicio, M., Durniak, T., Herman, P., Karuturi, J., Woods, C., Gilman, C., Ramnath, R. and Lam, H., NIIIP-SMART: An Investigation of Distributed Object Approaches to Support MES Development and Deployment in a Virtual Enterprise. In 2nd Intl Enterprise Distributed Computing Workshop, 1998, IEEE Press.
17. Foster, I. Internet Computing and the Emerging Grid. Nature Web Matters,(http://www.nature.com/nature/webmatters/grid/grid.html) 2000.
18. Aiken, R., Carey, M., Carpenter, B., Foster, I., Lynch, C., Mambretti, J., Moore, R., Strasnner, J. and Teitelbaum, B. Network Policy and Services: A Report of a Workshop on Middleware, IETF, RFC 2768, 2000.
http://www. ietf. org/rfc/rfc2768. txt.
19. Realizing the Information Future: The Internet and Beyond. National Academy Press, 1994. http://www.nap.edu/readingroom/books/rtif/.
20. Tierney, B., Johnston, W., Lee, J. and Hoo, G. Performance Analysis in High-Speed Wide Area IP over ATM Networks: Top-to-Bottom End-to-End Monitoring. IEEE Networking, 1996.
21. Beynon, M., Ferreira, R., Kurc, T., Sussman, A. and Saltz, J., DataCutter: Middleware for Filtering Very Large Scientific Datasets on Archival Storage Systems. In Proc. 8th Goddard Conference on Mass Storage Systems and Technologies/17th IEEE Symposium on Mass Storage Systems, 2000, 119-133.
22. Foster, I., Roy, A. and Sander, V., A Quality of Service Architecture that Combines Resource Reservation and Application Adaptation. In Proc. 8th International Workshop on Quality of Service, 2000.
23. Papakhian, M. Comparing Job-Management Systems: The User's Perspective. IEEE Computationial Science & Engineedng, (April-June) 1998.
24. Litzkow, M., Livny, M. and Mutka, M. Condor - A Hunter of Idle Workstations. In Proc. 8th Intl Conf. on Distributed Computing Systems, 1988, 104-111.
25. Livny, M. High-Throughput Resource Management. In Foster, I. and Kesselman, C. eds. The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999, 311-337.
26. Dierks, T. and Allen, C. The TLS Protocol Version 1.0, IETF, RFC 2246, 1999. http://www.ieff.org/rfc/rfc2246.txt.
27. Steiner, J., Neuman, B. C. and Schiller, J., Kerberos: An Authentication System for Open Network Systems. In Proc. Usenix Conference, 1988, 191-202.
28. Linn, J. Generic Security Service Application Program Interface Version 2,
Update 1, IETF, RFC 2743, 2000. http://www.ietf.org/rfc/rfc2743.
29. Foster, I. and Kesselman, C. (eds.). The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, 1999.
30. Foster, I. and Karonis, N. A Grid-Enabled MPI: Message Passing in Heterogeneous Distributed Computing Systems. In Proc. SC'98, 1998.
31. Berman, F. High-Performance Schedulers. In Foster, I. and Kesselman, C. eds. The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999, 279-309.
32. Berman, F., Wolski, R., Figueira, S., Schopf, J. and Shao, G. Application-Level Scheduling on Distributed Heterogeneous Networks. In Proc. Supercomputing '96, 1996.
33．褚兴军，徐志伟，企业门户框架的数据描述、控制及操作，计算机研究与发展，Vol．38增刊，2001．
34．徐志伟，李晓林，游赣梅，织女星信息网格的体系结构研究，计算机研究与发展，2002．
35．徐志伟，李伟．织女星网格的体系结构研究，计算机研究与发展，2002．
36. M. Handley, H. Schulzrinne, E. Schooler, and J. Rosenberg, "SIP: session initiation protocol," rfc2543, Internet Engineering Task Force, Mar. 1999.
37. B. Biggs, "A SIP Application Level Gateway for Network Address Translation", Internet Draft, Internet Engineering Task Force, March 2000.
38. M. Kaat, "Overview of 1999 IAB Network Layer Workshop", rfc2956, Internet Engineering Task Force. October 2000.
39. Rosenberg, Weinberger, Huitema, Mahy, "STUN - Simple Traversal of UDP Through Network Address Translators". Internet Draft, Internet Engineering Task Force, December 19, 2002
40. J. Rosenberg, J. Weinberger, H. Schulzrinne, " SIP Extensions for NAT Traversal". Internet Draft, Internet Engineering Task Force, August 28, 2001.
41. W3C 1.1标准．URL:http://www.w3.org/TR/SOAP
42. Erich Gamma, Richard Helm, Ralph Jhonson, Jhon Vlisside: Design Patton: Elements of Reusable Object-Orient Software
43. Acquie Barker, Java面向对象编程指南[M]．电子工业出版社，2001．11
44. Tarak Modi, Clean up you wire protocol with soap, UR: http://www.javaworld.com/javaworld/jw-03-2001/jw-0330-soap.html
45. Joshy Joseph, Handling attachment in soap URL: http://www-9OO.ibm.com/developerWorks/webservices/index.shtml
46. Gavin Bong, Apache SOAP type mapping, URL: http://www-900.ibm.com/developerWorks/webservices/index.shtml
2. The Globus Project: A Status Report. I. Foster, C. Kesselman, Proc. IPPS/SPDP '98 Heterogeneous Computing Workshop, pg. 4-18, 1998.
3. A National-Scale Authentication Infrastructure. R. Butler, D. Engert, I. Foster, C. Kesselman, S. Tuecke, J. Volmer, V. Welch. IEEE Computer, 33(12): 60-66, 2000.
4. A Security Architecture for Computational Grids. I. Foster, C. Kesselman, G. Tsudik, S. Tuecke, Proc. 5th ACM Conference on Computer and Communications Security Conference, pg. 83-92, 1998.
5. A Resource Management Architecture for Metacomputing Systems. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke. Proc. IPPS/SPDP '98 Workshop on Job Scheduling Strategies for Parallel Processing, 1998
6. A Directory Service for Configuring High-Performance Distributed Computations. S. Fitzgerald, I. Foster, C. Kesselman, G. von Laszewski, W. Smith, S. Tuecke. Proc. 6th IEEE Symp. on High-Performance Distributed Computing, pg. 365-375, 1997.
7. GASS: A Data Movement and Access Service for Wide Area Computing Systems. J. Bester, I. Foster, C. Kesselman, J. Tedesco, S. Tuecke. (Sixth Workshop on I/O in Parallel and Distributed Systems, May 5, 1999).
8. A Fault Detection Service for Wide Area Distributed Computations. P. Stelling, I. Foster, C. Kesselman, C. Lee, G. von Laszewski, Proc. 7th IEEE Symp. on High Performance Distributed Computing, 268-278, 1998.
9. Managing Multiple Communication Methods in High-Performance Networked Computing Systems. I. Foster, J. Geisler, C. Kesselman, S.
Tuecke. J. Parallel and Distributed Computing, 40: 35-48, 1997.
10. Application Experiences with the Globus Toolkit. S. Brunett, K. Czajkowski, S. Fitzgerald, I. Foster, A. Johnson, C. Kesselman, J. Leigh, S. Tuecke, 7th IEEE Symp. on High Performance Distributed Computing. July 1998.
11. L. Smarr and C. Catlett, Metacomputing, Communication of the ACM, 35(1992), pp. 44-52.
12. T. DeFanti, I. Foster, M. Papka, R. Stevents, and T. Kuhfuss, Overview of the I-WAY: Wide area visual supercomputing, International Journal of Supercomputer Applications, 10(1996), pp. 123-130.
13. C. E. Nugent, T. E. Vollman, and J. Ruml, An experimental comparison of techniques for the assignment of facilities to locations, Operations Research 16(1968) pp. 150-173.
14. J. P. Goux, S. Kulkami, J. Linderoth, and M. Yoder, An enabling framework for master-worker computing applications on the computational grid, available at URL: http://www. mcs. anl. gov/metaneos/papers/mw2.ps, June 2000.
15. Sculley, A. and Woods, W. B2B Exchanges: The Killer Application in the Business-to-Business Internet Revolution. ISI Publications, 2000.
16. Barry, J., Aparicio, M., Durniak, T., Herman, P., Karuturi, J., Woods, C., Gilman, C., Ramnath, R. and Lam, H., NIIIP-SMART: An Investigation of Distributed Object Approaches to Support MES Development and Deployment in a Virtual Enterprise. In 2nd Intl Enterprise Distributed Computing Workshop, 1998, IEEE Press.
17. Foster, I. Internet Computing and the Emerging Grid. Nature Web Matters,(http://www.nature.com/nature/webmatters/grid/grid.html) 2000.
18. Aiken, R., Carey, M., Carpenter, B., Foster, I., Lynch, C., Mambretti, J., Moore, R., Strasnner, J. and Teitelbaum, B. Network Policy and Services: A Report of a Workshop on Middleware, IETF, RFC 2768, 2000.
http://www. ietf. org/rfc/rfc2768. txt.
19. Realizing the Information Future: The Internet and Beyond. National Academy Press, 1994. http://www.nap.edu/readingroom/books/rtif/.
20. Tierney, B., Johnston, W., Lee, J. and Hoo, G. Performance Analysis in High-Speed Wide Area IP over ATM Networks: Top-to-Bottom End-to-End Monitoring. IEEE Networking, 1996.
21. Beynon, M., Ferreira, R., Kurc, T., Sussman, A. and Saltz, J., DataCutter: Middleware for Filtering Very Large Scientific Datasets on Archival Storage Systems. In Proc. 8th Goddard Conference on Mass Storage Systems and Technologies/17th IEEE Symposium on Mass Storage Systems, 2000, 119-133.
22. Foster, I., Roy, A. and Sander, V., A Quality of Service Architecture that Combines Resource Reservation and Application Adaptation. In Proc. 8th International Workshop on Quality of Service, 2000.
23. Papakhian, M. Comparing Job-Management Systems: The User's Perspective. IEEE Computationial Science & Engineedng, (April-June) 1998.
24. Litzkow, M., Livny, M. and Mutka, M. Condor - A Hunter of Idle Workstations. In Proc. 8th Intl Conf. on Distributed Computing Systems, 1988, 104-111.
25. Livny, M. High-Throughput Resource Management. In Foster, I. and Kesselman, C. eds. The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999, 311-337.
26. Dierks, T. and Allen, C. The TLS Protocol Version 1.0, IETF, RFC 2246, 1999. http://www.ieff.org/rfc/rfc2246.txt.
27. Steiner, J., Neuman, B. C. and Schiller, J., Kerberos: An Authentication System for Open Network Systems. In Proc. Usenix Conference, 1988, 191-202.
28. Linn, J. Generic Security Service Application Program Interface Version 2,
Update 1, IETF, RFC 2743, 2000. http://www.ietf.org/rfc/rfc2743.
29. Foster, I. and Kesselman, C. (eds.). The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, 1999.
30. Foster, I. and Karonis, N. A Grid-Enabled MPI: Message Passing in Heterogeneous Distributed Computing Systems. In Proc. SC'98, 1998.
31. Berman, F. High-Performance Schedulers. In Foster, I. and Kesselman, C. eds. The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999, 279-309.
32. Berman, F., Wolski, R., Figueira, S., Schopf, J. and Shao, G. Application-Level Scheduling on Distributed Heterogeneous Networks. In Proc. Supercomputing '96, 1996.
33．褚兴军，徐志伟，企业门户框架的数据描述、控制及操作，计算机研究与发展，Vol．38增刊，2001．
34．徐志伟，李晓林，游赣梅，织女星信息网格的体系结构研究，计算机研究与发展，2002．
35．徐志伟，李伟．织女星网格的体系结构研究，计算机研究与发展，2002．
36. M. Handley, H. Schulzrinne, E. Schooler, and J. Rosenberg, "SIP: session initiation protocol," rfc2543, Internet Engineering Task Force, Mar. 1999.
37. B. Biggs, "A SIP Application Level Gateway for Network Address Translation", Internet Draft, Internet Engineering Task Force, March 2000.
38. M. Kaat, "Overview of 1999 IAB Network Layer Workshop", rfc2956, Internet Engineering Task Force. October 2000.
39. Rosenberg, Weinberger, Huitema, Mahy, "STUN - Simple Traversal of UDP Through Network Address Translators". Internet Draft, Internet Engineering Task Force, December 19, 2002
40. J. Rosenberg, J. Weinberger, H. Schulzrinne, " SIP Extensions for NAT Traversal". Internet Draft, Internet Engineering Task Force, August 28, 2001.
41. W3C 1.1标准．URL:http://www.w3.org/TR/SOAP
42. Erich Gamma, Richard Helm, Ralph Jhonson, Jhon Vlisside: Design Patton: Elements of Reusable Object-Orient Software
43. Acquie Barker, Java面向对象编程指南[M]．电子工业出版社，2001．11
44. Tarak Modi, Clean up you wire protocol with soap, UR: http://www.javaworld.com/javaworld/jw-03-2001/jw-0330-soap.html
45. Joshy Joseph, Handling attachment in soap URL: http://www-9OO.ibm.com/developerWorks/webservices/index.shtml
46. Gavin Bong, Apache SOAP type mapping, URL: http://www-900.ibm.com/developerWorks/webservices/index.shtml