集群技术与网格技术在卫星数据动态可视化系统中的研究与应用
|
摘要
在国家气象卫星中心和华东师范大学的通力合作下，风云三号海量卫星数据动态可视化系统预研项目的一期工程已经圆满完成，作为项目一期主要研究成果的原型系统实现了海量卫星数据动态可视化系统的主要功能。进入二期阶段后，项目的主要任务是提高系统的数据处理能力和网格吞吐量。为此，本文分析和比较了各种潜在解决方案，最终选择了“服务器集群技术(方案)”和“网格技术(方案)”作为提高系统性能的技术手段。
本文对“服务器集群技术”和“网格技术”做了初步研究，针对卫星系统的应用特点，解决了应用过程中遇到的关键技术和难点，并在研究成果的基础之上对上述两种技术方案做了具体实现。其中，集群方案使用了Apache HTTP Server和Apache Tomcat搭建了成熟的开源解决方案。在对缓存模块的重构中，利用了Sticky Session技术避免了分布式缓存的使用。在网格方案中，考虑到网格技术的发展现状及卫星应用对性能的高要求，本文并未使用业界现有的网格解决方案，而是针对卫星系统的自身特点开发了专用的网格构架，并对负载监控、作业调度、数据管理、安全认证等方面进行了较为深入的研究和应用。最后，本文利用了有限的测试资源，使用了无线网络等新技术，模拟了集群和网格的测试环境，通过分析和比较测试数据，验证了“服务器集群方案”和“网格方案”的可行性和有效性。
尽管本文所实现的集群和网格方案针对的是“卫星数据可视化系统”，但对其他基于J2EE构架的高性能Web应用程序也具有一定的借鉴意义。
With the collaboration between National Satellite Meteorological Center and East China Normal University, the phase 1 of FY3 Satellite Data Visualization System Project has succefully completed. As the major achievement of phase 1, a demo system has been developed to implement most of the functions of FY3 Satellite Data Visualization System. In phase 2, the main objective of the project has turned to the improvement of the data processing capacity and network throughput. In order to do that, this essay analyses and compares a variety of potential solutions. As a result, cluster technology (solution) and grid technology (solution) have been choosen to achieve this goal.
This essay preliminarily studys and implements cluster technology (solution) and grid technology (solution). During the implementation, key technologies and difficulties have been solved based on the features of satellite system. Specifically, cluster solution is a combination of Apache HTTP Server and Apache Tomcat, which is the most widely-used open souce cluster solution. During the reflaction of cache module, Sticky Session has been picked to avoid distributed cache. In the grid solution, due to the development situation of grid technology and high demand for high performance, this essay has not chosen any of the existing grid solutions. Instead, an autonomous architecture has been built, which particularly targets at Satellite Data Visualization System. Within this architecture, a lot of aspects have been taken into consideration, such as load monitoring, job dispatching, data management, security authentication, etc. Finally, this essay utilizes available resources and new techniques as wireless network to simulate testing environment and performe performance tests for the system. Test results prove the feasibility and effectiveness of cluster solution and grid solution.
Although the solutions in this essay target at Satellite Data Visualization System, it can be refered by other J2EE based web applications that require high performance.
本文对“服务器集群技术”和“网格技术”做了初步研究，针对卫星系统的应用特点，解决了应用过程中遇到的关键技术和难点，并在研究成果的基础之上对上述两种技术方案做了具体实现。其中，集群方案使用了Apache HTTP Server和Apache Tomcat搭建了成熟的开源解决方案。在对缓存模块的重构中，利用了Sticky Session技术避免了分布式缓存的使用。在网格方案中，考虑到网格技术的发展现状及卫星应用对性能的高要求，本文并未使用业界现有的网格解决方案，而是针对卫星系统的自身特点开发了专用的网格构架，并对负载监控、作业调度、数据管理、安全认证等方面进行了较为深入的研究和应用。最后，本文利用了有限的测试资源，使用了无线网络等新技术，模拟了集群和网格的测试环境，通过分析和比较测试数据，验证了“服务器集群方案”和“网格方案”的可行性和有效性。
尽管本文所实现的集群和网格方案针对的是“卫星数据可视化系统”，但对其他基于J2EE构架的高性能Web应用程序也具有一定的借鉴意义。
With the collaboration between National Satellite Meteorological Center and East China Normal University, the phase 1 of FY3 Satellite Data Visualization System Project has succefully completed. As the major achievement of phase 1, a demo system has been developed to implement most of the functions of FY3 Satellite Data Visualization System. In phase 2, the main objective of the project has turned to the improvement of the data processing capacity and network throughput. In order to do that, this essay analyses and compares a variety of potential solutions. As a result, cluster technology (solution) and grid technology (solution) have been choosen to achieve this goal.
This essay preliminarily studys and implements cluster technology (solution) and grid technology (solution). During the implementation, key technologies and difficulties have been solved based on the features of satellite system. Specifically, cluster solution is a combination of Apache HTTP Server and Apache Tomcat, which is the most widely-used open souce cluster solution. During the reflaction of cache module, Sticky Session has been picked to avoid distributed cache. In the grid solution, due to the development situation of grid technology and high demand for high performance, this essay has not chosen any of the existing grid solutions. Instead, an autonomous architecture has been built, which particularly targets at Satellite Data Visualization System. Within this architecture, a lot of aspects have been taken into consideration, such as load monitoring, job dispatching, data management, security authentication, etc. Finally, this essay utilizes available resources and new techniques as wireless network to simulate testing environment and performe performance tests for the system. Test results prove the feasibility and effectiveness of cluster solution and grid solution.
Although the solutions in this essay target at Satellite Data Visualization System, it can be refered by other J2EE based web applications that require high performance.
引文
[1] http://www.cnsa.gov.cn/ (国家航天局)
[2] 风云三号气象卫星应用系统一期工程开发任务书V1.0，国家卫星气象中心，2007
[3] 风云三号海量卫星数据产品快速显示技术和原型软件系统研究项目技术报告，华东师范大学，2007
[4] http://www.globus.org/
[5] 林凡，集群的可扩展性及其分布式体系结构，IBM DeveloperWorks，2001
[6] http://windows.chinaitlab.com/
[7] 杨晔，集群服务器的性能分析与研究，西安电子科技大学硕士学位论文，2007
[8] 马双良，集群测控系统设计与关键性技术研究，西北工业大学硕士学位论文，2007
[9] 赵中楠，一种基于广域网分布式集群系统进程迁移的研究与实现，哈尔滨理工大学硕士学位论文，2006
[10] 杨振宇，Web集群服务器负载均衡与容错技术的研究与应用，西安电子科技大学硕士学位论文，2007
[11] Rajkumar Buyya, High Performance Cluster Computing: Architectures and Systems, Prentice Hall PTR, 1999
[12] 金戈，Linux高性能计算集群，IBM DeveloperWorks，2002
[13] http://www.ccidnet.com/
[14] 左国华，基于J2EE构架的分布式企业级Web应用的研究，浙江大学硕士学位论文，2005
[15] Hidayatullah Shaikh, High-availability middleware on Linux, Part 1: Heartbeat and Apache Web server, IBM Developerworks, 2004
[16] Jon Purdy, Session Management for Clustered Applications, BEA Dev2Dev, 2005
[17] http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html
[18] Budi Kumiawan, Using JMeter, O'Reilly ONJava, 2003
[19] 唐华，同步镜像服务器!让南北用户都高速冲浪，pconline, 2006
[20] http://www.chinagdd.net/
[21] http://www.ibm.com/developerworks/cn/grid/
[22] 袁文成，服务网格门户关键技术研究与设计，西北工业大学硕士学位论文，2007
[23] Martin C. Brown, Building a grid using Web services standards, IBM Developerworks, 2006
[24] 马骏，网管技术在气象系统中的应用研究，山东大学硕士学位论文，2006
[25] http://www.gridgain.com/
[26] I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastrucure[M], 1999
[27] 孔勋，计算服务网格资源管理技术研究，西北工业大学硕士学位论文，2007
[28] 邹德清，纵览网格服务体系结构的演变，赛迪网，2006
[29] 万维兵，互联网架构基础：谈什么是ChinaNet，IT世界网，2007
[30] 风云三号海量卫星数据产品快速显示技术和原型软件系统研究项目原型系统测试报告，华东师范大学，2007
[31] Wilson Mark, LoadRunner Architecture, http://www.wilsonmar.com/, 2007
[32] Oliver Kaven, Performance Tests: Wireless LANs, PC MAGAZINE, 2006
[33] http://www.51testing.com/
[2] 风云三号气象卫星应用系统一期工程开发任务书V1.0，国家卫星气象中心，2007
[3] 风云三号海量卫星数据产品快速显示技术和原型软件系统研究项目技术报告，华东师范大学，2007
[4] http://www.globus.org/
[5] 林凡，集群的可扩展性及其分布式体系结构，IBM DeveloperWorks，2001
[6] http://windows.chinaitlab.com/
[7] 杨晔，集群服务器的性能分析与研究，西安电子科技大学硕士学位论文，2007
[8] 马双良，集群测控系统设计与关键性技术研究，西北工业大学硕士学位论文，2007
[9] 赵中楠，一种基于广域网分布式集群系统进程迁移的研究与实现，哈尔滨理工大学硕士学位论文，2006
[10] 杨振宇，Web集群服务器负载均衡与容错技术的研究与应用，西安电子科技大学硕士学位论文，2007
[11] Rajkumar Buyya, High Performance Cluster Computing: Architectures and Systems, Prentice Hall PTR, 1999
[12] 金戈，Linux高性能计算集群，IBM DeveloperWorks，2002
[13] http://www.ccidnet.com/
[14] 左国华，基于J2EE构架的分布式企业级Web应用的研究，浙江大学硕士学位论文，2005
[15] Hidayatullah Shaikh, High-availability middleware on Linux, Part 1: Heartbeat and Apache Web server, IBM Developerworks, 2004
[16] Jon Purdy, Session Management for Clustered Applications, BEA Dev2Dev, 2005
[17] http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html
[18] Budi Kumiawan, Using JMeter, O'Reilly ONJava, 2003
[19] 唐华，同步镜像服务器!让南北用户都高速冲浪，pconline, 2006
[20] http://www.chinagdd.net/
[21] http://www.ibm.com/developerworks/cn/grid/
[22] 袁文成，服务网格门户关键技术研究与设计，西北工业大学硕士学位论文，2007
[23] Martin C. Brown, Building a grid using Web services standards, IBM Developerworks, 2006
[24] 马骏，网管技术在气象系统中的应用研究，山东大学硕士学位论文，2006
[25] http://www.gridgain.com/
[26] I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastrucure[M], 1999
[27] 孔勋，计算服务网格资源管理技术研究，西北工业大学硕士学位论文，2007
[28] 邹德清，纵览网格服务体系结构的演变，赛迪网，2006
[29] 万维兵，互联网架构基础：谈什么是ChinaNet，IT世界网，2007
[30] 风云三号海量卫星数据产品快速显示技术和原型软件系统研究项目原型系统测试报告，华东师范大学，2007
[31] Wilson Mark, LoadRunner Architecture, http://www.wilsonmar.com/, 2007
[32] Oliver Kaven, Performance Tests: Wireless LANs, PC MAGAZINE, 2006
[33] http://www.51testing.com/