面向大数据量的Web服务传输方案研究
|
摘要
Web服务是一种常用的面向服务架构实现方式,但在基于面向服务架构的网络系统中,存在大数据量传输效率不高的问题。如何提高从数据库到客户端的传输效率,即在使用中快速响应客户端的大数据量请求,从而提高用户使用体验,是Web服务传输方案研究的核心问题。
本文首先概述了Web服务传输方案的研究现状,包括目前国内外比较典型的传输方案优化策略,如:压缩思想、减少多次调用、XML解析器优化、XML标签简化、缓存机制等。进而,基于Web服务传输原理,分析了影响数据传输效率的相关因素。根据数据类型的不同而造成冗余的特点,提出了根据数据类型的不同进行选择性压缩的Web服务传输方案。实验结果表明,选择性压缩能有效提高Web服务传输效率。但这种方案对于压缩率已经较高的数据而言,其效果并不是很明显。针对该方案中存在的问题,本文提出了一种融入多线程思想的Web服务传输方案。该方案结合数据取模分类和多线程传输原理,从提高用户使用体验的角度出发,对于大数据量的传输,先根据所请求的数据特点进行取模分类,再通过多线程分别获取数据,分批向用户逐步展示数据。实验结果表明,该方案能更快响应客户端的大量数据请求,并降低传输总时间。
另一方面,随着基于海量数据应用的增加,传统集中式数据库已经越来越多地被分布式数据库取代。针对分布式架构,提出取模分类和Web服务相结合的查询方案,即分配多线程到各自对应的分布式数据库中获取相应数据。实验结果表明,该方案能有效提高传输效率,并增加系统架构的可扩展性。
Web Service is a common service-oriented architecture, but SOA system has a bottleneck of low efficiency in transmitting large amounts of data. How to improve transmission efficiency from database to client, should be considered seriously on system architecture design phrase. Improve transmission efficiency can enhance user experience, and respond client requests quickly when get large amounts of data.
This dissertation summarize the state of Web Service transmission solution research, include typical and optimization strategy both at home and abroad. For example: compress data, reduce multiple calls, optimize XML parser, simplify XML tags, and cache mechanism. Then, based on Web Service transfer principle, the data transmission efficiency is analyzed. Combining with the characteristics of different data, propose a solution that different compression methods are used for different data to improve the efficiency of data transmission. The experiment results show that the selective compress solution can improve data transmission efficiency in Web Service. But for some higher compressed data, this method is unsatisfactory. In order to solve the problem, this dissertation brings up a solution with multi-threading. The new solution based on modular classification and multi-threading transfer principle, in order to enhance user experience satisfaction, propose for the transmission of large amounts of data, how to classify modulus according to the data characteristics, and retrieve data through multi-threading respectively, show the data in batches to the user step by step. The experimental results show that modular classification and multi-threading transmission on network can respond client requests quickly.
On the other hand, with critical applications for the huge amount of data are more and more, the traditional centralized database has been overwhelmed, so widely used in distributed database has already begun. Based on distributed architecture and SOA, propose a query solution that modular classification combined with Web Service. This solution use multi-threading to obtain their respective data from corresponding distributed database. The experimental results show that it can improve transmission efficiency, and increase the scalability of system architecture.
本文首先概述了Web服务传输方案的研究现状,包括目前国内外比较典型的传输方案优化策略,如:压缩思想、减少多次调用、XML解析器优化、XML标签简化、缓存机制等。进而,基于Web服务传输原理,分析了影响数据传输效率的相关因素。根据数据类型的不同而造成冗余的特点,提出了根据数据类型的不同进行选择性压缩的Web服务传输方案。实验结果表明,选择性压缩能有效提高Web服务传输效率。但这种方案对于压缩率已经较高的数据而言,其效果并不是很明显。针对该方案中存在的问题,本文提出了一种融入多线程思想的Web服务传输方案。该方案结合数据取模分类和多线程传输原理,从提高用户使用体验的角度出发,对于大数据量的传输,先根据所请求的数据特点进行取模分类,再通过多线程分别获取数据,分批向用户逐步展示数据。实验结果表明,该方案能更快响应客户端的大量数据请求,并降低传输总时间。
另一方面,随着基于海量数据应用的增加,传统集中式数据库已经越来越多地被分布式数据库取代。针对分布式架构,提出取模分类和Web服务相结合的查询方案,即分配多线程到各自对应的分布式数据库中获取相应数据。实验结果表明,该方案能有效提高传输效率,并增加系统架构的可扩展性。
Web Service is a common service-oriented architecture, but SOA system has a bottleneck of low efficiency in transmitting large amounts of data. How to improve transmission efficiency from database to client, should be considered seriously on system architecture design phrase. Improve transmission efficiency can enhance user experience, and respond client requests quickly when get large amounts of data.
This dissertation summarize the state of Web Service transmission solution research, include typical and optimization strategy both at home and abroad. For example: compress data, reduce multiple calls, optimize XML parser, simplify XML tags, and cache mechanism. Then, based on Web Service transfer principle, the data transmission efficiency is analyzed. Combining with the characteristics of different data, propose a solution that different compression methods are used for different data to improve the efficiency of data transmission. The experiment results show that the selective compress solution can improve data transmission efficiency in Web Service. But for some higher compressed data, this method is unsatisfactory. In order to solve the problem, this dissertation brings up a solution with multi-threading. The new solution based on modular classification and multi-threading transfer principle, in order to enhance user experience satisfaction, propose for the transmission of large amounts of data, how to classify modulus according to the data characteristics, and retrieve data through multi-threading respectively, show the data in batches to the user step by step. The experimental results show that modular classification and multi-threading transmission on network can respond client requests quickly.
On the other hand, with critical applications for the huge amount of data are more and more, the traditional centralized database has been overwhelmed, so widely used in distributed database has already begun. Based on distributed architecture and SOA, propose a query solution that modular classification combined with Web Service. This solution use multi-threading to obtain their respective data from corresponding distributed database. The experimental results show that it can improve transmission efficiency, and increase the scalability of system architecture.
引文
[1]岳昆,王晓玲,周傲英. Web服务核心支撑技术:研究综述.软件学报, 2004, 15(3):428-442
[2] Wang Y, Nnaji B.O. Document-Driven Design for Distributed CAD Services in Service-Oriented Architecture. Journal of Computing and Information Science in Engineering, 2006, 6(2):127-138
[3]丁兆青,董传良.基于SOA的分布式应用集成研究.计算机工程, 2007, 33(10):246-248
[4]付燕宁,刘磊,金成植.基于服务链的Web服务组合方法.通信学报, 2007, 28(7):92-97
[5]张艳,周明天,佘堃.基于Web services的XML引擎安全模型研究.计算机应用研究, 2008, 25(7):2161-2163, 2170
[6]张健钦,龚建华,林珲等.基于Web Service构建分布式虚拟地理环境系统.高技术通讯, 2006, 16(3):296-300
[7]陈颖慧,邱雪松,高志鹏等.基于模型的Web Service性能测试方法.北京邮电大学学报, 2009, 32(z1), 44-48
[8]胡春明,怀进鹏,孙海龙.基于Web服务的网格体系结构及其支撑环境研究.软件学报, 2004, 15(7):1064-1073
[9]方其庆,刘庆华,彭晓明. QoS全局最优的多目标Web服务选择算法.计算机应用研究, 2009, 26(12):4442-4445, 4448
[10]张宇,宋顺林.基于XML的Web数据自动挖掘.计算机工程与设计, 2008, 29(22):5770-5773, 5813
[11]罗震霄,和菊珍,王晓玲等. XML压缩文档上的数值更新方法. 2007, 34(4):106-107, 144
[12]冯进,丁博,史殿习等. XML解析技术研究.计算机工程与科学, 2009, 31(2):120-124
[13]张锦,程京,陆绍飞等.传感器网络中基于密度控制的路由算法.华中科技大学学报, 2005, 33(z1):79-81
[14] Yasser Shohoud. Real world XML Web Services. Pearson Education, 2003, 247-248
[15]王云.提高Web Service数据传输效率的基本方法. http://www.ibm.com/developerworks/cn/webservices/0710_wangyun/index.html,2007
[16] Steve Graham, Simeon Simeonov, et al. Building Web Services with Java: Making Sense of XML, SOAP, WSDL, and UDDI. Sam Publishing, 2002, 66-67
[17] C.Lau, A. Ryman. Developing XML Web Services with websphere studio application developer. IBM Systems Journal, 2002, 41(2):77-78
[18] Rajesh Chawla, Vivek Chopin, Gary Damschen. Professional XML Web Services. Wrox Press, 2002, 205-206
[19] Scott Seely. SOAP Cross Platform Web Service Development Using XML. Pearson Education, 2001, 179
[20] Eirinaki M, Vazirgiannis M. Web Mining for Web Personalization. ACM Transactions on Internet Technology, 2003, 3(1):1-27
[21] ROBERT ABOOLIAN, YI SUN, GRAY J.KOEHLER. A location-allocation problem for a Web Services provider in a competitive marker. European Journal of Operational Research, 2009, 194, 64–77
[22] BUHWAN JEONG, DAEWON LEE, JAEWOOK LEE, HYUNBO CHO. Support for seamless data exchanges between Web Services through information mapping analysis using kernel methods. Expert Systems with Applications, 2009, 36(1):358-365
[23]李果,张锦,周鑫等.基于压缩思想的Web服务数据传输优化研究.计算机应用, 2009, 29(S2):34-36, 40
[24]郜震霄,肖田元,范文慧等.基于Web服务的协同优化平台.系统仿真学报, 2008, 20(19):5286-5290
[25]王安华,胡国林,班晓娟等.基于服务质量的Web服务发现研究与实现.计算机工程与设计, 2007, 28(21):5112-5114
[26]马巧梅,朱林泉.基于LZ77算法的文本压缩软件的实现.电脑开发与应用, 2008, 21(5):41-42
[27]王防修,周康.通过哈夫曼编码实现文件的压缩与解压.武汉工业学院学报, 2008, 27(4):46-49
[28]毛国勇,张晓斌,张武.基于序列化及压缩的网格数据传输性能改进.计算机工程, 2008, 34(16):89-91
[29]宋刚,蒋孟奇,张云泉等.基于共享存储和Gzip的并行压缩算法研究.计算机工程与设计, 2009, 30(4):781-784
[30]张胜,包晓玲,舒坚等. XML压缩方法的比较分析.计算机工程, 2009, 35(11):26-28, 31
[31]郭荷清,王增勋. XML数据绑定及对象序列化的应用研究.计算机应用与软件, 2006, 23(5):67-68
[32]汤自安.基于Web Services的工作流在SOA中的应用研究.电脑开发与应用, 2009, 22(8):57-60
[33] Zeng LZ, Benatallah B, Ngu AHH, et al. QoS-Aware middleware for Web Services composition. IEEE Trans. on Software Engineering, 2004, 30(5): 311-327
[34] Oh SC, Lee D, Kumara SRT. Web Service Planner (WsPr): An effective and scalable Web Service Web composition algorithm. Int’l Journal of Web Services Research, 2007, 4(1):1-23
[35] Zhang CW, Su S, Chen JL. Genetic algorithm on Web Services selection supporting QoS. Chinese Journal of Computers, 2006, 29(7):1029-1037
[36]邵凌霜,周立,赵俊峰等.一种Web Service的服务质量预测方法.软件学报, 2009, 20(8):2062-2073
[37]魏新红,张凯,吕凤琳.基于Web服务的质量分析与优化系统研究.微计算机信息, 2008, 24(13):279-280
[38]张勇,张轶博,刘军等.基于取模运算和哈希函数映射均衡性研究.重庆邮电学院学报, 2004, 16(2):108-111
[39]赵月爱,彭新光.异或和取模运算的负载均衡算法.计算机工程与设计, 2007, 28(6):1290-1292
[40]李刚,高峰,林凌.快速取模算法在BLACKF IN DSP上的实现.微计算机应用, 2008, 29(6):23-27
[41]郗英,王小尚.多线程和网络传输的数据采集系统在ERP中的应用.现代制造工程, 2009, 10(2):26-28
[42]林建素,钟勇,丁洁.基于动态反馈多线程的大数据驱动缓存的设计与实现.计算机应用, 2008, 28(4):877-880
[43]田小雄,蔡鸿明,姜丽红.基于Web服务的分布式实体建模框架.计算机工程, 2009, 35(4):63-65
[44]周傲英,周敏奇,钱卫宁等.大规模分布式系统中的多属性查询处理.计算机学报, 2008, 31(09):1563-1572
[45]乔晓强,魏峻,黄涛.基于分布式协调模型的服务协作方法研究.软件学报, 2009, 20(06):1470-1486
[46]徐署华,胡君.分布式数据流查询方案及优化.计算机应用, 2009, 29(07):1771-1774
[47]朱青,王珊,丁博麟等.基于数据网格面向服务的查询算法.计算机学报, 2006, 29(07):1234-1240
[48]张峰,邵秀丽,李耀芳.基于Web Service的异构集成查询方案.计算机工程, 2009, 35(12):50-53
[49]朱泓丞,徐志广.分布式数据库数据副本最优安置问题的研究.计算机应用, 2009, 29(09):2509-2511
[50]邬建锋,彭宇行.分布式环境下数据共享中的多表查询转换算法.计算机工程, 2009, 35(20):66-67, 70
[51]邱宁.利用Web Service保证分布式数据库数据一致的方法研究.计算机应用与软件, 2007, 24(12):101-103
[52]曹羽中.用Hadoop进行分布式并行编程. http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop1/index.html, 2008
[53]岑文初.分布式计算开源框架Hadoop介绍. http://www.infoq.com/cn/articles/hadoop-intro, 2008
[54] Tan W, Fan Y.S, Zhou M.C. A Petri net-based method for compatibility analysis and composition of Web Services in business process execution language . IEEE Transactions on Automation Science and Engineering, 2009, 6(01): 94-106
[55] Al-Omari R, Somani A K, Manimaran G. An adaptive scheme for fault-tolerant scheduling of soft real-time tasks in multiprocessor systems. Journal of Parallel and Distributed Computing, 2005, 65(05):595-608
[56] Robie J. XML processing and data integration with XQuery. Internet Computing, 2007, 11(04):62-67
[57]兰雨晴,赵同,高静等.基础软件平台质量评估.软件学报, 2009, 20(03):567-582
[2] Wang Y, Nnaji B.O. Document-Driven Design for Distributed CAD Services in Service-Oriented Architecture. Journal of Computing and Information Science in Engineering, 2006, 6(2):127-138
[3]丁兆青,董传良.基于SOA的分布式应用集成研究.计算机工程, 2007, 33(10):246-248
[4]付燕宁,刘磊,金成植.基于服务链的Web服务组合方法.通信学报, 2007, 28(7):92-97
[5]张艳,周明天,佘堃.基于Web services的XML引擎安全模型研究.计算机应用研究, 2008, 25(7):2161-2163, 2170
[6]张健钦,龚建华,林珲等.基于Web Service构建分布式虚拟地理环境系统.高技术通讯, 2006, 16(3):296-300
[7]陈颖慧,邱雪松,高志鹏等.基于模型的Web Service性能测试方法.北京邮电大学学报, 2009, 32(z1), 44-48
[8]胡春明,怀进鹏,孙海龙.基于Web服务的网格体系结构及其支撑环境研究.软件学报, 2004, 15(7):1064-1073
[9]方其庆,刘庆华,彭晓明. QoS全局最优的多目标Web服务选择算法.计算机应用研究, 2009, 26(12):4442-4445, 4448
[10]张宇,宋顺林.基于XML的Web数据自动挖掘.计算机工程与设计, 2008, 29(22):5770-5773, 5813
[11]罗震霄,和菊珍,王晓玲等. XML压缩文档上的数值更新方法. 2007, 34(4):106-107, 144
[12]冯进,丁博,史殿习等. XML解析技术研究.计算机工程与科学, 2009, 31(2):120-124
[13]张锦,程京,陆绍飞等.传感器网络中基于密度控制的路由算法.华中科技大学学报, 2005, 33(z1):79-81
[14] Yasser Shohoud. Real world XML Web Services. Pearson Education, 2003, 247-248
[15]王云.提高Web Service数据传输效率的基本方法. http://www.ibm.com/developerworks/cn/webservices/0710_wangyun/index.html,2007
[16] Steve Graham, Simeon Simeonov, et al. Building Web Services with Java: Making Sense of XML, SOAP, WSDL, and UDDI. Sam Publishing, 2002, 66-67
[17] C.Lau, A. Ryman. Developing XML Web Services with websphere studio application developer. IBM Systems Journal, 2002, 41(2):77-78
[18] Rajesh Chawla, Vivek Chopin, Gary Damschen. Professional XML Web Services. Wrox Press, 2002, 205-206
[19] Scott Seely. SOAP Cross Platform Web Service Development Using XML. Pearson Education, 2001, 179
[20] Eirinaki M, Vazirgiannis M. Web Mining for Web Personalization. ACM Transactions on Internet Technology, 2003, 3(1):1-27
[21] ROBERT ABOOLIAN, YI SUN, GRAY J.KOEHLER. A location-allocation problem for a Web Services provider in a competitive marker. European Journal of Operational Research, 2009, 194, 64–77
[22] BUHWAN JEONG, DAEWON LEE, JAEWOOK LEE, HYUNBO CHO. Support for seamless data exchanges between Web Services through information mapping analysis using kernel methods. Expert Systems with Applications, 2009, 36(1):358-365
[23]李果,张锦,周鑫等.基于压缩思想的Web服务数据传输优化研究.计算机应用, 2009, 29(S2):34-36, 40
[24]郜震霄,肖田元,范文慧等.基于Web服务的协同优化平台.系统仿真学报, 2008, 20(19):5286-5290
[25]王安华,胡国林,班晓娟等.基于服务质量的Web服务发现研究与实现.计算机工程与设计, 2007, 28(21):5112-5114
[26]马巧梅,朱林泉.基于LZ77算法的文本压缩软件的实现.电脑开发与应用, 2008, 21(5):41-42
[27]王防修,周康.通过哈夫曼编码实现文件的压缩与解压.武汉工业学院学报, 2008, 27(4):46-49
[28]毛国勇,张晓斌,张武.基于序列化及压缩的网格数据传输性能改进.计算机工程, 2008, 34(16):89-91
[29]宋刚,蒋孟奇,张云泉等.基于共享存储和Gzip的并行压缩算法研究.计算机工程与设计, 2009, 30(4):781-784
[30]张胜,包晓玲,舒坚等. XML压缩方法的比较分析.计算机工程, 2009, 35(11):26-28, 31
[31]郭荷清,王增勋. XML数据绑定及对象序列化的应用研究.计算机应用与软件, 2006, 23(5):67-68
[32]汤自安.基于Web Services的工作流在SOA中的应用研究.电脑开发与应用, 2009, 22(8):57-60
[33] Zeng LZ, Benatallah B, Ngu AHH, et al. QoS-Aware middleware for Web Services composition. IEEE Trans. on Software Engineering, 2004, 30(5): 311-327
[34] Oh SC, Lee D, Kumara SRT. Web Service Planner (WsPr): An effective and scalable Web Service Web composition algorithm. Int’l Journal of Web Services Research, 2007, 4(1):1-23
[35] Zhang CW, Su S, Chen JL. Genetic algorithm on Web Services selection supporting QoS. Chinese Journal of Computers, 2006, 29(7):1029-1037
[36]邵凌霜,周立,赵俊峰等.一种Web Service的服务质量预测方法.软件学报, 2009, 20(8):2062-2073
[37]魏新红,张凯,吕凤琳.基于Web服务的质量分析与优化系统研究.微计算机信息, 2008, 24(13):279-280
[38]张勇,张轶博,刘军等.基于取模运算和哈希函数映射均衡性研究.重庆邮电学院学报, 2004, 16(2):108-111
[39]赵月爱,彭新光.异或和取模运算的负载均衡算法.计算机工程与设计, 2007, 28(6):1290-1292
[40]李刚,高峰,林凌.快速取模算法在BLACKF IN DSP上的实现.微计算机应用, 2008, 29(6):23-27
[41]郗英,王小尚.多线程和网络传输的数据采集系统在ERP中的应用.现代制造工程, 2009, 10(2):26-28
[42]林建素,钟勇,丁洁.基于动态反馈多线程的大数据驱动缓存的设计与实现.计算机应用, 2008, 28(4):877-880
[43]田小雄,蔡鸿明,姜丽红.基于Web服务的分布式实体建模框架.计算机工程, 2009, 35(4):63-65
[44]周傲英,周敏奇,钱卫宁等.大规模分布式系统中的多属性查询处理.计算机学报, 2008, 31(09):1563-1572
[45]乔晓强,魏峻,黄涛.基于分布式协调模型的服务协作方法研究.软件学报, 2009, 20(06):1470-1486
[46]徐署华,胡君.分布式数据流查询方案及优化.计算机应用, 2009, 29(07):1771-1774
[47]朱青,王珊,丁博麟等.基于数据网格面向服务的查询算法.计算机学报, 2006, 29(07):1234-1240
[48]张峰,邵秀丽,李耀芳.基于Web Service的异构集成查询方案.计算机工程, 2009, 35(12):50-53
[49]朱泓丞,徐志广.分布式数据库数据副本最优安置问题的研究.计算机应用, 2009, 29(09):2509-2511
[50]邬建锋,彭宇行.分布式环境下数据共享中的多表查询转换算法.计算机工程, 2009, 35(20):66-67, 70
[51]邱宁.利用Web Service保证分布式数据库数据一致的方法研究.计算机应用与软件, 2007, 24(12):101-103
[52]曹羽中.用Hadoop进行分布式并行编程. http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop1/index.html, 2008
[53]岑文初.分布式计算开源框架Hadoop介绍. http://www.infoq.com/cn/articles/hadoop-intro, 2008
[54] Tan W, Fan Y.S, Zhou M.C. A Petri net-based method for compatibility analysis and composition of Web Services in business process execution language . IEEE Transactions on Automation Science and Engineering, 2009, 6(01): 94-106
[55] Al-Omari R, Somani A K, Manimaran G. An adaptive scheme for fault-tolerant scheduling of soft real-time tasks in multiprocessor systems. Journal of Parallel and Distributed Computing, 2005, 65(05):595-608
[56] Robie J. XML processing and data integration with XQuery. Internet Computing, 2007, 11(04):62-67
[57]兰雨晴,赵同,高静等.基础软件平台质量评估.软件学报, 2009, 20(03):567-582