分布式业务性能采集与海量数据分析系统的研究与实现
|
摘要
近年来,随着互联网技术的蓬勃发展,网络上的应用越来越多,网络拓扑以及网络环境日益复杂。网络上的负载逐渐加重,导致网络性能下降等网络安全和服务质量问题。这些问题引起了越来越多的关注。为解决这些问题,为新的网络业务进行规划,需要对网络的性能指标进行分析,对网络服务质量进行监控和评估,从而评估网络的总体性能。
如何对海量的测量数据进行统计分析,从中得出网络总体性能情况,是网络测量的研究重点。在现有技术中,网络测量的数据分析操作大多采用集中管理的平台。由独立的一个管理平台同时对多个测量探针的海量测量数据进行采集与分析,无法保证数据的完整性和可靠性。另一方面,在面对海量的测量数据时,因其计算能力有限,不具备对海量数据进行分析的能力。随着云计算技术的推广,利用云计算平台强大的计算能力来处理海量数据,已经成为当前网络测量的研究与发展趋势。
分布式业务性能采集与海量数据分析系统利用Hadoop平台的MapReduce计算模型,解决了海量数据的快速计算的问题,并运用性能评估算法,从海量测量数据中分析出网络的整体性能情况。在Hadoop框架的任务分配和失效容错原理的基础上,为系统添加了任务分配和节点失效容错机制。
本文首先介绍云计算和网络测量相关研究情况,简要介绍了网络性能指标和网络性能评价方法；其次,根据实际网络测量中遇到的问题,对系统做了需求分析,根据一定的设计原则进行功能划分和性能要求；再次,在需求分析的基础上,对系统从总体架构到各模块进行了由总体到部分的设计与实现；最后,搭建测试环境,对本文中实现的系统各功能进行了测试,并分析测试结果,证明本系统能够满足分布式环境下网络测量的需求。
Recent years, as the flourishing growth of Internet technology, more and more Internet applications came into being. The complication of network topology and network environment and the aggravation of network workload lead to problems of network security and the quality of service, such as the decline of network performance. To solve these problems and plan for new network traffic, it is necessary to analyse network performance indicators and monitor and assess the quality of service of the network, so as to evaluate the overall performance of the network.
Research of network measurement emphasizes on how to do statistical analysis on mass measurement data, from which the overall network performance can be derived. In current techniques, most of the operations of data analysis of network measurement use a centralized management platform. The integrity and reliability is not guaranteed by an isolated management platform collecting and analysing the mass measurement data from multiple measurement probes. On the other hand, when facing mass measurement data, the centralized platform is not capable for analysing mass data due to its limited computing ability. Nowadays, as the spread of cloud computing technology, using the powerful computing ability of cloud computing platform to process mass data has already become the trend of research and development in network measurement.
The Distributed Traffic Performance Collection and Mass Data Analysis System resolves the problem of rapid calculation of mass data by the MapReduce computation model of Hadoop platform, and utilizes performance evaluation algorithm to analyse the overall performance of network. Mechanisms of task assignment and fault-tolerance for node failure are added based on the principle of task assignment and fault-tolerance of Hadoop framework.
In this paper, first the statuses of relevant research in the field of cloud computing and network measurement are introduced, together with network performance metrix and network performance evaluation methods in a nutshell; second, requirements analysis is done according to problems encounterred in real network measurement, while functions are divided and performance demand is completed based on certain design principles; third, design and implementation are done in the light of requirements analysis from general framework deep into detailed modules; last but not least, testing environment is set up to test each funtion realized in this paper, and to prove the capability of this network measurement system in a distributed network environment by analyzing the results of tests.
如何对海量的测量数据进行统计分析,从中得出网络总体性能情况,是网络测量的研究重点。在现有技术中,网络测量的数据分析操作大多采用集中管理的平台。由独立的一个管理平台同时对多个测量探针的海量测量数据进行采集与分析,无法保证数据的完整性和可靠性。另一方面,在面对海量的测量数据时,因其计算能力有限,不具备对海量数据进行分析的能力。随着云计算技术的推广,利用云计算平台强大的计算能力来处理海量数据,已经成为当前网络测量的研究与发展趋势。
分布式业务性能采集与海量数据分析系统利用Hadoop平台的MapReduce计算模型,解决了海量数据的快速计算的问题,并运用性能评估算法,从海量测量数据中分析出网络的整体性能情况。在Hadoop框架的任务分配和失效容错原理的基础上,为系统添加了任务分配和节点失效容错机制。
本文首先介绍云计算和网络测量相关研究情况,简要介绍了网络性能指标和网络性能评价方法；其次,根据实际网络测量中遇到的问题,对系统做了需求分析,根据一定的设计原则进行功能划分和性能要求；再次,在需求分析的基础上,对系统从总体架构到各模块进行了由总体到部分的设计与实现；最后,搭建测试环境,对本文中实现的系统各功能进行了测试,并分析测试结果,证明本系统能够满足分布式环境下网络测量的需求。
Recent years, as the flourishing growth of Internet technology, more and more Internet applications came into being. The complication of network topology and network environment and the aggravation of network workload lead to problems of network security and the quality of service, such as the decline of network performance. To solve these problems and plan for new network traffic, it is necessary to analyse network performance indicators and monitor and assess the quality of service of the network, so as to evaluate the overall performance of the network.
Research of network measurement emphasizes on how to do statistical analysis on mass measurement data, from which the overall network performance can be derived. In current techniques, most of the operations of data analysis of network measurement use a centralized management platform. The integrity and reliability is not guaranteed by an isolated management platform collecting and analysing the mass measurement data from multiple measurement probes. On the other hand, when facing mass measurement data, the centralized platform is not capable for analysing mass data due to its limited computing ability. Nowadays, as the spread of cloud computing technology, using the powerful computing ability of cloud computing platform to process mass data has already become the trend of research and development in network measurement.
The Distributed Traffic Performance Collection and Mass Data Analysis System resolves the problem of rapid calculation of mass data by the MapReduce computation model of Hadoop platform, and utilizes performance evaluation algorithm to analyse the overall performance of network. Mechanisms of task assignment and fault-tolerance for node failure are added based on the principle of task assignment and fault-tolerance of Hadoop framework.
In this paper, first the statuses of relevant research in the field of cloud computing and network measurement are introduced, together with network performance metrix and network performance evaluation methods in a nutshell; second, requirements analysis is done according to problems encounterred in real network measurement, while functions are divided and performance demand is completed based on certain design principles; third, design and implementation are done in the light of requirements analysis from general framework deep into detailed modules; last but not least, testing environment is set up to test each funtion realized in this paper, and to prove the capability of this network measurement system in a distributed network environment by analyzing the results of tests.
引文
[1]孙小宁；康建初.基于Web Service的大规模网络测量平台的研究与实现[J].计算机工程与应用.2003,(12)
[2]Buyyaa R.; Yeo C. S.; Venugopal S.等. Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility[C]. Future Generation Computer Sys-teems, Australia: The University of Melbourne. 2009.
[3]胡光民；周亮；柯立新.基于Hadoop的网络日志分析系统研究[J].电脑知识与技术.2010,(22)
[4]张帆；李磊；杨成胡等.基于Eucalyptus构建私有云计算平台[J].电信科学.2011,(11)
[5]陈全.云计算及其关键技术[J].计算机应用,2009,(9)
[6]维基百科http://zh.wikipedia.org/wiki/
[7]董世晓.云计算开源先锋Hadoop——第四届Hadoop中国云计算大会纪实[J].程序员,2010,(10)
[8]田秀霞.基于Hadoop架构的分布与式计算和存储技术及其应用[.上海电力学院学报.2011,(27)
[9]霍树民.基于Hadoop的海量影像数据管理关键技术研究[学位论文].国防科学技术大学.2010
[10]杨秋杰.云计算的核心技术——粒度计算[J].时代经贸.2010,(18)
[11]杨家海；吴建平；安常青编著.互联网络测量理论与应用[M].人民邮电出版社.2009
[12]Paxson V.; Almes G; Mahdavi J等RFC2330. Framework for IP Performance Metrics[S].1988
[13]聂玉婷；高仲合.单向时延测量中的时间同步问题[J].通信技术.2009,(10)
[14]任波；雒江涛；滕欢IPTV测试仪媒体流传送指标(MDI)监测子系统的FPGA实现[.内江师范学院学报.2011,(2)
[15]张冬艳；胡铭曾；张宏莉.基于测量的网络性能评价方法研究[J].通信学报.2006,(10)
[16]何娜.基于云计算的分布民网络测量系统的研究与实现[学位论文].北京邮电大学.2012
[17]伊国力.基于节点性能的应用层组播模型[学位论文].南京邮电大学.2009
[18]White Tom著周傲英,曾大聃译Hadoop权威指南(中文版)[M].清华大学出版社.2010
[2]Buyyaa R.; Yeo C. S.; Venugopal S.等. Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility[C]. Future Generation Computer Sys-teems, Australia: The University of Melbourne. 2009.
[3]胡光民；周亮；柯立新.基于Hadoop的网络日志分析系统研究[J].电脑知识与技术.2010,(22)
[4]张帆；李磊；杨成胡等.基于Eucalyptus构建私有云计算平台[J].电信科学.2011,(11)
[5]陈全.云计算及其关键技术[J].计算机应用,2009,(9)
[6]维基百科http://zh.wikipedia.org/wiki/
[7]董世晓.云计算开源先锋Hadoop——第四届Hadoop中国云计算大会纪实[J].程序员,2010,(10)
[8]田秀霞.基于Hadoop架构的分布与式计算和存储技术及其应用[.上海电力学院学报.2011,(27)
[9]霍树民.基于Hadoop的海量影像数据管理关键技术研究[学位论文].国防科学技术大学.2010
[10]杨秋杰.云计算的核心技术——粒度计算[J].时代经贸.2010,(18)
[11]杨家海；吴建平；安常青编著.互联网络测量理论与应用[M].人民邮电出版社.2009
[12]Paxson V.; Almes G; Mahdavi J等RFC2330. Framework for IP Performance Metrics[S].1988
[13]聂玉婷；高仲合.单向时延测量中的时间同步问题[J].通信技术.2009,(10)
[14]任波；雒江涛；滕欢IPTV测试仪媒体流传送指标(MDI)监测子系统的FPGA实现[.内江师范学院学报.2011,(2)
[15]张冬艳；胡铭曾；张宏莉.基于测量的网络性能评价方法研究[J].通信学报.2006,(10)
[16]何娜.基于云计算的分布民网络测量系统的研究与实现[学位论文].北京邮电大学.2012
[17]伊国力.基于节点性能的应用层组播模型[学位论文].南京邮电大学.2009
[18]White Tom著周傲英,曾大聃译Hadoop权威指南(中文版)[M].清华大学出版社.2010