Internet网络流量预测
摘要
随着Internent的快速发展,人们的生活方式和传统的信息交换方式也受到巨大的影响。伴随着各种各样的应用需求和网络规模越来越大,网络管理工作也越来越繁重和困难。网络拥塞、网络安全、网络故障等一系列问题时刻影响着信息社会的正常发展。网络流量的测量和预测对大规模网络管理、规划和设计具有非常重要的意义。本论文主要研究内容和特色如下:
(1)通过查阅国内外相关文献资料,对网络流量测量方法进行研究。常用的3种测量技术是:SNMP测量、Packet Sniffing测量和Netflow测量。SNMP是通过提取网络设备(代理)提供的MIB(管理对象信息库)中收集的一些具体设备以及流量信息有关的变量进行测量的。Packet Sniffing是一种用网卡在链路层来获取网络流量的方法,使用时将它串接在需要捕捉流量的链路中,通过分流链路上的数字信号来获取流量信息。Netflow是一种数据交换方式,流量采集是基于网络设备提供的Netflow机制实现的。
(2)介绍常用预测方法。如最小均方误差、神经网络、ARMA模型以及基于统计学习理论的支持向量机等。
(3)通过NS2仿真器,结合支持向量机回归理论对模拟的网络流量进行预测实验。首先分析了NS2软件包的结构、仿真原理和流程,然后利用NS2模拟简单网络拓扑结构并产生流量数据,为了提高预测精度将原始数据进行差分处理,然后组织训练数据和检验数据,最后基于SVM建立预测模型,用训练数据训练模型参数,然后用检验数据进行流量预测,并分析结果,结果表明实验预测效果比较理想。
(4)在仿真实验可行的基础上,采用Netflow技术对真实环境下的网络流量进行测量,并进行预测分析。在某网管中心的核心路由器上进行Netflow配置,并在一台PC机上通过Netflow相关工具进行Netflow包收集和流量计算。最后对真实数据进行预测分析,结果表明SVM预测方法对高非线性、抖动剧烈的流量数据预测效果较好。
With the fast development of Internet, people's living style and methods of traditional information exchange have been affectted seriously. Various application requirements and Internet scale-up lead to much more difficulties on the network management. Network congestion, security and malfunction etc. have influence on the normal development of the information society. Internet traffic measure and forecast have huge significance on the management, layout and design of large-sized network. Nowadays, Netflow method provided by Cisco is used comprehensively and likely to be the factual standard on measure. Support Vector Machine, a newly machine learning algorithm, having some unique merits such as rapidness of solution and strong generalization performance, shows good performance on nonlinear regression.
The main research contents and feature of this paper are as follows:
(1) This paper introduces the traffic measure methods in common used. SNMP measure collects traffic through the variants in the MIB provided by SNMP network agents. Packet sniffing measure uses NIC (Network Interface Card) to get the traffic at the Link layer. And netflow measure gets traffic by the data exchange mechanisim.
(2) The paper introduces some forecast methods in common used. Such as Min-average error method, Artificial Nerual Network method, Self-regression glide average model method and ARMA model method etc. The paper mainly studys the SVM (Support Vector Machine) method based on the statistics theory.
(3) The paper introduces the structure of NS2 software pack, NS2 simulation principle and process. Generate traffic data by NS2 simple network topology. Make difference operation on those data and orgnize training data and checkout data. Build SVM forecasting model, get model parameters by traning data, and verify the forecast accuracy by the checkout data. The result on the precision of forecast is good and feasible.
(4) Based on the feasible of simulation, using Netflow measure to get traffic in the real circumstance, and forecasting. Through making Netflow configuration on router of Network Mangement Center, collects Netflow packets and computes traffic by related Netflow tools on an appointed PC. Forecasting on that traffic, the result shows SVM regression method having good effect even if the traffic is high nonlinear and huge oscillation.
(1)通过查阅国内外相关文献资料,对网络流量测量方法进行研究。常用的3种测量技术是:SNMP测量、Packet Sniffing测量和Netflow测量。SNMP是通过提取网络设备(代理)提供的MIB(管理对象信息库)中收集的一些具体设备以及流量信息有关的变量进行测量的。Packet Sniffing是一种用网卡在链路层来获取网络流量的方法,使用时将它串接在需要捕捉流量的链路中,通过分流链路上的数字信号来获取流量信息。Netflow是一种数据交换方式,流量采集是基于网络设备提供的Netflow机制实现的。
(2)介绍常用预测方法。如最小均方误差、神经网络、ARMA模型以及基于统计学习理论的支持向量机等。
(3)通过NS2仿真器,结合支持向量机回归理论对模拟的网络流量进行预测实验。首先分析了NS2软件包的结构、仿真原理和流程,然后利用NS2模拟简单网络拓扑结构并产生流量数据,为了提高预测精度将原始数据进行差分处理,然后组织训练数据和检验数据,最后基于SVM建立预测模型,用训练数据训练模型参数,然后用检验数据进行流量预测,并分析结果,结果表明实验预测效果比较理想。
(4)在仿真实验可行的基础上,采用Netflow技术对真实环境下的网络流量进行测量,并进行预测分析。在某网管中心的核心路由器上进行Netflow配置,并在一台PC机上通过Netflow相关工具进行Netflow包收集和流量计算。最后对真实数据进行预测分析,结果表明SVM预测方法对高非线性、抖动剧烈的流量数据预测效果较好。
With the fast development of Internet, people's living style and methods of traditional information exchange have been affectted seriously. Various application requirements and Internet scale-up lead to much more difficulties on the network management. Network congestion, security and malfunction etc. have influence on the normal development of the information society. Internet traffic measure and forecast have huge significance on the management, layout and design of large-sized network. Nowadays, Netflow method provided by Cisco is used comprehensively and likely to be the factual standard on measure. Support Vector Machine, a newly machine learning algorithm, having some unique merits such as rapidness of solution and strong generalization performance, shows good performance on nonlinear regression.
The main research contents and feature of this paper are as follows:
(1) This paper introduces the traffic measure methods in common used. SNMP measure collects traffic through the variants in the MIB provided by SNMP network agents. Packet sniffing measure uses NIC (Network Interface Card) to get the traffic at the Link layer. And netflow measure gets traffic by the data exchange mechanisim.
(2) The paper introduces some forecast methods in common used. Such as Min-average error method, Artificial Nerual Network method, Self-regression glide average model method and ARMA model method etc. The paper mainly studys the SVM (Support Vector Machine) method based on the statistics theory.
(3) The paper introduces the structure of NS2 software pack, NS2 simulation principle and process. Generate traffic data by NS2 simple network topology. Make difference operation on those data and orgnize training data and checkout data. Build SVM forecasting model, get model parameters by traning data, and verify the forecast accuracy by the checkout data. The result on the precision of forecast is good and feasible.
(4) Based on the feasible of simulation, using Netflow measure to get traffic in the real circumstance, and forecasting. Through making Netflow configuration on router of Network Mangement Center, collects Netflow packets and computes traffic by related Netflow tools on an appointed PC. Forecasting on that traffic, the result shows SVM regression method having good effect even if the traffic is high nonlinear and huge oscillation.
引文
[1] C. S. Hood and C. Ji. Beyond thresholds: an alternative method for extracting information from network measures[C]. In: Proceedings of IEEE Globecom Conference, Phoeenix, Arizona, 1997,487-491.
[2] R.Cronk, P.Callahan, and L.Bemstein. Rule-based expert systems for network management and operations: an introduction[C], IEEE Network, 1988,2(5):7-21.
[3] C. S. Chao, D. L. Yang and A. C. Liu. An Automated Fault Diagnosis System. Using Hierarchical Reasoning and Alarm Correlation[J]. Journal of Network and Systems Management, 2001,9(2):183-202.
[4] G. Jakobson and M. Weissman. Real-time telecommunication network management: extending event correlation with temporal constraints[C]. In IFIP/IEEE International Symposium on Integrated Network Management, IV(ISINM'95), 1995,290-301.
[5] Lundy Lewis. A case-based reasoning approach to the resolution of faults in communications networks[C]. In IFIP International Symposium on Integrated NetworkManagement, III(ISINM'93),1993,671-682.
[6] S. Waldbusser. Remote Network Monitoring Management Information Base. Request For Comments 1271, 1991.
[7] E. L. Madruga and L. M. R.Tarouco. Fault Management Tools for a Cooperative and Decentralized Network Operations Environment[J]. IEEE Journal on Selected Areas in Communications, 1994,12(6): 1121-1230.
[8] Robert H. Deng, Aurel A. Lazar and Weiguo Wang. A Probabilistic Approach to Fault Diagnosis in Liner Lightwave Networks[J]. IEEE Journal of Selected Areas in Communications, 1993, 11:1438-1448.
[9] G. Goldszmidt, Y. Yemini. Distributed Management by Delegation[C]. Proceedings of the 15~(th) International Conference on Distributed Computing System, 1995,233-238.
[10]王振龙．时间序列分析[M]：北京：中国统计出版社,2000．
[11] Roy A. Maxion and Frank E. Feather. A Case Study of Ethernet Anomalies in a Distributed Computing Environment[C]. IEEE Transaction on Reliability, 1990, 39(4):433-443.
[12]M.Thottan and C.Ji.Proactive Anomaly Detection Using Distributed Intelligent Agents[C].IEEE Network,1998,12(5):21-27.
[13]S.Hariri,Y.Kim,K.Varshney,R.Kaminski,D.Hague,C.Maciag.A framework for end-to-end proactive network management[C].IEEE NOMS 98,Network Operations and Management Symposium,1998,1:280-286.
[14]Qiming He and M.A.Shayman.Using Reinforcement Learning for Proactive Network Fault Management[C].Proceedings of the International Conference on Communication Technologies,Beijing,P.R.China,2000.
[15]Dongxu Shen,prediction of network traffic for network managementp,Master thesis,Rensselaer Polytechnic Institute,1998.
[16]王耀生.智能预测系统设计及其应用[M].气象出版社,1990.
[17]张有为.预测的数学方法[M].国防工业出版社.1991.
[18]A.Adas.Using Adaptive Linear Prediction to Support Real-Time VBR Video Under RCBR Network Service Model[J].IEEE/ACM Transaction on Networking,1998,6(5):635-644.
[19]D.Morato,J.Aracil,L.A.Diez,M.Izal,and E.Magana.On linear prediction of Intemet traffic for packet and burst switching networks[R].http://www.tlm.unavarra.es/～daniel/publicaciones/pdf/icccNo1.pdf.
[20]R.Grunenfelder et al.Characterization of Video Codecs as Autoregressive Moving Average Processes and Ralatged Quening System Performance[J].IEEE JSAC,1991,9:283-293.
[21]A.Sang and S.Q.Li.A Predictability Analysis of Network Traffic[C].Proceedings of IEEE INFOCOM 2000,2000,342-351.
[22]刘剑平,陆元鸿,曹霄临.概率论与数理统计方法[M].武汉:华东理工大学出版社,2004.
[23]何晓群.现代统计分析方法与应用[M].北京:中国人民大学出版社,1998.
[24]Edmund S.Yu and C.Y.Roger Chen.Global Telecommunications Conference,Technical Program Conference Record,IEEE in Houston,GLOBECOM'93,IEEE,1993,2:991-995.
[25]Ahmed A.Tarraf,lbrahim W.Habib,Tarek N.Saadawi,and Samir A.Ahmed.ATM Multimedia Traffic Prediction Using Neural Networks[C].Global Data Networking,1993,77-84.
[26]H TONG,K SLIM.Threshold auto-regressive limit cycles and.cyclical data[J],J of the Royal Statistical Society,1980,(3):245-292.
[27]邓聚龙.灰色理论基础[M].武汉:华中科技大学出版社,2002.
[28]边肇祺,张学工.模式识别[M].北京:清华大学出版社,2000.
[29]Vapnik V,Levin E,Lc C Y.Measuring the VC Dimension of a Learning Machine[C].Neural Computation.1994,(6):851-876.
[30]Scholkopf B.Statistical Learning and Kernel Methods[R].Microsoft Research Limited.http://research.microsoft.com/～bsc,2000.
[31]Byun H,Lee S W.Applications of Support Vector Machines for Pattern Recognition:A Survey[J].Lee S W,Verri A(Eds.):SVM.LNCS 2388,2002,213-236.
[32]Zhang Ling.The Relationship Between Kernel Functions Based SVM and Three-layer Feedforword Neural Networks[J].Chinese J.Computer,2002,25(7):1-5.
[33]Kevin Fall,kannan Varadhan.The ns Manual[R],http://www.isi.edu/nsnam/ns/.2005.
[34]E.C.Park,H.Lim,K.J.Park,and C.H.Choi."Analysis and design of the virtual rate control algorithm for stabilization queues on TCP networks"[J].Computer Networks,Elsevier Science,2003,(44):17-41.
[35]Fan Yanfei,Ren Fengyuan,and Lin Chuang."Design a P1D controller for active queue management"[C].Proceedings of the 8th IEEE International Symposium on Computers and Communication(ISCC 2003),Kemer-Antalya,2003,985-990.
[36]Suykens J A K,Branbanter J K,Lukas L,et al.Weighted least squares support vector machines:robustness and spare approximation[J].Neurocomputing,2002,48(1):85-105.
[37]邹柏贤,姚志强.一种网络流量平稳化方法[J].通信学报.2004,25(8):14-23.
[38]George C.Sackett 著 前导工作室译.Cisco路由器手册[M].机械工业出版社.2002.
[39]熊齐邦,黄明哲.基于Netflow和异步服务的网络流量监测系统[J].计算机工程.2006.32(13):144-146.
[40]麻海雷,静永健,王鑫.Netflow技术在校园网流量分析中的应用[J].中国教育网络.2008.1-2.
[2] R.Cronk, P.Callahan, and L.Bemstein. Rule-based expert systems for network management and operations: an introduction[C], IEEE Network, 1988,2(5):7-21.
[3] C. S. Chao, D. L. Yang and A. C. Liu. An Automated Fault Diagnosis System. Using Hierarchical Reasoning and Alarm Correlation[J]. Journal of Network and Systems Management, 2001,9(2):183-202.
[4] G. Jakobson and M. Weissman. Real-time telecommunication network management: extending event correlation with temporal constraints[C]. In IFIP/IEEE International Symposium on Integrated Network Management, IV(ISINM'95), 1995,290-301.
[5] Lundy Lewis. A case-based reasoning approach to the resolution of faults in communications networks[C]. In IFIP International Symposium on Integrated NetworkManagement, III(ISINM'93),1993,671-682.
[6] S. Waldbusser. Remote Network Monitoring Management Information Base. Request For Comments 1271, 1991.
[7] E. L. Madruga and L. M. R.Tarouco. Fault Management Tools for a Cooperative and Decentralized Network Operations Environment[J]. IEEE Journal on Selected Areas in Communications, 1994,12(6): 1121-1230.
[8] Robert H. Deng, Aurel A. Lazar and Weiguo Wang. A Probabilistic Approach to Fault Diagnosis in Liner Lightwave Networks[J]. IEEE Journal of Selected Areas in Communications, 1993, 11:1438-1448.
[9] G. Goldszmidt, Y. Yemini. Distributed Management by Delegation[C]. Proceedings of the 15~(th) International Conference on Distributed Computing System, 1995,233-238.
[10]王振龙．时间序列分析[M]：北京：中国统计出版社,2000．
[11] Roy A. Maxion and Frank E. Feather. A Case Study of Ethernet Anomalies in a Distributed Computing Environment[C]. IEEE Transaction on Reliability, 1990, 39(4):433-443.
[12]M.Thottan and C.Ji.Proactive Anomaly Detection Using Distributed Intelligent Agents[C].IEEE Network,1998,12(5):21-27.
[13]S.Hariri,Y.Kim,K.Varshney,R.Kaminski,D.Hague,C.Maciag.A framework for end-to-end proactive network management[C].IEEE NOMS 98,Network Operations and Management Symposium,1998,1:280-286.
[14]Qiming He and M.A.Shayman.Using Reinforcement Learning for Proactive Network Fault Management[C].Proceedings of the International Conference on Communication Technologies,Beijing,P.R.China,2000.
[15]Dongxu Shen,prediction of network traffic for network managementp,Master thesis,Rensselaer Polytechnic Institute,1998.
[16]王耀生.智能预测系统设计及其应用[M].气象出版社,1990.
[17]张有为.预测的数学方法[M].国防工业出版社.1991.
[18]A.Adas.Using Adaptive Linear Prediction to Support Real-Time VBR Video Under RCBR Network Service Model[J].IEEE/ACM Transaction on Networking,1998,6(5):635-644.
[19]D.Morato,J.Aracil,L.A.Diez,M.Izal,and E.Magana.On linear prediction of Intemet traffic for packet and burst switching networks[R].http://www.tlm.unavarra.es/～daniel/publicaciones/pdf/icccNo1.pdf.
[20]R.Grunenfelder et al.Characterization of Video Codecs as Autoregressive Moving Average Processes and Ralatged Quening System Performance[J].IEEE JSAC,1991,9:283-293.
[21]A.Sang and S.Q.Li.A Predictability Analysis of Network Traffic[C].Proceedings of IEEE INFOCOM 2000,2000,342-351.
[22]刘剑平,陆元鸿,曹霄临.概率论与数理统计方法[M].武汉:华东理工大学出版社,2004.
[23]何晓群.现代统计分析方法与应用[M].北京:中国人民大学出版社,1998.
[24]Edmund S.Yu and C.Y.Roger Chen.Global Telecommunications Conference,Technical Program Conference Record,IEEE in Houston,GLOBECOM'93,IEEE,1993,2:991-995.
[25]Ahmed A.Tarraf,lbrahim W.Habib,Tarek N.Saadawi,and Samir A.Ahmed.ATM Multimedia Traffic Prediction Using Neural Networks[C].Global Data Networking,1993,77-84.
[26]H TONG,K SLIM.Threshold auto-regressive limit cycles and.cyclical data[J],J of the Royal Statistical Society,1980,(3):245-292.
[27]邓聚龙.灰色理论基础[M].武汉:华中科技大学出版社,2002.
[28]边肇祺,张学工.模式识别[M].北京:清华大学出版社,2000.
[29]Vapnik V,Levin E,Lc C Y.Measuring the VC Dimension of a Learning Machine[C].Neural Computation.1994,(6):851-876.
[30]Scholkopf B.Statistical Learning and Kernel Methods[R].Microsoft Research Limited.http://research.microsoft.com/～bsc,2000.
[31]Byun H,Lee S W.Applications of Support Vector Machines for Pattern Recognition:A Survey[J].Lee S W,Verri A(Eds.):SVM.LNCS 2388,2002,213-236.
[32]Zhang Ling.The Relationship Between Kernel Functions Based SVM and Three-layer Feedforword Neural Networks[J].Chinese J.Computer,2002,25(7):1-5.
[33]Kevin Fall,kannan Varadhan.The ns Manual[R],http://www.isi.edu/nsnam/ns/.2005.
[34]E.C.Park,H.Lim,K.J.Park,and C.H.Choi."Analysis and design of the virtual rate control algorithm for stabilization queues on TCP networks"[J].Computer Networks,Elsevier Science,2003,(44):17-41.
[35]Fan Yanfei,Ren Fengyuan,and Lin Chuang."Design a P1D controller for active queue management"[C].Proceedings of the 8th IEEE International Symposium on Computers and Communication(ISCC 2003),Kemer-Antalya,2003,985-990.
[36]Suykens J A K,Branbanter J K,Lukas L,et al.Weighted least squares support vector machines:robustness and spare approximation[J].Neurocomputing,2002,48(1):85-105.
[37]邹柏贤,姚志强.一种网络流量平稳化方法[J].通信学报.2004,25(8):14-23.
[38]George C.Sackett 著 前导工作室译.Cisco路由器手册[M].机械工业出版社.2002.
[39]熊齐邦,黄明哲.基于Netflow和异步服务的网络流量监测系统[J].计算机工程.2006.32(13):144-146.
[40]麻海雷,静永健,王鑫.Netflow技术在校园网流量分析中的应用[J].中国教育网络.2008.1-2.