基于小波分析的网络异常检测系统
|
摘要
本文从宏观角度来考察大规模网络的流量，以一定的采样率将包汇聚成流，再将其以字节大小、流数或包数为单位对应到离散的时间轴上，它本质上属于非平稳的时间序列，具有周期性、趋势性、随机性、季节波动等特性。随着网络应用的丰富和网络用户的增多，安全问题也日益重要，而基于宏观流量的网络异常检测也成为研究重点。本文充分研究了各种现有的非平稳网络流量的模型以及各种实用的异常检测的方法，并对其适用层次和优缺点进行了总结。在此基础上提出将传统的基于统计的异常处理和信号处理的方法进行结合，将流量看成信号，并使用统计和信号处理的方法，如小波分析、量图分析等，来进行异常的定位和检测。同时本文还对网络中的常见异常进行了分类，并对其在数学和信号上的表现形式作了分析，以便可以利用检测的结果来归类异常。
本文主要从定性和定量的角度来考虑异常的检测，其中定性分析侧重于从网络流量的建模出发，利用描述模型奇异特性的几种参数，如反映正则性的李氏(Lipschitz)指数，反映自相似度的Hurst指数以及对应不同李氏指数的分形维数，试图从这些参数的变化轨迹中找出与异常出现的对应关系，进而来检测出异常；而定量分析则侧重于对奇异现象的量化，不需要人为的判断，通过一系列的统计分析、信号分析最终将异常突显，这方面主要包括能量比分布分析、小波多层分解、偏差值等方法，最终建立了自动在线实时异常检测系统。
影响网络的因素太多太复杂，甚至还涉及到网络用户的行为波动，因此不可能只通过某一种方法或某一种手段就达到异常的完美检测。本文从不同的角度不同的手段来考虑异常检测的方法，试图建立起一个完善的异常检测体系，而评价异常检测系统好坏的两个因素，即误报率和漏报率，是检测我们系统的唯一标准。本文主要对四段含有异常的流量进行分析，实验结果表明，对异常的检测和突显是有效的。
本文不论是定性或定量的异常检测，都是以小波分析作为算法的基础，因此本系统实际上是基于小波分析的网络异常检测系统。
This paper inspects the traffic of large-scale network from a macroscopic perspective. We aggregate packets into flows in a certain sampling rate, which can be mapped to the ordinate of bytes, flows, or packets. It is essentially non-stationary time series with the periodic trend, randomness, seasonal fluctuations and other characteristics. Along with the rich network applications and the increasing number of network users, security issues is becoming more and more important. Correspondingly, network anomaly detection based on flows from a macroscopic point of view has become the focus of the research. This paper studies various existing models of non-stationary network, as well as practical methods of anomaly detection and gives a summarization about their applications, advantages and disadvantages. This paper regards flow aggregation as a signal and combines statistical and signal processing methods, such as wavelet analysis, scalogram analysis to locate and detect anomalies. This paper also gives a classification of network anomalies and analyzes their manifestations in mathematics and the signal, which can be used to classify the result of anomaly detection.
This paper mainly studies anomaly detection from the qualitative and quantitative perspective. Qualitative analysis of anomaly detection focuses on models of network traffic and their parameters, which can be used to describe the singular characteristic of traffic such as Lipschitz exponent, Hurst exponent and fractal dimension. This paper tries to identify the relationship between the changing trace of these parameters and the presence of anomalies, which can be used to detect anomalies. Quantitative analysis focuses on the quantified singular phenomenon. Through a series of statistical analysis and signal processing, such as the energy ratio distribution analysis, multi-level wavelet decomposition and the deviation value, this paper establishes a automatic on-line real-time anomaly detection system, which can highlight and detect the anomalies under no human judgment.
The network is affected by too many and complicated factors, even the volatility of network users. It is not possible only through one method or one means to achieve the perfect anomaly detection. This paper studies the algorithms of anomaly detection from different perspectives and different methods, trying to build a comprehensive system of anomaly detection. This system can be evaluated by two factors: false positive rate and false negative rate, which are the only criterion. This paper experiments on four traffic flow samples, which contain anomalies. The results show that this system is effective to the detection and highlight of anomalies.
In this paper, both qualitative and quantitative detection of the anomalies are based on wavelet analysis. So the system can be called "Network Anomaly Detection System Based on Wavelet Analysis".
本文主要从定性和定量的角度来考虑异常的检测，其中定性分析侧重于从网络流量的建模出发，利用描述模型奇异特性的几种参数，如反映正则性的李氏(Lipschitz)指数，反映自相似度的Hurst指数以及对应不同李氏指数的分形维数，试图从这些参数的变化轨迹中找出与异常出现的对应关系，进而来检测出异常；而定量分析则侧重于对奇异现象的量化，不需要人为的判断，通过一系列的统计分析、信号分析最终将异常突显，这方面主要包括能量比分布分析、小波多层分解、偏差值等方法，最终建立了自动在线实时异常检测系统。
影响网络的因素太多太复杂，甚至还涉及到网络用户的行为波动，因此不可能只通过某一种方法或某一种手段就达到异常的完美检测。本文从不同的角度不同的手段来考虑异常检测的方法，试图建立起一个完善的异常检测体系，而评价异常检测系统好坏的两个因素，即误报率和漏报率，是检测我们系统的唯一标准。本文主要对四段含有异常的流量进行分析，实验结果表明，对异常的检测和突显是有效的。
本文不论是定性或定量的异常检测，都是以小波分析作为算法的基础，因此本系统实际上是基于小波分析的网络异常检测系统。
This paper inspects the traffic of large-scale network from a macroscopic perspective. We aggregate packets into flows in a certain sampling rate, which can be mapped to the ordinate of bytes, flows, or packets. It is essentially non-stationary time series with the periodic trend, randomness, seasonal fluctuations and other characteristics. Along with the rich network applications and the increasing number of network users, security issues is becoming more and more important. Correspondingly, network anomaly detection based on flows from a macroscopic point of view has become the focus of the research. This paper studies various existing models of non-stationary network, as well as practical methods of anomaly detection and gives a summarization about their applications, advantages and disadvantages. This paper regards flow aggregation as a signal and combines statistical and signal processing methods, such as wavelet analysis, scalogram analysis to locate and detect anomalies. This paper also gives a classification of network anomalies and analyzes their manifestations in mathematics and the signal, which can be used to classify the result of anomaly detection.
This paper mainly studies anomaly detection from the qualitative and quantitative perspective. Qualitative analysis of anomaly detection focuses on models of network traffic and their parameters, which can be used to describe the singular characteristic of traffic such as Lipschitz exponent, Hurst exponent and fractal dimension. This paper tries to identify the relationship between the changing trace of these parameters and the presence of anomalies, which can be used to detect anomalies. Quantitative analysis focuses on the quantified singular phenomenon. Through a series of statistical analysis and signal processing, such as the energy ratio distribution analysis, multi-level wavelet decomposition and the deviation value, this paper establishes a automatic on-line real-time anomaly detection system, which can highlight and detect the anomalies under no human judgment.
The network is affected by too many and complicated factors, even the volatility of network users. It is not possible only through one method or one means to achieve the perfect anomaly detection. This paper studies the algorithms of anomaly detection from different perspectives and different methods, trying to build a comprehensive system of anomaly detection. This system can be evaluated by two factors: false positive rate and false negative rate, which are the only criterion. This paper experiments on four traffic flow samples, which contain anomalies. The results show that this system is effective to the detection and highlight of anomalies.
In this paper, both qualitative and quantitative detection of the anomalies are based on wavelet analysis. So the system can be called "Network Anomaly Detection System Based on Wavelet Analysis".
引文
[1] 程光，龚俭，丁伟．基于抽样测量的高速网络实时异常检测模型．软件学报．2003(03)：594-599
[2] Denning DE. An intrusion-detection model. IEEE Transactions on Software Engineering. 1987(13): 222-232
[3] J Brutlag. Aberrant behavior detection in time series for network monitoring, in Proceedings of the USENIX Fourteenth System Administration Conference LISA ⅪⅤ. New Orleans, LA, December 2000
[4] Leland W E, Taqqu M S, Willinger W, Wilson D V. On the Self-Similar Nature of Ethernet Traffic(Extended Version). IEEE/ACM Transaction on Networking. 1994(1): 1-15
[5] Mandelbrot. Intermittent turbulence in self-similar cascades: Divergence of high moments and dimension of the carrier. J Fluid Mech. 1974: 331-358
[6] Paxson V et al. Wide-area traffic: The failure of Poisson modeling. In: Proc of the ACM SIGCOMM'94. London, 1994: 257-268
[7] Crovella M E et al. Self-similarity in World Wide Web traffic evidence and possible causes. IEEE/ACM Transactions on Networking. 1997(6): 835-846
[8] Rudolf H. Riedi, Matthew S. Crouse, Matthew S. Crouse, Richard G. Baraniuk. A Multifractal Wavelet. Model with Application to Network Traffic. IEEE TRANSACTIONS ON INFORMATION THEORY. 1999(3): 992-1018
[9] Yajuan Tang, Xiapu Luo, Zijie Yang. Fault Detection Through Multi-fractal Nature of Traffic. IEEE Transactions on Software Engineering. 2002: 695-699
[10] Stephane Mallat. A Wavelet Tour of Signal Processing. 2nd edn. New York: Acadmic Press, 1999
[11] Stephane Mallat, Wen Liang Hwang. Singularity Detection and Processing with Wavelets. IEEE TRANSACTIONS ON INFORMATION THEORY. 1992(2): 617-643
[12] Paul Barford, Jeffery Kline, David Plonka, Amos Ron. A Signal Analysis of Network Traffic Anomalies. ACM SIGCOMM Internet Measurement Workshop. 2002: 71-82
[13] Lan Li, Gyungho Lee. DDoS Attack Detection and Wavelets. IEEE TRANSACTIONS ON INFORMATION THEORY. 2003(3): 421-427
[14] 任勋益，王汝传，王海艳．基于自相似检测DDoS攻击的小波分析方法．通信学报．2006(27)：6-11
[15] 顾俊佳，李宁．网络DDoS攻击流的小波分析与检测．计算机工程与应用．2006：127-130
[16] LAN LI, GYUNGHO LEE. DDoS Attack Detection and Wavelets. Telecommunication Systems. 2005(28): 435-451
[17] 王伟，韩维桓．一种基于小波分析的大规模网络流量模拟．华中科技大学学报(自然科学版)．2003(31)：72-74
[18] Alarcon-Aquio V, Barria J A. Anomaly detection in communication networks using wavelet. IEE Proceeding-Communication, 2001.148(6): 355-362
[19] Polly Huang, Anja Feldmann, Walter Willinger. A non-intrusive, wavelet-based approach to detecting network performance problems. Proceedins of Internet Workshop, Nov. 2001, 15. http://citeseer.nj.nec.com/huang01nontrusive.html
[20] 第文军，薛丽军，蒋士奇．运用网络流量自相似分析的网络流量异常检测．网络信息技术．2003(22)：28-31
[21] 肖志新，杨岳湘，杨霖．基于小波技术的网络异常流量检测与实现．计算机科学．2006 (33)：116-118
[22] 任勋益，王汝传，张登银．R／S和小波分析法检测DDoS攻击的研究与比较．南京邮电大学学报(自然科学版)．2006(26)：48-51
[23] LILan, LEE Gyungho. DDoS Attack Detection and Wavelets. Computer Communications and Networks. The 12th International Conference on, 20-22 Oct. 2003: 421-427
[24] 田明．网络入侵检测系统检测算法的研究．南京航空航天大学．2004
[25] Herv Debar, Marc Dacier, Andreas Wespi. Towards a Taxonomy of Intrusion Detection Systems. IBM Technical Paper, Computer Networks. 1999(31): 805-822
[26] Pang-NingTan, Michael Steinbach. 数据挖掘导论．第一版．北京：人民邮电出版社，2006
[27] Cisco IOS NetFlow. http://www.cisco.com/go/netflow. 2002
[28] Yiming Gong. http://www.securityfocus.com/infocus/1796
[29] 胡昌华，张军波，夏军，等．基于MATLAB的系统分析与设计—小波分析[M]．西安：西安电子科技大学出版社．1999
[30] S. Mallat. Multiresolution approximations and wavelet orthonormal bases of L2 (R). Trans. Amer. Math. Soc. 1989: 69-87
[31] The Trous algorithm, http://jstarck.free.fr/cours.htm. 2002
[32] Mark J. Shensa. The discrete wavelet transform: wedding the trous and mallat algorithms. IEEE Transactions on Software Engineering. 1992(40): 2464-2482
[33] Mandelbrot B B, Wallis J R. Some Long-run Properties of Geophysical Records. Water Res Res. 1969(5): 321-340
[34] 李春峰．分形分析在测井数据应用中的几个问题．测井技术．2005(1)：15-20
[35] Li Chun Feng. Information Passage From Acoustic Impedance to Seismogram: Perspectives From Wavelet2based Multiscale Analysis. J Geophys Res. 2004
[36] 李庆谋，刘少华．分形噪声、(多维)分形滤波及地球物理测井曲线处理应用．物探化探计算技术．2001(1)：37-42
[37] Saucier A, Muller J. Use of Multif racial Analysis in the Characterization of Geological Formation. Fractals. 1993(1): 617-628
[38] Muller J, Bokn I, McCauley J L. Multifractal Analysis of Petrophysical Data. Ann Geophysicae. 1992(10): 735-761
[39] Seong Soo Kim, A. L. Narasimha Reddy, and Marina Vannucci. Detecting Traffic Anomalies Using Discrete Wavelet Transform. H.-K. Kahng and S. Goto (Eds.). 2004: 951-961
[40] 董长虹．Matlab小波分析工具箱原理与应用．第一版．北京：国防工业出版社，2004
[41] The MathWorks. http://www.mathworks.com
[42] Ingrid Daubechies．小波十讲．第一版．北京：国防工业出版社，2004
[43] 杨福生．小波变换的工程分析与应用．第一版．北京：科学出版社，1999
[2] Denning DE. An intrusion-detection model. IEEE Transactions on Software Engineering. 1987(13): 222-232
[3] J Brutlag. Aberrant behavior detection in time series for network monitoring, in Proceedings of the USENIX Fourteenth System Administration Conference LISA ⅪⅤ. New Orleans, LA, December 2000
[4] Leland W E, Taqqu M S, Willinger W, Wilson D V. On the Self-Similar Nature of Ethernet Traffic(Extended Version). IEEE/ACM Transaction on Networking. 1994(1): 1-15
[5] Mandelbrot. Intermittent turbulence in self-similar cascades: Divergence of high moments and dimension of the carrier. J Fluid Mech. 1974: 331-358
[6] Paxson V et al. Wide-area traffic: The failure of Poisson modeling. In: Proc of the ACM SIGCOMM'94. London, 1994: 257-268
[7] Crovella M E et al. Self-similarity in World Wide Web traffic evidence and possible causes. IEEE/ACM Transactions on Networking. 1997(6): 835-846
[8] Rudolf H. Riedi, Matthew S. Crouse, Matthew S. Crouse, Richard G. Baraniuk. A Multifractal Wavelet. Model with Application to Network Traffic. IEEE TRANSACTIONS ON INFORMATION THEORY. 1999(3): 992-1018
[9] Yajuan Tang, Xiapu Luo, Zijie Yang. Fault Detection Through Multi-fractal Nature of Traffic. IEEE Transactions on Software Engineering. 2002: 695-699
[10] Stephane Mallat. A Wavelet Tour of Signal Processing. 2nd edn. New York: Acadmic Press, 1999
[11] Stephane Mallat, Wen Liang Hwang. Singularity Detection and Processing with Wavelets. IEEE TRANSACTIONS ON INFORMATION THEORY. 1992(2): 617-643
[12] Paul Barford, Jeffery Kline, David Plonka, Amos Ron. A Signal Analysis of Network Traffic Anomalies. ACM SIGCOMM Internet Measurement Workshop. 2002: 71-82
[13] Lan Li, Gyungho Lee. DDoS Attack Detection and Wavelets. IEEE TRANSACTIONS ON INFORMATION THEORY. 2003(3): 421-427
[14] 任勋益，王汝传，王海艳．基于自相似检测DDoS攻击的小波分析方法．通信学报．2006(27)：6-11
[15] 顾俊佳，李宁．网络DDoS攻击流的小波分析与检测．计算机工程与应用．2006：127-130
[16] LAN LI, GYUNGHO LEE. DDoS Attack Detection and Wavelets. Telecommunication Systems. 2005(28): 435-451
[17] 王伟，韩维桓．一种基于小波分析的大规模网络流量模拟．华中科技大学学报(自然科学版)．2003(31)：72-74
[18] Alarcon-Aquio V, Barria J A. Anomaly detection in communication networks using wavelet. IEE Proceeding-Communication, 2001.148(6): 355-362
[19] Polly Huang, Anja Feldmann, Walter Willinger. A non-intrusive, wavelet-based approach to detecting network performance problems. Proceedins of Internet Workshop, Nov. 2001, 15. http://citeseer.nj.nec.com/huang01nontrusive.html
[20] 第文军，薛丽军，蒋士奇．运用网络流量自相似分析的网络流量异常检测．网络信息技术．2003(22)：28-31
[21] 肖志新，杨岳湘，杨霖．基于小波技术的网络异常流量检测与实现．计算机科学．2006 (33)：116-118
[22] 任勋益，王汝传，张登银．R／S和小波分析法检测DDoS攻击的研究与比较．南京邮电大学学报(自然科学版)．2006(26)：48-51
[23] LILan, LEE Gyungho. DDoS Attack Detection and Wavelets. Computer Communications and Networks. The 12th International Conference on, 20-22 Oct. 2003: 421-427
[24] 田明．网络入侵检测系统检测算法的研究．南京航空航天大学．2004
[25] Herv Debar, Marc Dacier, Andreas Wespi. Towards a Taxonomy of Intrusion Detection Systems. IBM Technical Paper, Computer Networks. 1999(31): 805-822
[26] Pang-NingTan, Michael Steinbach. 数据挖掘导论．第一版．北京：人民邮电出版社，2006
[27] Cisco IOS NetFlow. http://www.cisco.com/go/netflow. 2002
[28] Yiming Gong. http://www.securityfocus.com/infocus/1796
[29] 胡昌华，张军波，夏军，等．基于MATLAB的系统分析与设计—小波分析[M]．西安：西安电子科技大学出版社．1999
[30] S. Mallat. Multiresolution approximations and wavelet orthonormal bases of L2 (R). Trans. Amer. Math. Soc. 1989: 69-87
[31] The Trous algorithm, http://jstarck.free.fr/cours.htm. 2002
[32] Mark J. Shensa. The discrete wavelet transform: wedding the trous and mallat algorithms. IEEE Transactions on Software Engineering. 1992(40): 2464-2482
[33] Mandelbrot B B, Wallis J R. Some Long-run Properties of Geophysical Records. Water Res Res. 1969(5): 321-340
[34] 李春峰．分形分析在测井数据应用中的几个问题．测井技术．2005(1)：15-20
[35] Li Chun Feng. Information Passage From Acoustic Impedance to Seismogram: Perspectives From Wavelet2based Multiscale Analysis. J Geophys Res. 2004
[36] 李庆谋，刘少华．分形噪声、(多维)分形滤波及地球物理测井曲线处理应用．物探化探计算技术．2001(1)：37-42
[37] Saucier A, Muller J. Use of Multif racial Analysis in the Characterization of Geological Formation. Fractals. 1993(1): 617-628
[38] Muller J, Bokn I, McCauley J L. Multifractal Analysis of Petrophysical Data. Ann Geophysicae. 1992(10): 735-761
[39] Seong Soo Kim, A. L. Narasimha Reddy, and Marina Vannucci. Detecting Traffic Anomalies Using Discrete Wavelet Transform. H.-K. Kahng and S. Goto (Eds.). 2004: 951-961
[40] 董长虹．Matlab小波分析工具箱原理与应用．第一版．北京：国防工业出版社，2004
[41] The MathWorks. http://www.mathworks.com
[42] Ingrid Daubechies．小波十讲．第一版．北京：国防工业出版社，2004
[43] 杨福生．小波变换的工程分析与应用．第一版．北京：科学出版社，1999