大规模网络流量异常分析
|
摘要
随着网络规模的日益扩大和承载业务种类的逐渐增多,Internet的发展给人们带来了巨大方便。但是,这些也使网络中出现各种异常的机会大大增加,给网络监测带来了更大的挑战。网络流量异常分析是网络监测中的关键部分,能够准确、及时地检测出异常对提高网络的可用性和可靠性具有非常重要的意义。
大规模网络流量的特点是维数多、速度快、规模宏大,但现有的基于时间序列的统计分析和基于信号的小波分析对这类数据的处理能力有限,需要一种更简单高效的异常流量分析方法。本文就是探索大规模网络流量异常分析的新方法,以提高对网络流量异常的检测和分类的能力,并利用这种方法实现网络流量监测的功能。
首先对已提出的子空间方法进行了研究,并应用于实验环境中实现了大规模网络流量异常检测,通过对结果的对比分析证明了基于子空间方法的异常检测具有更高的检测精度。然后在改进基于信息熵的大规模网络流量异常检测和异常分类方法的基础上,提出了一种分布式的异常检测方法,经实验证明该方法操作简单,处理时间短,检测效果好。最后提出大规模网络流量异常检测系统框架模型,由流量采集模块和预处理模块、异常分析模块、综合分析及可视化模块组成,并通过实验验证了该模型的有效性,具有实际应用价值。
With the expansion of network size and the increase of services provided, the rapid development of the Internet brings us a lot of convenience. However, this also results in the menaces from various kinds of security incidents, which enable network-monitor to face the greater challenges. Network traffic anomaly analysis is a key part of network-monitor, whether the network anomaly is detected accurately or not is very import to improve network availability and reliability.
The characteristic of the large-scale network traffic data is many dimensions, rapid, large, while the availability of today’s statistic analysis based on time sequence and wavelet analysis based on signal dealing with is limit. So a simple and effective anomaly analysis method is in great need. This thesis presents some new methods to detect the large-scale network traffic anomaly analysis, which not only improve the capability to detect anomaly and anomaly classification, but also realize network-monitor function.
First of all, we study subspace method which has been presented, and in experiment environment realize the process of detecting the large-scale network traffic anomalies applying subspace method. Through the results’contrast analysis, it shows that the traffic anomaly detection based on subspace method has a higher detection precision. Then in terms of improvement of the large-scale network traffic anomaly detection and anomaly classification methods based on entropy, we present a new method of distributed anomaly detection. The experiment proves that this method operates simply and reduces the detection time greatly, which can satisfy the request of the online detection. Finally, we present the large-scale network traffic anomaly detection system frame model, which is made up of traffic data collection and pre-disposal model, anomaly analysis model, synthesis analysis and visualized model. Several practices have been made to prove that the model is effective and practical.
大规模网络流量的特点是维数多、速度快、规模宏大,但现有的基于时间序列的统计分析和基于信号的小波分析对这类数据的处理能力有限,需要一种更简单高效的异常流量分析方法。本文就是探索大规模网络流量异常分析的新方法,以提高对网络流量异常的检测和分类的能力,并利用这种方法实现网络流量监测的功能。
首先对已提出的子空间方法进行了研究,并应用于实验环境中实现了大规模网络流量异常检测,通过对结果的对比分析证明了基于子空间方法的异常检测具有更高的检测精度。然后在改进基于信息熵的大规模网络流量异常检测和异常分类方法的基础上,提出了一种分布式的异常检测方法,经实验证明该方法操作简单,处理时间短,检测效果好。最后提出大规模网络流量异常检测系统框架模型,由流量采集模块和预处理模块、异常分析模块、综合分析及可视化模块组成,并通过实验验证了该模型的有效性,具有实际应用价值。
With the expansion of network size and the increase of services provided, the rapid development of the Internet brings us a lot of convenience. However, this also results in the menaces from various kinds of security incidents, which enable network-monitor to face the greater challenges. Network traffic anomaly analysis is a key part of network-monitor, whether the network anomaly is detected accurately or not is very import to improve network availability and reliability.
The characteristic of the large-scale network traffic data is many dimensions, rapid, large, while the availability of today’s statistic analysis based on time sequence and wavelet analysis based on signal dealing with is limit. So a simple and effective anomaly analysis method is in great need. This thesis presents some new methods to detect the large-scale network traffic anomaly analysis, which not only improve the capability to detect anomaly and anomaly classification, but also realize network-monitor function.
First of all, we study subspace method which has been presented, and in experiment environment realize the process of detecting the large-scale network traffic anomalies applying subspace method. Through the results’contrast analysis, it shows that the traffic anomaly detection based on subspace method has a higher detection precision. Then in terms of improvement of the large-scale network traffic anomaly detection and anomaly classification methods based on entropy, we present a new method of distributed anomaly detection. The experiment proves that this method operates simply and reduces the detection time greatly, which can satisfy the request of the online detection. Finally, we present the large-scale network traffic anomaly detection system frame model, which is made up of traffic data collection and pre-disposal model, anomaly analysis model, synthesis analysis and visualized model. Several practices have been made to prove that the model is effective and practical.
引文
[1] JUAN M E, PEDRO G, JESUS E D, Anomaly detection methods in wired networks:a survey and taxonomy, Computer Communications, 2004, 27(16):1569-1584.
[2] C. S. Hood, C. Ji, Beyond thresholds: an alternative method for extracting information from network measures, In: Proceedings of IEEE Globecom Conference, Phoenix, Arizona, 1997, 487-491.
[3] Denning DE, An intrusion-detection model, IEEE Transactions on Software Engineering, 1987, SE-13:222-232.
[4] CarlaT L, Brodley E, Temporal sequence learning and data reduction for anomaly detection, Proceedings of the 5th Conference on Computer and Communications Security, New York:ACM Press, 1998, 150-158.
[5] Lee W, Stolfo S, Kui M, A data mining framework for adaptive intrusion detection, In:IEEE Computer Society, ed. Proceedings of the 1999 IEEE Symposium on Security and Privacy, Oakland, CA:IEEE Press, 1999, 120-132.
[6] Lunt TF, Tamaru A, Gilham F, A real-time intrusion detection expert system(IDES ), Technical Report, Menlo Park, CA:Computer Science Laboratory, SRI International,1992.
[7] 饶鲜, 董春曦, 杨绍全, 基于支持向量机的入侵检测系统, 软件学报, 第 14卷, 第 4 期, 2003.
[8] 陈光英, 张千里, 李星, 基于 SVM 分类机的入侵检测系统, 通信学报, 第 23卷, 第 5 期, 2002.
[9] 单征, 刘铁铭, 楚蓓蓓, 基于网络状态的入侵检测模型, 信息工程大学学报, 第 3 卷, 第 3 期, 2002.
[10] 龚俭, 陆晨, 大规模互联网络的入侵检测, 东南大学学报(自然科学版), 第32 卷, 第 3 期, 2001.
[11] 谭小彬, 王卫平, 奚宏生, 殷保群, 计算机系统入侵检测的隐马尔可夫模型, 计算机研究与发展, 第 40 卷, 第 2 期, 2003.
[12] Wang H, Zhang D, Kang S, Detecting SYN flooding attacks[A], Proceedings of the IEEE Infocom[C], New York, 2002, 123-132.
[13] 邹柏贤, 一种网络异常实时检测方法, 计算机学报, 2003, 26(8):940-948.
[14] Roy A. Maxion, Frank E. Feather, A Case Study of Ethernet Anomalies in aDistributed Computing Environment, IEEE Transaction on Reliability, 1990, 39(4): 433-443.
[15] E Featherr, Dan Siewiorek, R. Maxion, Fault detection in an Ethernet network using anomaly signature matching, Proceedings of ACM SIGCOMM, 1993, 23(4):279-288.
[16] Marina Thottan, C. Ji, Adaptive Thresholding for Proactive Network Problem Detection, In: IEEE International Workshop on Systems Management, Newport, Rhode Island, 1998, 108-116.
[17] M. Thottan, C. Ji, Proactive Anomaly Detection Using Distributed Intelligent Agents, IEEE Network, 1998, 12(5):21-27.
[18] Marina Thottan, C. Ji, Fault Prediction at The Network Layer Using Intelligent Agents, In IEEE/IFIP, Integrated Network Management VI, Boston, USA., 1999, 745-760.
[19] M. thottan, C. Ji, Using Network Fault Predictions to Enable II' Traffic Management, Journal of Network and Systems management, 2001, 19(3):327-346.
[20] V. Alarcon-Aquio, J. A. Barria, Anomaly detection in communication networks using wavelets, IEE Proceeding-Communication, 2001, 148(6):355-362.
[21] P. Barford, J. Kline, D. Plonka, A signal analysis of network traffic anomalies[A], Proceedings of ACM SIGCOMM Internet Measurement Workshop[C], Marseilles, France, 2002, 71-82.
[22] R. Weil, A. Garcia-Ortiz, J. Wooten, Detection of traffic anomalies using fuzzy logic based techniques, Fuzzy Systems Proceedings, 1998, 1176-1181.
[23] 薛丽军, 第文军, 蒋世奇, 一种新的网络流量异常检测方法, 燃气涡轮试验与研究, 第 16 卷, 第 3 期, 2003.
[24] 第文军, 薛丽军, 蒋士奇, 运用网络流量自相似分析的网络流量异常检测, 网络信息技术, 第 22 卷, 第 6 期, 2003.
[25] P. Barford, D. Plonka, Characteristics of Network Traffic Flow Anomalies, In Proceedings of ACMSIGCOMM Intemet Measurement Workshop (IMW) 2001, 2001.
[26] Ye N, Sean V, Chen Q, Computer intrusion detection through EWMA for autocorrelated and uncorrelated data[J], IEEE Transactions on Reliability, 2003, 52(1):75-82.
[27] 程光, 龚检, 丁伟, 基于抽样测量的高速网络实时检测模型, 软件学报, 2003, 14(3):594-599.
[28] 应明, 李建华, 铁玲, 基于条件规则库的流量异常检测系统设计, 通信技术, 第 10 期, 2003.
[29] 应明, 李建华, 铁玲, 因果网的非稳态流量异常检测系统设计, 上海交通大学学报, 第 38 卷, 第 5 期, 2004.
[30] 李忠诚, 邹柏贤, 基于 AR 模型的网络异常检测, 微电子学与计算机, 第 12期, 2002.
[31] P. Abry, D. Veitch, Wavelet analysis of long range dependent traffic, IEEE Transactions on Information Theory, 1998, 44(1).
[32] 郭永宁, 邹柏贤, 姚志强, 一种检测网络异常的小波方法, 福建师范大学学报(自然科学版), 第 2 期, 2003.
[33] Seong Soo Kim, A. L. Narasimha Reddy, A Study of Analyzing Network traffic as Images in Real-Time, In Proceedings of IEEE INFOCOM 2005, Miami, Florida, USA, 2005.
[34] T. Karagiannis, K. Papagiannaki, M. Faloutsos, BLINC: Multilevel traffic classification in the dark, ACM SIGCOMM 2005, 35 (4):229-240.
[35] 程光, 龚检, 丁伟, 大规模网络流量行为累加分解研究, 计算机工程与科学, 2002, 24(5):53-56.
[36] 程光, 龚检, 大规模网络流量宏观行为周期性分析研究, 小型微型计算机系统, 2003, 24(6):991-994.
[37] A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, ED. Kolaczyk, N. Taft, Structural Analysis of Network Traffic Flows, In ACM SIGMETRICS, 2004.
[38] LI. Smith, A tutorial on Principal Components Analysis, Maintained by Cornell University, USA, 2002, 2-20.
[39] 郑军, 胡铭曾, 云晓春, 郑仲, 基于数据流方法的大规模网络异常发现, 通信学报, 2006, 27(2):1-8.
[40] A. Lakhina, M. Crovella, C. Diot, Diagnosing Network-Wide Traffic Anomalies, In ACM SIGCOMM, Portland, 2004.
[41] A. Lakhina, M. Crovella, C. Diot, Characterization of Network-Wide Anomalies in Traffic Flows, Technical Report BUCS-2004-020, Boston University, 2004.
[42] A. Lakhina, M. Crovella, C. Diot, Mining Anomalies Using Traffic Feature Distributions, In ACM SIGCOMM, USA, 2005.
[43] J. E. Jackson, G. S. Mudholkar, Control Procedures for Residuals Associated with Principal Component Analysis, Technometrics,1979, 341-349.
[44] R. Dunia, S. J. Qin, Multi-dimensional Fault Diagnosis Using a Subspace Approach, In American Control Conference, 1997.
[45] A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, F. True, Deriving Traffic Demands for Operational IP networks:Methodology and Experience, In IEEE/ACM Transactions on Neworking, 2001.
[46] 肖志新, 杨岳湘, 杨霖, 基于小波技术的网络异常流量检测与实现, 计算机科学, 已录用, 2007(2).
[47] 余艳, 基于信息熵的粗糙集在故障诊断中的应用研究, 湖南大学硕士学位论文, 2004.
[48] 向继, 高能, 荆继武, 聚类算法在网络入侵检测中的应用, 计算机工程, 第29 卷, 第 16 期, 2003.
[49] 陈华, 陈书海, 张平, K-means 算法在遥感分类中的应用, 红外与激光工程, 第 29 卷, 第 2 期, 2000.
[50] C. Gates, M. Collins, M. Duggan, A. Kompanek, M. Thomas, More Netflow Tools for Performance and Security, Proceedings of the 18th USENIX conference on System administration, Atlanta, 2004, 121-132.
[51] 郝志宇, 大规模网络异常检测技术研究, 哈尔滨工业大学硕士学位论文, 2004.
[2] C. S. Hood, C. Ji, Beyond thresholds: an alternative method for extracting information from network measures, In: Proceedings of IEEE Globecom Conference, Phoenix, Arizona, 1997, 487-491.
[3] Denning DE, An intrusion-detection model, IEEE Transactions on Software Engineering, 1987, SE-13:222-232.
[4] CarlaT L, Brodley E, Temporal sequence learning and data reduction for anomaly detection, Proceedings of the 5th Conference on Computer and Communications Security, New York:ACM Press, 1998, 150-158.
[5] Lee W, Stolfo S, Kui M, A data mining framework for adaptive intrusion detection, In:IEEE Computer Society, ed. Proceedings of the 1999 IEEE Symposium on Security and Privacy, Oakland, CA:IEEE Press, 1999, 120-132.
[6] Lunt TF, Tamaru A, Gilham F, A real-time intrusion detection expert system(IDES ), Technical Report, Menlo Park, CA:Computer Science Laboratory, SRI International,1992.
[7] 饶鲜, 董春曦, 杨绍全, 基于支持向量机的入侵检测系统, 软件学报, 第 14卷, 第 4 期, 2003.
[8] 陈光英, 张千里, 李星, 基于 SVM 分类机的入侵检测系统, 通信学报, 第 23卷, 第 5 期, 2002.
[9] 单征, 刘铁铭, 楚蓓蓓, 基于网络状态的入侵检测模型, 信息工程大学学报, 第 3 卷, 第 3 期, 2002.
[10] 龚俭, 陆晨, 大规模互联网络的入侵检测, 东南大学学报(自然科学版), 第32 卷, 第 3 期, 2001.
[11] 谭小彬, 王卫平, 奚宏生, 殷保群, 计算机系统入侵检测的隐马尔可夫模型, 计算机研究与发展, 第 40 卷, 第 2 期, 2003.
[12] Wang H, Zhang D, Kang S, Detecting SYN flooding attacks[A], Proceedings of the IEEE Infocom[C], New York, 2002, 123-132.
[13] 邹柏贤, 一种网络异常实时检测方法, 计算机学报, 2003, 26(8):940-948.
[14] Roy A. Maxion, Frank E. Feather, A Case Study of Ethernet Anomalies in aDistributed Computing Environment, IEEE Transaction on Reliability, 1990, 39(4): 433-443.
[15] E Featherr, Dan Siewiorek, R. Maxion, Fault detection in an Ethernet network using anomaly signature matching, Proceedings of ACM SIGCOMM, 1993, 23(4):279-288.
[16] Marina Thottan, C. Ji, Adaptive Thresholding for Proactive Network Problem Detection, In: IEEE International Workshop on Systems Management, Newport, Rhode Island, 1998, 108-116.
[17] M. Thottan, C. Ji, Proactive Anomaly Detection Using Distributed Intelligent Agents, IEEE Network, 1998, 12(5):21-27.
[18] Marina Thottan, C. Ji, Fault Prediction at The Network Layer Using Intelligent Agents, In IEEE/IFIP, Integrated Network Management VI, Boston, USA., 1999, 745-760.
[19] M. thottan, C. Ji, Using Network Fault Predictions to Enable II' Traffic Management, Journal of Network and Systems management, 2001, 19(3):327-346.
[20] V. Alarcon-Aquio, J. A. Barria, Anomaly detection in communication networks using wavelets, IEE Proceeding-Communication, 2001, 148(6):355-362.
[21] P. Barford, J. Kline, D. Plonka, A signal analysis of network traffic anomalies[A], Proceedings of ACM SIGCOMM Internet Measurement Workshop[C], Marseilles, France, 2002, 71-82.
[22] R. Weil, A. Garcia-Ortiz, J. Wooten, Detection of traffic anomalies using fuzzy logic based techniques, Fuzzy Systems Proceedings, 1998, 1176-1181.
[23] 薛丽军, 第文军, 蒋世奇, 一种新的网络流量异常检测方法, 燃气涡轮试验与研究, 第 16 卷, 第 3 期, 2003.
[24] 第文军, 薛丽军, 蒋士奇, 运用网络流量自相似分析的网络流量异常检测, 网络信息技术, 第 22 卷, 第 6 期, 2003.
[25] P. Barford, D. Plonka, Characteristics of Network Traffic Flow Anomalies, In Proceedings of ACMSIGCOMM Intemet Measurement Workshop (IMW) 2001, 2001.
[26] Ye N, Sean V, Chen Q, Computer intrusion detection through EWMA for autocorrelated and uncorrelated data[J], IEEE Transactions on Reliability, 2003, 52(1):75-82.
[27] 程光, 龚检, 丁伟, 基于抽样测量的高速网络实时检测模型, 软件学报, 2003, 14(3):594-599.
[28] 应明, 李建华, 铁玲, 基于条件规则库的流量异常检测系统设计, 通信技术, 第 10 期, 2003.
[29] 应明, 李建华, 铁玲, 因果网的非稳态流量异常检测系统设计, 上海交通大学学报, 第 38 卷, 第 5 期, 2004.
[30] 李忠诚, 邹柏贤, 基于 AR 模型的网络异常检测, 微电子学与计算机, 第 12期, 2002.
[31] P. Abry, D. Veitch, Wavelet analysis of long range dependent traffic, IEEE Transactions on Information Theory, 1998, 44(1).
[32] 郭永宁, 邹柏贤, 姚志强, 一种检测网络异常的小波方法, 福建师范大学学报(自然科学版), 第 2 期, 2003.
[33] Seong Soo Kim, A. L. Narasimha Reddy, A Study of Analyzing Network traffic as Images in Real-Time, In Proceedings of IEEE INFOCOM 2005, Miami, Florida, USA, 2005.
[34] T. Karagiannis, K. Papagiannaki, M. Faloutsos, BLINC: Multilevel traffic classification in the dark, ACM SIGCOMM 2005, 35 (4):229-240.
[35] 程光, 龚检, 丁伟, 大规模网络流量行为累加分解研究, 计算机工程与科学, 2002, 24(5):53-56.
[36] 程光, 龚检, 大规模网络流量宏观行为周期性分析研究, 小型微型计算机系统, 2003, 24(6):991-994.
[37] A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, ED. Kolaczyk, N. Taft, Structural Analysis of Network Traffic Flows, In ACM SIGMETRICS, 2004.
[38] LI. Smith, A tutorial on Principal Components Analysis, Maintained by Cornell University, USA, 2002, 2-20.
[39] 郑军, 胡铭曾, 云晓春, 郑仲, 基于数据流方法的大规模网络异常发现, 通信学报, 2006, 27(2):1-8.
[40] A. Lakhina, M. Crovella, C. Diot, Diagnosing Network-Wide Traffic Anomalies, In ACM SIGCOMM, Portland, 2004.
[41] A. Lakhina, M. Crovella, C. Diot, Characterization of Network-Wide Anomalies in Traffic Flows, Technical Report BUCS-2004-020, Boston University, 2004.
[42] A. Lakhina, M. Crovella, C. Diot, Mining Anomalies Using Traffic Feature Distributions, In ACM SIGCOMM, USA, 2005.
[43] J. E. Jackson, G. S. Mudholkar, Control Procedures for Residuals Associated with Principal Component Analysis, Technometrics,1979, 341-349.
[44] R. Dunia, S. J. Qin, Multi-dimensional Fault Diagnosis Using a Subspace Approach, In American Control Conference, 1997.
[45] A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, F. True, Deriving Traffic Demands for Operational IP networks:Methodology and Experience, In IEEE/ACM Transactions on Neworking, 2001.
[46] 肖志新, 杨岳湘, 杨霖, 基于小波技术的网络异常流量检测与实现, 计算机科学, 已录用, 2007(2).
[47] 余艳, 基于信息熵的粗糙集在故障诊断中的应用研究, 湖南大学硕士学位论文, 2004.
[48] 向继, 高能, 荆继武, 聚类算法在网络入侵检测中的应用, 计算机工程, 第29 卷, 第 16 期, 2003.
[49] 陈华, 陈书海, 张平, K-means 算法在遥感分类中的应用, 红外与激光工程, 第 29 卷, 第 2 期, 2000.
[50] C. Gates, M. Collins, M. Duggan, A. Kompanek, M. Thomas, More Netflow Tools for Performance and Security, Proceedings of the 18th USENIX conference on System administration, Atlanta, 2004, 121-132.
[51] 郝志宇, 大规模网络异常检测技术研究, 哈尔滨工业大学硕士学位论文, 2004.