基于Lindeberg-Feller定理的网络异常检测算法
|
摘要
在网络运维管理领域,需要及时发现网络异常并进行提示。网络异常事件与正常数据相比较少,难以作为二分类问题进行处理。同时异常事件丰富多样,没有统一模式和规律。因此,需要对网络正常数据进行建模,利用待检数据与正常数据的偏离程度判定网络异常事件是否发生。通过对正常数据进行建模分析,以Lindeberg-Feller中心极限定理为基础,设计合理的假设检验统计量,对待检数据计算出的检验统计量是否在置信度对应的拒绝域给出异常事件判别结论。最后,用仿真实验证明算法原理,并给出算法在公开数据集和实际数据集上的检测性能,在选择合理的异常事件对应参数后,异常事件召回率可以达到90%以上。
In the fields of network maintenance and operation, it attracts much attention how to detect and prompt the network anomalies in time. Anomalous events are less in dataset than the normal ones, leading to the fact that it is difficult to use the two-class classifications for anomaly detection because of the imbalance of data labeled as normal or anomalous.Meanwhile, anomalous events are in various patterns and there is little prior information about the anomaly that the users are concerned with, therefore, it is necessary to model the normal data and use them for anomaly detection by comparing the received data with the normal model. Based on Lindeberg-Feller central limit theorem, a hypothesis test is designed to detect whether the data to be tested is anomalous or not, according to the refusing area calculated by the confidential parameter. Finally, the theorem of this algorithm is simulated and the performance is also tested both on the common and the actual datasets. When the users take the correlation features of the anomalous events as the algorithm input, the recall ratio reaches 90%.
In the fields of network maintenance and operation, it attracts much attention how to detect and prompt the network anomalies in time. Anomalous events are less in dataset than the normal ones, leading to the fact that it is difficult to use the two-class classifications for anomaly detection because of the imbalance of data labeled as normal or anomalous.Meanwhile, anomalous events are in various patterns and there is little prior information about the anomaly that the users are concerned with, therefore, it is necessary to model the normal data and use them for anomaly detection by comparing the received data with the normal model. Based on Lindeberg-Feller central limit theorem, a hypothesis test is designed to detect whether the data to be tested is anomalous or not, according to the refusing area calculated by the confidential parameter. Finally, the theorem of this algorithm is simulated and the performance is also tested both on the common and the actual datasets. When the users take the correlation features of the anomalous events as the algorithm input, the recall ratio reaches 90%.
引文
[1]刘楚舒,王伟平,刘鹏飞.结合资源特征的Android恶意应用检测方法[J].计算机工程与应用,2018,54(15):67-73.
[2]Varun C,Arindam B,Vipin K.Anomaly detection:a survey[J].ACM Computing Surveys,2009,41(3):1-58.
[3]Cheng H,Tan P N,Potter C,et al.A robust graph-based algorithm for detection and characterization of anomalies in noisy multivariate time series[C]//IEEE International Conference on Data Mining Workshops,2008:349-358.
[4]Wang Z,Zhao Z,Weng S,et al.Incremental multiple instance outlier detection[J].Neural Computing&Applications,2015,26(4):957-968.
[5]Guan H,Li Q,Yan Z,et al.SLOF:identify density-based local outliers in big data[C]//Web Information System and Application Conference,2016:61-66.
[6]Khashei M,Bijari M.A novel hybridization of artificial neural networks and ARIMA models for time series forecasting[J].Applied Soft Computing Journal,2011,11(2):2664-2675.
[7]Wang X,Mueen A,Ding H,et al.Experimental comparison of representation methods and distance measures for time series data[J].Data Mining&Knowledge Discovery,2013,26(2):275-309.
[8]Saini S,Dua S.Temporal pattern mining for multivarite clinical decision support[C]//Proceedings of International Conference on Making Sense of Converging Media,2013:99.
[9]Weng X,Shen J.Outlier mining for multivariate time series based on local sparsity coefficient[C]//Sixth World Congress on Intelligent Control and Automation,2006:5957-5961.
[10]陈湘涛,李明亮,陈玉娟.基于时间序列相似性聚类的应用研究综述[J].计算机工程与设计,2010,31(3):577-581.
[11]Fountoulakis N.On the evolution of random graphs on spaces of negative curvature[J].arXiv:1205.2923,2012.
[12]Barabasi A L,Albert R.Emergence of scaling in random networks[J].Science,1999,286(5439):509-512.
[13]Jure L.Graphs over time:densification laws,shrinking diameters,explanations and realistic generators[C]//KDD,2005:177-187.
[14]孟啸.动态复杂网络中的异常检测问题的研究[D].哈尔滨:哈尔滨工业大学计算机科学与技术学院,2010.
[15]Bunke H,Dickinson P J,Kraetzl M,et al.A graph-theoretic approach to enterprise network dynamics(progress in computer science&applied logic)[M].Boston:Birkhauser,2007.
[16]Mayank L,Tanya Y B Periodic subgraph mining in dynamic networks[J].Knowledge and Information Systems,2010,24(3):467-497.
[17]Scharw?chter E,Müller E,Donges J,et al.Detecting change processes in dynamic networks by frequent graph evolution rule mining[C]//IEEE International Conference on Data Mining,2017:1191-1196.
[18]Wager S,Fithian W,Liang P.Data augmentation via Levy processes[J/OL].(2016-03-21)[2018-10-24].https://arxiv.org/pdf/1603.06340.pdf.
[19]Um T T,Pfister F M J,Pichler D,et al.Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks[J].arXiv:1706.00527,2017.
[20]Goodfellow I J,Pouget-Abadie J,Mirza M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems,2014:2672-2680.
[21]Arjovsky M,Chintala S,Bottou L.Wasserstein GAN[J/OL].(2017-12-06)[2018-10-24].https://arxiv.org/pdf/1701.07875.pdf.
[22]Gulrajani I,Ahmed F,Arjovsky M,et al.Improved training of Wasserstein GANs[J/OL].(2017-12-25)[2018-10-24].https://arxiv.org/pdf/1704.00028.pdf.
[23]Schlegl T,Seeb?ck P,Waldstein S M,et al.Unsupervised anomaly detection with generative adversarial networks to guide marker discovery[C]//International Conference on Information Processing in Medical Imaging.Berlin:Springer,2017:146-157.
[24]Baumgartner C F,Koch L M,Tezcan K C,et al.Visual feature attribution using Wasserstein GANs[J/OL].(2018-06-26)[2018-10-24].https://arxiv.org/pdf/1711.08998.pdf.
[2]Varun C,Arindam B,Vipin K.Anomaly detection:a survey[J].ACM Computing Surveys,2009,41(3):1-58.
[3]Cheng H,Tan P N,Potter C,et al.A robust graph-based algorithm for detection and characterization of anomalies in noisy multivariate time series[C]//IEEE International Conference on Data Mining Workshops,2008:349-358.
[4]Wang Z,Zhao Z,Weng S,et al.Incremental multiple instance outlier detection[J].Neural Computing&Applications,2015,26(4):957-968.
[5]Guan H,Li Q,Yan Z,et al.SLOF:identify density-based local outliers in big data[C]//Web Information System and Application Conference,2016:61-66.
[6]Khashei M,Bijari M.A novel hybridization of artificial neural networks and ARIMA models for time series forecasting[J].Applied Soft Computing Journal,2011,11(2):2664-2675.
[7]Wang X,Mueen A,Ding H,et al.Experimental comparison of representation methods and distance measures for time series data[J].Data Mining&Knowledge Discovery,2013,26(2):275-309.
[8]Saini S,Dua S.Temporal pattern mining for multivarite clinical decision support[C]//Proceedings of International Conference on Making Sense of Converging Media,2013:99.
[9]Weng X,Shen J.Outlier mining for multivariate time series based on local sparsity coefficient[C]//Sixth World Congress on Intelligent Control and Automation,2006:5957-5961.
[10]陈湘涛,李明亮,陈玉娟.基于时间序列相似性聚类的应用研究综述[J].计算机工程与设计,2010,31(3):577-581.
[11]Fountoulakis N.On the evolution of random graphs on spaces of negative curvature[J].arXiv:1205.2923,2012.
[12]Barabasi A L,Albert R.Emergence of scaling in random networks[J].Science,1999,286(5439):509-512.
[13]Jure L.Graphs over time:densification laws,shrinking diameters,explanations and realistic generators[C]//KDD,2005:177-187.
[14]孟啸.动态复杂网络中的异常检测问题的研究[D].哈尔滨:哈尔滨工业大学计算机科学与技术学院,2010.
[15]Bunke H,Dickinson P J,Kraetzl M,et al.A graph-theoretic approach to enterprise network dynamics(progress in computer science&applied logic)[M].Boston:Birkhauser,2007.
[16]Mayank L,Tanya Y B Periodic subgraph mining in dynamic networks[J].Knowledge and Information Systems,2010,24(3):467-497.
[17]Scharw?chter E,Müller E,Donges J,et al.Detecting change processes in dynamic networks by frequent graph evolution rule mining[C]//IEEE International Conference on Data Mining,2017:1191-1196.
[18]Wager S,Fithian W,Liang P.Data augmentation via Levy processes[J/OL].(2016-03-21)[2018-10-24].https://arxiv.org/pdf/1603.06340.pdf.
[19]Um T T,Pfister F M J,Pichler D,et al.Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks[J].arXiv:1706.00527,2017.
[20]Goodfellow I J,Pouget-Abadie J,Mirza M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems,2014:2672-2680.
[21]Arjovsky M,Chintala S,Bottou L.Wasserstein GAN[J/OL].(2017-12-06)[2018-10-24].https://arxiv.org/pdf/1701.07875.pdf.
[22]Gulrajani I,Ahmed F,Arjovsky M,et al.Improved training of Wasserstein GANs[J/OL].(2017-12-25)[2018-10-24].https://arxiv.org/pdf/1704.00028.pdf.
[23]Schlegl T,Seeb?ck P,Waldstein S M,et al.Unsupervised anomaly detection with generative adversarial networks to guide marker discovery[C]//International Conference on Information Processing in Medical Imaging.Berlin:Springer,2017:146-157.
[24]Baumgartner C F,Koch L M,Tezcan K C,et al.Visual feature attribution using Wasserstein GANs[J/OL].(2018-06-26)[2018-10-24].https://arxiv.org/pdf/1711.08998.pdf.