通信管理网告警控制机制的研究应用
摘要
故障管理是电信管理网中最基本的五大管理功能之一。现在的电信网络中,网络设备的类型繁多,多种网管系统并存,使大数据量告警产生问题成为了影响网管性能和系统稳定性的关键。
本文研究的目的是为了解决告警量巨大,对无关和冗余的告警进行压缩、抑制和过滤,以提高故障管理的性能。通过对电信网管中的告警进行研究,对各种情况进行分析,并针对各种不同情况进行告警的控制和过滤。其中详细阐述了告警关联的必要性,比较了国内外现有的告警关联分析方法后,通过分析典型的Apriori算法,将数据挖掘技术应用到电信告警中来,并设计构建了告警关联知识库。
最后将此告警模块通过模拟真实环境测试,通过测试,上报到服务器的数据只有原始数据的30%,告警量比没有处理以前压缩了70%,效果比较满意。
本课题从实际项目需要出发,基于市场需求,应用数据挖掘技术,采用目前居于主流的基于规则的方法解决了电信网中最急需解决的告警相关性分析问题。通过对告警关联、告警震荡、告警闪断、告警风暴方面对告警进行控制和过滤,达到压缩或删除无关和冗余告警的目的。
Today, with the booming development of telecommunication industry, the telecommunication network management technology tends to be comprehensive, distributed and intelligent. As a prevailing international standard architecture of telecommunication network, Telecommunications Management Network (TMN) has been widely adopted by more and more network management systems. TMN is presented by ITU-T measuring up the M.3010 series standard. It partly inherits some ideas of the OSI, such as the concepts of Manager and Agent, reflects the idea of the separation between the management network and the managed network, and the idea of hierarchical management, and also realizes the effective management on fault, configuration, performance, safety and account.
Fault management, one of the most basic functions of network management, is to manage network condition and the exception of TMN operation and device installation environment. It can be divided into fault measuring, fault diagnosis and fault recovery. The big amount of alarm data is the key problem to affect network management performance and stability of the whole system. If alarm is filtered by one single way, thus the quantity of alarm can’t be controlled and fault can’t be fixed, there is a hidden danger to come to bottleneck of the performance. So, in order to avoid a large number of repeated, redundant and related alarm data were stored into the database and sent to the client under the special circumstances and unusual situation, through analyzing several kinds of typical special states of network equipments, compressing or deleting the alarm, and improving the performance of fault management module and client, the goal of rectifying and improving and optimizing fault management module could be achieved by controlling the alarm. This study has following purposes:
l Because the domestic research and development of thecomprehensive network management of TMN start in recent years, this study is a pioneer work.
l The related rule in the Data Mining finds out that this technology is employed in analyzing history alarm data, automatically establishing alarm related analysis rule, applying the related rule to active alarm and thus realizing the intellectuality of the related rule module. This technology is innovative to some extent.
l Alarm storm, alarm vibration, and alarm flash, which take place often in the fault management, are discussed and the effective solution is brought out to fast find, fix and deal with fault accurately, and can improve efficiency and performance of the module, enhance service quality. Therefore, this study has very strong use value.
This dissertation first carries out a detailed analysis on the communication management network and fault management. This dissertation first carries out a detailed analysis on the communication management network and fault management. The fault management is responsible for gathering various kinds of device alarms and reports of network issues caused by various kinds of network equipment (NE) in the range of data network. Then the fault management filters faults according to the time. Then the dissertation analyzes the definition of alarm and the characteristics of data in the fault management, and states the content of general fault management and business treatment procedure.
Then the dissertation studies three alarm special circumstances in the network management: alarm storm, alarm vibration, and alarm flash. Through the description to phenomenon of these alarm special circumstances, the author analyzes the reasons of each circumstance, puts forward the solutions to each alarm special circumstance, structure the alarm controlling model with various functions.
Moreover, the author also introduces technology, procedure andcommon methods of data mining. Through using the data mining to analyze the amount of alarm data, the author does the research on the inner rules and characters of professional alarm data, analyzes the output alarm related rules and applies these rules to bring the data mining in the alarm control. The basic procedures of data mining and the basic mining concepts of the related rule are introduced in the dissertation. The basic procedures of alarm related rule mining are brought after the difficulties of alarm related rule mining are analyzed by the author.
Finally,based on the above analysis, the author proposes an alarm controlling mechanism. The function of this mechanism mainly includes two aspects: first, it can realize the control to alarm characteristics. The detailed illustration and procedure graphs of the dealing methods to alarm vibration, alarm storm, and alarm flash are put forth in the dissertation. Second, it can realize the related alarm control, including the way to use experts’experience to analyze the root alarm and correlative alarm, the way to shield correlative alarm, and the way to use the classic data mining algorithm Apriori to realize the related rule. An example is used to illustrate the mining procedure of alarm related rule.
The main innovation of the research is embodied by the following:
l Analysis to various kinds of characteristics of alarm is carried out and solution to these characteristics is put forth by the author.
l The alarm controlling model is defined through analyzing the characteristics of alarm data, the characteristics of alarm and the relationship among alarms.
l Through analyzing classical Apriori algorithm, the author applies it to telecommunication fault management.
l The author designs and structures an alarm related rule database, and uses these rules in the practical way.
Due to the short time of this research, there are some unavoidable
disadvantages:
l The bottleneck performance question of Apriori algorithm turns to be scanning the database several times and produces a large number of candidate items, and expenses a lot in time and space while the related rule is mined in the amount of alarms.
l The enlargement of the alarm related rule database asks for the further improvement of the maintenance of the alarm related rule database.
l Within the alarm related rules, some repeated and invalid related rules, need to be analyzed, judged and dealed with.
本文研究的目的是为了解决告警量巨大,对无关和冗余的告警进行压缩、抑制和过滤,以提高故障管理的性能。通过对电信网管中的告警进行研究,对各种情况进行分析,并针对各种不同情况进行告警的控制和过滤。其中详细阐述了告警关联的必要性,比较了国内外现有的告警关联分析方法后,通过分析典型的Apriori算法,将数据挖掘技术应用到电信告警中来,并设计构建了告警关联知识库。
最后将此告警模块通过模拟真实环境测试,通过测试,上报到服务器的数据只有原始数据的30%,告警量比没有处理以前压缩了70%,效果比较满意。
本课题从实际项目需要出发,基于市场需求,应用数据挖掘技术,采用目前居于主流的基于规则的方法解决了电信网中最急需解决的告警相关性分析问题。通过对告警关联、告警震荡、告警闪断、告警风暴方面对告警进行控制和过滤,达到压缩或删除无关和冗余告警的目的。
Today, with the booming development of telecommunication industry, the telecommunication network management technology tends to be comprehensive, distributed and intelligent. As a prevailing international standard architecture of telecommunication network, Telecommunications Management Network (TMN) has been widely adopted by more and more network management systems. TMN is presented by ITU-T measuring up the M.3010 series standard. It partly inherits some ideas of the OSI, such as the concepts of Manager and Agent, reflects the idea of the separation between the management network and the managed network, and the idea of hierarchical management, and also realizes the effective management on fault, configuration, performance, safety and account.
Fault management, one of the most basic functions of network management, is to manage network condition and the exception of TMN operation and device installation environment. It can be divided into fault measuring, fault diagnosis and fault recovery. The big amount of alarm data is the key problem to affect network management performance and stability of the whole system. If alarm is filtered by one single way, thus the quantity of alarm can’t be controlled and fault can’t be fixed, there is a hidden danger to come to bottleneck of the performance. So, in order to avoid a large number of repeated, redundant and related alarm data were stored into the database and sent to the client under the special circumstances and unusual situation, through analyzing several kinds of typical special states of network equipments, compressing or deleting the alarm, and improving the performance of fault management module and client, the goal of rectifying and improving and optimizing fault management module could be achieved by controlling the alarm. This study has following purposes:
l Because the domestic research and development of thecomprehensive network management of TMN start in recent years, this study is a pioneer work.
l The related rule in the Data Mining finds out that this technology is employed in analyzing history alarm data, automatically establishing alarm related analysis rule, applying the related rule to active alarm and thus realizing the intellectuality of the related rule module. This technology is innovative to some extent.
l Alarm storm, alarm vibration, and alarm flash, which take place often in the fault management, are discussed and the effective solution is brought out to fast find, fix and deal with fault accurately, and can improve efficiency and performance of the module, enhance service quality. Therefore, this study has very strong use value.
This dissertation first carries out a detailed analysis on the communication management network and fault management. This dissertation first carries out a detailed analysis on the communication management network and fault management. The fault management is responsible for gathering various kinds of device alarms and reports of network issues caused by various kinds of network equipment (NE) in the range of data network. Then the fault management filters faults according to the time. Then the dissertation analyzes the definition of alarm and the characteristics of data in the fault management, and states the content of general fault management and business treatment procedure.
Then the dissertation studies three alarm special circumstances in the network management: alarm storm, alarm vibration, and alarm flash. Through the description to phenomenon of these alarm special circumstances, the author analyzes the reasons of each circumstance, puts forward the solutions to each alarm special circumstance, structure the alarm controlling model with various functions.
Moreover, the author also introduces technology, procedure andcommon methods of data mining. Through using the data mining to analyze the amount of alarm data, the author does the research on the inner rules and characters of professional alarm data, analyzes the output alarm related rules and applies these rules to bring the data mining in the alarm control. The basic procedures of data mining and the basic mining concepts of the related rule are introduced in the dissertation. The basic procedures of alarm related rule mining are brought after the difficulties of alarm related rule mining are analyzed by the author.
Finally,based on the above analysis, the author proposes an alarm controlling mechanism. The function of this mechanism mainly includes two aspects: first, it can realize the control to alarm characteristics. The detailed illustration and procedure graphs of the dealing methods to alarm vibration, alarm storm, and alarm flash are put forth in the dissertation. Second, it can realize the related alarm control, including the way to use experts’experience to analyze the root alarm and correlative alarm, the way to shield correlative alarm, and the way to use the classic data mining algorithm Apriori to realize the related rule. An example is used to illustrate the mining procedure of alarm related rule.
The main innovation of the research is embodied by the following:
l Analysis to various kinds of characteristics of alarm is carried out and solution to these characteristics is put forth by the author.
l The alarm controlling model is defined through analyzing the characteristics of alarm data, the characteristics of alarm and the relationship among alarms.
l Through analyzing classical Apriori algorithm, the author applies it to telecommunication fault management.
l The author designs and structures an alarm related rule database, and uses these rules in the practical way.
Due to the short time of this research, there are some unavoidable
disadvantages:
l The bottleneck performance question of Apriori algorithm turns to be scanning the database several times and produces a large number of candidate items, and expenses a lot in time and space while the related rule is mined in the amount of alarms.
l The enlargement of the alarm related rule database asks for the further improvement of the maintenance of the alarm related rule database.
l Within the alarm related rules, some repeated and invalid related rules, need to be analyzed, judged and dealed with.
引文
[1] Xiao Debao, Li Birong, Chen Chunhong, Event-driven Domain Managersfor Open Management Environments, Chinese Journal of Advanced Software Research, Vol. 6, No. 4, 1999 P392-401
[2] A. Dupuy, S. Sengupta and O. Wolfson, A Network Management Environment, IEEE Network: special issue on network operations and management, February 1991,P148-174
[3] Ledri, Network Management for the 90s, Proceedings of ECOOP '9E Workshop on Systems and Network Management, July 1996
[4] 孔令萍,李建国.电信管理网.人民邮电出版社,1997
[5] 夏海涛,詹志强.新一代网络管理技术.北京邮电大学出版社,2003
[6] ITU-T Rec.M.30I00:GeneriC Network Information Model [S].1992
[7] ITU-T Rec.M.3400:TNN management functions[S].2001
[8] ITU-T Rec. M.3100:Generic network information model. 2000
[9] ITU-T Rec. M.3200:TMN management services and communications managed areas:overview. 2001
[10] 毛京丽.现代通信网[M].北京邮电大学出版社,2002
[11] 唐纯贞,严建民,鲁碧英.电信网与电信业务〔M],人民邮电出版社,2003
[12] ITU-T Rec. M.3010:Principles for a telecommunication management network. 2000
[13] Malheiros Maria D , Marcos Silva Nogueira J. Modeling a telecommunication network for fault management applications Network Operations and Management Symposium, NOMS 1998,3
[14] 彭东斌,汪纪锋.智能化的电信网故障管理技术研究,数字通信.2000 (3); 20-23
[15] 郑庆国.吕卫峰.诵信网络中的告誉相关性研究(TI.计算和工程与应用.2002.
[16] 中国电信集团公司部门文件.本地通信网集中告警系统功能需求规范V1[1].10[S],2005
[17] 杨家海、任宪坤、王沛瑜,网络管理原理与实现技术第一版,清华大学出版社,2000年9月
[18] 刘康平,李增智.网络告警序列中的频繁情景规则挖掘算法闭.小型微型计算机系统,2003
[19] M. Chen et al. Data mining: An overview from a database perspective. IEEE Trans. On Knowledge and Data Engineering. 1996, Vo1.8: 866-883
[20] 李建东,信息网络理论基础[M]西安电子科技大学出版社,2001
[21] Jacobson G, Weissman M D,Alarm Correlation, IEEE Network, 1999,7(6):52-59
[22] Liu G, Mok A K, Yang E J, Composite events for network event correlation Proceedings of IEEE IM' 99. Boston. MA. USA. 1999. 247- 260
[23] U.Fayyad&G. Piatesky-Shapiro,P. Smyth. From data mining to knowledge discover: An overview.ACM KDD,1994
[24] Jiawei Han Micheline Kamber.数据挖掘概念与技术[M].机械工业出版社,2001
[25] 王运峰.数据库中关联规则的并行挖掘算法[M].北京:计算机工程与应用.2001 .37(16)
[26] Interpol Bandar. Data Mining and Knowledge Discovery in NBA Data. DateMining and Knowledge Discovery. Boston:Kluwer Academic Publishers,1999
[27] 王小虎.关联规则挖掘综述[D].计算机工程与应用,2003 33: 190-193
[28] RAgrawal and R-Srikant. Fast algorithm for mining association rules.In Proc.1994 Int.Conf. Very Large Data Base, pp. 487-499, Santiago, Chile, Sept.,1994
[29] R Agrawal,T.Imielinski, A. N. Swamy. Mining association noesbetween sets of items in large databases. In Proceedings of 1993 ACM SIGMOD International Conference on Management of Data, pages 207-216. Washington, D.C. USA, 1993.
[30] J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufrnanr Publishers- August 2000.
[31] 温磊,基于有项集图的关联规则挖掘算法研究与应用[博士论文],天津大学,2002,11.
[32] J. Han, M. Kamber 著.范明,孟小峰译.轰畴嗜掘楞忿与岌术机械工业出版社2001.8.
[33] R Srikant and R Agrawal. Mining quantitative association rules in large relational tables. In Proceedings of 1996 ACM-SIGMOD International Conference on Management of Data (SIGMOD95), pages1-12, Montreal, Canada, June 1996.
[34] 苑森森,程晓青.数量关联规则发现中的聚类方法研究。计算机学报2000, 23(8) 866-871.
[35] T.Fukuda, Y Morimoto, S. Morishita, and T Tokuyama. Mining optimized association rules for numeric attributes. In Proceedings of the 15th ACM SIGACTSIGMOD-SIGART Symp on Principles of Database Systems (PODS96), pages 182-191,Montreal,Canada, June 1996, ACM Press.
[36] C.M. Kuol- A. Fu- and M.H. Wone. Fuzzv association niles in late databases with quantitative attributes. In ACMSIGMOD Records, March, 1998
[37] 段云峰李剑威基于数量的关联规则挖掘.北京邮电大学学报2002,25(4). 56-60
[38] 陈富赞,寇纪淞等.基于网络的数值关联规则挖掘方法.系统工程与理论2002, 22(4).1-9.
[39] J. Han and Fu. Discovery of multiple-level association rules from large databases. In Proceedings of VLDB-95, pages 420-431, Zurich,Switzerland, September 1995.
[40] 程继华,施鹏飞.快速多层次关联规则的挖掘.计算机学报1998,21(11).1037-1041.
[41] 范明,牛常勇等.一种挖掘多维关联规则的有效算法.; 计算机科学2001,28(11).44-47.
[42] 杨学民蔡庆生.基于数据立方体的维内关联规则挖掘算法.北京科技大学学报2003,25(1).83-86.
[43] Frank, P M et al. New developments using Al in fault diagnosis[J] .Engineering Application of Artificial Intelligence ,1997;10(1);3-14
[44] H. Mannila, H. Toivonen, and A. 1. Verkamo. Efficient algorithms for discovering association rules. In Proceedings of AAAI'94 Workshop Knowledge Discovery in Databases (KDD'94), pages 181-192, Seattle, WA, July 1994.
[45] Jakobson G.,Weissman M. Alarm Cornelation[J]. IEEE Network, 1993
[46] 张岗山、李建东、吴宇红,基于 SNMP 故障管理中的告警振荡和抵消算法,北京邮电大学学报Vol.27 31-35, 2004.12
[47] 戴华,矩阵论,科学出版社.2001 年:139-142 页.
[48] 范明.数据挖掘概念与技术「川.北京:机械工业出版社,200 1. 8: 15.149-175,304
[2] A. Dupuy, S. Sengupta and O. Wolfson, A Network Management Environment, IEEE Network: special issue on network operations and management, February 1991,P148-174
[3] Ledri, Network Management for the 90s, Proceedings of ECOOP '9E Workshop on Systems and Network Management, July 1996
[4] 孔令萍,李建国.电信管理网.人民邮电出版社,1997
[5] 夏海涛,詹志强.新一代网络管理技术.北京邮电大学出版社,2003
[6] ITU-T Rec.M.30I00:GeneriC Network Information Model [S].1992
[7] ITU-T Rec.M.3400:TNN management functions[S].2001
[8] ITU-T Rec. M.3100:Generic network information model. 2000
[9] ITU-T Rec. M.3200:TMN management services and communications managed areas:overview. 2001
[10] 毛京丽.现代通信网[M].北京邮电大学出版社,2002
[11] 唐纯贞,严建民,鲁碧英.电信网与电信业务〔M],人民邮电出版社,2003
[12] ITU-T Rec. M.3010:Principles for a telecommunication management network. 2000
[13] Malheiros Maria D , Marcos Silva Nogueira J. Modeling a telecommunication network for fault management applications Network Operations and Management Symposium, NOMS 1998,3
[14] 彭东斌,汪纪锋.智能化的电信网故障管理技术研究,数字通信.2000 (3); 20-23
[15] 郑庆国.吕卫峰.诵信网络中的告誉相关性研究(TI.计算和工程与应用.2002.
[16] 中国电信集团公司部门文件.本地通信网集中告警系统功能需求规范V1[1].10[S],2005
[17] 杨家海、任宪坤、王沛瑜,网络管理原理与实现技术第一版,清华大学出版社,2000年9月
[18] 刘康平,李增智.网络告警序列中的频繁情景规则挖掘算法闭.小型微型计算机系统,2003
[19] M. Chen et al. Data mining: An overview from a database perspective. IEEE Trans. On Knowledge and Data Engineering. 1996, Vo1.8: 866-883
[20] 李建东,信息网络理论基础[M]西安电子科技大学出版社,2001
[21] Jacobson G, Weissman M D,Alarm Correlation, IEEE Network, 1999,7(6):52-59
[22] Liu G, Mok A K, Yang E J, Composite events for network event correlation Proceedings of IEEE IM' 99. Boston. MA. USA. 1999. 247- 260
[23] U.Fayyad&G. Piatesky-Shapiro,P. Smyth. From data mining to knowledge discover: An overview.ACM KDD,1994
[24] Jiawei Han Micheline Kamber.数据挖掘概念与技术[M].机械工业出版社,2001
[25] 王运峰.数据库中关联规则的并行挖掘算法[M].北京:计算机工程与应用.2001 .37(16)
[26] Interpol Bandar. Data Mining and Knowledge Discovery in NBA Data. DateMining and Knowledge Discovery. Boston:Kluwer Academic Publishers,1999
[27] 王小虎.关联规则挖掘综述[D].计算机工程与应用,2003 33: 190-193
[28] RAgrawal and R-Srikant. Fast algorithm for mining association rules.In Proc.1994 Int.Conf. Very Large Data Base, pp. 487-499, Santiago, Chile, Sept.,1994
[29] R Agrawal,T.Imielinski, A. N. Swamy. Mining association noesbetween sets of items in large databases. In Proceedings of 1993 ACM SIGMOD International Conference on Management of Data, pages 207-216. Washington, D.C. USA, 1993.
[30] J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufrnanr Publishers- August 2000.
[31] 温磊,基于有项集图的关联规则挖掘算法研究与应用[博士论文],天津大学,2002,11.
[32] J. Han, M. Kamber 著.范明,孟小峰译.轰畴嗜掘楞忿与岌术机械工业出版社2001.8.
[33] R Srikant and R Agrawal. Mining quantitative association rules in large relational tables. In Proceedings of 1996 ACM-SIGMOD International Conference on Management of Data (SIGMOD95), pages1-12, Montreal, Canada, June 1996.
[34] 苑森森,程晓青.数量关联规则发现中的聚类方法研究。计算机学报2000, 23(8) 866-871.
[35] T.Fukuda, Y Morimoto, S. Morishita, and T Tokuyama. Mining optimized association rules for numeric attributes. In Proceedings of the 15th ACM SIGACTSIGMOD-SIGART Symp on Principles of Database Systems (PODS96), pages 182-191,Montreal,Canada, June 1996, ACM Press.
[36] C.M. Kuol- A. Fu- and M.H. Wone. Fuzzv association niles in late databases with quantitative attributes. In ACMSIGMOD Records, March, 1998
[37] 段云峰李剑威基于数量的关联规则挖掘.北京邮电大学学报2002,25(4). 56-60
[38] 陈富赞,寇纪淞等.基于网络的数值关联规则挖掘方法.系统工程与理论2002, 22(4).1-9.
[39] J. Han and Fu. Discovery of multiple-level association rules from large databases. In Proceedings of VLDB-95, pages 420-431, Zurich,Switzerland, September 1995.
[40] 程继华,施鹏飞.快速多层次关联规则的挖掘.计算机学报1998,21(11).1037-1041.
[41] 范明,牛常勇等.一种挖掘多维关联规则的有效算法.; 计算机科学2001,28(11).44-47.
[42] 杨学民蔡庆生.基于数据立方体的维内关联规则挖掘算法.北京科技大学学报2003,25(1).83-86.
[43] Frank, P M et al. New developments using Al in fault diagnosis[J] .Engineering Application of Artificial Intelligence ,1997;10(1);3-14
[44] H. Mannila, H. Toivonen, and A. 1. Verkamo. Efficient algorithms for discovering association rules. In Proceedings of AAAI'94 Workshop Knowledge Discovery in Databases (KDD'94), pages 181-192, Seattle, WA, July 1994.
[45] Jakobson G.,Weissman M. Alarm Cornelation[J]. IEEE Network, 1993
[46] 张岗山、李建东、吴宇红,基于 SNMP 故障管理中的告警振荡和抵消算法,北京邮电大学学报Vol.27 31-35, 2004.12
[47] 戴华,矩阵论,科学出版社.2001 年:139-142 页.
[48] 范明.数据挖掘概念与技术「川.北京:机械工业出版社,200 1. 8: 15.149-175,304