基于插件技术的网络数据采集器的研究与实现
摘要
基于插件技术的网络数据采集器是信息安全综合管理平台的关键部分之一,在其中担任日志采集工作,负责从网络安全设备中采集日志,然后将日志格式化为统一的格式,最后发送至数据库。网络数据采集器采用的是日志归一化技术,即将日志中不同内容描述方法、不同字段顺序按照事先制定的标准格式来转换。由于企业中的网络安全设备(IDS、防火墙、路由器、交换机等)往往来自不同的厂商,各自为政,相互之间缺乏关联,这些设备的日志又都基于厂商自己定义的格式,如果不对这些日志进行归一化处理,将难以进行关联分析,使得网络安全管理工作事倍功半。
本课题针对现有技术中数据采集技术存在的一些问题,提出了一种改进方案,主要研究内容包括：
1)在现有技术中,每格式化一份日志都需要经过查找判断过程,这在有大量日志产生的网络环境中,会影响系统的执行效率。本课题设计了一种改进架构,将插件与设备和端口进行绑定,使得无需经过查找判断过程便可进行日志格式化,从而提高了系统架构的执行效率。
2)在现有技术中,数据采集器在遇到设备日志类型发生改变或者加入了新的设备时,会导致查找判断过程失败,从而丢弃当前日志。本课题设计了自动更新模块,在遇到上述情况时,能够自动从插件库下载对应的插件继续完成日志格式化,避免了重要安全信息的丢失。
3)使用PHP、MySql Perl对设计的数据采集器进行开发。开发的内容包括插件、自动更新程序、插件调用程序、后台管理界面。
基于本课题自己定义的标准格式,输入数据采集器的是原始的设备日志,从数据采集器输出的是格式统一的日志,并作为信息安全综合管理平台的数据源。虽然本课题设计的基于插件技术的网络数据采集器是信息安全综合管理平台中的一部分,但也可用于其他的数据采集环境。
The data acquisition device based on plug-in technology belongs to one of the important part of the information security integrated management platform and works as a log collector which is responsible for data acquisition from network security devices and log format then sends the unified format logs to the central database. The data acquisition device uses a log format technique by which the different ways of content representation and different order of fields in logs are translated according to the pre-established stander format. Currently varieties of network security devices(IDS, firewalls, routers, switches, etc.) in enterprises may come from different manufacturers. Therefore the devices deal with the problems in their own way but lack of correlation among them. In addition the logs from them are based on the format defined by manufacturers. Without formatting these logs security experts will analyze security incidents doubly difficult. Accordingly network security management gets half the result with twice the effort.
Aimming at some problems in current data acquisition device, this project presents an improved scheme which are listed in the aspects below:
1) In the existing technology each log format has to go through to find the process to judge which will affect the efficiency of the system implementation when a large number of logs arise. This project presents an improved scheme in which log format can run without the judging process by the bind of the plug and the device and port. Accordingly the efficiency of the system implementation is improved.
2) When log type changed or new equipment added, the judging process will fail and discard the current log. This project designs a automatic update module which can automatically download the corresponding plug-in from library to complete the log format in the face of the above which avoids the loss of important information.
3) Developing the data acquisition device including plug-in, automatic updates module, plug-in called procedures, admin interface by PHP, Mysql and Perl.
Based on the standard format defined by the project, the input into the data acquisition device is the original log and the output from the data collector is a unified log format, which work as the data source in information security integrated management platform. Although the data acquisition device based on plug-in technology belongs to information security integrated management platform, but can also be used for other data acquisition environment.
本课题针对现有技术中数据采集技术存在的一些问题,提出了一种改进方案,主要研究内容包括：
1)在现有技术中,每格式化一份日志都需要经过查找判断过程,这在有大量日志产生的网络环境中,会影响系统的执行效率。本课题设计了一种改进架构,将插件与设备和端口进行绑定,使得无需经过查找判断过程便可进行日志格式化,从而提高了系统架构的执行效率。
2)在现有技术中,数据采集器在遇到设备日志类型发生改变或者加入了新的设备时,会导致查找判断过程失败,从而丢弃当前日志。本课题设计了自动更新模块,在遇到上述情况时,能够自动从插件库下载对应的插件继续完成日志格式化,避免了重要安全信息的丢失。
3)使用PHP、MySql Perl对设计的数据采集器进行开发。开发的内容包括插件、自动更新程序、插件调用程序、后台管理界面。
基于本课题自己定义的标准格式,输入数据采集器的是原始的设备日志,从数据采集器输出的是格式统一的日志,并作为信息安全综合管理平台的数据源。虽然本课题设计的基于插件技术的网络数据采集器是信息安全综合管理平台中的一部分,但也可用于其他的数据采集环境。
The data acquisition device based on plug-in technology belongs to one of the important part of the information security integrated management platform and works as a log collector which is responsible for data acquisition from network security devices and log format then sends the unified format logs to the central database. The data acquisition device uses a log format technique by which the different ways of content representation and different order of fields in logs are translated according to the pre-established stander format. Currently varieties of network security devices(IDS, firewalls, routers, switches, etc.) in enterprises may come from different manufacturers. Therefore the devices deal with the problems in their own way but lack of correlation among them. In addition the logs from them are based on the format defined by manufacturers. Without formatting these logs security experts will analyze security incidents doubly difficult. Accordingly network security management gets half the result with twice the effort.
Aimming at some problems in current data acquisition device, this project presents an improved scheme which are listed in the aspects below:
1) In the existing technology each log format has to go through to find the process to judge which will affect the efficiency of the system implementation when a large number of logs arise. This project presents an improved scheme in which log format can run without the judging process by the bind of the plug and the device and port. Accordingly the efficiency of the system implementation is improved.
2) When log type changed or new equipment added, the judging process will fail and discard the current log. This project designs a automatic update module which can automatically download the corresponding plug-in from library to complete the log format in the face of the above which avoids the loss of important information.
3) Developing the data acquisition device including plug-in, automatic updates module, plug-in called procedures, admin interface by PHP, Mysql and Perl.
Based on the standard format defined by the project, the input into the data acquisition device is the original log and the output from the data collector is a unified log format, which work as the data source in information security integrated management platform. Although the data acquisition device based on plug-in technology belongs to information security integrated management platform, but can also be used for other data acquisition environment.
引文
[1]网御神州.安全管理平台(SOC)的发展趋势分析.[2010-05-05].http://www.legendsec.com/view.php?cid=10&tid=517&up=7
[2]无线移动环境下信息安全综合管理项目资金申请报告.华为技术有限公司.2008
[3]安氏中国.安氏中国发展历程.[2010-05-05]. http://bj.is-one.net/aboutus/intro/
[4]安氏中国.安氏安全管理中心.[2010-05-05]. http://bj.is-one.net/product/soc/
[5]天融信.天融信网络卫士安全管理系统.[2010-05-05].http://www.topsec.com.cn/aqli.php?id=141
[6]格尔软件.格尔信息安全综合监控与管理平台.[2010-05-05].http://www.koal.com/products/products product others02.htm
[7]ArcSight. ArcSight Loggger-Log Management. [2009-12-02] http://www.arcsight.com/products/products-logger/
[8]徐君.安全管理中心系统中的日志格式化单元及方法.中国,发明专利,200610036617.4.2007-01-03
[9]王海靖.设备日志实时解析的方法、装置和系统.中国,发明专利,200810007607.7.2008-08-06
[10]缪峰,张坤.基于LDAP的分布式网络安全管理平台系统.计算机工程与设计,2009,(7)：1605-1607
[11]彭琪.统一网络安全管理系统中数据采集关键技术的研究：[硕士学位论文].武汉：华中师范大学,2008
[12]罗睿.统一网络安全管理平台关键技术的研究：[硕士学位论文].武汉：华中师范大学,2007
[13]范宝峰.统一网络安全管理平台技术研究：[硕士学位论文].四川：四川大学,2005
[14]薛静锋,曹元大.基于XMLRPC的网络安全管理平台的设计与实现.计算机应用,2005,25(5)：1130-1132.
[15]冯新勇.基于XML的网络安全日志管理与系统优化配置：[硕士学位论文].四川：四川大学,2004.5
[16]RFC4765. The Intrusion Detection Message Exchange Format (IDMEF). http://www.ietf.org/rfc/rfc4765.txt
[17]无线移动信息安全综合管理项目验收总结报告.华为技术有限公司.2009.10
[18]Valdes A, Skinner K. Probabilistic Alert Correlation. Proceedings of the 4th international Symposium on Recent Advances in Intrusion Detection(RAID 2001). 2001:54-68
[19]Cuppens and Miege 2002 CUPPENS, F. AND MIEGE, A.2002. Alert correlation in a cooperative intrusion detection framework. In: Proceedings of the 2002 IEEE Symposium on Security and Privacy,2002. security Education. Printing House,2004. 84-91.
[20]Steven J, Templeton,Karl Levitt. A Requires/Provides Model for Computer Attacks. New Security Paradigm Workshop 9/00 Ballycotton, Co. Cork, Ireland.2001 ACM ISBN 1-58113-260-3101/0002
[21]NING P, CUIY, REEVES DS. Analyzing intensive intrusion alerts via correlation. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection. Zurich, Switzerland,2002.
[22]胡文,黄皓.自动入侵响应技术研究.计算机工程,2009,31(18)：143-145.
[23]晏丰.基于风险的入侵响应决策技术研究：[硕士学位论文].北京：北京交通大学,2006.
[24]郭代飞,杨义先,胡正明.基于大规模网络的白适应入侵响应模型研究.北京邮电大学学报,2004,27(1).
[25]刘合富.基于syslog技术的防火墙日志数据采集方法的研究：[硕士学位论文].武汉：华中师范大学,2007
[26]RFC3195. Reliable Delivery for syslog. http://www.ietf.org/rfc/rfc3195.txt
[27]Matos VD, Silva JLDCE, etc. Design and implementation of an ANTS-based test bed for collecting data in active framework. Lcture Notes in Computer Science(LNCS)3124, 2004:251-256.
[28]RFC2570. Introduction to Version 3 of the Internet-standard Network Management Framework, http://www.ietf.org/rfc/rfc2570.txt
[29]肖诗松,陈涛.基于插件技术的日志采集Agent系统的设计与实现.东南大学学报(自然科学版),2008,(38)：90-93.
[30]SNMP协议.http://baike.baidu.com/view/21938.htm
[31]The CEE Board. Common Event Expression. MITRE,2008.
[32]Michael Rash. Linux防火墙.北京：人民邮电出版社,2009.
[33]Snort. Snort Users Manual. http://www.snort.org/docs
[34]Perl. Perl Programming Documentation. http://perldoc.perl.org/perl.html#NAME
[35]MySql. MySQL 5.5 Reference Manual. http://dev.mysql.com/doc/refman/5.5/en/
[36]Qu HP, Li DQ, Su PR, Feng DG.. An IP trace back scheme with packet marking in blocks. Journal of Computer Research and Development,2005,42(12):2084-2092(in Chinese with English abstract).
[37]韩锐生,赵彬,徐开勇.基于策略的一体化网络安全管理系统.计算机工程,2009,35(8)：201-204.
[38]王锡强.基于日志的网络安全审计系统的设计与实现：[硕士学位论文].山东：山东大学,2007
[39]周琪锋.基于网络日志的安全审计系统的研究设计.计算机技术与发展,2009,19(11)：139-142.
[40]刘必雄,魏连,许榕生.基于Agent技术的多源日志采集系统的设计与实现.计算机系统应用,2008,(2)：71-74.
[41]倪俊,陈晓苏,吴金华.基于域的策略网管中安全管理模型的研究.计算机工程与设计,2009,(7)：1586-1589.
[42]单智勇,石文昌.多级分布式网络安全管理系统的体系结构.计算机工程与设计,2007,28(14)：3316-3320.
[43]KAREN KENT, MURUGIAH SOUPPAYA. Guide to computer security log management. USA:National Institute of Standards and Technology,2006.
[2]无线移动环境下信息安全综合管理项目资金申请报告.华为技术有限公司.2008
[3]安氏中国.安氏中国发展历程.[2010-05-05]. http://bj.is-one.net/aboutus/intro/
[4]安氏中国.安氏安全管理中心.[2010-05-05]. http://bj.is-one.net/product/soc/
[5]天融信.天融信网络卫士安全管理系统.[2010-05-05].http://www.topsec.com.cn/aqli.php?id=141
[6]格尔软件.格尔信息安全综合监控与管理平台.[2010-05-05].http://www.koal.com/products/products product others02.htm
[7]ArcSight. ArcSight Loggger-Log Management. [2009-12-02] http://www.arcsight.com/products/products-logger/
[8]徐君.安全管理中心系统中的日志格式化单元及方法.中国,发明专利,200610036617.4.2007-01-03
[9]王海靖.设备日志实时解析的方法、装置和系统.中国,发明专利,200810007607.7.2008-08-06
[10]缪峰,张坤.基于LDAP的分布式网络安全管理平台系统.计算机工程与设计,2009,(7)：1605-1607
[11]彭琪.统一网络安全管理系统中数据采集关键技术的研究：[硕士学位论文].武汉：华中师范大学,2008
[12]罗睿.统一网络安全管理平台关键技术的研究：[硕士学位论文].武汉：华中师范大学,2007
[13]范宝峰.统一网络安全管理平台技术研究：[硕士学位论文].四川：四川大学,2005
[14]薛静锋,曹元大.基于XMLRPC的网络安全管理平台的设计与实现.计算机应用,2005,25(5)：1130-1132.
[15]冯新勇.基于XML的网络安全日志管理与系统优化配置：[硕士学位论文].四川：四川大学,2004.5
[16]RFC4765. The Intrusion Detection Message Exchange Format (IDMEF). http://www.ietf.org/rfc/rfc4765.txt
[17]无线移动信息安全综合管理项目验收总结报告.华为技术有限公司.2009.10
[18]Valdes A, Skinner K. Probabilistic Alert Correlation. Proceedings of the 4th international Symposium on Recent Advances in Intrusion Detection(RAID 2001). 2001:54-68
[19]Cuppens and Miege 2002 CUPPENS, F. AND MIEGE, A.2002. Alert correlation in a cooperative intrusion detection framework. In: Proceedings of the 2002 IEEE Symposium on Security and Privacy,2002. security Education. Printing House,2004. 84-91.
[20]Steven J, Templeton,Karl Levitt. A Requires/Provides Model for Computer Attacks. New Security Paradigm Workshop 9/00 Ballycotton, Co. Cork, Ireland.2001 ACM ISBN 1-58113-260-3101/0002
[21]NING P, CUIY, REEVES DS. Analyzing intensive intrusion alerts via correlation. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection. Zurich, Switzerland,2002.
[22]胡文,黄皓.自动入侵响应技术研究.计算机工程,2009,31(18)：143-145.
[23]晏丰.基于风险的入侵响应决策技术研究：[硕士学位论文].北京：北京交通大学,2006.
[24]郭代飞,杨义先,胡正明.基于大规模网络的白适应入侵响应模型研究.北京邮电大学学报,2004,27(1).
[25]刘合富.基于syslog技术的防火墙日志数据采集方法的研究：[硕士学位论文].武汉：华中师范大学,2007
[26]RFC3195. Reliable Delivery for syslog. http://www.ietf.org/rfc/rfc3195.txt
[27]Matos VD, Silva JLDCE, etc. Design and implementation of an ANTS-based test bed for collecting data in active framework. Lcture Notes in Computer Science(LNCS)3124, 2004:251-256.
[28]RFC2570. Introduction to Version 3 of the Internet-standard Network Management Framework, http://www.ietf.org/rfc/rfc2570.txt
[29]肖诗松,陈涛.基于插件技术的日志采集Agent系统的设计与实现.东南大学学报(自然科学版),2008,(38)：90-93.
[30]SNMP协议.http://baike.baidu.com/view/21938.htm
[31]The CEE Board. Common Event Expression. MITRE,2008.
[32]Michael Rash. Linux防火墙.北京：人民邮电出版社,2009.
[33]Snort. Snort Users Manual. http://www.snort.org/docs
[34]Perl. Perl Programming Documentation. http://perldoc.perl.org/perl.html#NAME
[35]MySql. MySQL 5.5 Reference Manual. http://dev.mysql.com/doc/refman/5.5/en/
[36]Qu HP, Li DQ, Su PR, Feng DG.. An IP trace back scheme with packet marking in blocks. Journal of Computer Research and Development,2005,42(12):2084-2092(in Chinese with English abstract).
[37]韩锐生,赵彬,徐开勇.基于策略的一体化网络安全管理系统.计算机工程,2009,35(8)：201-204.
[38]王锡强.基于日志的网络安全审计系统的设计与实现：[硕士学位论文].山东：山东大学,2007
[39]周琪锋.基于网络日志的安全审计系统的研究设计.计算机技术与发展,2009,19(11)：139-142.
[40]刘必雄,魏连,许榕生.基于Agent技术的多源日志采集系统的设计与实现.计算机系统应用,2008,(2)：71-74.
[41]倪俊,陈晓苏,吴金华.基于域的策略网管中安全管理模型的研究.计算机工程与设计,2009,(7)：1586-1589.
[42]单智勇,石文昌.多级分布式网络安全管理系统的体系结构.计算机工程与设计,2007,28(14)：3316-3320.
[43]KAREN KENT, MURUGIAH SOUPPAYA. Guide to computer security log management. USA:National Institute of Standards and Technology,2006.