网络安全监测数据集成关键技术的研究与实现
|
摘要
互联网正在成为国家关键信息基础设施,事关国家和全社会的根本利益。随着互联网技术的飞速发展,针对网络信息系统的恶意攻击向着分布化、规模化、复杂化、间接化等趋势发展。因此迫切需要研究新的技术来对网络攻击进行主动的防护,而网络安全监控技术作为网络安全防护的重要手段之一,已成为当前研究的热点。本文在深入分析了相关的数据抽取与集成以及数据流技术的基础上,对网络安全监测中数据集成关键技术进行了深入研究,本文的主要工作概述如下:
1、针对网络安全监测中数据抽取过程比较复杂的问题,设计了一种基于正则表达式的数据抽取与集成转换方法,从而能够抽取各种安全探测工具中的网络攻击数据,并且支持异构探针的动态接入。
2、针对在对网络安全态势展示中数据处理出现的延迟性,设计了一种数据流与数据库的混合连接查询算法,对数据流进行预处理,加快了后续的查询速度,从而在网络安全态势展示中达到近实时的效果。
3、基于上述的研究基础,在网络安全态势分析与预测系统YH-SOC中实现了上述算法,并对以上技术的有效性进行了验证。
The Internet is a national critical information infrastructure is vital to state and the fundamental interests of society as a whole. With the rapid development of Internet technology for network information system towards the distribution of malicious attacks, large-scale, complex, indirect and so on trend. There is an urgent need to study new technologies to take the initiative to attack the network protection, and network security monitoring is the basic technology of network security protection, which has become a hotspot of current research. This paper in-depth analysis of the data extraction and integration techniques, and data flow techniques, and then we will in-depth research the key technologies of data integration of network security monitoring data, the major work of this paper are summarized as follows:
First, for network security monitoring data extraction process is complex, design of a data extraction and integration method which is based on regular expression. It can extract the data and complete the format conversion from a variety of security network attack detection tools, and it also supports dynamics access of the heterogeneous probe.
Second, targeted at the security situation on the network show delays in data processing, design a hybrid connection query algorithms between data flow and database, it can pre-process of data streams to speed up the follow-up of the query speed, resulting in to achieve near real-time display network security posture.
Third, based on the above research base, implementing the above algorithm in YH-SOC system which is a system to analysis and forecast of the network security situation, as well as the effectiveness of the above techniques has been verified.
1、针对网络安全监测中数据抽取过程比较复杂的问题,设计了一种基于正则表达式的数据抽取与集成转换方法,从而能够抽取各种安全探测工具中的网络攻击数据,并且支持异构探针的动态接入。
2、针对在对网络安全态势展示中数据处理出现的延迟性,设计了一种数据流与数据库的混合连接查询算法,对数据流进行预处理,加快了后续的查询速度,从而在网络安全态势展示中达到近实时的效果。
3、基于上述的研究基础,在网络安全态势分析与预测系统YH-SOC中实现了上述算法,并对以上技术的有效性进行了验证。
The Internet is a national critical information infrastructure is vital to state and the fundamental interests of society as a whole. With the rapid development of Internet technology for network information system towards the distribution of malicious attacks, large-scale, complex, indirect and so on trend. There is an urgent need to study new technologies to take the initiative to attack the network protection, and network security monitoring is the basic technology of network security protection, which has become a hotspot of current research. This paper in-depth analysis of the data extraction and integration techniques, and data flow techniques, and then we will in-depth research the key technologies of data integration of network security monitoring data, the major work of this paper are summarized as follows:
First, for network security monitoring data extraction process is complex, design of a data extraction and integration method which is based on regular expression. It can extract the data and complete the format conversion from a variety of security network attack detection tools, and it also supports dynamics access of the heterogeneous probe.
Second, targeted at the security situation on the network show delays in data processing, design a hybrid connection query algorithms between data flow and database, it can pre-process of data streams to speed up the follow-up of the query speed, resulting in to achieve near real-time display network security posture.
Third, based on the above research base, implementing the above algorithm in YH-SOC system which is a system to analysis and forecast of the network security situation, as well as the effectiveness of the above techniques has been verified.
引文
[1]应向荣.网络攻击新趋势下主动防御系统的重要性.计算机安全, 2003(7):53~55.
[2] Martin Roesch. Snort– Lightweight Intrusion Detection for Networks. Proceedings of LISA '99: 13th Systems Administration Conference. Seattle, Washington, USA, November 7–12, 1999:229~238.
[3] G. Hulten, L. Spencer, P. Domingos. Mining time—changing data streams.In: Proc of the International Conference on Knowledge Discovery and Data Mining (SIGKDD01). New York: ACM Press, 2001:97~106.
[4] H. Wang, W. Fan, P. S. Yu, J. Han. Mining concept—drifting data streams using ensemble classifiers. In: Proc of the lnt’l Conf on Knowledge Discovery and Data Mining (SIGKDD03). New York: ACM Press, 2003:226~35.
[5] Y. Kim, W. C. Lau, M. C. Chuah, H. J. Chao. PacketScore: A Statistics-Based Packet Filtering Scheme against Distributed Denial-of-Service Attacks.IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING. APRIL- JUNE 2006, 3(2):141~155.
[6] A. Lall, V. Sekar, M. Ogihara, et al. Data Streaming Algorithms for Estimating Entropy of Network Traffic. SIGMETRICS/Performance’06, Saint Malo, France. June, 26-30, 2006:145~156.
[7] Y. Chen, K. Hwang, W. S. Ku. Distributed Change-Point Detection of DDoS Attacks over Multiple Network Domains. IEEE Transaction on Parallel and Distributed Systems, 2007:7~14.
[8] M. Cai, K. Hwang, J. P. Pan, C. Papadopoulos. WormShield: Fast Worm Signature Generation with Distributed Fingerprint Aggregation. IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING. APRIL-JUNE, 2007, 4(2):88~104.
[9] R. Stone. Center track: An IP overlay network for tracking dos floods. In: Proc 9th Usenix Security Symp. Usenix Assoc, 2000:199~212.
[10] H. Burch, B. Cheswick. Tracing anonymous packets to their approximate source. In: Proc 14th Conf Systems Administration. Usenix Assoc, 2000:313~322.
[11] D. X. Song, A. Perrig. Advanced and authenticated marking scheme for IP traceback. In: Proceedings of 2001 IEEE INFOCOM Conference. California: ACM, 2001:878~886.
[12] A. C. Snoeren, C. Partridge, L. A. Sanchez, et al. Hash-based IP traceback. In: Proceedings of 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM). California: ACM, 2001:3~14.
[13] S. Song, C. N. Manikopoulos. A Control Theoretical Approach for Flow Control to Mitigate Bandwidth Attacks. In Proc of 2006 IEEE Information Assurance Workshop. 2006:348~360.
[14] P. Naldurg, R. H. Campbell. Dynamic Access Control: Preserving Safety and Trust for Network Defense Operations. In: Proc of the eighth ACM symposium on Access control models and technologies. ACM Press. June, 2003:231~237.
[15] H. J. Kim, J. C. Na, S. W. Sohn. Response to distributed denial-of-service attack using active technology. In: Proceedings of the Eighth IASTED International Conference on Internet and Multimedia Systems and Applications. Kauai: Acta Press. 2004:244~248.
[16] J. W. Freeman, T. C. Darr, R. B. Neely. Risk Assessment for Large Heterogeneous Systems. Proceedings of 13th Annual Computer Security Applications Conference, San Diego, CA, USA, 1997:44~52.
[17] Y. J. Han, J. S. Yang, B. H. Chang, et al. The Vulnerability Assessment for Active Networks; Model, Policy, Procedures, and Performance Evaluations. In: Proceedings of 2004 International Conference on Computational Science and Its Applications, Lecture Notes in Computer Science 3043, Springer Verlag, Assisi, Italy, 2004:191~198.
[18] S. Kamara, S. Fahmy, E. Schultz, et al. Analysis of Vulnerabilities in Internet Firewalls. Computers & Security. 2003, 22(3):214~232.
[19] R. Ortalo, Y. Deswarte, M. Kaaniche. Experimenting with quantitative evaluation tools for monitoring operational security. IEEE Trans. on Software Engineering, 1999, 25(5):633~651.
[20] B. Babcock, S. Babu, M. Datar, R. Motwani, J. Widom. Models and Issues in Data Streams. In Proc. ACM Symp. On Principles of Database Systems, 2002:1~16.
[21] Sirish. Chandrasekaran, Michael J. Franklin. Streaming Queries over Streaming Data. Proceedings of the 28th VLDB Conference, Hong Kong, China, 2002:203~214.
[22] Lukasz Golab, M. Tamer Ozsu. Processing Sliding Window Multi-Joins in Continuous Queries over Data Streams. Proceedings of the 29th VLDB Conference, Berlin, Germany, 2003:500~511.
[23] J. Kang, J. Naughton, S. Viglas. Evaluating Window Joins over Unbounded Streams. In Proc. 19th Int.Conf. on Data Engineering, 2003:341~352.
[24]金澈清.流数据分析与管理综述.软件学报, 2004,15(8):134~138.
[25] A. N. Wilschut, P. M. G. Apers. Dataflow Query Execution in a Parallel Main-memory Environment. Proc. of the 1st Int’l. Conf. on Parallel and Distributed Information Systems, Miami Beach, FL, 1991:68~77.
[26] T. Urhan, M. J. Franklin. XJoin: A Reactively-scheduled Pipelined Join Operator.IEEE Data Engineering Bulletin, 2000, 23(2):27~33.
[27] HAASPJ, J. M. HELLERSTEIN. Ripple Joins for Online Aggregation. SIGMOD Conference, 1999:287~298.
[28] S. VIGLAS, J. NAUGHTON. Rate-Based Query Optimization for Stream Information Source. In Proc. ACM Int .Conf. On Management of Data. 2002:37~48.
[29]钱江波,徐宏炳,王永利,刘学军,董逸生.多数据流滑动窗口并发连接方法.计算机研究与发展. 2005:1771~1778.
[30]刘学军,钱江波.分布式数据流连接查询算法.计算机工程. 2006.11:41~43.
[31]郭庆平.一种分布式数据流连接查询算法.武汉理工大学学报. 2009.2.29-32.
[32]王帅.异构数据转换与同步技术的研究与实现.学位论文.长沙.国防科学技术计算机学院.2005.
[33] http://ww.rpbourret.com/xmldbms/index.html
[34] M. F. Fernandez, W. C. Tan, D. Suciu. SilkRoute: Trading between Relations and XML. In Int’l World Wide Web Conf. (WWW),AmsTERDAM, Netherlands, May 2000:723~745.
[35] M. Carey, D. Florescu, Z. Ives, Y. Lu, J. Shanmugasundaram, E. Shekita, and S. Subramanian. XPERANTO: Publishing Object-Relational Data as XML. In Int’l Workshop on the web and Database (Web DB), Dallas, TX, May 2000:105~110.
[36] Josephine Cheng, Jane Xu. IBM DB2 XML Extender: An end-to-end solution for storing and retrieving XML documents. ICDE’00 Conference, San Diego, February 2000:128~136.
[37] Sirish Chandrasekaran, Owen Cooper, Amol Deshpande, Michael J. Franklin, Joseph M. Hellerstein. TelegraphCQ: Continuous Dataflow Processing for an uncertain world.CIDR Conference. Asilomar, CA, USA, January 5-8, 2003:269~280.
[38] Instruction Detection System [EB/OL]. http://en.wikipedia.org/wiki/Intrusion_detection_system,2009-11-17.
[39] Introduction of VScanner [EB/OL]. http://www.esrf.eu/UsersAndScience/Experiments/TBS/ISG/vscanner,2008-9-26.
[40] Introduction of ArpWatch [EB/OL]. http://en.wikipedia.org/wiki/Arpwatch,2009-11-15.
[41] The new p0f[EB/OL].http://lcamtuf.coredump.cx/p0f.shtml,2006-9-6.
[42] Regular Expression [EB/OL]. http://en.wikipedia.org/wiki/Regular_expression,2009-11-17.
[43]安天网络病毒监控系统[EB/OL].http://www.antiy.com/cn/product/vds.htm.
[44] Introduction of Firewall [EB/OL]. http://en.wikipedia.org/wiki/Firewall, 2009-11-17.
[45] L. Golab, M. T. Ozsu. Issues in Data stream management. SIGMOD Record, 2003, 32(2):5~14.
[2] Martin Roesch. Snort– Lightweight Intrusion Detection for Networks. Proceedings of LISA '99: 13th Systems Administration Conference. Seattle, Washington, USA, November 7–12, 1999:229~238.
[3] G. Hulten, L. Spencer, P. Domingos. Mining time—changing data streams.In: Proc of the International Conference on Knowledge Discovery and Data Mining (SIGKDD01). New York: ACM Press, 2001:97~106.
[4] H. Wang, W. Fan, P. S. Yu, J. Han. Mining concept—drifting data streams using ensemble classifiers. In: Proc of the lnt’l Conf on Knowledge Discovery and Data Mining (SIGKDD03). New York: ACM Press, 2003:226~35.
[5] Y. Kim, W. C. Lau, M. C. Chuah, H. J. Chao. PacketScore: A Statistics-Based Packet Filtering Scheme against Distributed Denial-of-Service Attacks.IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING. APRIL- JUNE 2006, 3(2):141~155.
[6] A. Lall, V. Sekar, M. Ogihara, et al. Data Streaming Algorithms for Estimating Entropy of Network Traffic. SIGMETRICS/Performance’06, Saint Malo, France. June, 26-30, 2006:145~156.
[7] Y. Chen, K. Hwang, W. S. Ku. Distributed Change-Point Detection of DDoS Attacks over Multiple Network Domains. IEEE Transaction on Parallel and Distributed Systems, 2007:7~14.
[8] M. Cai, K. Hwang, J. P. Pan, C. Papadopoulos. WormShield: Fast Worm Signature Generation with Distributed Fingerprint Aggregation. IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING. APRIL-JUNE, 2007, 4(2):88~104.
[9] R. Stone. Center track: An IP overlay network for tracking dos floods. In: Proc 9th Usenix Security Symp. Usenix Assoc, 2000:199~212.
[10] H. Burch, B. Cheswick. Tracing anonymous packets to their approximate source. In: Proc 14th Conf Systems Administration. Usenix Assoc, 2000:313~322.
[11] D. X. Song, A. Perrig. Advanced and authenticated marking scheme for IP traceback. In: Proceedings of 2001 IEEE INFOCOM Conference. California: ACM, 2001:878~886.
[12] A. C. Snoeren, C. Partridge, L. A. Sanchez, et al. Hash-based IP traceback. In: Proceedings of 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM). California: ACM, 2001:3~14.
[13] S. Song, C. N. Manikopoulos. A Control Theoretical Approach for Flow Control to Mitigate Bandwidth Attacks. In Proc of 2006 IEEE Information Assurance Workshop. 2006:348~360.
[14] P. Naldurg, R. H. Campbell. Dynamic Access Control: Preserving Safety and Trust for Network Defense Operations. In: Proc of the eighth ACM symposium on Access control models and technologies. ACM Press. June, 2003:231~237.
[15] H. J. Kim, J. C. Na, S. W. Sohn. Response to distributed denial-of-service attack using active technology. In: Proceedings of the Eighth IASTED International Conference on Internet and Multimedia Systems and Applications. Kauai: Acta Press. 2004:244~248.
[16] J. W. Freeman, T. C. Darr, R. B. Neely. Risk Assessment for Large Heterogeneous Systems. Proceedings of 13th Annual Computer Security Applications Conference, San Diego, CA, USA, 1997:44~52.
[17] Y. J. Han, J. S. Yang, B. H. Chang, et al. The Vulnerability Assessment for Active Networks; Model, Policy, Procedures, and Performance Evaluations. In: Proceedings of 2004 International Conference on Computational Science and Its Applications, Lecture Notes in Computer Science 3043, Springer Verlag, Assisi, Italy, 2004:191~198.
[18] S. Kamara, S. Fahmy, E. Schultz, et al. Analysis of Vulnerabilities in Internet Firewalls. Computers & Security. 2003, 22(3):214~232.
[19] R. Ortalo, Y. Deswarte, M. Kaaniche. Experimenting with quantitative evaluation tools for monitoring operational security. IEEE Trans. on Software Engineering, 1999, 25(5):633~651.
[20] B. Babcock, S. Babu, M. Datar, R. Motwani, J. Widom. Models and Issues in Data Streams. In Proc. ACM Symp. On Principles of Database Systems, 2002:1~16.
[21] Sirish. Chandrasekaran, Michael J. Franklin. Streaming Queries over Streaming Data. Proceedings of the 28th VLDB Conference, Hong Kong, China, 2002:203~214.
[22] Lukasz Golab, M. Tamer Ozsu. Processing Sliding Window Multi-Joins in Continuous Queries over Data Streams. Proceedings of the 29th VLDB Conference, Berlin, Germany, 2003:500~511.
[23] J. Kang, J. Naughton, S. Viglas. Evaluating Window Joins over Unbounded Streams. In Proc. 19th Int.Conf. on Data Engineering, 2003:341~352.
[24]金澈清.流数据分析与管理综述.软件学报, 2004,15(8):134~138.
[25] A. N. Wilschut, P. M. G. Apers. Dataflow Query Execution in a Parallel Main-memory Environment. Proc. of the 1st Int’l. Conf. on Parallel and Distributed Information Systems, Miami Beach, FL, 1991:68~77.
[26] T. Urhan, M. J. Franklin. XJoin: A Reactively-scheduled Pipelined Join Operator.IEEE Data Engineering Bulletin, 2000, 23(2):27~33.
[27] HAASPJ, J. M. HELLERSTEIN. Ripple Joins for Online Aggregation. SIGMOD Conference, 1999:287~298.
[28] S. VIGLAS, J. NAUGHTON. Rate-Based Query Optimization for Stream Information Source. In Proc. ACM Int .Conf. On Management of Data. 2002:37~48.
[29]钱江波,徐宏炳,王永利,刘学军,董逸生.多数据流滑动窗口并发连接方法.计算机研究与发展. 2005:1771~1778.
[30]刘学军,钱江波.分布式数据流连接查询算法.计算机工程. 2006.11:41~43.
[31]郭庆平.一种分布式数据流连接查询算法.武汉理工大学学报. 2009.2.29-32.
[32]王帅.异构数据转换与同步技术的研究与实现.学位论文.长沙.国防科学技术计算机学院.2005.
[33] http://ww.rpbourret.com/xmldbms/index.html
[34] M. F. Fernandez, W. C. Tan, D. Suciu. SilkRoute: Trading between Relations and XML. In Int’l World Wide Web Conf. (WWW),AmsTERDAM, Netherlands, May 2000:723~745.
[35] M. Carey, D. Florescu, Z. Ives, Y. Lu, J. Shanmugasundaram, E. Shekita, and S. Subramanian. XPERANTO: Publishing Object-Relational Data as XML. In Int’l Workshop on the web and Database (Web DB), Dallas, TX, May 2000:105~110.
[36] Josephine Cheng, Jane Xu. IBM DB2 XML Extender: An end-to-end solution for storing and retrieving XML documents. ICDE’00 Conference, San Diego, February 2000:128~136.
[37] Sirish Chandrasekaran, Owen Cooper, Amol Deshpande, Michael J. Franklin, Joseph M. Hellerstein. TelegraphCQ: Continuous Dataflow Processing for an uncertain world.CIDR Conference. Asilomar, CA, USA, January 5-8, 2003:269~280.
[38] Instruction Detection System [EB/OL]. http://en.wikipedia.org/wiki/Intrusion_detection_system,2009-11-17.
[39] Introduction of VScanner [EB/OL]. http://www.esrf.eu/UsersAndScience/Experiments/TBS/ISG/vscanner,2008-9-26.
[40] Introduction of ArpWatch [EB/OL]. http://en.wikipedia.org/wiki/Arpwatch,2009-11-15.
[41] The new p0f[EB/OL].http://lcamtuf.coredump.cx/p0f.shtml,2006-9-6.
[42] Regular Expression [EB/OL]. http://en.wikipedia.org/wiki/Regular_expression,2009-11-17.
[43]安天网络病毒监控系统[EB/OL].http://www.antiy.com/cn/product/vds.htm.
[44] Introduction of Firewall [EB/OL]. http://en.wikipedia.org/wiki/Firewall, 2009-11-17.
[45] L. Golab, M. T. Ozsu. Issues in Data stream management. SIGMOD Record, 2003, 32(2):5~14.