可扩展网络流量分析平台的研究与实现
详细信息    本馆镜像全文|  推荐本文 |  |   获取CNKI官网全文
摘要
随着现代网络技术的发展,网络已经成为人们生活中不可缺少的一部分,人们对网络的依赖也与日俱增。计算机网络的普及,为人们的工作、学习和生活带来了莫大的便利,我们也已经进入了网络时代,我们的生活、工作和学习,已经离不开网络,但我们在享受着它的便利的同时,也在为我们信息的安全担忧,常常会有网络用户信息被非法窃取、篡改的网络安全事件发生,普通用户则对这些事件闻风色变。
     计算机网络是一把双刃剑,随着网络信息传输量的急剧增长和计算机网络的广泛普及,一些机构和部门在得益于网络带来的便利与方便之余,也存在自身的网络数据安全以及各方面的利益受到威胁的风险,如近期闹得沸沸扬扬的维基解密事件,再加上网络用户的日渐多样化,网络安全已经成为保障用户正常生活、工作与学习的头等大事。
     提到网络安全,我们一般首先想到的是接触较多的firewall(防火墙),防火墙可以屏蔽掉一些访问特定端口的请求,但很多存在于所执行的代码中的恶意入侵企图,而防火墙通常情况下是检测不到的,因此,我们就需要入侵检测系统(IDS, Intrusion Detection System)来弥补这个缺陷。无论是相对独立的个人还是相关的企业以及部门机构,对网络进行管理是必要更是必须的,而网络监控作为网络管理的一个重要方面,也成为研究的热点,但是NIDS (Network Intrusion Detection System,网络监控系统)的实现可以有不同的方式,现在也有比较成熟的像Sniffer之类的系统,但是由于用户的具体应用场景不同,因此不免就会产生一些不必要的开销,而网络流量分析监控系统的研究则可以减少用户在这方面的顾虑,因此网络流量监测系统的研究与实现就变得更加的有意义。在这样的情况下,可扩展网络流量监测平台的研究地位就更加突出,用户可以根据具体应用设计不同的应用功能进行响应的网络监测功能,不仅可以实现通常网络监测系统应该具备的功能,如通过网络流量测量获取网络运行参数,并在此基础上进行分析、检测,最终判断网络运行状况,同时也可以通过网络流量监测及时地发现问题,克服网络传输过程中的瓶颈,以及检测网络攻击,另外,也可以在可扩展平台基础上进一步扩展优化,以满足不同用户的需求。
     越来越多的普通人已经进入到了网络中,包括日常生活、工作、学习在内的社会生活的各个领域已经离不开网络技术的应用及发展,甚至连对国计民生意义重大的国防、金融等关键领域,也更加地对网络的应用及发展依赖。随着网络规模的不断扩大,很多SN (Special Network,专用网络)和PN (Public Network,公众网络)已经互通连接,要保证如此庞大的网络的有序运行,实现安全高效,以便能够保证普通用户的正常工作、学习,进一步保证一个民族各项事业、生活的有条不紊,仅仅靠普通的网络管理是很难实现以上目标的,我们更需要的是能够主动地去发现网络运行过程中可能存在的问题,以网络监控、网络流量分析检测作为更进一步的方式,来实现对网络环境的有序、安全保障。
     然而随着网络规模以及网络传输速度和传输数据量的增长,传统的对于低带宽网络的监测已经很难满足日益增长的网络数据量的要求,在这样的情况下,如何在低开销的情况下,更加快速的实现对高速网络环境中的大规模数据量进行分析检测,为我们提出了很高的要求,因此,我们发现对网络流量进行监测分析研究的地位更显突出。
     本文主要目标是对于可扩展网络流量分析监测平台的研究和实现,在已基本实现的网络流量监控平台的基础上,实现可扩展插件平台的研究和实现,使得用户能够根据不同的应用,通过简单的编码设计相应功能的插件,该平台是基本是以libnids框架的思想为原型进行开发的,能够实现对出入口流量进行分析、检测,以便实现能够保障网络的正常、高效运转的目的;通过已有的端口镜像采集流经网络出入口的网络数据包,在网络数据包捕获过程中,为降低系统开销,采用零拷贝技术;能够统计流量中包的大小、实现IP协议包流量分类、IP协议包的数量分类、TCP端口流量统计、TCP传输控制包数量、网络错误数量等,为使平台具有较充分的可扩展性,方便加入新的功能模块,对插件模块部分做优化,并实现图形化显示统计和分析数据。另外,我们提出CHHFR(Cashing Heavy-hiiter Flows with Replacement)算法来快速并准确地识别heavy-hitter(大流)对象,并以插件形式在该可扩展平台上实现,结果显示该算法能达到理想的效果。
With the rapid widespread of network, our life has become much more convenient. Network is everywhere whatever we do and wherever we go indeed. Just as the coin has two sides, it is not so secure for us to use network sometime. It is not strange for us to know that our information on the internet may be stolen, modified and even falsified illegally.
     The network also brings convenience and benefits for some organizations and departments. But some of them are anxious about the security of their private data which includes their user information and financial records because of the existence of hackers and attackers. As the reasons listed above, network security is becoming more and more important. When we talk about the network security, some may think of the firewall. But as we know that the firewall can't check the hostile attacks hidden in the codes, though it can shield the unexpected access. So to avoid this defect, it's necessary for us to do research on IDS (Intrusion Detection System).
     The network has become an indispensable part of our life. Our dependency on the internet is growing no matter whoever we are and whatever we do. It's much more necessary for us to manage the network and know the status of our network. As an important factor of the network management, the network monitoring becomes research hotspots. However, the detection of the network flow is an effective way for network monitoring. Though there are some comparative mature tools such as sniffer, they are not so convenient for normal users for different application. So research on the extensible platform for the detection of the network flow is also much more important. Users can develop different application tools on the extensible platform for different application backgrounds. In this process we can also reduce the cost of the system. We can use this platform for the normal detection, such as some important parameters of the network, the status of the network, the dubious attackers and so on. In a word, we can use it for different applications. So, it has a new and deeper meaning for the maintenance of the network security and the improvement of the quality of services provided by the network.
     The main goal of our paper is the research and realization of the extensible platform for network flow measurement. Our research is mainly on the basic existed network monitoring system, and then do the research and realization of the extensible platform for network flow measurement. The users can apply new application plugin to this platform, and this platform is based on libnids which provides API for the network monitoring application developers. This platform can be used to measuring the rate of the accessing flow and detecting the dubious flow items. We also can use this system to record the size of the flow packets, classify the packets by size and type of IP or TCP protocol. Users also can know the status of the network by user interface browser. We also propose an algorithm called CHHFR (Cashing Heavy-hiiter Flows with Replacement) to identifying heavy-hitter flows fast and accurately. It demonstrated that the algorithm can work well and can achieve the expected effects.
引文
[1]于小红,曾文方,严鹏网络管理软件的选择和应用计算机应用研究,2001,(02):89-93
    [2]http://baike.baidu.com/view/1648705.htm
    [3]IBM.IBM Netview White papers [EB/OL], http://www-03.ibm.com/support/atsmastr.nsf/WebIndex/WP 100806,2006
    [4]Cisco Systems Inc.Cisco device MIBs[EB/OL], http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml,2007
    [5]钟健松基于HP Open View的网管系统的开发通信世界,1999,(12):57-60
    [6]H3C Technologies Co. Limited.H3C Quidview Network Management System [EB/OL].http://www.huawei-3com.com/portal/Products-Solutions/Products/Net
    [7]周红雷校园网监测系统的设计与实现:[硕士学位论文].郑州:郑州大学,2003
    [8]温建平面向园区网络管理系统的设计与实现:[硕士学位论文].济南:山东大学,2009
    [9]王风宇,云小春等高速网络监控中大流量对象的提取:软件学报Vol.18,:3061-3069No.12, December 2007
    [10]Keys K, Moore D, Estan C. A robust system for accurate real-time summaries of Internet traffic. SIGMETRICS Performance Evaluation Review,2005, 33(1):85-96.
    [111]黄君基于J2EE的分层分布式网络管理的研究与应用:[硕士学位论文].哈尔滨:哈尔滨工程大学,2005
    [12]Schweller R, Gupta A, Parsons E, Chen Y. Reversible sketches for efficient and accurate change detection over network data streams. In:Proc. of the ACM SIGCOMM Conference on Internet Measurement. Taormina:ACM Press,2004. 207-212.
    [13]Estan C, Varghese G. New directions in traffic measurement and accounting: Focusing on elephants, ignoring the mice. ACM Trans. on Computer Systems, 2003,21(3):270-313.
    [14]CiscoNetFlow.2006.http://www.cisco.com/en/US/productslps6601/products_ios_protocol_group_home.html
    [15]Brownlee N, Mills C, Ruth G. Traffic flow measurement:Architecture. RFC 2722,1999.
    [16]Estan C, Keys K, Moore D, Varghese G. Building a better NetFlow. In:ACM SIGCOMM Computer Communication Review,2004,34(4):245-256.
    [17]Duffield N, Lund C, Thorup M. Flow sampling under hard resource constraints. In:ACM. SIGMETRICS 2004, New York:ACM Press,2004.85-96.
    [18]Cheng G, Gong J, Ding W. Network traffic sampling measurement model on packet identification. Acta Electronica Sinica,2002,30(12A):1986-1990 (in Chinese with English abstract).
    [19]Wang HB, Wei AM, Lin Y, Cheng SD. Time stratified packet sampling based on measurement buffer for flow measurement. Journal of Software,2006,17(8):1775-1784 (in Chinese with English abstract).
    [20]Kodialam M, Lakshman TV, Mohanty S. Runs based traffic estimator (RATE):A simple, memory efficient scheme for per-flow rate estimation. In:IEEE INFOCOM 2004. Hongkong:IEEE Press,2004.1808-1818.
    [21]Hao F, Kodialam M, Lakshman TV. ACCEL-RATE:A faster mechanism for memory efficient per-flow traffic estimation. In:ACM SIGMETRICS. New York: ACM Press,2004.155-166.
    [22]Hao F, Kodialam M, Lakshman TV, Zhang H. Fast, memory-efficient traffic estimation by coincidence counting. In:INFOCOM 2005. Miami:IEEE Press, 2005.2080-2090.
    [23]Kumar A, Xu J, Wang J, Spatschek O, Li L. Space-Code Bloom filter for efficient per-flow traffic measurement. In:IEEE INFOCOM. Hang Kong:IEEE Press,2004.1762-1773.
    [24]Gong J, Peng YB,Yang W, Liu WJ. Reconstructing the parameter for massive abnormal TCP connections with Bloom filter. Journal of Software, 2006,17(3):434—44 (in Chinese with English abstract).
    [25]Yin Z, Sumeet S, Subhabrata S, et al. Online identification of hierarchical heavy hitters:Algorithms, evaluation, and applications. In:Internet Measurement Conf. (IMC 2004). Taormina:ACM Press,2004.101-104.
    [26]Kim I. Analyzing network traces to identify long-term high rate flows [MS. Thesis], Texas A&M Univ.,2001.
    [27]http://baike.baidu. com/view/4130801.htm
    [28]吴君怡基于Web的网络管理及IP流量计费系统的研究与实现:[硕士学位论文].合肥合肥工业大学,2002
    [29]朱平一种基于中小型网络的监控系统:[硕士学位论文].北京北京交通大学,2006.12
    [30]http://baike.baidu.com/view/786548.htm
    [31]http://pg.zhku.edu.cn/inforwork/kejian/COURSE/chll/4_2.htm
    [32]钟旺伟统一入侵检测框架的设计与实现:微计算机信息Vol.22,第9-3期P128-130,2006
    [33]http://baike.baidu.com/view/1193864.htm
    [34]http:blog.csdn.net/joshua_yu/archive/2006/02/02/591044.aspx
    [35]http://mail.gnome.org/archives/gnome-announce-list/2002-August/msg00003.htm 1
    [36]http://sourceforge.net/projects/libnet-dev/
    [37]http://sourceforge.net/projects/libpcap/
    [38]R. Mahajan, S. Floyd and D. Wetherall, Controlling high-bandwidth flows at the congested router, in the Ninth International Conference on Network Protocols (ICNP'01), Riverside, California, USA, Novermber 2001, pp.192.
    [39]Sven Anderson and Dieter Hogrefe, Mouse Trapping:A Flow Data Reduction Method, in the Third International Conference on Internet Monitoring and Protection (ICIMP 2008), Bucharest, Romania, June 2008, pp.17-22.
    [40]孙昂Linux内核网络流量监测系统:[硕士学位论文].杭州浙江大学,2006.5
    [41]Luca Deri, Finsiel S.p.A. Stefano Suin. Effective Traffic Measurement Using ntop. IEEE Communications Magazine, vol.38, pp.138—143. May 2000.
    [42]V. Paxson. Bro:A System for Detecting Network Intruders in Real-Time. In 7th Annual USENIX Security Symposium, January 1998.
    [43]Marcus J. Ranum, Kent Landfield, Mike Stolarchuk, Mark Sienkiewicz. Implementing a Generalized Tool for Network Monitoring. Proc. LISA'97, USENIX 11 th Systems Administration Conference, San Diego, Oct.1997.
    [44]http://blogold.chinaunix.net/ul/45976/showart_360546.html
    [45]http://sourceforge.net/projects/libnids/,2011.02.21
    [46]http://hi.baidu.com/sbzhysh/blog/item/8421e4fa48f2622a4e4aeaac.html

© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号

地址:北京市海淀区学院路29号 邮编:100083

电话:办公室:(+86 10)66554848;文献借阅、咨询服务、科技查新:66554700