网络流量控制系统的设计与实现
|
摘要
网络流量控制系统的作用是根据用户需求对网络上指定的应用流量进行精确的控制。当今网络上的各种应用日益繁多复杂,用户对资源的获取也更加的便捷,拿网络应用常见的P2P下载来说,在这几年中,依靠独有的技术优势和下载质量P2P得到了迅速的发展,用户越来越多,目前P2P应用已经占到普通网络流量的60%以上,超过了HTTP应用成为网络带宽最大的消耗者,P2P网络在文件资源共享以及分布式计算等很多方面已经充分显示出了其便捷之处和强大的技术优势。与此同时P2P也给人们带来了诸多负面的问题。P2P的用户大都以很高的速度下载各种资源文件,由此导致网络带宽被急剧缩减,很容易造成各网络链路的堵塞,其他用户的关键应用就得不到很好的保障。另一方面,P2P应用的特殊性还在于这种协议上下行的流量是基本对称的,这就容易造成传统的非对称网络(下行带宽远远大于上行带宽)产生上行方向流量的堵塞。为了有效的监控和分析网络带宽的使用情况,防止BT下载、网络电视等占用网络带宽应用的过度使用,同时实现对被监控设备的业务进行精细的控制,在本文中提出并实现了适用于局域网等小型网络环境的网络流量控制系统的实现方案。
本文重点研究了网络流量控制的关键技术方案,其中着重研究了P2P加密流量的识别技术和依托Linux系统Netfilter架构的流量控制方案,并在此基础上实现了一个新的网络流量控制系统。该系统支持常用的深度负载层检测技术和行为识别技术,可以基本满足对网络各类流量的识别和控制。具体本文主要完成了以下几方面的工作：
1、研究了各种现有的网络流量识别技术和控制技术,并详细分析了各种技术的实现方案和各自的优缺点。
2、研究分析了P2P流量的特点,对现有技术识别情况不好的P2P流量提出了两种行为识别方案。
3、研究并提出了一种新的定位于局域网的流量控制系统的系统架构和网络部署方案。本系统能够实现对网络流量的准确识别,并可以根据用户自定义的流量管理策略对目标流量进行有效的控制,同时提供实时流量和历史流量查询功能。
4、设计并实现网络流量控制系统的后台流量处理模块,对Linux系统的Netfilter框架进行了深入的研究,对网络报文在内核中存储转发过程进行了详尽的分析,结合流量识别技术对从内核态转发到用户态的网络报文依据流量管理策略进行有效的控制。提出了针对流量控制的限流算法,保证了流量控制效果的准确性。
5、设计并实现网络流量控制系统的前台模块,对前台所实现的功能进行了详尽的阐述。
6、对系统的性能和功能进行了具体分析,很好的达到了各项设计需求,对网络流量的控制效果较好。达到了良好的扩展性和移植性的设计目标。
The role of network traffic control system is the accurately control of the network traffic based on the needs of users. Applications on the network today keep on growing at a high speed, user access to resources more and more convenient. Take the P2P download traffic for example, in the past few years, P2P technology with more and more users is developing rapidly for its unique advantage and high speed, and according to statistics, P2P applications occupied 60% of normal network traffic, which exceed HTTP traffic and become the biggest consumer of network bandwidth, P2P has been already show a strong advantage in distributed computing and file sharing. At the same time P2P has brought us a lot of negative issues. Most P2P users download various resources with a high speed, which led to a sharp reduction in network bandwidth, so it could easily lead to block the network bandwidth; the other applications are not well protected. On the other hand, P2P applications is still particularity in the symmetric of upload traffic and download traffic, which easily lead to the block of traditional non-symmetric network (download traffic bandwidth much larger than the upload traffic bandwidth,). In order to monitor and analyze network bandwidth usage effectively, to control the BT download, online TV network from excessive using, and achieving an accurate control of the different applications, the network control system which targeted on local networks is proposed in this paper.
This paper focuses on the key technologies of network traffic control system, especially on the encrypted P2P traffic identification proposal and the Linux Netfilter structure, and put forward a new network traffic control system on this basis. The system supports the commonly used DPI detection techniques and behavior detection technologies, which can basically meet the need of all kinds of network traffic identification and control. In this paper, the primary work includes:
1. The paper researched and analyzed technology of network flow identification and network flow control, then contrasts their advantages and disadvantages.
2. The paper studied and analyzed the development and features of P2P streams, then proposed two proposals in order to identify encrypted traffic of P2P traffic which are not well identified.
3. The paper proposed a new traffic control system targeted on the local network. The system can achieve an accurate identification of network traffic, and can effective control the flows based on traffic management strategies, and it can also provide real-time and historical traffic queries.
4. The paper designed and achieved the flow handle module, and also studied Netfilter structure of the Linux system, then had a detailed analysis of the packet storage and forwarding process, achieved an effective control combined with flow identification technology. The paper proposed the traffic limit algorithm to ensure the accuracy of the traffic control.
5. The paper designed and achieved the web module, and have an detailed description of its functions.
6. The paper analyzed the system in performance and function, the design achieved well in the demand. It also achieved good scalability and portability.
本文重点研究了网络流量控制的关键技术方案,其中着重研究了P2P加密流量的识别技术和依托Linux系统Netfilter架构的流量控制方案,并在此基础上实现了一个新的网络流量控制系统。该系统支持常用的深度负载层检测技术和行为识别技术,可以基本满足对网络各类流量的识别和控制。具体本文主要完成了以下几方面的工作：
1、研究了各种现有的网络流量识别技术和控制技术,并详细分析了各种技术的实现方案和各自的优缺点。
2、研究分析了P2P流量的特点,对现有技术识别情况不好的P2P流量提出了两种行为识别方案。
3、研究并提出了一种新的定位于局域网的流量控制系统的系统架构和网络部署方案。本系统能够实现对网络流量的准确识别,并可以根据用户自定义的流量管理策略对目标流量进行有效的控制,同时提供实时流量和历史流量查询功能。
4、设计并实现网络流量控制系统的后台流量处理模块,对Linux系统的Netfilter框架进行了深入的研究,对网络报文在内核中存储转发过程进行了详尽的分析,结合流量识别技术对从内核态转发到用户态的网络报文依据流量管理策略进行有效的控制。提出了针对流量控制的限流算法,保证了流量控制效果的准确性。
5、设计并实现网络流量控制系统的前台模块,对前台所实现的功能进行了详尽的阐述。
6、对系统的性能和功能进行了具体分析,很好的达到了各项设计需求,对网络流量的控制效果较好。达到了良好的扩展性和移植性的设计目标。
The role of network traffic control system is the accurately control of the network traffic based on the needs of users. Applications on the network today keep on growing at a high speed, user access to resources more and more convenient. Take the P2P download traffic for example, in the past few years, P2P technology with more and more users is developing rapidly for its unique advantage and high speed, and according to statistics, P2P applications occupied 60% of normal network traffic, which exceed HTTP traffic and become the biggest consumer of network bandwidth, P2P has been already show a strong advantage in distributed computing and file sharing. At the same time P2P has brought us a lot of negative issues. Most P2P users download various resources with a high speed, which led to a sharp reduction in network bandwidth, so it could easily lead to block the network bandwidth; the other applications are not well protected. On the other hand, P2P applications is still particularity in the symmetric of upload traffic and download traffic, which easily lead to the block of traditional non-symmetric network (download traffic bandwidth much larger than the upload traffic bandwidth,). In order to monitor and analyze network bandwidth usage effectively, to control the BT download, online TV network from excessive using, and achieving an accurate control of the different applications, the network control system which targeted on local networks is proposed in this paper.
This paper focuses on the key technologies of network traffic control system, especially on the encrypted P2P traffic identification proposal and the Linux Netfilter structure, and put forward a new network traffic control system on this basis. The system supports the commonly used DPI detection techniques and behavior detection technologies, which can basically meet the need of all kinds of network traffic identification and control. In this paper, the primary work includes:
1. The paper researched and analyzed technology of network flow identification and network flow control, then contrasts their advantages and disadvantages.
2. The paper studied and analyzed the development and features of P2P streams, then proposed two proposals in order to identify encrypted traffic of P2P traffic which are not well identified.
3. The paper proposed a new traffic control system targeted on the local network. The system can achieve an accurate identification of network traffic, and can effective control the flows based on traffic management strategies, and it can also provide real-time and historical traffic queries.
4. The paper designed and achieved the flow handle module, and also studied Netfilter structure of the Linux system, then had a detailed analysis of the packet storage and forwarding process, achieved an effective control combined with flow identification technology. The paper proposed the traffic limit algorithm to ensure the accuracy of the traffic control.
5. The paper designed and achieved the web module, and have an detailed description of its functions.
6. The paper analyzed the system in performance and function, the design achieved well in the demand. It also achieved good scalability and portability.
引文
[1]孙贤淑.IP网络流量测量的研究与应用：[硕士学位论文].北京邮电大学,2004.
[2]程光,龚俭.大规模高速网络流量测量研究[M].计算机工程与应用,2002,38(5)：17-19
[3]周世杰,秦志光,吴春江.对等网络流量检测技术研究[EB/OL] http://www.qkzz.net/magazine/1009-6868/2007/05/1695937.htm,2007.
[4]佚名.业务识别与管理系统和网络流量的管理[EB/OL] http://network.51cto.com/art/200809/90119.html,2008
[5]Allot Traffic management [EB/OL].http://www.sandi.co.jp/semi-nar/report-040610/pdf/azi.pdf
[6]汤昊,李之棠基于DPI的P2P流量控制系统的设计与实现[J]通信技术第6期,2007 13,20-22
[7]BraehaShaPira, PeretzShoval, UriHanani, ExPerimentationwithan Information Filtering system that combine seognitive and soeiologieal Filtering Integrated with users tereoty Pes, DeeisionSuPPortSystems27 1995 22-24.
[8]Karagiannis Thomas Broido Andre Faloutsos Michalis et al.Transport Layer Identification of P2P Traffic,In:Proceedings of the 2004 ACM SIGCOMM Internet Measurement Conference,ACM Press,2004:121-134
[9]Nicoll J R,Bateman M,Ruddle A et al,Challenges in Measurement and Analysis of the BitTorrent Contenet Distribution Model,In:Proceedings of International Postgraduate Symposium on the Convergence of Telecommunications,Networking and Broadcasting,Liverpool John Moores University,2004:27-30
[10]Sen S, Wang J. Analyzing peer-to-peer traffic across large net-works. IEEE/ACM Trans Netw,2004,12(2):219-232 [11] Schollmeier R, Dumanois A. Peer-to-Peer Traffic Characteris-tics. In:Proceedings of the 9th Eunice Open European SummerSchool, Budapest, Hungary,2003:35-37[12]郑勇涛,刘玉树.支持向量机解决多分类问题研究[J].计算机工程与应用,2005. 23,190-192.[13] Sen S, Wang J. Analyzing peer-to-peer traffic across large net-works. In:Proceedings of ACM SIGCOMM Internet Measure-mentWorkshop, Marseilles, France,2002 [14] Netfilter Architecture. Http://www..Netfilter.org/documentation/HOWTO/Netfilter hacking-HOWTO-3.html,2004.[15]米淑云.IP网络流量监控系统的设计与实现[D].北京邮电大学,2009[16] Suehring S, Ziegier R L1 Linux防火墙[M]1.何泾沙等译1北京,机械工业出版社,2006117-631[17]李晓峰,张玉清,李星Linux2.4内核防火墙底层结构分析[M].计算机工程与应用,2002,(14)：138-144.[18]李霞丽.基于Linux的网络流量测量技术研究和Netfilter Prober的设计与实现：[硕士学位论文].西安交通大学,2004[19]姚晓宇,赵晨Linux内核防火墙Netfilte实现与应用研究.计算机工程,2003,29(8)：112-113[20]练书成,徐敬东,咎世刚.基于Linux防火墙连接跟踪机制的应用层协议过滤方法的研究,计算机]：程与应用,2005,41(13)：129-131[21]王永杰,刘京菊,孙乐昌Linux可装载模块的开发与应用.计算机应用研究,2002,19(7)：143-146[22]毛德操,胡希明[M]1.杭州,Linux内核源代码情景分析.浙江大学出版社2001.[23] Sen Subhabrata Wang Jia Analyzing Peer-to-Peer traffic across large networks. ACM/IEEE Transactions on Networking,2004,12(2):219-232 [24] Paxson Vern,Floyd Sally.Wide-Area Traffic:The Failure of Poisson Modeling, IEEE/ACM Transactions on Networking,1995,3(3):226-244 [25] Leland Will,Taqqu Murad,Willinger Walter et al,On the Self Similar Nature of Ethernet Traffic(Extended Version),IEEE/ACM Transactions on Networking,1994, 2(1):1-5 [26] M. Jakobsson, M. Yung, Proving without knowing:On oblivious, agnostic and blindfolded provers. Advances in Cryptology CRYPTO'96, volume 1109 of Lecture Notes in Computer Science, pages 186-200, Berlin,1996. Springer-Verlag.
[2]程光,龚俭.大规模高速网络流量测量研究[M].计算机工程与应用,2002,38(5)：17-19
[3]周世杰,秦志光,吴春江.对等网络流量检测技术研究[EB/OL] http://www.qkzz.net/magazine/1009-6868/2007/05/1695937.htm,2007.
[4]佚名.业务识别与管理系统和网络流量的管理[EB/OL] http://network.51cto.com/art/200809/90119.html,2008
[5]Allot Traffic management [EB/OL].http://www.sandi.co.jp/semi-nar/report-040610/pdf/azi.pdf
[6]汤昊,李之棠基于DPI的P2P流量控制系统的设计与实现[J]通信技术第6期,2007 13,20-22
[7]BraehaShaPira, PeretzShoval, UriHanani, ExPerimentationwithan Information Filtering system that combine seognitive and soeiologieal Filtering Integrated with users tereoty Pes, DeeisionSuPPortSystems27 1995 22-24.
[8]Karagiannis Thomas Broido Andre Faloutsos Michalis et al.Transport Layer Identification of P2P Traffic,In:Proceedings of the 2004 ACM SIGCOMM Internet Measurement Conference,ACM Press,2004:121-134
[9]Nicoll J R,Bateman M,Ruddle A et al,Challenges in Measurement and Analysis of the BitTorrent Contenet Distribution Model,In:Proceedings of International Postgraduate Symposium on the Convergence of Telecommunications,Networking and Broadcasting,Liverpool John Moores University,2004:27-30
[10]Sen S, Wang J. Analyzing peer-to-peer traffic across large net-works. IEEE/ACM Trans Netw,2004,12(2):219-232 [11] Schollmeier R, Dumanois A. Peer-to-Peer Traffic Characteris-tics. In:Proceedings of the 9th Eunice Open European SummerSchool, Budapest, Hungary,2003:35-37[12]郑勇涛,刘玉树.支持向量机解决多分类问题研究[J].计算机工程与应用,2005. 23,190-192.[13] Sen S, Wang J. Analyzing peer-to-peer traffic across large net-works. In:Proceedings of ACM SIGCOMM Internet Measure-mentWorkshop, Marseilles, France,2002 [14] Netfilter Architecture. Http://www..Netfilter.org/documentation/HOWTO/Netfilter hacking-HOWTO-3.html,2004.[15]米淑云.IP网络流量监控系统的设计与实现[D].北京邮电大学,2009[16] Suehring S, Ziegier R L1 Linux防火墙[M]1.何泾沙等译1北京,机械工业出版社,2006117-631[17]李晓峰,张玉清,李星Linux2.4内核防火墙底层结构分析[M].计算机工程与应用,2002,(14)：138-144.[18]李霞丽.基于Linux的网络流量测量技术研究和Netfilter Prober的设计与实现：[硕士学位论文].西安交通大学,2004[19]姚晓宇,赵晨Linux内核防火墙Netfilte实现与应用研究.计算机工程,2003,29(8)：112-113[20]练书成,徐敬东,咎世刚.基于Linux防火墙连接跟踪机制的应用层协议过滤方法的研究,计算机]：程与应用,2005,41(13)：129-131[21]王永杰,刘京菊,孙乐昌Linux可装载模块的开发与应用.计算机应用研究,2002,19(7)：143-146[22]毛德操,胡希明[M]1.杭州,Linux内核源代码情景分析.浙江大学出版社2001.[23] Sen Subhabrata Wang Jia Analyzing Peer-to-Peer traffic across large networks. ACM/IEEE Transactions on Networking,2004,12(2):219-232 [24] Paxson Vern,Floyd Sally.Wide-Area Traffic:The Failure of Poisson Modeling, IEEE/ACM Transactions on Networking,1995,3(3):226-244 [25] Leland Will,Taqqu Murad,Willinger Walter et al,On the Self Similar Nature of Ethernet Traffic(Extended Version),IEEE/ACM Transactions on Networking,1994, 2(1):1-5 [26] M. Jakobsson, M. Yung, Proving without knowing:On oblivious, agnostic and blindfolded provers. Advances in Cryptology CRYPTO'96, volume 1109 of Lecture Notes in Computer Science, pages 186-200, Berlin,1996. Springer-Verlag.