IPFIX双向流生成系统的研究与实现
摘要
随着互联网的飞速发展,网络上各种应用业务在为人们提供方便生活的同时,带来的网络流量监控和管理的要求也日益受到人们的关注。一个良好的网络业务流量测量系统能对网络承载的各类业务进行及时和准确的流量分析,并为网络流量控制、计费、安全检测方面提供有效的实施手段。目前对网络测量的研究大多集中在有向性流、即单边流量的统计特性方面,这种流信息具有更明确的数据信息语义,却隔离了双边通信的来往数据关联。然而,有研究表明当前网络环境中基于面向连接的、不对称的双向数据通信占用了网络应用的大多数份额,关联了正反数据流特性的双向流将比单向流更有利于从业务应用和网络安全的角度去分析网络中的数据流量。
IPFIX是IETF为了统一流输出格式而定义的标准流输出协议,它是一种基于模板的单向流生成标准,以一组流特征属性来识别并统计网络中的数据包,具有很强的可扩展性。然而,未经扩展的IPFIX完成的是单向流的输出,因此,在该标准上进行双向流生成的实现将对网络业务分析具有重大意义。
本文从IPFIX流生成定义出发,首先介绍了IPFIX的数据格式及模板特征,并以IPFIX标准系统架构为基础描述了流产生输出的过程。通过对比双向流与单向流的生成差异,引出了双向流的设计方案及实施难点。基于单向流生成系统模型,本文设计并实现了IPFIX的双向流生成系统,重点阐述了双向流生成阶段的流程设计,以及相关的算法与数据结构。最后,对系统进行测试验证了该设计方案的可行性,并通过对单向流与双向流生成结果的分析,说明了双向流在网络流量测量方面的优势。
Accompanied by the increasing development of Internet, a variety of services and applications based on network have been provided to people for the convenience of daily life. This development also causes more requirements on the management and supervision of network traffic. A network measurement system should keep collecting and analyzing network traffic features for the purpose of QoS monitoring, usage-based accounting, attack/intrusion detection and so on. But researches of network measurement are mostly focused on the unidirectional-flow, or uniflow. Though this kind of flow has a more accurate semantic for the description on a single traffic, it cuts off the relationship of forward and reverse traffic between two endpoints. While, many flow analysis tasks benefit from association of the upstream and downstream flows of a bidirectional communication. One recent study shows that connection-oriented and anti-symmetrical bidirectional communication has counted the majority proportion of network traffic. That means, bidirectional-flow, rather than uiflow, should be a more effective way for network measurement and analysis, especially for the security and application parts.
The IPFIX protocol is the IETF standard for IP flow information export. It is a unidirectional, template-based data transport protocol that provides flexible flow selection; a flow can be defined by an arbitrary number of packet fields, which compose the flow key. Since the original IPFIX does not export biflow, the study and implementations of biflow export using IPFIX standard will be significant for a better and efficient measurement of nowadays network.
This paper begins with the definition of IPFIX export. It first introduces the IPFIX message format and it's template, then the process of flow classifying and export will be clarified based on the standard IPFIX architecture. Second, the feature of bidirectional communication is presented by comparing the difference of uniflow and biflow. After that, based on the single flow export architecture, this paper designs and implements the biflow export system using IPFIX,. The process of biflow generation and related algorithm and data structure is the core of this paper. Last, this system runs some tests to prove its feasibility. Result is also given by analyzing the difference between uniflow and biflow records, which shows the superiority of biflow on network measurement.
IPFIX是IETF为了统一流输出格式而定义的标准流输出协议,它是一种基于模板的单向流生成标准,以一组流特征属性来识别并统计网络中的数据包,具有很强的可扩展性。然而,未经扩展的IPFIX完成的是单向流的输出,因此,在该标准上进行双向流生成的实现将对网络业务分析具有重大意义。
本文从IPFIX流生成定义出发,首先介绍了IPFIX的数据格式及模板特征,并以IPFIX标准系统架构为基础描述了流产生输出的过程。通过对比双向流与单向流的生成差异,引出了双向流的设计方案及实施难点。基于单向流生成系统模型,本文设计并实现了IPFIX的双向流生成系统,重点阐述了双向流生成阶段的流程设计,以及相关的算法与数据结构。最后,对系统进行测试验证了该设计方案的可行性,并通过对单向流与双向流生成结果的分析,说明了双向流在网络流量测量方面的优势。
Accompanied by the increasing development of Internet, a variety of services and applications based on network have been provided to people for the convenience of daily life. This development also causes more requirements on the management and supervision of network traffic. A network measurement system should keep collecting and analyzing network traffic features for the purpose of QoS monitoring, usage-based accounting, attack/intrusion detection and so on. But researches of network measurement are mostly focused on the unidirectional-flow, or uniflow. Though this kind of flow has a more accurate semantic for the description on a single traffic, it cuts off the relationship of forward and reverse traffic between two endpoints. While, many flow analysis tasks benefit from association of the upstream and downstream flows of a bidirectional communication. One recent study shows that connection-oriented and anti-symmetrical bidirectional communication has counted the majority proportion of network traffic. That means, bidirectional-flow, rather than uiflow, should be a more effective way for network measurement and analysis, especially for the security and application parts.
The IPFIX protocol is the IETF standard for IP flow information export. It is a unidirectional, template-based data transport protocol that provides flexible flow selection; a flow can be defined by an arbitrary number of packet fields, which compose the flow key. Since the original IPFIX does not export biflow, the study and implementations of biflow export using IPFIX standard will be significant for a better and efficient measurement of nowadays network.
This paper begins with the definition of IPFIX export. It first introduces the IPFIX message format and it's template, then the process of flow classifying and export will be clarified based on the standard IPFIX architecture. Second, the feature of bidirectional communication is presented by comparing the difference of uniflow and biflow. After that, based on the single flow export architecture, this paper designs and implements the biflow export system using IPFIX,. The process of biflow generation and related algorithm and data structure is the core of this paper. Last, this system runs some tests to prove its feasibility. Result is also given by analyzing the difference between uniflow and biflow records, which shows the superiority of biflow on network measurement.
引文
[1]马华.基于Flow的IP QoS流量分析系统设计与实现
[2]T. Zseby, B. Claise, S. Zander. RFC 3917. October 2004. Requirements for IP Flow Information Export (IPFIX).
[3]B. Claise, Ed. RFC 5101. January 2008. Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information.
[4]单华琦.IP网络流量和业务质量监测的研究与实现.北京邮电大学
[5]Elisa Boschi, Brian Trammell. Bidirectional Flow Measurement, IPFIX, and Security Analysis.
[6]李树旺,杨路明,张超,邵济平.Linux下基于flow-tools的网络异常流量分析.网络安全技术与应用2005.10
[7]吴桦,龚俭,张晓宇.基于Netflow的网络服务监测系统.东华大学学报(自然科学版).2008年9月
[8]IPFIX技术白皮书.锐捷网络有限公司
[9]Arne (?)sleb(?)& Olav Kvittem. Extending the IPFIX protocol for better QoS monitoring
[10]胡际涛.基于IPFIX协议的数据传输系统.华东师范大学
[11]Ganesh Sadasivan, Nevil Brownlee, Benoit Claise, Juergen Quittek.RFC 5470. March 2009. Architecture for IP Flow Information Export.
[12]Martin Stiemerling, Paul Aitken. RFC 5153. April 2008. IP Flow Information Export (IPFIX) Implementation Guidelines.
[13]Hitoshi Irino, Masaru Katayama, Shinichiro Chaki. Flow-based Network Measurement—NetFlow&IPFIX
[14]Per Juvhaugen. Exporting IP flows using IPFIX
[15]N. Brownlee, C. Mills, G. Ruth. Traffic Flow Measurement:Architecture. RFC 2722. October 1999
[16]T. Zseby, E. Boschi, N. Brownlee, B. Claise. RFC 5472, March 2009. IP Flow Information Export (IPFIX) Applicability.
[17]D. Harrington, R. Presuhn, B. Wijnen. RFC 2571. April 1999. An Architecture for Describing SNMP Management Frameworks.
[18]张欣.一种NetFlow流量分析器的设计.《现代电子技术》2006年第21期总第236期
[19]Nakjung Choil, Hyeongu Son, Youngseok Lee, Yanghee Choi. Experiences with IPFIX-based Traffic Measurement for IPv6 Networks
[20]Youngseok Lee, Seongho Shin, Soonbyoung Choi, Hyeon-gu Son. IPv6 Anomaly Traffic Monitoring with IPFIX. ICIMP 2007
[21]Cisco-Netflow Overview
[22]Brian H. Trammell, Elisa Boschi. RFC 5103 January 2008. Bidirectional Flow Export Using IP Flow Information Export (IPFIX).
[23]Rohmad, Mohd Saufy; Azmat, Farok; Manaf, Mazani; Manan, Jamalul-Lail Abdul Enhanced Netflow Version 9 (e-Netflow v9) For Network. International Symposium on Information Technology 2008
[24]Robin Summer and Anja Feldmann. NetFlow:Information loss or win?
[25]张志刚.IPFIX完善流量分析.中国计算机报／2003年／09月／08日／第C10版
[26]IPFIX简化网络流报告.网络世界／2006年／7月／17日／第019版
[27]网络测量系统NTop。
[28]PF_RING User Guide. January 2008.2004-08 ntop.org
[29]Zhang Pei, Lin Zhaowen, Huang Xiaohong, Ma Yan. An Extensible Flow Information Export Engine For Traffic Measurement.
[30]Woo-jin Yang, Tae-il Kim, Hae-won Jung.Optimizing Hash Table Structure of Flow Exporting. Software. Feb.20-22,2006. ICACT2006.
[31]冯国华,蔡家麟,王友平,代世宝.IP流信息输出协议—IPFIX.科技咨询导报.2007 No.11
[32]DongJin Lee, Nevil Brownlee.Passive Measurement of One-way and Two-way Flow Lifetimes. ACM SIGCOMM Computer Communication Review 19. Volume 37, Number 3, July 2007
[2]T. Zseby, B. Claise, S. Zander. RFC 3917. October 2004. Requirements for IP Flow Information Export (IPFIX).
[3]B. Claise, Ed. RFC 5101. January 2008. Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information.
[4]单华琦.IP网络流量和业务质量监测的研究与实现.北京邮电大学
[5]Elisa Boschi, Brian Trammell. Bidirectional Flow Measurement, IPFIX, and Security Analysis.
[6]李树旺,杨路明,张超,邵济平.Linux下基于flow-tools的网络异常流量分析.网络安全技术与应用2005.10
[7]吴桦,龚俭,张晓宇.基于Netflow的网络服务监测系统.东华大学学报(自然科学版).2008年9月
[8]IPFIX技术白皮书.锐捷网络有限公司
[9]Arne (?)sleb(?)& Olav Kvittem. Extending the IPFIX protocol for better QoS monitoring
[10]胡际涛.基于IPFIX协议的数据传输系统.华东师范大学
[11]Ganesh Sadasivan, Nevil Brownlee, Benoit Claise, Juergen Quittek.RFC 5470. March 2009. Architecture for IP Flow Information Export.
[12]Martin Stiemerling, Paul Aitken. RFC 5153. April 2008. IP Flow Information Export (IPFIX) Implementation Guidelines.
[13]Hitoshi Irino, Masaru Katayama, Shinichiro Chaki. Flow-based Network Measurement—NetFlow&IPFIX
[14]Per Juvhaugen. Exporting IP flows using IPFIX
[15]N. Brownlee, C. Mills, G. Ruth. Traffic Flow Measurement:Architecture. RFC 2722. October 1999
[16]T. Zseby, E. Boschi, N. Brownlee, B. Claise. RFC 5472, March 2009. IP Flow Information Export (IPFIX) Applicability.
[17]D. Harrington, R. Presuhn, B. Wijnen. RFC 2571. April 1999. An Architecture for Describing SNMP Management Frameworks.
[18]张欣.一种NetFlow流量分析器的设计.《现代电子技术》2006年第21期总第236期
[19]Nakjung Choil, Hyeongu Son, Youngseok Lee, Yanghee Choi. Experiences with IPFIX-based Traffic Measurement for IPv6 Networks
[20]Youngseok Lee, Seongho Shin, Soonbyoung Choi, Hyeon-gu Son. IPv6 Anomaly Traffic Monitoring with IPFIX. ICIMP 2007
[21]Cisco-Netflow Overview
[22]Brian H. Trammell, Elisa Boschi. RFC 5103 January 2008. Bidirectional Flow Export Using IP Flow Information Export (IPFIX).
[23]Rohmad, Mohd Saufy; Azmat, Farok; Manaf, Mazani; Manan, Jamalul-Lail Abdul Enhanced Netflow Version 9 (e-Netflow v9) For Network. International Symposium on Information Technology 2008
[24]Robin Summer and Anja Feldmann. NetFlow:Information loss or win?
[25]张志刚.IPFIX完善流量分析.中国计算机报／2003年／09月／08日／第C10版
[26]IPFIX简化网络流报告.网络世界／2006年／7月／17日／第019版
[27]网络测量系统NTop。
[28]PF_RING User Guide. January 2008.2004-08 ntop.org
[29]Zhang Pei, Lin Zhaowen, Huang Xiaohong, Ma Yan. An Extensible Flow Information Export Engine For Traffic Measurement.
[30]Woo-jin Yang, Tae-il Kim, Hae-won Jung.Optimizing Hash Table Structure of Flow Exporting. Software. Feb.20-22,2006. ICACT2006.
[31]冯国华,蔡家麟,王友平,代世宝.IP流信息输出协议—IPFIX.科技咨询导报.2007 No.11
[32]DongJin Lee, Nevil Brownlee.Passive Measurement of One-way and Two-way Flow Lifetimes. ACM SIGCOMM Computer Communication Review 19. Volume 37, Number 3, July 2007