P2P实时流媒体服务器的设计与实现
摘要
P2P流媒体技术是近年流行的网络多媒体传播技术。流媒体以其边播放边下载的特点,大大减小了等待时延,同时也降低了对系统缓存的需求量。随着Internet的普及和web访问的日益频繁,出现了视频点播VOD(Video On Demand)和实时流媒体传输服务。但这两种服务很多仍然以传统的客户端/服务器(C/S)体系结构为基础,这种体系结构下,所有用户的流媒体服务完全只由服务器提供,但随着用户的逐步增加,服务器的负载和压力也将急剧增加。在解决实时流媒体传输的问题上,许多研究者提出了IP组播技术,这个技术利用“一路发送,多路共享”的思想来减少数据包在网络中(IP层)的传输,从而减少网络带宽的耗用。但因为是基于IP层的组播,所以该技术必须要得到底层路由器的支持。可是基于简单原则和分层体系结构的要求,现实中很多路由器仍然不支持IP组播,因为这会增加路由器用于保存分组信息导致的额外性能开销,从而影响报文转发这个主流业务。
为了解决该问题,出现了基于P2P技术的实时流媒体。它利用P2P系统的高容量、易扩展、廉价以及自组织等特点再与流媒体技术有机结合,从而得到了广泛应用,比如当前主流的QQLive,PPLive,UUSEE等。通过这种技术,可以大大缓解中心服务器的压力,并具有良好的服务和扩展能力。但是和普通的P2P应用不同,在流媒体技术中,实时性占据着很重要的地位,所以P2P流媒体技术需要更好地分割、快速重传丢失的资源来满足实时性的要求。本文描述的P2P实时流媒体系统由P2P实时流媒体服务器、TCP代理和P2P客户端构成。
本文主要对P2P实时流媒体服务器的设计、实现和系统测试做较为全面的阐述。该服务器既是数据源又是资源索引服务器,它着重从资源管理的角度实现了流媒体数据的保存、分配和发送,并借鉴流媒体技术中的应用层组播模型以及NAT穿透模型来提供P2P的传输支持。同时,作为原始数据源,该服务器在多通道服务以及C/S和P2P混合传输的问题上也进行了有益的探索。该系统能够应用于许多场合,例如IPTV,视频会议,以及现场直播等实时性要求高的地方。
P2P streaming technology developed quickly in recent years. Streaming technology can significantly reduce time delay and memory requirement for the whole media file need not completely download on the local disk when streaming media is played. As the Internet and web service become a commonplace, video on demand (VOD) and real time streaming services appear. Unfortunately, many streaming systems providing the above services still follow the traditional Client-Server(C/S) principle in which all the media service is provided by the server only. Under C/S architecture, the server's overload would be worsened by more clients joint in system. Under this circumstance, taking the weak points of C/S into account, a number of researchers presented several solutions for real time streaming system, for instance, IP multicast. In IP multicast, the network keeps only one copy of data packet on IP layer of TCP/IP, each client requires the data that can get the copy without affecting the original one. However, this method does not get enough support from rooter for violating the principle of simplicity on IP layer.
Considering the aspects above, the P2P-based streaming system comes into being. This kind of system absorbs the advantages of both P2P and streaming system. The high capacity of service, ease of scalability, lower price and self-organization, those features plus NAT enable the P2P-based streaming system widely used on Internet, such as QQLive, PPLive and UUSEE. Every client in the system can either be data supplier or data consumer, and it can even be the both which is decided by the node's service capability and requirement on streaming media. In the same time, system's real time feature is quite emphasized. The P2P real time streaming server described in this dissertation together with TCP agent and P2P client compose the P2P real time streaming system. This system introduces the mechanism of quick retransfer of lost data and proper partition of data piece in order to help meeting real time requirement.
The main task in this dissertation includes the design, implementation and test of the P2P real-time streaming server. As the data source and allocator of P2P media, the system emphasizes the save, allocation and transmission of streaming media and it also makes a good attempt in combining transmission of C/S and P2P and it could be widely used in the high real time requiring place, such as video conference and live TV. In addition, the system also put NAT which is very common in P2P application into consideration.
为了解决该问题,出现了基于P2P技术的实时流媒体。它利用P2P系统的高容量、易扩展、廉价以及自组织等特点再与流媒体技术有机结合,从而得到了广泛应用,比如当前主流的QQLive,PPLive,UUSEE等。通过这种技术,可以大大缓解中心服务器的压力,并具有良好的服务和扩展能力。但是和普通的P2P应用不同,在流媒体技术中,实时性占据着很重要的地位,所以P2P流媒体技术需要更好地分割、快速重传丢失的资源来满足实时性的要求。本文描述的P2P实时流媒体系统由P2P实时流媒体服务器、TCP代理和P2P客户端构成。
本文主要对P2P实时流媒体服务器的设计、实现和系统测试做较为全面的阐述。该服务器既是数据源又是资源索引服务器,它着重从资源管理的角度实现了流媒体数据的保存、分配和发送,并借鉴流媒体技术中的应用层组播模型以及NAT穿透模型来提供P2P的传输支持。同时,作为原始数据源,该服务器在多通道服务以及C/S和P2P混合传输的问题上也进行了有益的探索。该系统能够应用于许多场合,例如IPTV,视频会议,以及现场直播等实时性要求高的地方。
P2P streaming technology developed quickly in recent years. Streaming technology can significantly reduce time delay and memory requirement for the whole media file need not completely download on the local disk when streaming media is played. As the Internet and web service become a commonplace, video on demand (VOD) and real time streaming services appear. Unfortunately, many streaming systems providing the above services still follow the traditional Client-Server(C/S) principle in which all the media service is provided by the server only. Under C/S architecture, the server's overload would be worsened by more clients joint in system. Under this circumstance, taking the weak points of C/S into account, a number of researchers presented several solutions for real time streaming system, for instance, IP multicast. In IP multicast, the network keeps only one copy of data packet on IP layer of TCP/IP, each client requires the data that can get the copy without affecting the original one. However, this method does not get enough support from rooter for violating the principle of simplicity on IP layer.
Considering the aspects above, the P2P-based streaming system comes into being. This kind of system absorbs the advantages of both P2P and streaming system. The high capacity of service, ease of scalability, lower price and self-organization, those features plus NAT enable the P2P-based streaming system widely used on Internet, such as QQLive, PPLive and UUSEE. Every client in the system can either be data supplier or data consumer, and it can even be the both which is decided by the node's service capability and requirement on streaming media. In the same time, system's real time feature is quite emphasized. The P2P real time streaming server described in this dissertation together with TCP agent and P2P client compose the P2P real time streaming system. This system introduces the mechanism of quick retransfer of lost data and proper partition of data piece in order to help meeting real time requirement.
The main task in this dissertation includes the design, implementation and test of the P2P real-time streaming server. As the data source and allocator of P2P media, the system emphasizes the save, allocation and transmission of streaming media and it also makes a good attempt in combining transmission of C/S and P2P and it could be widely used in the high real time requiring place, such as video conference and live TV. In addition, the system also put NAT which is very common in P2P application into consideration.
引文
[1]罗杰文.Peer-to-Peer(P2P)综述.中科院计算机研究所,2005.
[2]Peter Maymounkov,David Mazieres.Kademlia:A peer-to-peer information system based on the XOR metric.In:Proc.Of the 1~(st) Int'l workshop on Peer-to-Peer Systems(IPTPS 2002).2002.153-161.
[3]Salman A,Baset,Henning Schulzrinne.An Analysis of the Skype Peer-to-Peer Internet Telephony Protocol.Department of Computer Science,Columbia University,New York NY 10027.September 15,2004.
[4]Dapeng Wu,Yiwei Thomas Hou,Wenwu Zhn,Streaming Video over the Internet:Approaches and Directions.IEEE Transactions on Cricuits and System for Video Technology,2001.
[5]ITU-T Recommendation H.332.Packet-based multimedia communications systems[S].1998.
[6]Stephen E.D,David.C.Multicast Routing in Datagram Internet works and Extended LANS.ACM Transactions on Computer Systems,May 1990,85-110.
[7]Stevens,W.R.著.TCP/IP详解卷1:协议.北京:机械工业出版社.2002,78-90.
[8]Handley.M,Floyd.S,Pahdye.J et,al.TCP Friendly Rate Control(TFRC):Protocol Specificatoin.RFC 3448,January 2003.
[9]Y.H.Chu,S.G.Rao,H.Zhang.A Case for End System Multicast.In Proceedings of ACM SIGMETRICS,California,United States,June 2000,35-38.
[10]BaneRJee S,Bhattacharjee B,Kommareddy C,and Varghese G.Scalable application layer multicast.In Proc.Of ACM SIG-COMM'02,Pittsburgh,PA,USA,August,2002,205-220.
[11]Saint Louis,Missourt.Design Of Overlay Networks for Internet Multicast[dissertation].Univ.Washington,Saint Louis,USA,August 2002.
[12]Duc.A.T,Kien.A,Hua,et,al.A Peer-to-Peer Architecture for Media Streaming.To appear in IEEE JSAC Special Issue on Advances in Overlay Networks,2003.
[13]Karl-Andre Skevik,Vera Goebel,Thomas Plagemann,"Analysis of BitTorrent and its use for the Design of P2P based Streaming Protocol for a Hybrid CDN",Department of Informatics University of Osl.
[14]德加拉音视频压缩卡开发技术文档,版本:V1.8.10,2005.
[15]Napster.http://www.napster.com.
[16]Open Source Community.Gnutella.In http://gnutella.wego.com,2001.
[17]A.Rowstron and P.Druschel."Pastry:Scalable Distributed Object Location and Routing for Large-scale Peer-to-Peer Systems".Proceedings of IFIP/ACM Middleware 2001,Nov.2001.
[18]F.Dabek,B.Zhao,P.Druschel,et,al.Towards a common API for structured peer-to-peer overlays.In Proc.IPTPS 2003,Berkeley,California,2003.
[19]Rowstron A,Druschel P.Pastry:Scalable,distributed object location and routing for large-scale Peer-to-Peer systems[A].Proceedings of Middleware 2001[C]Gemany,2001.
[20]Bittorrent.http://www.bittorrent.com.
[21]KaZaA file sharing network.KaZaA.In http://www.kazaa.com.2002.
[22]Jon Stoica,Robert Morris,David Karger,M.Frans Kaashoek,Hari Balakrishnan."Chord A Scalable Peer-to-peer Lookup Service for Internet Applications".In Proceeding of ACM SIGGOMM 2001,San Diego,California,USA.
[23]Chen Dongfeng,Yang Shoubao,Peng Xiaoyan.TaChord:a Chord System Using Topology-Aware Routing and Super Peers.Department of Computer Science,University Science and Technology of China,Hefei 230026,China.
[24]Anjali Harren,Barbara Liskov,Rodrigo Rodigues.One Hop Lookups for Peer-to-Peer Overlays.9~(th) Workshop on Hot Topics in Operating Systems(HOTOS Ⅸ).May 2003.
[25]Wu.D,Hou.Y.T,Zhang.Y.Q.Transporting real-time video over Internet:Challenges and approaches.Proc.IEEE,December 2000.1855-1875.
[26]董海韬,基于对等网络的流媒体传输系统技术研究.清华大学硕士论文,2005年5月.
[27]H.Deshpande,M.Bawa,and H.Garcia-Molina."Streaming Live Media over a Peer-to-Peer Network",Technical Report,Stanford University,August 2001.
[28]H.Schulzrinne A.Rao R.Lanphier RFC2326 2326-1998 Real Time Streaming Protocal IETF Network Working Group,1998.
[29]Handbook of Communication Technologies:The Next Decade-Multimedia over IP:RSVP,RTP,RTCP,RTSP,R.Osso,Ed.CRC Press,BocaRaton,1999,29-46.
[30]AVS Video Expert Group,Information Technology-Advanced Audio Video Coding Standard Part2:Video,in Audio Video Coding Standard Group of China(AVS),Doe.AVS-N1063,Dec.2003
[31]James F.Kurose,Keith W.Ross.Computer Networking:A Top-Down Approach Featuring the Internet,Third Edition,2005,110-120.
[32]J.Jannotti,D.K.Giord,K.L.Johnson,M.F.Kaashoek and J.W.O'Toole,"Overcast:Reliable Multicasting with an Overlay Network",in Proceedings of the Fourth Symposium on Operating Systems Design and Implementation,2000.
[33]M.Heffeeda,A.Habib,B.Botev,D.Xu,and B.Bhargava.PROMISE:peer-to-peer media streaming using CollectCast.In Proc.www'03,Budapest,CA,Jun,2003.
[34]A.J.Ganesh,A.M.Kermarrec,and L.Massoulie.Peer-to-peer membership management for gossip-based protocols.IEEE Transactions on Computers,Feb,2003.
[35]A.Nicolosi and S.Annapureddy,"P2PCast:A Peer-to-Peer Multieast Scheme for Streaming Data",in IRIS Student Workshop,MIT,2003.
[36]Gou Xiantai,Jin Weidong.Distributed Broker-Agent Architecture for Multimedia Communicatoins Traversing NAT/Firewall in NGN.Journal of Southwest.Jiaotong University.Nov.2004.
[37]P.Srisuresh,M.Holdrege IP Network Address Translator(NAT) Terminology and Considerations RFC2663 Network Working Group August 1999.
[38]J.Rosenberg Interactive Connectivity Establishment(ICE):A Methodology for Network Address Translator(NAT) Traversal for Offer/Answer Protocals MMUSIC Internet-Draft,October 19,2005,Expires:April 22,2006.
[39]Jack Lee Y B and Leng W T.Study of a Server-less Architecture for Video-on-Demand Applications.Proe IEEE International Conference on Multimedia and Expot.Lansanne,Switzerland,August 2002,26-29,233-236.
[40]肖自美.图像信息理论与压缩编码技术.中山大学出版社,1994.
[41]ITU-T Recommendation H.264-ISO/IEC 14496-10 AVC.Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification.Mar.2003.
[42]J.Ostermann,J.Bormans,P.List,et al.Video coding with H.264/AVC:Tools,performance,and complexity.IEEE Circuits and Systems Magazine.2004,4(1),7-28.
[43]Anthony Jones,Jim Ohlund.Network Programming for Microsoft windows second edition.
[44]侯俊桀,深入浅出MFC,松岗电脑图资料股份有限公司,1998,178-233.
[45]张宝社,张宝峰,王艳辉.Windows界面下的网络编程.中国科学技术大学出版社,1997,255-287.
[46]Bruce Eckel.Thinking in Patterns,Problem-Solving Techniques using Java.President,MindView,Inc.2003,120-152.
[47]施炜,李铮,秦颖.Windows Sockets规范及应用,Windows网络编程接口.1996,30-49.
[48]Mircosoft.MSDN Libaray.July 2001.
[49]汤子瀛.计算机操作系统(修订版).西安电子科技大学.2002,177-200.
[50]Stanley B Lippman,Josee Lajoie.C++ Primer the 3~(rd) edition.2002,366-410.
[2]Peter Maymounkov,David Mazieres.Kademlia:A peer-to-peer information system based on the XOR metric.In:Proc.Of the 1~(st) Int'l workshop on Peer-to-Peer Systems(IPTPS 2002).2002.153-161.
[3]Salman A,Baset,Henning Schulzrinne.An Analysis of the Skype Peer-to-Peer Internet Telephony Protocol.Department of Computer Science,Columbia University,New York NY 10027.September 15,2004.
[4]Dapeng Wu,Yiwei Thomas Hou,Wenwu Zhn,Streaming Video over the Internet:Approaches and Directions.IEEE Transactions on Cricuits and System for Video Technology,2001.
[5]ITU-T Recommendation H.332.Packet-based multimedia communications systems[S].1998.
[6]Stephen E.D,David.C.Multicast Routing in Datagram Internet works and Extended LANS.ACM Transactions on Computer Systems,May 1990,85-110.
[7]Stevens,W.R.著.TCP/IP详解卷1:协议.北京:机械工业出版社.2002,78-90.
[8]Handley.M,Floyd.S,Pahdye.J et,al.TCP Friendly Rate Control(TFRC):Protocol Specificatoin.RFC 3448,January 2003.
[9]Y.H.Chu,S.G.Rao,H.Zhang.A Case for End System Multicast.In Proceedings of ACM SIGMETRICS,California,United States,June 2000,35-38.
[10]BaneRJee S,Bhattacharjee B,Kommareddy C,and Varghese G.Scalable application layer multicast.In Proc.Of ACM SIG-COMM'02,Pittsburgh,PA,USA,August,2002,205-220.
[11]Saint Louis,Missourt.Design Of Overlay Networks for Internet Multicast[dissertation].Univ.Washington,Saint Louis,USA,August 2002.
[12]Duc.A.T,Kien.A,Hua,et,al.A Peer-to-Peer Architecture for Media Streaming.To appear in IEEE JSAC Special Issue on Advances in Overlay Networks,2003.
[13]Karl-Andre Skevik,Vera Goebel,Thomas Plagemann,"Analysis of BitTorrent and its use for the Design of P2P based Streaming Protocol for a Hybrid CDN",Department of Informatics University of Osl.
[14]德加拉音视频压缩卡开发技术文档,版本:V1.8.10,2005.
[15]Napster.http://www.napster.com.
[16]Open Source Community.Gnutella.In http://gnutella.wego.com,2001.
[17]A.Rowstron and P.Druschel."Pastry:Scalable Distributed Object Location and Routing for Large-scale Peer-to-Peer Systems".Proceedings of IFIP/ACM Middleware 2001,Nov.2001.
[18]F.Dabek,B.Zhao,P.Druschel,et,al.Towards a common API for structured peer-to-peer overlays.In Proc.IPTPS 2003,Berkeley,California,2003.
[19]Rowstron A,Druschel P.Pastry:Scalable,distributed object location and routing for large-scale Peer-to-Peer systems[A].Proceedings of Middleware 2001[C]Gemany,2001.
[20]Bittorrent.http://www.bittorrent.com.
[21]KaZaA file sharing network.KaZaA.In http://www.kazaa.com.2002.
[22]Jon Stoica,Robert Morris,David Karger,M.Frans Kaashoek,Hari Balakrishnan."Chord A Scalable Peer-to-peer Lookup Service for Internet Applications".In Proceeding of ACM SIGGOMM 2001,San Diego,California,USA.
[23]Chen Dongfeng,Yang Shoubao,Peng Xiaoyan.TaChord:a Chord System Using Topology-Aware Routing and Super Peers.Department of Computer Science,University Science and Technology of China,Hefei 230026,China.
[24]Anjali Harren,Barbara Liskov,Rodrigo Rodigues.One Hop Lookups for Peer-to-Peer Overlays.9~(th) Workshop on Hot Topics in Operating Systems(HOTOS Ⅸ).May 2003.
[25]Wu.D,Hou.Y.T,Zhang.Y.Q.Transporting real-time video over Internet:Challenges and approaches.Proc.IEEE,December 2000.1855-1875.
[26]董海韬,基于对等网络的流媒体传输系统技术研究.清华大学硕士论文,2005年5月.
[27]H.Deshpande,M.Bawa,and H.Garcia-Molina."Streaming Live Media over a Peer-to-Peer Network",Technical Report,Stanford University,August 2001.
[28]H.Schulzrinne A.Rao R.Lanphier RFC2326 2326-1998 Real Time Streaming Protocal IETF Network Working Group,1998.
[29]Handbook of Communication Technologies:The Next Decade-Multimedia over IP:RSVP,RTP,RTCP,RTSP,R.Osso,Ed.CRC Press,BocaRaton,1999,29-46.
[30]AVS Video Expert Group,Information Technology-Advanced Audio Video Coding Standard Part2:Video,in Audio Video Coding Standard Group of China(AVS),Doe.AVS-N1063,Dec.2003
[31]James F.Kurose,Keith W.Ross.Computer Networking:A Top-Down Approach Featuring the Internet,Third Edition,2005,110-120.
[32]J.Jannotti,D.K.Giord,K.L.Johnson,M.F.Kaashoek and J.W.O'Toole,"Overcast:Reliable Multicasting with an Overlay Network",in Proceedings of the Fourth Symposium on Operating Systems Design and Implementation,2000.
[33]M.Heffeeda,A.Habib,B.Botev,D.Xu,and B.Bhargava.PROMISE:peer-to-peer media streaming using CollectCast.In Proc.www'03,Budapest,CA,Jun,2003.
[34]A.J.Ganesh,A.M.Kermarrec,and L.Massoulie.Peer-to-peer membership management for gossip-based protocols.IEEE Transactions on Computers,Feb,2003.
[35]A.Nicolosi and S.Annapureddy,"P2PCast:A Peer-to-Peer Multieast Scheme for Streaming Data",in IRIS Student Workshop,MIT,2003.
[36]Gou Xiantai,Jin Weidong.Distributed Broker-Agent Architecture for Multimedia Communicatoins Traversing NAT/Firewall in NGN.Journal of Southwest.Jiaotong University.Nov.2004.
[37]P.Srisuresh,M.Holdrege IP Network Address Translator(NAT) Terminology and Considerations RFC2663 Network Working Group August 1999.
[38]J.Rosenberg Interactive Connectivity Establishment(ICE):A Methodology for Network Address Translator(NAT) Traversal for Offer/Answer Protocals MMUSIC Internet-Draft,October 19,2005,Expires:April 22,2006.
[39]Jack Lee Y B and Leng W T.Study of a Server-less Architecture for Video-on-Demand Applications.Proe IEEE International Conference on Multimedia and Expot.Lansanne,Switzerland,August 2002,26-29,233-236.
[40]肖自美.图像信息理论与压缩编码技术.中山大学出版社,1994.
[41]ITU-T Recommendation H.264-ISO/IEC 14496-10 AVC.Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification.Mar.2003.
[42]J.Ostermann,J.Bormans,P.List,et al.Video coding with H.264/AVC:Tools,performance,and complexity.IEEE Circuits and Systems Magazine.2004,4(1),7-28.
[43]Anthony Jones,Jim Ohlund.Network Programming for Microsoft windows second edition.
[44]侯俊桀,深入浅出MFC,松岗电脑图资料股份有限公司,1998,178-233.
[45]张宝社,张宝峰,王艳辉.Windows界面下的网络编程.中国科学技术大学出版社,1997,255-287.
[46]Bruce Eckel.Thinking in Patterns,Problem-Solving Techniques using Java.President,MindView,Inc.2003,120-152.
[47]施炜,李铮,秦颖.Windows Sockets规范及应用,Windows网络编程接口.1996,30-49.
[48]Mircosoft.MSDN Libaray.July 2001.
[49]汤子瀛.计算机操作系统(修订版).西安电子科技大学.2002,177-200.
[50]Stanley B Lippman,Josee Lajoie.C++ Primer the 3~(rd) edition.2002,366-410.