流媒体服务器迁移技术研究
|
摘要
近十几年来,计算机网络技术得到迅猛的发展,流媒体传输技术在各行业领域得以广泛的应用和发展,如视频会议、视频点播、电话和远程教育等,流媒体已经从实验室走进人们的日常生活中。
集群式流媒体系统能够用相对较为廉价的方式提供较强的可扩展性和良好的吞吐性能,给流媒体用户提供更好更优质的服务。在集群流媒体系统中,由于影片的流行度有差异,用户对流媒体的访问存在一定的倾斜度,存储不同类型影片的各服务器节点负载往往达不到合理的分布,流行度高的影片所在的服务器接收到的请求数远远大于流行度低的影片所在的服务器,服务迁移技术是解决这一问题的有效手段之一。服务迁移技术,即把负载高的服务器在线请求实时迁移到低负载的服务器上,通过它可以重新改变系统各节点的负载分布,使各节点之间达到一定的负载均衡度,优化负载资源配置,从而提高系统服务质量。
本文针对当今集群流媒体系统中的负载均衡和实时迁移需求,并结合集群流媒体系统自身的特殊要求,实现了两类不同流行度影片服务请求的迁移技术,并成功的应用在国家“863”项目《多节点协作的集群流媒体服务器系统设计》中,为本项目所需要的实时迁移技术提出了完整的解决方案。对于热门影片的请求迁移过程,分为迁移“走”出源服务器和迁移“到”目标服务器两个步骤,文章先详细分析了这两步骤的细节设计过程,再用性能分析和实验结果证明了此方法的可行性和实用性。较热门文件的请求迁移而言,冷门文件的请求迁移过程实现相对复杂,不仅涵盖了迁移“走”和迁移“到”两个基本流程,还包括数据流的缓存机制。缓存在实现上是个难题,本文提出一种实用的缓存方法,同时给出数据流的缓存和发送时参数修改的规则,确保迁移前后数据流参数的一致性,并用实验结果证明了其正确性。
本文另一项工作是对迁移算法的研究,调研和分析了现有随即早期迁移REM算法,并针对其不足之处,给出了一种基于负载均衡的请求迁移算法。仿真结果证明,基于负载均衡的请求迁移算法,能有效的提高系统负载均衡度,优化了各节点之间的资源分配,从而提高了平均请求接收率,改进了系统的服务能力。
本文的创新点在于迁移实现的实用可靠和透明性,不需要修改客户端。这种灵活的迁移技术,为先进的服务器系统优化方法的提出和应用提供了巨大的空间。
Over the past ten years, with the rapid development of computer network, streaming media technology has been playing an important role in various industries, such as video conferencing, video on demand, tele-education and so on. Streaming media technology has applied successfully in daily life from the laboratory.
Streaming media cluster system can provide better scalability and good throughput performance in a way of relatively low cost and streaming media users can enjoy better service. However, in streaming media cluster system, because of the different popularity of streaming media files, users visit the various streaming media files in an inclined way. Thus the servers storaging different types of meida receive quite different load from each other. Servers with hot media copies receive much more client requests than other servers and are most likely to be overloaded. Service migration technology is one of effective solutions to this problem. Through service migration, we can move some on-line requests from the high loaded servers to low loaded servers. Service migration can redistribute the system load, achieve a certain degree of load balance, optimize the load allocation of resources and enhance the quality of service.
This paper aim to the demand for load balancing and real-time migration in today's streaming media cluster system, with consideration of the special requirements of the system itself, and realize the request migration technology in two types of media files. What is more, it has successfully used in the country“863”project,“multi-node streaming media cluster server system design”, and gives a complete solution to the demand for real-time migration in the project. The process of request migration of hot media can be divided into two steps: migrating from the source server and migrating to the target sever. This paper first analyses the detailed design of the two steps, and then does experiments to prove the feasibility and practicality of the method proposed. Compared with the request migration of the hot media files, the request migration for the unpopular file is more complicated. The process for the unpopular file covers not only the two steps mentioned above, but also the data stream caching mechanisms. Caching is a problem in the realization. This article presents a practical method for caching, including changing the parameters when caching and sending data stream, and the design of the caching file format. Experiment results show that the data stream maintains consistency between the migration operation.
Another important work of this paper is the study of request migration algorithm. Base on the existing REM (Random Early Migration) algorithm, and analysis of its deficiency, this paper proposes a new request migration algorithm based on load balancing. Simulation results show that the algorithm based on load balancing can effectively improve system load balancing, optimize the load allocation of resources between nodes, thereby increase the average rate of requests accepted, and improve the service capacity of the system.
The innovation of this paper is the practicality and reliability of the migration technology proposed, without needing to modify the client code. Such a flexible migration technology provides a great space to optimization in advanced streaming server system.
集群式流媒体系统能够用相对较为廉价的方式提供较强的可扩展性和良好的吞吐性能,给流媒体用户提供更好更优质的服务。在集群流媒体系统中,由于影片的流行度有差异,用户对流媒体的访问存在一定的倾斜度,存储不同类型影片的各服务器节点负载往往达不到合理的分布,流行度高的影片所在的服务器接收到的请求数远远大于流行度低的影片所在的服务器,服务迁移技术是解决这一问题的有效手段之一。服务迁移技术,即把负载高的服务器在线请求实时迁移到低负载的服务器上,通过它可以重新改变系统各节点的负载分布,使各节点之间达到一定的负载均衡度,优化负载资源配置,从而提高系统服务质量。
本文针对当今集群流媒体系统中的负载均衡和实时迁移需求,并结合集群流媒体系统自身的特殊要求,实现了两类不同流行度影片服务请求的迁移技术,并成功的应用在国家“863”项目《多节点协作的集群流媒体服务器系统设计》中,为本项目所需要的实时迁移技术提出了完整的解决方案。对于热门影片的请求迁移过程,分为迁移“走”出源服务器和迁移“到”目标服务器两个步骤,文章先详细分析了这两步骤的细节设计过程,再用性能分析和实验结果证明了此方法的可行性和实用性。较热门文件的请求迁移而言,冷门文件的请求迁移过程实现相对复杂,不仅涵盖了迁移“走”和迁移“到”两个基本流程,还包括数据流的缓存机制。缓存在实现上是个难题,本文提出一种实用的缓存方法,同时给出数据流的缓存和发送时参数修改的规则,确保迁移前后数据流参数的一致性,并用实验结果证明了其正确性。
本文另一项工作是对迁移算法的研究,调研和分析了现有随即早期迁移REM算法,并针对其不足之处,给出了一种基于负载均衡的请求迁移算法。仿真结果证明,基于负载均衡的请求迁移算法,能有效的提高系统负载均衡度,优化了各节点之间的资源分配,从而提高了平均请求接收率,改进了系统的服务能力。
本文的创新点在于迁移实现的实用可靠和透明性,不需要修改客户端。这种灵活的迁移技术,为先进的服务器系统优化方法的提出和应用提供了巨大的空间。
Over the past ten years, with the rapid development of computer network, streaming media technology has been playing an important role in various industries, such as video conferencing, video on demand, tele-education and so on. Streaming media technology has applied successfully in daily life from the laboratory.
Streaming media cluster system can provide better scalability and good throughput performance in a way of relatively low cost and streaming media users can enjoy better service. However, in streaming media cluster system, because of the different popularity of streaming media files, users visit the various streaming media files in an inclined way. Thus the servers storaging different types of meida receive quite different load from each other. Servers with hot media copies receive much more client requests than other servers and are most likely to be overloaded. Service migration technology is one of effective solutions to this problem. Through service migration, we can move some on-line requests from the high loaded servers to low loaded servers. Service migration can redistribute the system load, achieve a certain degree of load balance, optimize the load allocation of resources and enhance the quality of service.
This paper aim to the demand for load balancing and real-time migration in today's streaming media cluster system, with consideration of the special requirements of the system itself, and realize the request migration technology in two types of media files. What is more, it has successfully used in the country“863”project,“multi-node streaming media cluster server system design”, and gives a complete solution to the demand for real-time migration in the project. The process of request migration of hot media can be divided into two steps: migrating from the source server and migrating to the target sever. This paper first analyses the detailed design of the two steps, and then does experiments to prove the feasibility and practicality of the method proposed. Compared with the request migration of the hot media files, the request migration for the unpopular file is more complicated. The process for the unpopular file covers not only the two steps mentioned above, but also the data stream caching mechanisms. Caching is a problem in the realization. This article presents a practical method for caching, including changing the parameters when caching and sending data stream, and the design of the caching file format. Experiment results show that the data stream maintains consistency between the migration operation.
Another important work of this paper is the study of request migration algorithm. Base on the existing REM (Random Early Migration) algorithm, and analysis of its deficiency, this paper proposes a new request migration algorithm based on load balancing. Simulation results show that the algorithm based on load balancing can effectively improve system load balancing, optimize the load allocation of resources between nodes, thereby increase the average rate of requests accepted, and improve the service capacity of the system.
The innovation of this paper is the practicality and reliability of the migration technology proposed, without needing to modify the client code. Such a flexible migration technology provides a great space to optimization in advanced streaming server system.
引文
Anker T,Dolev D,Keidar I.Fault Tolerant Video on DemandServices. 19th IEEE ICDCS,1999。
Cui Y,Nahrsted t K,Xun,Seamless User-level Handoff in Ubiquitous Multimedia Service Delivery.Multimedia Tools and Applcations,2004,22:137～170。
D.Wu,T Hou,andY-Q,Zhang.Transporting real-time video over the Internet:C hallenges and approaches. Proc.of the IEEE,vo l.88 ,pp.1855-1875,December2 000.
H. Schulzrinne,S.Casner,RTP Profile for Audio and Video Conferences with Minimal Control,IETF RFC3551,2003。
H. Schulzrinne,S.Casner,RTCP:Real Time Control ProtocoI.IETF RFC3550。June 2003。
H. Schulzrinne,S.Casner,RFC2326, RTSP: Real Time Streaming ProtocoI.IETF RFC2326。June 2003。
Iain E. G. Richardson , H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia,John Wiley & Sons, Ltd,2003
L. Qiu, V. N. Padmanabhan, and G. M. Voelker,“On the placement of web server replicas,”in INFOCOM,pp. 1587–1596, 2001.
Minglong Zhang , Boqin Feng, A Novel Migration Algorithm Based-On“States-Balancing”in a Distributed Multimedia Services System[J], Multimedia and Ubiquitous Engineering, IEEE conference April 2008
J. Kangasharju, J. Roberts, and K. Ross,“Object replication strategies in content distribution networks,”in Sixth Web Caching and Content Distribution Workshop, 2001.
J. L. Wolf, P. S. Yu, and H. Shachinai. Disk load balancing for video-on-demand systems[J]. ACM/Springer Multimedia Systems Journal, 5(6):358–370, 1997。
P. Radoslavov, R. Govindan, and D. Estrin,“Topology-informed internet replica placement,”in Sixth Web Caching and Content Distribution Workshop, 2001.
Ross Carter,Microsoft Real-Time Communications: Protocols and Technologies,Microsoft Corporation,2003。
Roy S,Shen B,Sundaram V.Application 1.evel Hand—off Support for Mobile Media Transcoding Sessions.NOSSDAV'02.Mar。
S. Casner,R. Frederick,V. Jacobson,RFC3550,RTP: A Transport Protocol for Real-Time Applications,2003。
S. Tsao, M. C. Chen, M. Ko, J. Ho, and Y. M. Huang, Data allocation and dynamic load balancingfor distributed video storage server[J], Journal of Visual Communication and Image Representation, vol. 10, pp. 197–218, 1999.
S. Jamin, C. Jin, A. R. Kurc, D. Raz, and Y. Shavitt,“Constrained mirror placement on the internet,”inINFOCOM, pp. 31–40, 2001.
Yinqing Zhao, C.-C. Jay Kuo , video-on-demand server system design with random early migration,Circuits and Systems[J], 2003. ISCAS '03. Proceedings of the 2003 International Symposium on Volume 2, May 2003
Akamai. http://www.akamai.com.
Digital Island. http://www.digisle.net.
MPEG Industry Forum , MPEG-4– The Media Standard,http://www.m4if.org/public /documents/vault/m4-out-20027.pdf,2002。
MPEG-4 Applications, ISO/IEC JTC1/SC29/WG11 MPEG 99/N2724,March,1999。
The Real System SDK。http://www.realnetworks.com/resources/sdk/index.Html www.microsoft.com/windows/windowsmedialdefault.aspx
www.apple.com/quicktime/streamingserver
陈桦,张尧学,多媒体服务质量协商控制系统,《清华人学学报》.Vol.38,36-39,1998。
李涛,基于WEB的网络视频直播系统的设计与实现,西南交通大学, 2004。
孔英会,李庆国,常喜茂译,流媒体技术及商机揭密,电子工业出版社,2004。
刘衡竹,陈旭灿,陈福按,视频点播服务器中服务策略的研究,软件学报,V0l.10,No.I,Jan.1999.
谭海燕,实时视频流传输与控制的研究,中山大学, 2004: 5~6, 9~10.
魏恒义,程竹林,刘伟娜,等,多媒体会议系统中的视音频传输模块,计算机工程,2003,第13期:150~152。
吴国勇,邱学刚,万燕仔,网络视频—流媒体技术与应用,北京,北京邮电大学出版社, 2001: 9~12。
余海燕,郑笑飞,几种负载均衡解决方案的比较信息系统工程2000 (9) P28
赵水宁,邵军力,多WEB服务器负载均衡技术的研究电信科学2001 P6-8
钟玉琢,王琪,赵黎,杨小勤,MPEG-2运动图像压缩编码国际标准及MPEG的新进展清华大学出版社,2002年7月。
朱鹏,李春文,基于RTP的网络视频传输系统的设计与实现,计算机工程与应用,2003。
《流媒体技术的应用与发展》,流媒体论坛,http://www.liumeiti.org/tech/technical/2005-07/11911.html,
流媒体技术基础, http://tips.cqbluedon.com/search/newsdisplay4.asp?id=566&sort=4。
Cui Y,Nahrsted t K,Xun,Seamless User-level Handoff in Ubiquitous Multimedia Service Delivery.Multimedia Tools and Applcations,2004,22:137～170。
D.Wu,T Hou,andY-Q,Zhang.Transporting real-time video over the Internet:C hallenges and approaches. Proc.of the IEEE,vo l.88 ,pp.1855-1875,December2 000.
H. Schulzrinne,S.Casner,RTP Profile for Audio and Video Conferences with Minimal Control,IETF RFC3551,2003。
H. Schulzrinne,S.Casner,RTCP:Real Time Control ProtocoI.IETF RFC3550。June 2003。
H. Schulzrinne,S.Casner,RFC2326, RTSP: Real Time Streaming ProtocoI.IETF RFC2326。June 2003。
Iain E. G. Richardson , H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia,John Wiley & Sons, Ltd,2003
L. Qiu, V. N. Padmanabhan, and G. M. Voelker,“On the placement of web server replicas,”in INFOCOM,pp. 1587–1596, 2001.
Minglong Zhang , Boqin Feng, A Novel Migration Algorithm Based-On“States-Balancing”in a Distributed Multimedia Services System[J], Multimedia and Ubiquitous Engineering, IEEE conference April 2008
J. Kangasharju, J. Roberts, and K. Ross,“Object replication strategies in content distribution networks,”in Sixth Web Caching and Content Distribution Workshop, 2001.
J. L. Wolf, P. S. Yu, and H. Shachinai. Disk load balancing for video-on-demand systems[J]. ACM/Springer Multimedia Systems Journal, 5(6):358–370, 1997。
P. Radoslavov, R. Govindan, and D. Estrin,“Topology-informed internet replica placement,”in Sixth Web Caching and Content Distribution Workshop, 2001.
Ross Carter,Microsoft Real-Time Communications: Protocols and Technologies,Microsoft Corporation,2003。
Roy S,Shen B,Sundaram V.Application 1.evel Hand—off Support for Mobile Media Transcoding Sessions.NOSSDAV'02.Mar。
S. Casner,R. Frederick,V. Jacobson,RFC3550,RTP: A Transport Protocol for Real-Time Applications,2003。
S. Tsao, M. C. Chen, M. Ko, J. Ho, and Y. M. Huang, Data allocation and dynamic load balancingfor distributed video storage server[J], Journal of Visual Communication and Image Representation, vol. 10, pp. 197–218, 1999.
S. Jamin, C. Jin, A. R. Kurc, D. Raz, and Y. Shavitt,“Constrained mirror placement on the internet,”inINFOCOM, pp. 31–40, 2001.
Yinqing Zhao, C.-C. Jay Kuo , video-on-demand server system design with random early migration,Circuits and Systems[J], 2003. ISCAS '03. Proceedings of the 2003 International Symposium on Volume 2, May 2003
Akamai. http://www.akamai.com.
Digital Island. http://www.digisle.net.
MPEG Industry Forum , MPEG-4– The Media Standard,http://www.m4if.org/public /documents/vault/m4-out-20027.pdf,2002。
MPEG-4 Applications, ISO/IEC JTC1/SC29/WG11 MPEG 99/N2724,March,1999。
The Real System SDK。http://www.realnetworks.com/resources/sdk/index.Html www.microsoft.com/windows/windowsmedialdefault.aspx
www.apple.com/quicktime/streamingserver
陈桦,张尧学,多媒体服务质量协商控制系统,《清华人学学报》.Vol.38,36-39,1998。
李涛,基于WEB的网络视频直播系统的设计与实现,西南交通大学, 2004。
孔英会,李庆国,常喜茂译,流媒体技术及商机揭密,电子工业出版社,2004。
刘衡竹,陈旭灿,陈福按,视频点播服务器中服务策略的研究,软件学报,V0l.10,No.I,Jan.1999.
谭海燕,实时视频流传输与控制的研究,中山大学, 2004: 5~6, 9~10.
魏恒义,程竹林,刘伟娜,等,多媒体会议系统中的视音频传输模块,计算机工程,2003,第13期:150~152。
吴国勇,邱学刚,万燕仔,网络视频—流媒体技术与应用,北京,北京邮电大学出版社, 2001: 9~12。
余海燕,郑笑飞,几种负载均衡解决方案的比较信息系统工程2000 (9) P28
赵水宁,邵军力,多WEB服务器负载均衡技术的研究电信科学2001 P6-8
钟玉琢,王琪,赵黎,杨小勤,MPEG-2运动图像压缩编码国际标准及MPEG的新进展清华大学出版社,2002年7月。
朱鹏,李春文,基于RTP的网络视频传输系统的设计与实现,计算机工程与应用,2003。
《流媒体技术的应用与发展》,流媒体论坛,http://www.liumeiti.org/tech/technical/2005-07/11911.html,
流媒体技术基础, http://tips.cqbluedon.com/search/newsdisplay4.asp?id=566&sort=4。