基于P2P流媒体点播系统的磁盘缓存策略研究
|
摘要
近年来,随着宽带网的高速发展,流媒体内容的服务日益受到宽带用户的青睐,其中基于P2P网络的流媒体技术成为当前研究的热点。而流媒体数据对网络的巨大带宽消耗,使得运营商的网络资源越来越紧张,服务质量也越来越差。而P2P技术采用完全分布式结构,这种技术弱化了服务器的作用,甚至取消服务器,网络中的节点既可以获取其他节点的资源或服务,同时又是资源或服务的提供者。
本文针对大规模P2P流媒体点播中的一项关键技术——磁盘缓存进行了深入的研究。本文主要完成以下工作:
首先,针对提高用户观看质量的需求与降低骨干网带宽消耗的矛盾,在P2P文件共享多源协同流调度思想的启发下,提出了一种基于经济模型的分布式缓存策略,策略通过采用分布式一致性算法计算视频分段的流行度,并根据分段内容流行度高低对缓存分段进行缓存置换,实验结果表明,这种算法有效降低了服务器的负载,显著降低了启动延时,比传统的分段缓存方法有更好的性能。
其次,在衡量被请求的数据分段的存储位置与结点对之间的通信频率的基础上,研究并设计了与缓存策略相关的缓存副本分布策略和缓存副本概率一致性策略,详细介绍了设计思想和策略特点。
最后,针对目前仿真软件不能有效模拟点播行为,在分析了传统的仿真器的基础上,设计并开发了适用于本策略的仿真器,详细说明了各模块的作用,并对所需参数的设置进行了说明。通过用其对传统缓存策略和本文策略对比,说明其有效性。
In recent years, with the rapid development of broadband network, streaming media content services are increasingly welcomed by broadband customers. P2P network streaming media technology has become the focus of current research. The streaming media data consumes huge network bandwidth, which causes more shortage of the operator's network resources, and worse quality of service. And P2P technology uses fully distributed architecture, which weakens and even cancels the function of the server, network nodes can access the resources of other nodes or service, at the same time, they are also providers.
In this paper, we did a deep research on the key technology of the large-scale P2P streaming media VOD system, which is the disk cache. The main work is as following:
Firstly, in view of contradiction between improving quality to meet the user watch and reducing the backbone bandwidth consumption, and inspired by P2P file-sharing in multi-source collaboration flow scheduling thought, this paper presents a caching strategy based on economy model for distributed streaming media. It uses distributed consistency algorithm computes the prevalence of video segments, and in accordance with the contents of the level of popularity, which gives the corresponding cache replacement strategy. Experimental results show that the algorithm can effectively reduce the load of the server, and significantly reduce the startup delay, and has better performance than the traditional segmented-cache method.
Secondly, considering both data storage location and the distance between requestor and responder, this paper research and design the strategies associated with the cache the cached copy of a copy of the distribution strategy and the probability of cache coherence strategy, introduced the characteristics of design thought and strategy in detail.
Finally, according to the situation that the current simulation software can not effectively simulate the video on-demand behavior, this paper designs and develops a the simulator suitable for the strategy on the basis of analyzing traditional simulator, and explains the role of each module in detail and also makes an explanation of setting up of required parameter. This paper illustrates its effectiveness by comparing traditional strategy of caching and strategy of this paper.
本文针对大规模P2P流媒体点播中的一项关键技术——磁盘缓存进行了深入的研究。本文主要完成以下工作:
首先,针对提高用户观看质量的需求与降低骨干网带宽消耗的矛盾,在P2P文件共享多源协同流调度思想的启发下,提出了一种基于经济模型的分布式缓存策略,策略通过采用分布式一致性算法计算视频分段的流行度,并根据分段内容流行度高低对缓存分段进行缓存置换,实验结果表明,这种算法有效降低了服务器的负载,显著降低了启动延时,比传统的分段缓存方法有更好的性能。
其次,在衡量被请求的数据分段的存储位置与结点对之间的通信频率的基础上,研究并设计了与缓存策略相关的缓存副本分布策略和缓存副本概率一致性策略,详细介绍了设计思想和策略特点。
最后,针对目前仿真软件不能有效模拟点播行为,在分析了传统的仿真器的基础上,设计并开发了适用于本策略的仿真器,详细说明了各模块的作用,并对所需参数的设置进行了说明。通过用其对传统缓存策略和本文策略对比,说明其有效性。
In recent years, with the rapid development of broadband network, streaming media content services are increasingly welcomed by broadband customers. P2P network streaming media technology has become the focus of current research. The streaming media data consumes huge network bandwidth, which causes more shortage of the operator's network resources, and worse quality of service. And P2P technology uses fully distributed architecture, which weakens and even cancels the function of the server, network nodes can access the resources of other nodes or service, at the same time, they are also providers.
In this paper, we did a deep research on the key technology of the large-scale P2P streaming media VOD system, which is the disk cache. The main work is as following:
Firstly, in view of contradiction between improving quality to meet the user watch and reducing the backbone bandwidth consumption, and inspired by P2P file-sharing in multi-source collaboration flow scheduling thought, this paper presents a caching strategy based on economy model for distributed streaming media. It uses distributed consistency algorithm computes the prevalence of video segments, and in accordance with the contents of the level of popularity, which gives the corresponding cache replacement strategy. Experimental results show that the algorithm can effectively reduce the load of the server, and significantly reduce the startup delay, and has better performance than the traditional segmented-cache method.
Secondly, considering both data storage location and the distance between requestor and responder, this paper research and design the strategies associated with the cache the cached copy of a copy of the distribution strategy and the probability of cache coherence strategy, introduced the characteristics of design thought and strategy in detail.
Finally, according to the situation that the current simulation software can not effectively simulate the video on-demand behavior, this paper designs and develops a the simulator suitable for the strategy on the basis of analyzing traditional simulator, and explains the role of each module in detail and also makes an explanation of setting up of required parameter. This paper illustrates its effectiveness by comparing traditional strategy of caching and strategy of this paper.
引文
[1] CRANOR C D,GREEN M,KALMANEK CHUCK.Enhanced Streaming Services in A Content Distribution Network[J].IEEE Internet Computing,2001,5(4):66-75.
[2] FOX C.Peer-to-Peer Networks[J].Computing in Science & Engineering, 2001,3(3):75-77.
[3] DESHPANDE H,BAWA M,GARCIA-MOLINA H.Streaming Live Media Over a Peer-to-Peer Network[R].USA:Stanford InfoLab,2002.
[4] YANG G,KYOUNGWON S,JIM K.A Peer-to-Peer on Demand Streaming Service and Its Performance Evaluation[C] . 2003 IEEE International Conference on Multimedia and Ex,Baltimore,MD,USA,2003:649~652.
[5] VENKATA N,PADMANABH,HELEN J.Distributing Streaming Media Content Using Cooperative Networking[C] . Proceedings of the 12th International Workshop on Network and Operating Systems Support for Digital Audio and Video:NOSSDAV 2002,Miami,FL,United states,2002:177~186.
[6] MIGUEL C,PETER D,KERMARREC A M.SplitStream:High-Bandwidth Content Distribution in Cooperative Environments[C].Berkeley,CA,USA,2003:292-303.
[7] DANA C,LI D,HARRISON,D CHUAH.BASS:BitTorrent Assisted Streaming System for Video-on-Demand Multimedia Signal Processing[C].2005 IEEE 7th Workshop on Multimedia Signal Processing,Shanghai,China,2005:1-4.
[8] LIAO X F,JIN H,LIU Y H.AnySee:Peer-to-Peer Live Streaming[C].IEEE INFOCOM’06,Barcelona,Spain,2006:1-10.
[9] S SAROIU.Measuring and Analyzing the Characteristics of Napster and Gnutella Hosts[J].Multimedia Systems Journal,2003,9(2):170-184.
[10]傅锡天,孙景鳌,蔡安妮.分布式VoD系统的分级存储调度[J],通信学报,2000,21(3):64-69.
[11] HIDEHARU S , KAZUTOSHI N , AKIRA U . Storage Hierarchy for Video-on-Demand Systems[C].In proceedings of SPIE,San Jose,CA,USA,1994:198-207.
[12] QIAO L,ZHANG Z,LI M Y.An Empirical Study of Collusion Behavior in the Maze P2P File-Sharing System[C] . 27th International Conference on Distributed Computing Systems (ICDCS '07),Toronto,ON,Canada,2006:10-11.
[13] XIANG Z,ZHANG Q,ZHU W.Peer-to-peer Based Multimedia Distribution Service[J].IEEE Trans on Multimedia.2004,6(2):343–355.
[14] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams[C].IEEE INFOCOM '99 Conference on Computer Communications,New York,NY,USA,1999:1310-1319.
[15] JáNOS I.Some Practical Aspects of Fitting and Testing the Zipf-Mandelbrot Model[J].Scientometrics,2006,1(67):107-120.
[16] JOAN B.Theoretical Evidence for the Superiority of LRU-2 over LRU for the Paging Problem[C].In Fourth Workshop on Approximation and Online Algorithms(WAOA),London,2006:95-107.
[17] LEONID B,SOKOLINSKY.LFU-K:An Effective Buffer Management Replacement Algorithm[C].Proceedings of the 9th Int'l Conf,Berlin,2004:670-681.
[18]田小波,陈蜀宇.基于最小效用的流媒体缓存替换算法[J].计算机应用,2007,27(3):733-736.
[19]廖鑫.一种基于LRU算法改进的缓存方案研究与实现[J].电子工程师,2008,34(7):46-48.
[20] XU D , HEFEEDA M , HAMBRUSH . On Peer-to-Peer Media Streaming[C].Proceedings of IEEE International Conference on Distributed Computing System(ICDCS).Vienna Austria,2002:363-371.
[21] WANG B,SEN S,ADLER M.Optimal Proxy Cache Allocation for Efficient Streaming Media Distribution[J].IEEE Transactions on Multimedia,2004,6(2):366-374.
[22] CHEN S Q,SHEN B,WEE S.Adaptive and Lazy Segmentation Based Proxy Caching for Streaming Media Delivery[C] . Proceedings of the 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video,Monterey,California,USA,2003:22-31.
[23] WU K L,YU P S,WOLF J L.Segmentation of Multimedia Streams for Proxy Caching[J].IEEE Transactions on Multimedia,2004,6(5):770- 780.
[24] WANG J.A Survey of Web Caching Schemes for the Internet[J].ACM Computer Communication Review,1999,29(5):36-46.
[25] GWETZMAN,SELTZERM.World Wide Web Cache Consistency[C].USA:Proceedings of the USEN IX Technical Conference,San Diego,1996:141-152.
[26] SAITO Y , KARAMANOLIS C, KARLSSON M . Taming Aggressive Replication in the Pangaea Wide-Area Files System[C].Proceedings of the 5th Symposium on Operating Systems Design and Implementation,Boston,USA,2002:15-30.
[27]张超群.代理缓存关键技术的研究[J].计算机工程与科学,2007,29(2):142-144.
[28]魏苏林,乐红冰.基于P2P模式的新型视频点播系统[J].微计算机信息(管控一体化),2005,(26):7-9.
[29] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams [C].Proceedings of IEEE INFOCOM,New York,1999:1310-1319.
[30] REJAIE R,HANDLEY M,YU H.Proxy Caching Mechanism for Multimedia Playback Streams in The Internet[C].Proceedings of International Web Caching Workshop,San Diego,CA,1999:12-19.
[31] WU K,YU P S,WOLF J.Segment-Based Proxy Caching of Multimedia Streams[C].Proceedings of WWW,Hong Kong,China,2001:36-44.
[32] REJAIE R,YU H,HANDELY M.Multimedia Proxy Caching Mechanism for Quality Adaptive Streaming Applications in the Internet[C].Proceedings of IEEE INFOCOM,Tel-Aviv,2000:980-989.
[33] KIM T,AMMAR M H.A Comparison of Layering and Stream Replication Video Multicast Schemes[C] . Port Jefferson . Proceedings of ACM NOSSDAV,NY, 2001:63-72.
[34] TAKANO R,YOSHIZAWA Y.Offloading VoD Server Organized Dynamically Distributed Cache Using P2P Delivery[C] . International Conference on Information Networking,Tokyo,2008:1-5.
[35] CHEN S Q,SHEN B,WEE S.Adaptive and Lazy Segmentation Based Proxy Caching for Streaming Media Delivery[C] . Proceedings of the 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video,Monterey,California,USA,2003:22-31.
[36]罗治国,孙巍,王行刚.一种基于传输成本的流媒体缓存替换算法及其性能评价[J].通信学报,2004,25(2):61-67.
[37] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams[C].IEEE INFOCOM.IEEE Press,NewYork,USA,1999:1310-1319.
[38] GAO W.Recent Advances in Peer-to-Peer Media Streaming Systems[J].China Comminications,2006,5(13):52-57.
[39]徐臻,高仲合.基于CDN缓存技术和组播技术的视频点播研究[J].电脑知识与技术,2007,(11):1403-1404.
[40] MEHYAR M,SPANOS D,PONGSAJAPAN J,et al.Asynchronous Distributed Averaging on Communication Networks[J] . IEEE/ACM Transactions on Networking,2007,15(3):512-520.
[41]孙名松,唐亮,周红敏.P2P点播系统的客户端磁盘缓存策略[J].计算机工程,2008,34(20):71-73.
[42] DABEK F,KAASHOEK M F,KARGER D.Wide-Area Cooperative Storage with CFS[C].Proceedings of the ACM Symposium on Operating System Principles,New York,2001:202-215.
[43]吴琼.基于P2P网络的分布式存储系统缓存策略研究[D].大连:大连理工大学,2006:27-29.
[44] SHIM J,SCHEDERMANN P,VINGRALEK R.Proxy Cache Design:Algorithms,Implementation and Performance[J].IEEE Trans Knowledge and Data Engineering,1999,11(4):549-562.
[45]马杰,樊建平.LittleDuck流媒体缓存模拟器[J].计算机工程,2006,32(14): 208-210.
[46]张学魁.基于DHT的P2P网络路由算法的研究[D].成都:西华大学,2008:43-46.
[47]郑婕,张松,齐洁.P2P流媒体节点选择机制的研究与仿真[J].计算机工程与设计,2007,28(22):5396-5399.
[48]颉伟,张喆韬，张贝，曹勇．基于OPNET的网络并行仿真实现[J].电子科技,2008,21(2):72-75.
[2] FOX C.Peer-to-Peer Networks[J].Computing in Science & Engineering, 2001,3(3):75-77.
[3] DESHPANDE H,BAWA M,GARCIA-MOLINA H.Streaming Live Media Over a Peer-to-Peer Network[R].USA:Stanford InfoLab,2002.
[4] YANG G,KYOUNGWON S,JIM K.A Peer-to-Peer on Demand Streaming Service and Its Performance Evaluation[C] . 2003 IEEE International Conference on Multimedia and Ex,Baltimore,MD,USA,2003:649~652.
[5] VENKATA N,PADMANABH,HELEN J.Distributing Streaming Media Content Using Cooperative Networking[C] . Proceedings of the 12th International Workshop on Network and Operating Systems Support for Digital Audio and Video:NOSSDAV 2002,Miami,FL,United states,2002:177~186.
[6] MIGUEL C,PETER D,KERMARREC A M.SplitStream:High-Bandwidth Content Distribution in Cooperative Environments[C].Berkeley,CA,USA,2003:292-303.
[7] DANA C,LI D,HARRISON,D CHUAH.BASS:BitTorrent Assisted Streaming System for Video-on-Demand Multimedia Signal Processing[C].2005 IEEE 7th Workshop on Multimedia Signal Processing,Shanghai,China,2005:1-4.
[8] LIAO X F,JIN H,LIU Y H.AnySee:Peer-to-Peer Live Streaming[C].IEEE INFOCOM’06,Barcelona,Spain,2006:1-10.
[9] S SAROIU.Measuring and Analyzing the Characteristics of Napster and Gnutella Hosts[J].Multimedia Systems Journal,2003,9(2):170-184.
[10]傅锡天,孙景鳌,蔡安妮.分布式VoD系统的分级存储调度[J],通信学报,2000,21(3):64-69.
[11] HIDEHARU S , KAZUTOSHI N , AKIRA U . Storage Hierarchy for Video-on-Demand Systems[C].In proceedings of SPIE,San Jose,CA,USA,1994:198-207.
[12] QIAO L,ZHANG Z,LI M Y.An Empirical Study of Collusion Behavior in the Maze P2P File-Sharing System[C] . 27th International Conference on Distributed Computing Systems (ICDCS '07),Toronto,ON,Canada,2006:10-11.
[13] XIANG Z,ZHANG Q,ZHU W.Peer-to-peer Based Multimedia Distribution Service[J].IEEE Trans on Multimedia.2004,6(2):343–355.
[14] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams[C].IEEE INFOCOM '99 Conference on Computer Communications,New York,NY,USA,1999:1310-1319.
[15] JáNOS I.Some Practical Aspects of Fitting and Testing the Zipf-Mandelbrot Model[J].Scientometrics,2006,1(67):107-120.
[16] JOAN B.Theoretical Evidence for the Superiority of LRU-2 over LRU for the Paging Problem[C].In Fourth Workshop on Approximation and Online Algorithms(WAOA),London,2006:95-107.
[17] LEONID B,SOKOLINSKY.LFU-K:An Effective Buffer Management Replacement Algorithm[C].Proceedings of the 9th Int'l Conf,Berlin,2004:670-681.
[18]田小波,陈蜀宇.基于最小效用的流媒体缓存替换算法[J].计算机应用,2007,27(3):733-736.
[19]廖鑫.一种基于LRU算法改进的缓存方案研究与实现[J].电子工程师,2008,34(7):46-48.
[20] XU D , HEFEEDA M , HAMBRUSH . On Peer-to-Peer Media Streaming[C].Proceedings of IEEE International Conference on Distributed Computing System(ICDCS).Vienna Austria,2002:363-371.
[21] WANG B,SEN S,ADLER M.Optimal Proxy Cache Allocation for Efficient Streaming Media Distribution[J].IEEE Transactions on Multimedia,2004,6(2):366-374.
[22] CHEN S Q,SHEN B,WEE S.Adaptive and Lazy Segmentation Based Proxy Caching for Streaming Media Delivery[C] . Proceedings of the 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video,Monterey,California,USA,2003:22-31.
[23] WU K L,YU P S,WOLF J L.Segmentation of Multimedia Streams for Proxy Caching[J].IEEE Transactions on Multimedia,2004,6(5):770- 780.
[24] WANG J.A Survey of Web Caching Schemes for the Internet[J].ACM Computer Communication Review,1999,29(5):36-46.
[25] GWETZMAN,SELTZERM.World Wide Web Cache Consistency[C].USA:Proceedings of the USEN IX Technical Conference,San Diego,1996:141-152.
[26] SAITO Y , KARAMANOLIS C, KARLSSON M . Taming Aggressive Replication in the Pangaea Wide-Area Files System[C].Proceedings of the 5th Symposium on Operating Systems Design and Implementation,Boston,USA,2002:15-30.
[27]张超群.代理缓存关键技术的研究[J].计算机工程与科学,2007,29(2):142-144.
[28]魏苏林,乐红冰.基于P2P模式的新型视频点播系统[J].微计算机信息(管控一体化),2005,(26):7-9.
[29] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams [C].Proceedings of IEEE INFOCOM,New York,1999:1310-1319.
[30] REJAIE R,HANDLEY M,YU H.Proxy Caching Mechanism for Multimedia Playback Streams in The Internet[C].Proceedings of International Web Caching Workshop,San Diego,CA,1999:12-19.
[31] WU K,YU P S,WOLF J.Segment-Based Proxy Caching of Multimedia Streams[C].Proceedings of WWW,Hong Kong,China,2001:36-44.
[32] REJAIE R,YU H,HANDELY M.Multimedia Proxy Caching Mechanism for Quality Adaptive Streaming Applications in the Internet[C].Proceedings of IEEE INFOCOM,Tel-Aviv,2000:980-989.
[33] KIM T,AMMAR M H.A Comparison of Layering and Stream Replication Video Multicast Schemes[C] . Port Jefferson . Proceedings of ACM NOSSDAV,NY, 2001:63-72.
[34] TAKANO R,YOSHIZAWA Y.Offloading VoD Server Organized Dynamically Distributed Cache Using P2P Delivery[C] . International Conference on Information Networking,Tokyo,2008:1-5.
[35] CHEN S Q,SHEN B,WEE S.Adaptive and Lazy Segmentation Based Proxy Caching for Streaming Media Delivery[C] . Proceedings of the 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video,Monterey,California,USA,2003:22-31.
[36]罗治国,孙巍,王行刚.一种基于传输成本的流媒体缓存替换算法及其性能评价[J].通信学报,2004,25(2):61-67.
[37] SEN S,REXFORD J,TOWSLEY D.Proxy Prefix Caching for Multimedia Streams[C].IEEE INFOCOM.IEEE Press,NewYork,USA,1999:1310-1319.
[38] GAO W.Recent Advances in Peer-to-Peer Media Streaming Systems[J].China Comminications,2006,5(13):52-57.
[39]徐臻,高仲合.基于CDN缓存技术和组播技术的视频点播研究[J].电脑知识与技术,2007,(11):1403-1404.
[40] MEHYAR M,SPANOS D,PONGSAJAPAN J,et al.Asynchronous Distributed Averaging on Communication Networks[J] . IEEE/ACM Transactions on Networking,2007,15(3):512-520.
[41]孙名松,唐亮,周红敏.P2P点播系统的客户端磁盘缓存策略[J].计算机工程,2008,34(20):71-73.
[42] DABEK F,KAASHOEK M F,KARGER D.Wide-Area Cooperative Storage with CFS[C].Proceedings of the ACM Symposium on Operating System Principles,New York,2001:202-215.
[43]吴琼.基于P2P网络的分布式存储系统缓存策略研究[D].大连:大连理工大学,2006:27-29.
[44] SHIM J,SCHEDERMANN P,VINGRALEK R.Proxy Cache Design:Algorithms,Implementation and Performance[J].IEEE Trans Knowledge and Data Engineering,1999,11(4):549-562.
[45]马杰,樊建平.LittleDuck流媒体缓存模拟器[J].计算机工程,2006,32(14): 208-210.
[46]张学魁.基于DHT的P2P网络路由算法的研究[D].成都:西华大学,2008:43-46.
[47]郑婕,张松,齐洁.P2P流媒体节点选择机制的研究与仿真[J].计算机工程与设计,2007,28(22):5396-5399.
[48]颉伟,张喆韬，张贝，曹勇．基于OPNET的网络并行仿真实现[J].电子科技,2008,21(2):72-75.