基于云存储的视频信息分布式优化处理系统的研究与设计
|
摘要
随着科学技术的进步,视频处理系统虽然得到一定发展,但面对需要满足大量访问、快速响应等高质量视频服务时,传统方案在体系结构和负载均衡等方面还不够成熟,已经不能再满足当今的需求。本文所讨论的基于云存储的视频信息分布式优化处理系统,其中云存储作为一个新兴的研究和应用领域,其具有快速部署,低成本,灵活调整规模等优势,但云存储同样也受到了一定的限制,原因在于我们虽然拥有一系列对负载均衡进行衡量的算法,但是由于不能提前对负载进行预算度量,这就使负载均衡失去了基础,限制了整个系统性能。基于小波神经网络的负载均衡具有可预测性和自学习性,使负载均衡达到合理应用性。
本文就如何构建云存储环境、如何优化视频信息处理技术和如何运用小波神经网络来处理负载均衡策略这三个方面,给出了基于云存储的视频信息分布式优化处理系统的设计方案,在某种程度上解决传统视频信息处理系统技术上的不足,大大简化其应用环节,实现视频信息资源充分共享,提高其利用效率。本课题的主要研究工作如下：
(1)云存储构架的研究与设计。基于云存储概念及特点,设计了云存储四层存储服务器模型,从底层到上层依次是：云存储层,数据管理层,应用接口层(也叫数据服务层)以及用户访问层。本文提供的设计方案为：利用普通PC机群搭建云存储中的底层-云存储层,采用多种功能模块分块管理进行数据管理层的设计,在应用接口层针对相应功能开发一些实际接口,方便与访问层用户操作的交互。
(2)视频信息分布式优化处理。基于云存储环境,系统分别从视频信息传输、调度、存储等方面进行优化设计。对接收到的视频信息进行重组及H.264解码,采用TCP与RTP相结合的方式进行传输。在调度方面,选择一种新调度算法-最强能力优先调度算法,存储策略则是采用基于时间序列的视频文件热度进行有效存储。
(3)负载均衡的研究与设计。针对传统算法的局限性,文章提出了一种基于小波神经网络预测模型的改进算法,并在MATLAB环境下进行仿真实验,证明优越性。并以此为基础,设计了系统负载均衡策略。
With the advances in science and technology, the video processing system, a certain development, but face the need to meet a lot of access, rapid response and other high-quality video services, traditional systems programs in architecture and load balancing is not mature enough, it can not no longer meet today's needs. Optimized processing system discussed in this paper based on the video information distributed cloud storage, in which cloud storage as an emerging research and application areas, with rapid deployment, low cost, flexibility to adjust the scale and other advantages, but the cloud storage has also been subjected to some restrictions, the reason is that although we have a series of measure load balancing algorithm, but due to the load can not advance the budget measure, which makes load balancing lost, limiting the overall system performance. Predictable and self-learning, wavelet neural network-based load balancing and load balancing to achieve a reasonable applied.
The article on how to build a cloud storage environment, how to optimize video processing technology and how to improve the video information distributed cloud-based storage to optimize the design of the processing system based on wavelet neural network load balancing strategies, given, in some extent, solve the technical deficiencies of the traditional video information processing system, greatly simplifies the application links, and video information resources to fully share and increase its use efficiency. The main research of this topic is as follows:
(1) Cloud storage architecture and design. Designed based on the concept and characteristics of cloud storage, cloud storage, four-story storage server model from the bottom to the top are:cloud storage layer, data management, data services layer (also called the Application Interface Layer) as well as access layer. This design are:ordinary PC group to build the underlying cloud storage-cloud storage layer, block management using a variety of functional modules for data management design, develop some of the actual interface for some of the features in the application interface layer, convenient and operations access layer user interaction.
(2) Video information distributed optimization. Cloud-based storage environments, video messaging, scheduling, storage optimization. The received video information to restructure and H.264decoding, using a combination of TCP and RTP transmission. Choose a new scheduling algorithm-the strongest ability to give priority scheduling algorithm in the dispatch, storage strategy is based on the time series of video files Redu effective storage.
(3) Load balancing and design. Address the limitations of the traditional algorithm, the article chose the improved algorithm based on wavelet neural network prediction model, and to prove the superiority of simulation in the MATLAB environment.
本文就如何构建云存储环境、如何优化视频信息处理技术和如何运用小波神经网络来处理负载均衡策略这三个方面,给出了基于云存储的视频信息分布式优化处理系统的设计方案,在某种程度上解决传统视频信息处理系统技术上的不足,大大简化其应用环节,实现视频信息资源充分共享,提高其利用效率。本课题的主要研究工作如下：
(1)云存储构架的研究与设计。基于云存储概念及特点,设计了云存储四层存储服务器模型,从底层到上层依次是：云存储层,数据管理层,应用接口层(也叫数据服务层)以及用户访问层。本文提供的设计方案为：利用普通PC机群搭建云存储中的底层-云存储层,采用多种功能模块分块管理进行数据管理层的设计,在应用接口层针对相应功能开发一些实际接口,方便与访问层用户操作的交互。
(2)视频信息分布式优化处理。基于云存储环境,系统分别从视频信息传输、调度、存储等方面进行优化设计。对接收到的视频信息进行重组及H.264解码,采用TCP与RTP相结合的方式进行传输。在调度方面,选择一种新调度算法-最强能力优先调度算法,存储策略则是采用基于时间序列的视频文件热度进行有效存储。
(3)负载均衡的研究与设计。针对传统算法的局限性,文章提出了一种基于小波神经网络预测模型的改进算法,并在MATLAB环境下进行仿真实验,证明优越性。并以此为基础,设计了系统负载均衡策略。
With the advances in science and technology, the video processing system, a certain development, but face the need to meet a lot of access, rapid response and other high-quality video services, traditional systems programs in architecture and load balancing is not mature enough, it can not no longer meet today's needs. Optimized processing system discussed in this paper based on the video information distributed cloud storage, in which cloud storage as an emerging research and application areas, with rapid deployment, low cost, flexibility to adjust the scale and other advantages, but the cloud storage has also been subjected to some restrictions, the reason is that although we have a series of measure load balancing algorithm, but due to the load can not advance the budget measure, which makes load balancing lost, limiting the overall system performance. Predictable and self-learning, wavelet neural network-based load balancing and load balancing to achieve a reasonable applied.
The article on how to build a cloud storage environment, how to optimize video processing technology and how to improve the video information distributed cloud-based storage to optimize the design of the processing system based on wavelet neural network load balancing strategies, given, in some extent, solve the technical deficiencies of the traditional video information processing system, greatly simplifies the application links, and video information resources to fully share and increase its use efficiency. The main research of this topic is as follows:
(1) Cloud storage architecture and design. Designed based on the concept and characteristics of cloud storage, cloud storage, four-story storage server model from the bottom to the top are:cloud storage layer, data management, data services layer (also called the Application Interface Layer) as well as access layer. This design are:ordinary PC group to build the underlying cloud storage-cloud storage layer, block management using a variety of functional modules for data management design, develop some of the actual interface for some of the features in the application interface layer, convenient and operations access layer user interaction.
(2) Video information distributed optimization. Cloud-based storage environments, video messaging, scheduling, storage optimization. The received video information to restructure and H.264decoding, using a combination of TCP and RTP transmission. Choose a new scheduling algorithm-the strongest ability to give priority scheduling algorithm in the dispatch, storage strategy is based on the time series of video files Redu effective storage.
(3) Load balancing and design. Address the limitations of the traditional algorithm, the article chose the improved algorithm based on wavelet neural network prediction model, and to prove the superiority of simulation in the MATLAB environment.
引文
[1]高宏卿,郭文鹭,翟炎杰,基于云存储的流媒体教育服务研究[J],现代教育技术,2011(02)：108-111
[2]叶雄杰,基于云存储的移动视频监控系统研究[D],广州,广东工业大学,2011(05)
[3]周可等,云存储技术及其应用[J].中兴通讯技术,2010
[4]SHVACHKO K, HAIRONG K, RADIA S, et al. The Hadoop Distributed File System[C]//IEEE 26th Symposium on Mass Storage Systems and Technologies, Inclie Village, NV:[s.n.],2010:1-10.
[5]胡涛,许胤龙,分布存储VOD系统的负载均衡设计及其仿真[J],计算机仿真,2009.26(04)
[6]Wenying Zeng, Yuelong Zhao. Kairi Ou, Wei Song. Research on cloud storage architecture and key technologies[C].ETRI, KISTI, AICIT.2nd International Conference on Interaction Sciences:Information Technology, Culture and Human.New York, United States:Association for Computing Machinery,2009:1044-1048.
[7]王永亮等,VOD系统的负载均衡存储策略及调度算法研究[J].电视技术,2004(11)
[8]陈荣征等,基于遗传神经网络的主机负载预测方法研究[J].计算机时代,2009(10)
[9]昌玉芳,分布式VOD系统中集群视频服务器的设计与实现[J].2006
[10]陈志刚等,一种基于预测的动态负载均衡模型及算法研究[J].计算机工程,2004(12)
[11]张迪等,基于WEB的移动端云存储技术研究[J].计算机工程与应用,2010(36)
[12]汪洋,基于集群的VOD系统视频服务器的负载均衡算法改进[J].电脑与信息技术,2009
[13]汪洋,基于集群的VOD系统视频服务器的设计实现[D].华中科技大学,2006
[14]黄晓云,基于HDFS的云存储服务系统研究[J].大连海事大学,2010
[15]崔力升,C／S模式下分布式文件系统中数据高度的应用研究[D],成都理工大学,2011
[16]史文明等,CSCW支撑环境-基于调度算法的360度组播视频子系统研究实现[J].计算机工程与应用,2004
[17JRABINOVICH M, RABINOVICH I, et al. A Dynamic Object Replication and Migration Protocol for an Internet Hosting Service[C]//19th IEEE International Conference on Distributed Computing Systems, Austin, Texas, USA:[s.n.],1999: 101-113.
[18]Daniel P. Bovet& Marco Cesati.Understanding the Linux Kernel,Third Edition[M].the United States of America:O'REILLY Media.Inc,2006.
[19]ZHONG Hai, ZHANG Zehua, ZHANG Xuejie. A Dynamic Relica Management Strategy Based on Data Grid[C]//2010 9th International Conference on Grid and Cloud Computing, Nanjing:[s.n.],2010:18-23.
[20]王永亮,刘峰,张春,VOD系统的负载均衡设计策略及调度算法研究[J],电视技术.2004年第11期
[21]许伟,分布式系统中的主机负载预测与动态负载均衡研究[D].中南大学,2004(06)
[22]李春辉,分布式系统主机负载预测方法的评估[J].科技创新导报,2009
[23]QAISAR R, LI Jianzhong, YANG Donghua. A Load Balancing Replica Placement Strategy in Data Grid[C]//3th International Conference on Digital Information Management, London:[s.n.],2008:751-756.
[24]张冬青,云计算对未来电子商务发展的影响[J].学术交流,2010
[25]李荣茂,浅谈电子封装技术的重要性及发展趋势[J].科技传播,2010
[26]肖霄,,可扩展逻辑云ELC系统的研究与设计[D].电子科技大学,20lO
[27]刘凡茂,基于云计算的乡镇卫生院信息化研究[D].中南大学,2009
[28]王忠儒,云环境下的虚拟机监控和服务部署关键技术研究[D].国防科技大学,2010
[29]祝建武等,云存储在企业容灾备份中全新模式探析[J].现代商贸工业,20ll
[30]曹健等,云计算及其发展策略.软件产业与工程[J].20lO
[31]吴晓伟,MapReduce并行编程模式的应用与研究[D].中国科学技术大学,2009
[32]叶钰等,基SimpleDB进行分布式数据存储[J].泰州职业技术学院学报,2010
[33]SASHI K, LECTURER S. A New Replica Creation and Placement Algorithm for Data Grid Environment[C].2010 International Conference on Data Storage and Data Engineering, Bangalore:[s.n.],2010:265-269.
[34]程宏兵等,一种基于自动回归的改进网格主机负载预测模型[J].计算机应用,2005
[35]廖志坚,基于历史运行轨迹的时间约束参数预测的研究[D].广东工业大学,2007
[36]方巍,EJB容器集群关键技术的研究与设计[D].苏州大学,2006
[37]吴杰伟,·网络视频服务系统中服务机制及算法研究[D].北方工业大学,2007
[38]郑荔敏,基于Symbian只能手机的H.264视频播放器的设计和仿真实现[D].厦门大学,2009
[39]田军,抚顺石化公司公分布式视频服务系统研究与实现[D].东北大学,2008
[40]刘洁彬,面相实时监控的流媒体播放器的设计与实现[D].北京邮电大学,20lO
[41]黄详书,基于网格的教学资源存储策略与调度算法的研究[D].大连交通大学,2008
[42]郭小林,基于主动和被动模式的分布式代理缓存服务器的设计与实现[D].电子科技大学,2010
[43]刘鹏程,物联网标准体系构建研究[D].北京交通大学,2011
[44]陈志刚等,Web集群系统QoS模式的预测与分类[J].计算机应用,2003
[45]谢芳清等,基于主机负载预测的动态任务调度算法研究[J].电脑知识与技术,2010
[46]周开利,康耀红,神经网络模型及其MATLAB仿真程序设计[M].清华大学出版社,2004
[47]阳景文,高美娟人工神经网络算法研究及应用[M].北京理工大学出版社,2006
[48]MATLAB中文论坛,MATLAB神经网络30个案例分析[M].北京航空航天大学出版社,2010
[2]叶雄杰,基于云存储的移动视频监控系统研究[D],广州,广东工业大学,2011(05)
[3]周可等,云存储技术及其应用[J].中兴通讯技术,2010
[4]SHVACHKO K, HAIRONG K, RADIA S, et al. The Hadoop Distributed File System[C]//IEEE 26th Symposium on Mass Storage Systems and Technologies, Inclie Village, NV:[s.n.],2010:1-10.
[5]胡涛,许胤龙,分布存储VOD系统的负载均衡设计及其仿真[J],计算机仿真,2009.26(04)
[6]Wenying Zeng, Yuelong Zhao. Kairi Ou, Wei Song. Research on cloud storage architecture and key technologies[C].ETRI, KISTI, AICIT.2nd International Conference on Interaction Sciences:Information Technology, Culture and Human.New York, United States:Association for Computing Machinery,2009:1044-1048.
[7]王永亮等,VOD系统的负载均衡存储策略及调度算法研究[J].电视技术,2004(11)
[8]陈荣征等,基于遗传神经网络的主机负载预测方法研究[J].计算机时代,2009(10)
[9]昌玉芳,分布式VOD系统中集群视频服务器的设计与实现[J].2006
[10]陈志刚等,一种基于预测的动态负载均衡模型及算法研究[J].计算机工程,2004(12)
[11]张迪等,基于WEB的移动端云存储技术研究[J].计算机工程与应用,2010(36)
[12]汪洋,基于集群的VOD系统视频服务器的负载均衡算法改进[J].电脑与信息技术,2009
[13]汪洋,基于集群的VOD系统视频服务器的设计实现[D].华中科技大学,2006
[14]黄晓云,基于HDFS的云存储服务系统研究[J].大连海事大学,2010
[15]崔力升,C／S模式下分布式文件系统中数据高度的应用研究[D],成都理工大学,2011
[16]史文明等,CSCW支撑环境-基于调度算法的360度组播视频子系统研究实现[J].计算机工程与应用,2004
[17JRABINOVICH M, RABINOVICH I, et al. A Dynamic Object Replication and Migration Protocol for an Internet Hosting Service[C]//19th IEEE International Conference on Distributed Computing Systems, Austin, Texas, USA:[s.n.],1999: 101-113.
[18]Daniel P. Bovet& Marco Cesati.Understanding the Linux Kernel,Third Edition[M].the United States of America:O'REILLY Media.Inc,2006.
[19]ZHONG Hai, ZHANG Zehua, ZHANG Xuejie. A Dynamic Relica Management Strategy Based on Data Grid[C]//2010 9th International Conference on Grid and Cloud Computing, Nanjing:[s.n.],2010:18-23.
[20]王永亮,刘峰,张春,VOD系统的负载均衡设计策略及调度算法研究[J],电视技术.2004年第11期
[21]许伟,分布式系统中的主机负载预测与动态负载均衡研究[D].中南大学,2004(06)
[22]李春辉,分布式系统主机负载预测方法的评估[J].科技创新导报,2009
[23]QAISAR R, LI Jianzhong, YANG Donghua. A Load Balancing Replica Placement Strategy in Data Grid[C]//3th International Conference on Digital Information Management, London:[s.n.],2008:751-756.
[24]张冬青,云计算对未来电子商务发展的影响[J].学术交流,2010
[25]李荣茂,浅谈电子封装技术的重要性及发展趋势[J].科技传播,2010
[26]肖霄,,可扩展逻辑云ELC系统的研究与设计[D].电子科技大学,20lO
[27]刘凡茂,基于云计算的乡镇卫生院信息化研究[D].中南大学,2009
[28]王忠儒,云环境下的虚拟机监控和服务部署关键技术研究[D].国防科技大学,2010
[29]祝建武等,云存储在企业容灾备份中全新模式探析[J].现代商贸工业,20ll
[30]曹健等,云计算及其发展策略.软件产业与工程[J].20lO
[31]吴晓伟,MapReduce并行编程模式的应用与研究[D].中国科学技术大学,2009
[32]叶钰等,基SimpleDB进行分布式数据存储[J].泰州职业技术学院学报,2010
[33]SASHI K, LECTURER S. A New Replica Creation and Placement Algorithm for Data Grid Environment[C].2010 International Conference on Data Storage and Data Engineering, Bangalore:[s.n.],2010:265-269.
[34]程宏兵等,一种基于自动回归的改进网格主机负载预测模型[J].计算机应用,2005
[35]廖志坚,基于历史运行轨迹的时间约束参数预测的研究[D].广东工业大学,2007
[36]方巍,EJB容器集群关键技术的研究与设计[D].苏州大学,2006
[37]吴杰伟,·网络视频服务系统中服务机制及算法研究[D].北方工业大学,2007
[38]郑荔敏,基于Symbian只能手机的H.264视频播放器的设计和仿真实现[D].厦门大学,2009
[39]田军,抚顺石化公司公分布式视频服务系统研究与实现[D].东北大学,2008
[40]刘洁彬,面相实时监控的流媒体播放器的设计与实现[D].北京邮电大学,20lO
[41]黄详书,基于网格的教学资源存储策略与调度算法的研究[D].大连交通大学,2008
[42]郭小林,基于主动和被动模式的分布式代理缓存服务器的设计与实现[D].电子科技大学,2010
[43]刘鹏程,物联网标准体系构建研究[D].北京交通大学,2011
[44]陈志刚等,Web集群系统QoS模式的预测与分类[J].计算机应用,2003
[45]谢芳清等,基于主机负载预测的动态任务调度算法研究[J].电脑知识与技术,2010
[46]周开利,康耀红,神经网络模型及其MATLAB仿真程序设计[M].清华大学出版社,2004
[47]阳景文,高美娟人工神经网络算法研究及应用[M].北京理工大学出版社,2006
[48]MATLAB中文论坛,MATLAB神经网络30个案例分析[M].北京航空航天大学出版社,2010