基于内容分析的无线视频适配关键技术的研究
|
摘要
进入信息化社会以来,移动数字媒体业务的应用越来越广泛。传统的2G时代的业务,如即时通信、手机音乐,已不能满足现代用户多样化的要求。随着3G网络的陆续建成,移动带宽增加,各种移动终端技术在逐渐加强,如CPU处理能力、电池容量等都在提高,这些为移动视频业务提供了现实可能性。从用户来看,移动视频业务,如手机电视,在线MV等,符合现代人们的心理需求。手机作为移动终端的一种,应用广泛,功能先进的智能手机也逐渐普及。因此,本文把手机视频作为研究焦点。由于无线网络带宽有限,各手机设备之间的差异性较大,如支持不同的视频格式等,所以,视频资源要想顺利传输并显示在手机终端上,必须进行相应的适配。本文在对视频进行内容分析的基础上,结合不同的终端条件对视频进行自适应传输和适配。
首先,我们在综合考虑视频静态特征和动态特征的基础上,对视频资源进行基于内容的视频分析(CBVA, Content-Based Video Analyze)。静态特征方面,利用互信息量的方法提取视频关键帧,并对关键帧进行感兴趣区域分析。关键帧对重构视频内容有着重要的作用,在对视频进行适配时,加强保护,对优化用户的视觉感受有极大的帮助。动态特征方面,以块匹配的三步搜索法为基础,计算视频帧间运动能量差(MEDF, Motion Energy Difference between Frames),用量化的方式去区分各帧的重要程度。其次,依视频资源各帧的重要程度,对视频帧进行质量分层。根据各视频帧是否是关键帧、运动能量的大小等特征,可把视频分成四层。最后,模拟测试环境搭建。以手机模拟器NOKIA n97作为测试对象,建立适配主页,对用户请求的视频资源进行适配。当视频格式以及分辨率等不符合手机终端参数要求时,利用开源代码FFmpeg对其进行调整。本文最后的实验部分,显示了相关视频适配的效果。
In the information society, mobile digital media business has been using more widely. The traditional2G era of business, such as instant communication, mobile phone music, already cannot satisfy the requirement of modern user diversification. With the succession of3G network, the increase of wireless bandwidth and the strengthening of mobile terminal technology, mobile video business development is increasing rapidly. From users'perspective, mobile video services, such as mobile television, online MV and so on, conforms to modern people's psychological need. Mobile phone, as a kind of mobile terminal, has wide application and the function of advanced Smartphone is also gradually popular. Therefore, we focus on mobile phone video. Since wireless network bandwidth is limited and differences of phone equipments are large, such as supporting different video format, video resources need appropriate adaption in order to transfer and display on a mobile terminal successfully. Combining with different network characteristics and terminal conditions, we provide video content with adaptive transmission and adaptation on the basis of content-based video analysis.
Firstly, we analyze video resources based on content from two aspects of video static and dynamic characteristics. In static characteristics, we extract video key frames using the method of mutual information and analyze the interested region of key frames. The key frame plays an important role in the reconstruction of video content. This will be a great help to optimize the user's visual experience if strengthening the protection of key frames in the process of video adapter. In dynamic characteristics, we calculate the motion energy difference between the video frames based on the three-step search block-matching method, in order to distinguish the importance degree of each frame in quantitative way. Secondly, according to the degree of importance of each frame in video resources, video frames can be divided into several quality layers. According to the video features, such as key frames and the value of MEDF, video frames can be divided into four level layers. Finally, we set up simulation test environment, using NOKIA n97emulator as test subject and establish home page of video adaptation for users'request. When the video format and resolution does not meet the requirements of the mobile terminal parameters, they can be adjusted using open source code of FFmpeg. In the last part of the experiment, we show some examples of video adaptation.
首先,我们在综合考虑视频静态特征和动态特征的基础上,对视频资源进行基于内容的视频分析(CBVA, Content-Based Video Analyze)。静态特征方面,利用互信息量的方法提取视频关键帧,并对关键帧进行感兴趣区域分析。关键帧对重构视频内容有着重要的作用,在对视频进行适配时,加强保护,对优化用户的视觉感受有极大的帮助。动态特征方面,以块匹配的三步搜索法为基础,计算视频帧间运动能量差(MEDF, Motion Energy Difference between Frames),用量化的方式去区分各帧的重要程度。其次,依视频资源各帧的重要程度,对视频帧进行质量分层。根据各视频帧是否是关键帧、运动能量的大小等特征,可把视频分成四层。最后,模拟测试环境搭建。以手机模拟器NOKIA n97作为测试对象,建立适配主页,对用户请求的视频资源进行适配。当视频格式以及分辨率等不符合手机终端参数要求时,利用开源代码FFmpeg对其进行调整。本文最后的实验部分,显示了相关视频适配的效果。
In the information society, mobile digital media business has been using more widely. The traditional2G era of business, such as instant communication, mobile phone music, already cannot satisfy the requirement of modern user diversification. With the succession of3G network, the increase of wireless bandwidth and the strengthening of mobile terminal technology, mobile video business development is increasing rapidly. From users'perspective, mobile video services, such as mobile television, online MV and so on, conforms to modern people's psychological need. Mobile phone, as a kind of mobile terminal, has wide application and the function of advanced Smartphone is also gradually popular. Therefore, we focus on mobile phone video. Since wireless network bandwidth is limited and differences of phone equipments are large, such as supporting different video format, video resources need appropriate adaption in order to transfer and display on a mobile terminal successfully. Combining with different network characteristics and terminal conditions, we provide video content with adaptive transmission and adaptation on the basis of content-based video analysis.
Firstly, we analyze video resources based on content from two aspects of video static and dynamic characteristics. In static characteristics, we extract video key frames using the method of mutual information and analyze the interested region of key frames. The key frame plays an important role in the reconstruction of video content. This will be a great help to optimize the user's visual experience if strengthening the protection of key frames in the process of video adapter. In dynamic characteristics, we calculate the motion energy difference between the video frames based on the three-step search block-matching method, in order to distinguish the importance degree of each frame in quantitative way. Secondly, according to the degree of importance of each frame in video resources, video frames can be divided into several quality layers. According to the video features, such as key frames and the value of MEDF, video frames can be divided into four level layers. Finally, we set up simulation test environment, using NOKIA n97emulator as test subject and establish home page of video adaptation for users'request. When the video format and resolution does not meet the requirements of the mobile terminal parameters, they can be adjusted using open source code of FFmpeg. In the last part of the experiment, we show some examples of video adaptation.
引文
[1]Verto, C.ChristoPulos, H.F.Sun, "Video transcoding architectuers and techniques:An Overview", IEEE Signal Process magazine, vol.20, no.2, pp.18-29, Mar.2003.
[2]柳辉.可伸缩性视频编码的转码及其应用[D].中国科学技术大学,2009.5.
[3]孙晓艳,高文,文枫等.基于宏块的具有时域和SNR精细可伸缩性的视频编码[J].计算机学报,2003.3.
[4]Fang Ling, Weiping Li, Hongqiao Sun. Bitplane coding of DCT coefficients for image and video compression. In:Proceedings of SPIE VCIP'99, Jan.1999.
[5]Min Xu, Jesse S.Jin, Suhuai Luo. "Personalized video adaptation based on video content analysis", Proceedings of the 9th International Workshop on Multimedia Data Mining, pp.26-35, August.2008.
[6]刘昕雨.基于视频特征的火灾火焰识别算法[D],西安电子科技大学,2009.
[7]张巧荣,李淑红,肖会敏.基于视觉注意的医学图像感兴趣区域提取[J].计算机应用研究,2009(26).
[8]陈素霞.图像感兴趣区域技术研究[D].河南理工大学,2009.
[9]Itti L, Koch C, Niebur E. A Model of Saliency-Based Visual Attention for Rapid Scene Analysis [J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998,20(11):1255.
[10]Itti L., Koch C. Computational Modeling of Visual Attention [J]. Nature Reviews Neuroscience,2001,2(3):194-203.
[11]Navalpakkam V, Itti L. Modeling the Influence of Task on Attention [J]. Vision Research,2005,45(2):205-231.
[12]Chernyak D A, Stark L W. Top-Down Guided Eye Movements [J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B,2001,31(4):514-522.
[13]Rutishauser U., Walther D., Koch C., et al. Is Bottom-Up Attention Useful for Object Recognition[C]. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington DC,2004.
[14]高静静,张菁,沈兰荪.基于视觉注意力模型的图像检索的研究.中国科技论文在线http://www.paper.edu.cn/.
[15]甘新胜.基于码书的运动目标检测方法[J].中国图象图形学报,2008(13).
[16]郭曦.基于视觉的运动检测与手部动作识别[D].上海交通大学,2010.
[17]陶阳,刘志明,彭宇行等.一种有效的匹配准则及块匹配算法[J].计算机工程与科学,2007,29(2).
[18]耿强.《多媒体数字影视技术》[M].电子工业出版社,2005年03月.
[19]程赓,严德聪,杨宗凯.运动图像压缩的新标准—H.264/AVC[J].数字电视与数字视频,2003(6).
[20]王方石,须德,吴伟鑫.基于自适应阈值的自动提取关键帧的聚类算法[J].计算机研究与发展,2005,42(10)：1752-1757.
[21]吕庆文,陈武凡.基于互信息量的图像分割[J].计算机学报,2006,29(2)：296-301.
[22]孔娟,田丽.基于互信息量的视频关键帧提取算法[J].安阳工学院学报,2010,9(4)：55-58.
[23]王昊.基于内容的视频检索关键技术研究[D].西安电子科技大学,2006.
[24]D. Walther, C. Koch, Modeling attention to salient proto-objects. Neural Networks, vol.19, no.9, pp.1395-1407,2006.
[25]D. Walther. Saliency Toolbox, [Online], Available:http://www.saliencytoolbox.net/, December 20,2008.
[26]曲仁慧,邸鑫,邸朝生.基于H1264的RCPC信道编码非平等误码保护[J].吉林大学学报(信息科学版),2008,26(4).
[27]Yao Qinghua, Cao Miao, "Study of Unequal Error Protection Method Based on JPEG2000 for Surveying& Mapping Image," 2008 International Conference on Computer Science and Software Engineering, Wuhan, Hubei, China, Dec.2008.pp.951-954.
[28]Davoud Arasteh, Teaching Convolutional Coding using MATLAB in Communication Systems Course, [Online], Available: http://www.aseegsw.Org/Proceedings/5.pdf,2006.
[29]A.GNguyen, J.N.Hwang. A novel hybrid HVPC/mathematical model rate control for low bit-rate streaming video [J]. Signal Processing:Image Communication,2002, 17(5):423-440.
[30]A.GNguyen, J.N.Hwang. Scene context dependent rate control, ACM Conference on Multimedia 2001, Ottawa, Ontario, Canada,30 September-5 October 2001.
[31]Song H, Kuo C C Jay. Rate control for low-bit rate video via variable-encoding frame rates [J].IEEE Trans. Circuits Syst. Video Technol.,2001,11(4):512-521.
[32]王辉柏,张春田.一种基于图像内容重要性的缓存器控制策略[J].通信学报,2000,21(8)：21-26.
[33]赵刚.移动互联网上自适应多媒体传输研究[D].华中科技大学,2007.
[34]K.V. Sridhar. Implementation of Prioritised ROI Coding for Medical Image Archiving using JPEG2000 [C]. International Conference on Signals and Electronic System, KRAKOW,, September 14-17,2008. pp 239-242.
[35]戴若愚.基于帧间运动能量差的跟踪算法研究与实现[D].华中科技大学,2007.
[36]蔡奕奕.基于运动特征的视频检索技术研究[D].广西大学,2008.
[37]章毓晋.基于语义视觉信息检索的进展[J].科学技术与工程,2004,4(4).
[38]刘袁缘.基于帧间运动能量差小目标运动检测算子模型研究[D].华中科技大学,2007.
[39]史凯雒,江涛,张治中.基于RTP的h.264无线视频传输和Qos控制[J].微计算机信息(管控一体化),2009,25(2-3).
[40]周祥东.视频特征提取和视频镜头分析[D].国防科学技术大学,2002.
[41]王崇骏,杨育彬,陈世福.基于高层语义的图像检索算法[J].软件学报,2004,15(10).
[42]俞天力,章毓晋.基于全局运动信息的视频检索技术[J].电子学报,2001,29(12).
[43]曾玮,薛向阳.基于运动特征的视频信息检索综述[J].计算机科学,2004,3(2).
[44]尹明,王宏远.基于H.264的数据分割和不等差错保护方案[J].华中科技大学学报(自然科学版),2006,34(6).
[45]王海巍,叶春,郭晶.基于DM642实验平台的H.264视频编码算法研究[J].计算机应用与研究,2010,52(1).
[46]邓玉春,姜昱明,张建荣.视频序列图像中运动对象分割综述[J].计算机应用研究,2008(1).
[47]范欣.针对移动设备的跨媒体网络信息检索及自适应信息显示研究[D].中国科学技术大学,2007.
[2]柳辉.可伸缩性视频编码的转码及其应用[D].中国科学技术大学,2009.5.
[3]孙晓艳,高文,文枫等.基于宏块的具有时域和SNR精细可伸缩性的视频编码[J].计算机学报,2003.3.
[4]Fang Ling, Weiping Li, Hongqiao Sun. Bitplane coding of DCT coefficients for image and video compression. In:Proceedings of SPIE VCIP'99, Jan.1999.
[5]Min Xu, Jesse S.Jin, Suhuai Luo. "Personalized video adaptation based on video content analysis", Proceedings of the 9th International Workshop on Multimedia Data Mining, pp.26-35, August.2008.
[6]刘昕雨.基于视频特征的火灾火焰识别算法[D],西安电子科技大学,2009.
[7]张巧荣,李淑红,肖会敏.基于视觉注意的医学图像感兴趣区域提取[J].计算机应用研究,2009(26).
[8]陈素霞.图像感兴趣区域技术研究[D].河南理工大学,2009.
[9]Itti L, Koch C, Niebur E. A Model of Saliency-Based Visual Attention for Rapid Scene Analysis [J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998,20(11):1255.
[10]Itti L., Koch C. Computational Modeling of Visual Attention [J]. Nature Reviews Neuroscience,2001,2(3):194-203.
[11]Navalpakkam V, Itti L. Modeling the Influence of Task on Attention [J]. Vision Research,2005,45(2):205-231.
[12]Chernyak D A, Stark L W. Top-Down Guided Eye Movements [J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B,2001,31(4):514-522.
[13]Rutishauser U., Walther D., Koch C., et al. Is Bottom-Up Attention Useful for Object Recognition[C]. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington DC,2004.
[14]高静静,张菁,沈兰荪.基于视觉注意力模型的图像检索的研究.中国科技论文在线http://www.paper.edu.cn/.
[15]甘新胜.基于码书的运动目标检测方法[J].中国图象图形学报,2008(13).
[16]郭曦.基于视觉的运动检测与手部动作识别[D].上海交通大学,2010.
[17]陶阳,刘志明,彭宇行等.一种有效的匹配准则及块匹配算法[J].计算机工程与科学,2007,29(2).
[18]耿强.《多媒体数字影视技术》[M].电子工业出版社,2005年03月.
[19]程赓,严德聪,杨宗凯.运动图像压缩的新标准—H.264/AVC[J].数字电视与数字视频,2003(6).
[20]王方石,须德,吴伟鑫.基于自适应阈值的自动提取关键帧的聚类算法[J].计算机研究与发展,2005,42(10)：1752-1757.
[21]吕庆文,陈武凡.基于互信息量的图像分割[J].计算机学报,2006,29(2)：296-301.
[22]孔娟,田丽.基于互信息量的视频关键帧提取算法[J].安阳工学院学报,2010,9(4)：55-58.
[23]王昊.基于内容的视频检索关键技术研究[D].西安电子科技大学,2006.
[24]D. Walther, C. Koch, Modeling attention to salient proto-objects. Neural Networks, vol.19, no.9, pp.1395-1407,2006.
[25]D. Walther. Saliency Toolbox, [Online], Available:http://www.saliencytoolbox.net/, December 20,2008.
[26]曲仁慧,邸鑫,邸朝生.基于H1264的RCPC信道编码非平等误码保护[J].吉林大学学报(信息科学版),2008,26(4).
[27]Yao Qinghua, Cao Miao, "Study of Unequal Error Protection Method Based on JPEG2000 for Surveying& Mapping Image," 2008 International Conference on Computer Science and Software Engineering, Wuhan, Hubei, China, Dec.2008.pp.951-954.
[28]Davoud Arasteh, Teaching Convolutional Coding using MATLAB in Communication Systems Course, [Online], Available: http://www.aseegsw.Org/Proceedings/5.pdf,2006.
[29]A.GNguyen, J.N.Hwang. A novel hybrid HVPC/mathematical model rate control for low bit-rate streaming video [J]. Signal Processing:Image Communication,2002, 17(5):423-440.
[30]A.GNguyen, J.N.Hwang. Scene context dependent rate control, ACM Conference on Multimedia 2001, Ottawa, Ontario, Canada,30 September-5 October 2001.
[31]Song H, Kuo C C Jay. Rate control for low-bit rate video via variable-encoding frame rates [J].IEEE Trans. Circuits Syst. Video Technol.,2001,11(4):512-521.
[32]王辉柏,张春田.一种基于图像内容重要性的缓存器控制策略[J].通信学报,2000,21(8)：21-26.
[33]赵刚.移动互联网上自适应多媒体传输研究[D].华中科技大学,2007.
[34]K.V. Sridhar. Implementation of Prioritised ROI Coding for Medical Image Archiving using JPEG2000 [C]. International Conference on Signals and Electronic System, KRAKOW,, September 14-17,2008. pp 239-242.
[35]戴若愚.基于帧间运动能量差的跟踪算法研究与实现[D].华中科技大学,2007.
[36]蔡奕奕.基于运动特征的视频检索技术研究[D].广西大学,2008.
[37]章毓晋.基于语义视觉信息检索的进展[J].科学技术与工程,2004,4(4).
[38]刘袁缘.基于帧间运动能量差小目标运动检测算子模型研究[D].华中科技大学,2007.
[39]史凯雒,江涛,张治中.基于RTP的h.264无线视频传输和Qos控制[J].微计算机信息(管控一体化),2009,25(2-3).
[40]周祥东.视频特征提取和视频镜头分析[D].国防科学技术大学,2002.
[41]王崇骏,杨育彬,陈世福.基于高层语义的图像检索算法[J].软件学报,2004,15(10).
[42]俞天力,章毓晋.基于全局运动信息的视频检索技术[J].电子学报,2001,29(12).
[43]曾玮,薛向阳.基于运动特征的视频信息检索综述[J].计算机科学,2004,3(2).
[44]尹明,王宏远.基于H.264的数据分割和不等差错保护方案[J].华中科技大学学报(自然科学版),2006,34(6).
[45]王海巍,叶春,郭晶.基于DM642实验平台的H.264视频编码算法研究[J].计算机应用与研究,2010,52(1).
[46]邓玉春,姜昱明,张建荣.视频序列图像中运动对象分割综述[J].计算机应用研究,2008(1).
[47]范欣.针对移动设备的跨媒体网络信息检索及自适应信息显示研究[D].中国科学技术大学,2007.