3D视频编码算法研究
摘要
3D视频,又叫立体视频,包括双目立体视频和多视点视频,因其具有立体感和交互性的特点,越来越受到学术界和工业界的重视,随着视频技术的发展进步,立体视频将最终将取代HDTV,成为未来视频的发展方向。
然而,由于立体视频需要同时传送多个视频流,在给用户带来立体感的同时,也造成了数据量的急剧增加,如果没有高效的编解码方法,将很难存储和传输。因此,本文的重点是对3D视频编解码器性能改进的研究。
本文首先介绍了立体视频的概念、应用,现有的立体视频编解码方法,以及多视点视频的标准化过程,阐述了H.264视频编解码的关键技术,对其与立体视频有关的技术进行了重点研究。
其次,介绍了基于H.264的多视点视频编码(MVC)方法,该方法联合使用视点预测和运动预测,具有很高的压缩性能,并概括了MVC的研究内容,对其预测结构进行了深入的分析。
最后,提出了MVC预测结构的调整策略,调整了I视点的位置并去除了部分视点问预测,测试结果表明,本文提出的预测结构可以在略微影响多视点视频编解码效率的前提下,提高多视点视频的随机访问性能,并降低MVC的编码复杂度。
3D video includes stereo video and multi-view video, which can provide viewers with benefits of realism and selective viewing. Researchers in video field become more and more interested in it. With the development of video coding technology,3D video will replace HDTV and become the novel research area in video technology.
However, compared to single view video, the data amount of multi-view video increases along with the increase of the view number. So more effective method to encode the video data must be attained in order to make 3D video storage and transition easier. Thus we focus on improving the performance of codec in this paper.
Firstly, there will be some talk about conception of 3D video, the application of it and following it, some key technology of H.264 video coding will be introduced.
Secondly, multi-view video coding method based on H.264 will be discussed. It uses disparity prediction and motion prediction, so it proves to be a good method for 3D video coding. There will be a discussion about research of MVC. It's coding structure will be analysed.
Lastly, some improvements about the existing coding structure are made, the selection of I-view is adjusted and some inter-view prediction is removed. Based on the test result, we can say that the improved structure can improve the random access performance and reduce coding complexity without affecting the performance of MVC too much.
然而,由于立体视频需要同时传送多个视频流,在给用户带来立体感的同时,也造成了数据量的急剧增加,如果没有高效的编解码方法,将很难存储和传输。因此,本文的重点是对3D视频编解码器性能改进的研究。
本文首先介绍了立体视频的概念、应用,现有的立体视频编解码方法,以及多视点视频的标准化过程,阐述了H.264视频编解码的关键技术,对其与立体视频有关的技术进行了重点研究。
其次,介绍了基于H.264的多视点视频编码(MVC)方法,该方法联合使用视点预测和运动预测,具有很高的压缩性能,并概括了MVC的研究内容,对其预测结构进行了深入的分析。
最后,提出了MVC预测结构的调整策略,调整了I视点的位置并去除了部分视点问预测,测试结果表明,本文提出的预测结构可以在略微影响多视点视频编解码效率的前提下,提高多视点视频的随机访问性能,并降低MVC的编码复杂度。
3D video includes stereo video and multi-view video, which can provide viewers with benefits of realism and selective viewing. Researchers in video field become more and more interested in it. With the development of video coding technology,3D video will replace HDTV and become the novel research area in video technology.
However, compared to single view video, the data amount of multi-view video increases along with the increase of the view number. So more effective method to encode the video data must be attained in order to make 3D video storage and transition easier. Thus we focus on improving the performance of codec in this paper.
Firstly, there will be some talk about conception of 3D video, the application of it and following it, some key technology of H.264 video coding will be introduced.
Secondly, multi-view video coding method based on H.264 will be discussed. It uses disparity prediction and motion prediction, so it proves to be a good method for 3D video coding. There will be a discussion about research of MVC. It's coding structure will be analysed.
Lastly, some improvements about the existing coding structure are made, the selection of I-view is adjusted and some inter-view prediction is removed. Based on the test result, we can say that the improved structure can improve the random access performance and reduce coding complexity without affecting the performance of MVC too much.
引文
[1]Pei S C, Lai C L. Very low bit-rate coding algorithm for stereo video with spatio-temporal HVS model and binary correlation disparity estimator [J]. IEEE journal on Selected Areas in Communications,16(1):98-107,1998.
[2]S A joscha, M C Chen.3DAV exploration of video based rendering technology in MPEG[J]. IEEE Transaction on Circuit and Systems for Video Technology.2004,14(3).
[3]Jose M. Martinez-Ibanez, MPEG 3DAV AhG Activities Report[R], TUM-Munich University of Technology, Trondheim-Norway,65th MPEG Meeting, July.2003.
[4]ISO/IEC JTCI/SC29/WG11. N5169:Description of exploration experiments in 3DAV[S]. Shanghai, China,2002.
[5]ISO/IEC JTCI/SC29/WG11. N5558:Report on status of 3DAV exploration [S]. Pattaya, Thailand,2003.
[6]ISO/IEC JTCI/SC29/WG11. N5678:Report on 3DAV exploration [S]. Trondheim, Norway, 2003.
[7]MPEG Video Subgroup. Draft call for proposals on multi-view video coding[A]. W6910, 71st MPEG Meeting[C]. Hong Kong, China,2005.
[8]Description of Core Experiments in MVC[R], ISO/IEC JTC1/SC29/Wg11 T100, April 2006
[9]Anthony Vetro, Purvin Pandit. JMVM Software Manual [EB/OL]. http://ftp3. itu. ch/av-areh/jvt-site/2007_06_Geneva/JVT-X008. zip,2007.
[10]Smolic A and Kimata H. Report on 3DAV exploration [A], ISO/IEC JTC1/SC29/WG11, Doc. N5878,2003.
[11]候紫峰.计算机立体视觉[J].程序员技术.1994,3：87-90.
[12]梁栋,韦穗,周敏彤.双眼立体感知几何模型的研究[J].中国图象图形学报.1998,3(8)：679-683.
[13]Magnor M, Ramanathan P and Girod B., Multi-view coding for image-based rendering using 3-d scene geometry [J], IEEE Transactions on Circuits and Systems for Video Technology, 2003,13(11):1092-1106.
[14]F Pedersini, A Sarti and S Tubaro. Multi-camera systems[J]. IEEE signal Processing Magazine. May 1999:55-65.
[15]荆其诚,焦书兰,纪桂萍.人类的视觉[M],北京：科学出版社,1987.
[16]Hartmut Emst, Robert Larice, Stephan Breide. Mixing of computer graphics and high-quality stereographic video[J]. IEEE Transactions. Consumer Electronics, 42(3):795-799,1996.
[17]朱仲杰,蒋刚毅,郁梅等.多视点视频编码中的视频目标提取与视差匹配[J].电子学报.2004,32(51)：848-850.
[18]余兆明,查日勇,黄磊,周海骄.图像编码标准H.264技术[M].人民邮电出版社,2006.
[19]R William, W Sherman, and B Craig. Understanding virtual reality:interface, application, and design[M]. San Francisco, CA:Morgan Kaufmann Publishers.2004:429-431.
[20]NVIDIA Corporation. NVIDIA 3D Stereo User's Guide[EB].2001.
[21]Narang S, Narang NK. Network oriented motion estimation for video-on—demand over wireless networks[C]. Vehicular Technology Conference.1999,2(2):1535-1539.
[22]李世平,郁梅,蒋刚毅.基于H.264的立体视频编码算法[J].计算机工程与应用.2005,1：77-79.
[23]Yoshio Matsumoto, Tomohiro Shibata, Katsuhiro Sakai. Real-time color stereo vision system for a mobile robot based on field multiplexing[C]. IEEE International Conference on Robotics and Automation,3:1934-1939,1997.
[24]D Marr. A Computational Investigation into the Human Representation and Processing of Visual Information[M]. WH. Freeman and Company San Francisco,1982.
[25]Thomas Sikora, Seinor Member. The MPEG-7 Visual Standard for Content Description:An Overview[J]. IEEE Transactions on Circuits and Systems for Video Technology,11(6), 1-7,2001.
[26]ISO/IEC JTC1/SC29/WGl 1 N6909, Survey of Algorithms used for Multi-view Video Coding(MVC)[S]. Hong Kong, China,2005.
[27]Mueller K, Merkle P and Schwarz H, Multi-view video coding based on H.264/MPEG4-AVC using hierarchical B Pictures[C]. Picture Coding Symposium, Beijing, China,2006.
[28]Sullivan G J, Topiwala P. The H.264/AVC advanced video coding standard:Overview and introduction to the fidelity range extensions[C]. Proceedings of SPIE-The International Society for Optical Engineering 5558(PART 1).2004:454-474.
[29]Luo Y, Zhang Z Y. An P. Stereo video coding based on flame estimation and interpolation[J]. IEEE Transactions on Broadcasting,2003,49(1):14-21.
[30]熊文娟.基于H.264的多视角视频及全局视差估计算法的研究[D],通信与信息系统,2008.
[31]Nikolaidis N, Pitas l. Copyright protection of images using robust digital signatures[J]. IEEE Proceeding on International Conference on Acoustics, Speech, and Signal Processing, Atlanta, IEEE Press.1996:2168-2171.
[32]CHEN Y, WANG Y K, UGUR K. The emerging MVC standard for 3D video services [J]. EURASIP Journal on Advances in Signal Processing,2009,(1):1-13.
[33]M Soriano, M Fernandez, E Sayrol. Multimedia copyright protection platform demonstrator[J]. Lecture Notes in Computer Science. vol.3477,2005:411-414.
[34]Markus P, Aditya M, Bernd G. Motion and disparity compensated coding for multiview video[J]. IEEE Trans on Circuits and Systems for Video Technology,2007,17(11): 1474-1484.
[35]Naemura T, Kaneko M, Harashima H. Compression and representation of 3-D images [J]. IEICE Trans INF&SYST.1999, E82-D(3):558-565.
[36]A Puri, H Hang. Adaptive schemes for motion-compression coding[C]. SPIE VCIP, vol.1001, 1988:925-934.
[37]Gangyi Jiang, Yu Mei, Qiumin Xu. A new multi-view video coding scheme for 3DAV systems [C]. Picture Coding Symposium, Beijing, China,2006.
[38]Cox I J, Mat L Miller. A review of watermarking and the importance of perceptual model ing[J]. SPIE Proceeding On Human Vision and Electronic Imaging.1997(3016): 92-99.
[39]Mobasseri B G. Direct sequence watermarking of digital Video using m-frames [C]. Image Processing. ICIP98 Proceedings.1998.
[40]L. Onural. An overview of a new European consortium:integrated three-dimensional television-capture, transmission and display(3DTV)[J]. Proceedings of European workshop on the Integration of Knowledge, Semantics and Digital Media Technology, 2004.
[41]蒋刚毅,张云,郁梅.基于相关性分析的多模式多视点视频编码[J].计算机学报,2007,30(12)：2205-2211.
[42]LI D X, ZHENG W, XIE X H. Optimizing inter-view prediction structure for multiview video coding with minimum spanning tree[J]. Electronics Letters,2007 43(23): 1269-1271.
[43]K00 H S, JEON Y J, JEON B M. Motion information inferring scheme for multi-view video coding[J]. IEICE Transactions on Communications,2008, E91-B(4):1247-1250.
[44]Yang H T, Chang Y L, Huo J Y. CE1:fine motion matching for motion skip mode in MVC[A]. JVT-Z021,26th JVT Meeting[C]. Antalya, TR,2008.
[45]Fecker U, Barkowsk M, Kaup A. Histogram-based prefiltering for luminance and chrominance compensation of multiview video[J]. IEEE Transactions on Circuits and Systems for Video Technology,2008,18(9):1258-1267.
[46]Narang S, Narang N K. Network oriented motion estimation for video-on-demand over wireless networks[C]. Vehicular Technology Conference.1999,2(2):1535-1539.
[47]HUR J H, CHO S, LEE Y L. Adaptive local illumination change compensation method for H.264/AVC-based multiview video coding[J]. IEEE Trans Circuits and Systems for Video Technology,2007,17(11):1496-1505.
[48]YEA S, VETRO A. RD-optimized view synthesis prediction for multiview video coding[C]. IEEE International Conference on Image Processing 2007. San Antonio, Texas, USA,2007. 209-212.
[49]Chen WangK, Ugur K. The emerging MVC standard for 3D video services [J]. EURASIP Journal on Advances in Signal Processing,2009,(1):1-13.
[50]Redert A, Hendriks E, Biemond J. Correspondence estimation in image pairs [J] IEEE Signal Processing Magazine.1999,16(3).
[51]Wiegand T, Sdlwan G J. Overview of the H.264/AVC video coding standard[J]. IEEE Transactions on Circuits and Systems for Video Technology.2003.13(7):560-576.
[52]HE Y, Ostermann J, Tanimoto M. Introduction to the special section on multiview video coding[J]. IEEE Trans Circuits and Systems for Video Technology,2007,17(11): 1433-1435.
[53]Triantafyllidis G A, Cetin A E, Somlic A.3DTV:capture, transmission, and display of 3D video[J]. EURASIP Journal on Advances in Signal Processing,2009, (1):1-2.
[54]陈建乐.多视点视频编码方法的研究[D].浙江：浙江大学.2006.
[55]Sullivan G J, Topiwala P. The H.264/AVC advanced video coding standard:Overview and introduction to the fidelity range extensions[C]. Proceedings of SPIE-The International Society for Optical Engineering 5558(PART 11.2004:454-474.
[2]S A joscha, M C Chen.3DAV exploration of video based rendering technology in MPEG[J]. IEEE Transaction on Circuit and Systems for Video Technology.2004,14(3).
[3]Jose M. Martinez-Ibanez, MPEG 3DAV AhG Activities Report[R], TUM-Munich University of Technology, Trondheim-Norway,65th MPEG Meeting, July.2003.
[4]ISO/IEC JTCI/SC29/WG11. N5169:Description of exploration experiments in 3DAV[S]. Shanghai, China,2002.
[5]ISO/IEC JTCI/SC29/WG11. N5558:Report on status of 3DAV exploration [S]. Pattaya, Thailand,2003.
[6]ISO/IEC JTCI/SC29/WG11. N5678:Report on 3DAV exploration [S]. Trondheim, Norway, 2003.
[7]MPEG Video Subgroup. Draft call for proposals on multi-view video coding[A]. W6910, 71st MPEG Meeting[C]. Hong Kong, China,2005.
[8]Description of Core Experiments in MVC[R], ISO/IEC JTC1/SC29/Wg11 T100, April 2006
[9]Anthony Vetro, Purvin Pandit. JMVM Software Manual [EB/OL]. http://ftp3. itu. ch/av-areh/jvt-site/2007_06_Geneva/JVT-X008. zip,2007.
[10]Smolic A and Kimata H. Report on 3DAV exploration [A], ISO/IEC JTC1/SC29/WG11, Doc. N5878,2003.
[11]候紫峰.计算机立体视觉[J].程序员技术.1994,3：87-90.
[12]梁栋,韦穗,周敏彤.双眼立体感知几何模型的研究[J].中国图象图形学报.1998,3(8)：679-683.
[13]Magnor M, Ramanathan P and Girod B., Multi-view coding for image-based rendering using 3-d scene geometry [J], IEEE Transactions on Circuits and Systems for Video Technology, 2003,13(11):1092-1106.
[14]F Pedersini, A Sarti and S Tubaro. Multi-camera systems[J]. IEEE signal Processing Magazine. May 1999:55-65.
[15]荆其诚,焦书兰,纪桂萍.人类的视觉[M],北京：科学出版社,1987.
[16]Hartmut Emst, Robert Larice, Stephan Breide. Mixing of computer graphics and high-quality stereographic video[J]. IEEE Transactions. Consumer Electronics, 42(3):795-799,1996.
[17]朱仲杰,蒋刚毅,郁梅等.多视点视频编码中的视频目标提取与视差匹配[J].电子学报.2004,32(51)：848-850.
[18]余兆明,查日勇,黄磊,周海骄.图像编码标准H.264技术[M].人民邮电出版社,2006.
[19]R William, W Sherman, and B Craig. Understanding virtual reality:interface, application, and design[M]. San Francisco, CA:Morgan Kaufmann Publishers.2004:429-431.
[20]NVIDIA Corporation. NVIDIA 3D Stereo User's Guide[EB].2001.
[21]Narang S, Narang NK. Network oriented motion estimation for video-on—demand over wireless networks[C]. Vehicular Technology Conference.1999,2(2):1535-1539.
[22]李世平,郁梅,蒋刚毅.基于H.264的立体视频编码算法[J].计算机工程与应用.2005,1：77-79.
[23]Yoshio Matsumoto, Tomohiro Shibata, Katsuhiro Sakai. Real-time color stereo vision system for a mobile robot based on field multiplexing[C]. IEEE International Conference on Robotics and Automation,3:1934-1939,1997.
[24]D Marr. A Computational Investigation into the Human Representation and Processing of Visual Information[M]. WH. Freeman and Company San Francisco,1982.
[25]Thomas Sikora, Seinor Member. The MPEG-7 Visual Standard for Content Description:An Overview[J]. IEEE Transactions on Circuits and Systems for Video Technology,11(6), 1-7,2001.
[26]ISO/IEC JTC1/SC29/WGl 1 N6909, Survey of Algorithms used for Multi-view Video Coding(MVC)[S]. Hong Kong, China,2005.
[27]Mueller K, Merkle P and Schwarz H, Multi-view video coding based on H.264/MPEG4-AVC using hierarchical B Pictures[C]. Picture Coding Symposium, Beijing, China,2006.
[28]Sullivan G J, Topiwala P. The H.264/AVC advanced video coding standard:Overview and introduction to the fidelity range extensions[C]. Proceedings of SPIE-The International Society for Optical Engineering 5558(PART 1).2004:454-474.
[29]Luo Y, Zhang Z Y. An P. Stereo video coding based on flame estimation and interpolation[J]. IEEE Transactions on Broadcasting,2003,49(1):14-21.
[30]熊文娟.基于H.264的多视角视频及全局视差估计算法的研究[D],通信与信息系统,2008.
[31]Nikolaidis N, Pitas l. Copyright protection of images using robust digital signatures[J]. IEEE Proceeding on International Conference on Acoustics, Speech, and Signal Processing, Atlanta, IEEE Press.1996:2168-2171.
[32]CHEN Y, WANG Y K, UGUR K. The emerging MVC standard for 3D video services [J]. EURASIP Journal on Advances in Signal Processing,2009,(1):1-13.
[33]M Soriano, M Fernandez, E Sayrol. Multimedia copyright protection platform demonstrator[J]. Lecture Notes in Computer Science. vol.3477,2005:411-414.
[34]Markus P, Aditya M, Bernd G. Motion and disparity compensated coding for multiview video[J]. IEEE Trans on Circuits and Systems for Video Technology,2007,17(11): 1474-1484.
[35]Naemura T, Kaneko M, Harashima H. Compression and representation of 3-D images [J]. IEICE Trans INF&SYST.1999, E82-D(3):558-565.
[36]A Puri, H Hang. Adaptive schemes for motion-compression coding[C]. SPIE VCIP, vol.1001, 1988:925-934.
[37]Gangyi Jiang, Yu Mei, Qiumin Xu. A new multi-view video coding scheme for 3DAV systems [C]. Picture Coding Symposium, Beijing, China,2006.
[38]Cox I J, Mat L Miller. A review of watermarking and the importance of perceptual model ing[J]. SPIE Proceeding On Human Vision and Electronic Imaging.1997(3016): 92-99.
[39]Mobasseri B G. Direct sequence watermarking of digital Video using m-frames [C]. Image Processing. ICIP98 Proceedings.1998.
[40]L. Onural. An overview of a new European consortium:integrated three-dimensional television-capture, transmission and display(3DTV)[J]. Proceedings of European workshop on the Integration of Knowledge, Semantics and Digital Media Technology, 2004.
[41]蒋刚毅,张云,郁梅.基于相关性分析的多模式多视点视频编码[J].计算机学报,2007,30(12)：2205-2211.
[42]LI D X, ZHENG W, XIE X H. Optimizing inter-view prediction structure for multiview video coding with minimum spanning tree[J]. Electronics Letters,2007 43(23): 1269-1271.
[43]K00 H S, JEON Y J, JEON B M. Motion information inferring scheme for multi-view video coding[J]. IEICE Transactions on Communications,2008, E91-B(4):1247-1250.
[44]Yang H T, Chang Y L, Huo J Y. CE1:fine motion matching for motion skip mode in MVC[A]. JVT-Z021,26th JVT Meeting[C]. Antalya, TR,2008.
[45]Fecker U, Barkowsk M, Kaup A. Histogram-based prefiltering for luminance and chrominance compensation of multiview video[J]. IEEE Transactions on Circuits and Systems for Video Technology,2008,18(9):1258-1267.
[46]Narang S, Narang N K. Network oriented motion estimation for video-on-demand over wireless networks[C]. Vehicular Technology Conference.1999,2(2):1535-1539.
[47]HUR J H, CHO S, LEE Y L. Adaptive local illumination change compensation method for H.264/AVC-based multiview video coding[J]. IEEE Trans Circuits and Systems for Video Technology,2007,17(11):1496-1505.
[48]YEA S, VETRO A. RD-optimized view synthesis prediction for multiview video coding[C]. IEEE International Conference on Image Processing 2007. San Antonio, Texas, USA,2007. 209-212.
[49]Chen WangK, Ugur K. The emerging MVC standard for 3D video services [J]. EURASIP Journal on Advances in Signal Processing,2009,(1):1-13.
[50]Redert A, Hendriks E, Biemond J. Correspondence estimation in image pairs [J] IEEE Signal Processing Magazine.1999,16(3).
[51]Wiegand T, Sdlwan G J. Overview of the H.264/AVC video coding standard[J]. IEEE Transactions on Circuits and Systems for Video Technology.2003.13(7):560-576.
[52]HE Y, Ostermann J, Tanimoto M. Introduction to the special section on multiview video coding[J]. IEEE Trans Circuits and Systems for Video Technology,2007,17(11): 1433-1435.
[53]Triantafyllidis G A, Cetin A E, Somlic A.3DTV:capture, transmission, and display of 3D video[J]. EURASIP Journal on Advances in Signal Processing,2009, (1):1-2.
[54]陈建乐.多视点视频编码方法的研究[D].浙江：浙江大学.2006.
[55]Sullivan G J, Topiwala P. The H.264/AVC advanced video coding standard:Overview and introduction to the fidelity range extensions[C]. Proceedings of SPIE-The International Society for Optical Engineering 5558(PART 11.2004:454-474.