自然三维电视系统中的深度提取技术研究
|
摘要
电视技术经过黑白电视、彩色电视和数字高清晰度电视三个阶段,逐渐向着大屏幕、彩色化、高清晰度、多媒体方向发展,而三维电视(3DTV)具有深度感和临场感,因而必将成为下一个新的发展方向,因此研究高效的三维电视系统具有很大的现实意义。
论文一方面围绕3DTV的研究现状,提出一种新的3DTV系统架构方式——自然三维电视系统。该系统强调的是自然的场景和自然的无需佩戴眼镜式的立体感受,采用的编码结构是当前比较实用的“一路视频+深度”的方式,同时引入了“遮挡信息”,在保证清晰的主视点立体显示效果的同时,提供了丰富的侧向和遮挡信息;同时该系统具有良好的可伸缩性,通过配置摄像头数量及遮挡信息的丰富程度,控制传输带宽和存储代价,获得不同质量的三维效果。
另一方面在3DTV系统的关键难题——深度提取算法这块做了大量工作。研究了立体匹配算法的研究现状和算法原理,在编写了立体匹配算法的系统框架调试软件的同时实现了多种深度提取算法。
最后,论文在现有深度提取算法各自独立的启发下,分析了立体匹配多种算法结合的可能性,并提出一种并行优化模型,适用于任意两种深度生成算法的优化。具体来说,该模型包括以下4个步骤:1)可靠像素点判定;2)可靠视差层选择;3)基于CROSS聚合的非可靠区域的视差填充;4)基于Canny边界检测的轮廓拟合。实验结果显示,该优化模型能充分发挥两种算法的优点,适用于任意两种深度提取算法的优化。并且非常高效,耗时不到1s,因此,这也可以作为两种实时算法的后期优化方案。
With the development of television(TV) which from black and white TV, color TV to digital high-definition TV, the TV technology has gradually towards the direction of large-screen, color, high-definition and multimedia. The three-dimensional television (3DTV) has a depth and ambiance sense, so it will surely to be the next direction of TV. It has a great practical significance to research the 3DTV system.
Around the status of 3DTV research, the paper propose a new 3DTV system - a natural three-dimensional television system. The system is stressed that the natural scene and natural 3D feeling without wearing glasses, and using the currently practical coding structure of "one video + depth" approach, while introduced the concept of "occlusion information". With the "occlusion information", not only do the audiences have the clear 3D display of main view, but also providing a rich side and occlusion information. Additional, this system has good scalability, which can obtain the different quality of 3D effect by configuring the camera, blocking information and control bandwidth and storage costs.
Then, I have done a lot of work in the key challenges of depth extracting of the natural 3DTV system. After researching the status and theory of stereo matching algorithm, I programmed a experimental platform software for stereo matching algorithms and achieved a wide range of stereo matching algorithm.
Finally, Inspired by the defects of single stereo matching algorithm, I propose a novel parallel optimization model for two arbitrary stereo matching algorithms. It includes four steps: Firstly, reliable disparity pixel marking; Secondly, reliable disparity layer selecting; And then, unreliable region filling based on cross aggregation; Finally, border handling based on canny edge detection. The experimental results show that the model can make full use of the advantages of two methods, especially for both poor methods. Additionally, our model is very efficient and it takes less than one second for a typical image. Therefore, it can be used as an optimization model for both real-time stereo parallel algorithms.
论文一方面围绕3DTV的研究现状,提出一种新的3DTV系统架构方式——自然三维电视系统。该系统强调的是自然的场景和自然的无需佩戴眼镜式的立体感受,采用的编码结构是当前比较实用的“一路视频+深度”的方式,同时引入了“遮挡信息”,在保证清晰的主视点立体显示效果的同时,提供了丰富的侧向和遮挡信息;同时该系统具有良好的可伸缩性,通过配置摄像头数量及遮挡信息的丰富程度,控制传输带宽和存储代价,获得不同质量的三维效果。
另一方面在3DTV系统的关键难题——深度提取算法这块做了大量工作。研究了立体匹配算法的研究现状和算法原理,在编写了立体匹配算法的系统框架调试软件的同时实现了多种深度提取算法。
最后,论文在现有深度提取算法各自独立的启发下,分析了立体匹配多种算法结合的可能性,并提出一种并行优化模型,适用于任意两种深度生成算法的优化。具体来说,该模型包括以下4个步骤:1)可靠像素点判定;2)可靠视差层选择;3)基于CROSS聚合的非可靠区域的视差填充;4)基于Canny边界检测的轮廓拟合。实验结果显示,该优化模型能充分发挥两种算法的优点,适用于任意两种深度提取算法的优化。并且非常高效,耗时不到1s,因此,这也可以作为两种实时算法的后期优化方案。
With the development of television(TV) which from black and white TV, color TV to digital high-definition TV, the TV technology has gradually towards the direction of large-screen, color, high-definition and multimedia. The three-dimensional television (3DTV) has a depth and ambiance sense, so it will surely to be the next direction of TV. It has a great practical significance to research the 3DTV system.
Around the status of 3DTV research, the paper propose a new 3DTV system - a natural three-dimensional television system. The system is stressed that the natural scene and natural 3D feeling without wearing glasses, and using the currently practical coding structure of "one video + depth" approach, while introduced the concept of "occlusion information". With the "occlusion information", not only do the audiences have the clear 3D display of main view, but also providing a rich side and occlusion information. Additional, this system has good scalability, which can obtain the different quality of 3D effect by configuring the camera, blocking information and control bandwidth and storage costs.
Then, I have done a lot of work in the key challenges of depth extracting of the natural 3DTV system. After researching the status and theory of stereo matching algorithm, I programmed a experimental platform software for stereo matching algorithms and achieved a wide range of stereo matching algorithm.
Finally, Inspired by the defects of single stereo matching algorithm, I propose a novel parallel optimization model for two arbitrary stereo matching algorithms. It includes four steps: Firstly, reliable disparity pixel marking; Secondly, reliable disparity layer selecting; And then, unreliable region filling based on cross aggregation; Finally, border handling based on canny edge detection. The experimental results show that the model can make full use of the advantages of two methods, especially for both poor methods. Additionally, our model is very efficient and it takes less than one second for a typical image. Therefore, it can be used as an optimization model for both real-time stereo parallel algorithms.
引文
[1] BENZIE P, WATSON J, SURMAN P,et al. "A Survey of 3DTV Displays: Techniques and Technologies". IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17 (11): 1647-1658.
[2] C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," in Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, USA, Jan. 2004
[3] Stereoscopic Displays and Applications. http://www.stereoscopic.org/.
[4] Aljoscha Smolic, Karsten Mueller, Joern Ostermann, Georgios Triantafyllidis, "Coding Algorithms for 3DTV-A Survey", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
[5] JAVIDI B, OKANO F. "Three-dimensional Television, Video, and Display Technologies". Springer, 2002.
[6] SON J Y, SMIRNOV V V, CHUN Y S, et al. "Non-glasses type stereoscopic display system based on polarization". Stereoscopic Displays and Virtual Reality Systems, 1999, SPIE, 3639: 132-136.
[7] Philipp Merkle, Aljoscha Smolic, Karsten Mueller and Thomas Wiegand, "Comparative Study of MVC Prediction Structures", JVT-V132, Jan. 2007
[8] DODGSON N A. "Autostereoscopic 3D Displays". Computer, 2005, 38 (8): 31-36
[9] 3DTV-Conference. http://www.3dtv-con2009.org/.
[10]C. Fehn, "3D-TV using depth-image-based rendering (DIBR)," in Proc. Picture Coding Symp., San Francisco, CA, USA, Dec. 2004;
[11]Y. Ho, K. Oh, "Overview of Multi-view Video Coding", 14th International Workshop on Communications and Services,pp:5 - 12, June 2007
[12]P. Merkle, K. Müller, A. Smolic, and T. Wiegand, "Efficient compression of multiview video exploiting inter-view dependencies based on H.264/MPEG4-AVC," in Proc. ICME 2006 IEEE Int. Conf. on Multimedia and Expo,Toronto,Ontario,Canada,Jul.2006;
[13]Joint Draft 8.0 on Multiview Video Coding.JVT-AB204.2008
[14]郑东华,于瀛洁,程维明,“三维立体显示技术的研究新进展”,光学技术,2008,34(3):426-430,434.
[15]吕朝辉,董跃,“立体电视研究现状与展望”,电视技术,2006(8):39-41.
[16]ISO/IEC International Standard 13818-2.Information Technology-Generic coding of Moving Pictures and Associated Audio Information:Video.1994
[17]ISO/IEC 13818-2 Amendment 3(Multi-View Profile),MPEG96 N1366.1996.
[18]Dong-Xiao Li,Wei Zheng,Xian-Hai Xie and Ming Zhang."Optimizing the inter-view prediction structure for multiview video coding with minimum spanning tree",Electronics Letters,Vol.43,No.23,pp.1269-1271,2007.
[19]张明,李东晓,谢贤海,王维东,谢磊,徐元欣,陈惠芳,郑伟。“一种自然三维电视系统”。国家发明专利,申请号:CN200810062854.7,2008年11月26号公开,公开号:CN101312542。
[20]HYUN L,KUGJIN Y,NAMHOH,et al."A Structure for 2D/3D Mixed Service Based on Terrestrial DMB System".3DTV Conference,2007:1-4
[21]FEHN C,KAUFF P,BEECK M O D,et al."An evolutionary and optimised approach on 3D-TV".International Broadcase Conference,2002:357-365.
[22]L.Stelmach,W.Tam,D.Meegan,A.Vincent,and P.Corfiveau,"Human perception of mismatched stereoscopic 3D inputs," in International Conference on Image Processing,vol.1,10-13 Sept.2000,pp.5-8
[23]LIU Zhan-wei,AN Ping,LIU Su-xing,ZHANG Zhao-yang,"Arbitrary View Generation Based On DIBR",Proceeding of 2007 Interational Symposium on Intelligent Signal Processing and Communication Systems Nov.28-Dec.1,2007Xiamen,China
[24]P.Seuntiens,L.Meesters,and W.Ijsselsteijn,"Perceived quality of compressed stereoscopic images:Effects of symmetric and asymmetric JPEG coding and camera separation," ACM Transactions on Applied Perception,vol.3,no.2,pp.95-109,Jan.2006
[25]R.Krishnamurthy,B.-B.Chai,H.Tao,and S.Sethuraman,"Compression and transmission of depth maps for image-based rendering," in International Conference on Image Processing,vol.3,7-10 Oct.2001,pp.828-831
[26]L.Zhang and W.Tam,"Stereoscopic image generation based on depth images for 3D TV," IEEE Transactions on Broadcasting,vol.51,no.2,pp.191-199,June 2005
[27]Yannick Morvan,Dirk Farin and Peter,"Incorporating Depth-image Based View-Prediction Into H.264 Multiview-image Coding",ICIP 2007
[28]D.Scharstein and R.Szeliski,"A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," Int'1 J.Computer Vision,Vol.47,no.1,pp.7-42,2002
[29]张金苓,王琰,”基于特征约束的图像点匹配算法”,沈阳理工大学学报,2006年25卷6期
[30]H.Tao,H.S.Sawhney and R.Kumar,"A Global Matching Framework for Stereo Computation," Proc.Int'1 Conf.Computer Vision 2001
[31]Liang Wang,Mingwei Gong,Minglun Gong,Ruigang Yang,"How Far Can We Go with Local Optimization in Real-Time Stereo Matching," 3D Data Processing,Visualization,and Transmission,2006
[32]K.-J.Yoon,I.-S.Kweon,"Locally adaptive support-weight approach for visual correspondence search," Proc.CVPR,pp.924-931,2005
[33]Zeng-Fu Wang,Zhi-Gang Zheng,"A Region Based Stereo Matching Algorithm Using Cooperative Optimization," 2007
[34]J.-C.Kim,K.M.Lee,B.-T.Choi,and S.U.Lee,"A dense stereo matching using two-pass dynamic programming with generalized ground control points," Proc.CVPR,pp.1075-1082,2005.
[35]Yang Q.,Wang L.,Yang R.,Wang S.,Liao M.,and Nister,"Real-time global stereo matching using hierarchical belief propagation",In Proc.British Machine Vision Conference,Edinburgh,UK,2006
[36]Jiangbo Lu,Ke Zhang,Lafruit,G.,Catthoor,F.,"Real-time stereo matching:A cross-based local approach," Acoustics,Speech and Signal Processing,2009.ICASSP 2009.IEEE International Conference on vol.,no.,pp.733-736,19-24April 2009.
[37]Liang Wang,Miao Liao,Minglun Gong,Ruigang Yang,Nister,D.,"High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming,"3D Data Processing,Visualization,and Transmission,Third International Symposium on,vol.,no.,pp.798-805,14-16 June 2006.
[38]Kuk-Jin Yoon;In So Kweon,"Adaptive support-weight approach for correspondence search," Pattern Analysis and Machine Intelligence,IEEE Transactions on,vol.28,no.4,pp.650-656,April 2006.
[39]S.El-Etriby,A.Al-Hamadi,and B.Michaelis,"Dense stereo correspondence with slanted surface using phase-based algorithm",IEEE ISIE 2007.
[40]O.Veksler,"Stereo correspondence by dynamic programming on a tree",CVPR 2005.
[41]尹传历,刘冬梅,宋建中,“改进的基于图像分割的立体匹配算法”,计算机辅助设计与图形学学报,2008
[42]C.Strecha,R.Fransens,and L.Van Gool,"Combined depth and outlier estimation in multi-view stereo",CVPR 2006
[43]D.Scharstein and R.Szelisk,Middlebury Stereo Vision Research Page,http://vision.middlebury.edu/stereo/.
[44]C.Strecha,R.Fransens,and L.Van Gool.Combined depth and outlier estimation in multi-view stereo.CVPR 2006.
[45]DISTIMA.European RACE 2045 Project.
[46]Kai Luo,Dong-xiao Li,Ya-mei Feng,Ming Zhang,"Depth-aided inpainting for disocclusion restoration of multi-view images using Depth-Image-Based Rendering(DIBR)",Journal of Zhejiang University,Science-A.
[47]http://www.3d4you.eu/index.php.
[2] C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," in Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, USA, Jan. 2004
[3] Stereoscopic Displays and Applications. http://www.stereoscopic.org/.
[4] Aljoscha Smolic, Karsten Mueller, Joern Ostermann, Georgios Triantafyllidis, "Coding Algorithms for 3DTV-A Survey", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
[5] JAVIDI B, OKANO F. "Three-dimensional Television, Video, and Display Technologies". Springer, 2002.
[6] SON J Y, SMIRNOV V V, CHUN Y S, et al. "Non-glasses type stereoscopic display system based on polarization". Stereoscopic Displays and Virtual Reality Systems, 1999, SPIE, 3639: 132-136.
[7] Philipp Merkle, Aljoscha Smolic, Karsten Mueller and Thomas Wiegand, "Comparative Study of MVC Prediction Structures", JVT-V132, Jan. 2007
[8] DODGSON N A. "Autostereoscopic 3D Displays". Computer, 2005, 38 (8): 31-36
[9] 3DTV-Conference. http://www.3dtv-con2009.org/.
[10]C. Fehn, "3D-TV using depth-image-based rendering (DIBR)," in Proc. Picture Coding Symp., San Francisco, CA, USA, Dec. 2004;
[11]Y. Ho, K. Oh, "Overview of Multi-view Video Coding", 14th International Workshop on Communications and Services,pp:5 - 12, June 2007
[12]P. Merkle, K. Müller, A. Smolic, and T. Wiegand, "Efficient compression of multiview video exploiting inter-view dependencies based on H.264/MPEG4-AVC," in Proc. ICME 2006 IEEE Int. Conf. on Multimedia and Expo,Toronto,Ontario,Canada,Jul.2006;
[13]Joint Draft 8.0 on Multiview Video Coding.JVT-AB204.2008
[14]郑东华,于瀛洁,程维明,“三维立体显示技术的研究新进展”,光学技术,2008,34(3):426-430,434.
[15]吕朝辉,董跃,“立体电视研究现状与展望”,电视技术,2006(8):39-41.
[16]ISO/IEC International Standard 13818-2.Information Technology-Generic coding of Moving Pictures and Associated Audio Information:Video.1994
[17]ISO/IEC 13818-2 Amendment 3(Multi-View Profile),MPEG96 N1366.1996.
[18]Dong-Xiao Li,Wei Zheng,Xian-Hai Xie and Ming Zhang."Optimizing the inter-view prediction structure for multiview video coding with minimum spanning tree",Electronics Letters,Vol.43,No.23,pp.1269-1271,2007.
[19]张明,李东晓,谢贤海,王维东,谢磊,徐元欣,陈惠芳,郑伟。“一种自然三维电视系统”。国家发明专利,申请号:CN200810062854.7,2008年11月26号公开,公开号:CN101312542。
[20]HYUN L,KUGJIN Y,NAMHOH,et al."A Structure for 2D/3D Mixed Service Based on Terrestrial DMB System".3DTV Conference,2007:1-4
[21]FEHN C,KAUFF P,BEECK M O D,et al."An evolutionary and optimised approach on 3D-TV".International Broadcase Conference,2002:357-365.
[22]L.Stelmach,W.Tam,D.Meegan,A.Vincent,and P.Corfiveau,"Human perception of mismatched stereoscopic 3D inputs," in International Conference on Image Processing,vol.1,10-13 Sept.2000,pp.5-8
[23]LIU Zhan-wei,AN Ping,LIU Su-xing,ZHANG Zhao-yang,"Arbitrary View Generation Based On DIBR",Proceeding of 2007 Interational Symposium on Intelligent Signal Processing and Communication Systems Nov.28-Dec.1,2007Xiamen,China
[24]P.Seuntiens,L.Meesters,and W.Ijsselsteijn,"Perceived quality of compressed stereoscopic images:Effects of symmetric and asymmetric JPEG coding and camera separation," ACM Transactions on Applied Perception,vol.3,no.2,pp.95-109,Jan.2006
[25]R.Krishnamurthy,B.-B.Chai,H.Tao,and S.Sethuraman,"Compression and transmission of depth maps for image-based rendering," in International Conference on Image Processing,vol.3,7-10 Oct.2001,pp.828-831
[26]L.Zhang and W.Tam,"Stereoscopic image generation based on depth images for 3D TV," IEEE Transactions on Broadcasting,vol.51,no.2,pp.191-199,June 2005
[27]Yannick Morvan,Dirk Farin and Peter,"Incorporating Depth-image Based View-Prediction Into H.264 Multiview-image Coding",ICIP 2007
[28]D.Scharstein and R.Szeliski,"A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," Int'1 J.Computer Vision,Vol.47,no.1,pp.7-42,2002
[29]张金苓,王琰,”基于特征约束的图像点匹配算法”,沈阳理工大学学报,2006年25卷6期
[30]H.Tao,H.S.Sawhney and R.Kumar,"A Global Matching Framework for Stereo Computation," Proc.Int'1 Conf.Computer Vision 2001
[31]Liang Wang,Mingwei Gong,Minglun Gong,Ruigang Yang,"How Far Can We Go with Local Optimization in Real-Time Stereo Matching," 3D Data Processing,Visualization,and Transmission,2006
[32]K.-J.Yoon,I.-S.Kweon,"Locally adaptive support-weight approach for visual correspondence search," Proc.CVPR,pp.924-931,2005
[33]Zeng-Fu Wang,Zhi-Gang Zheng,"A Region Based Stereo Matching Algorithm Using Cooperative Optimization," 2007
[34]J.-C.Kim,K.M.Lee,B.-T.Choi,and S.U.Lee,"A dense stereo matching using two-pass dynamic programming with generalized ground control points," Proc.CVPR,pp.1075-1082,2005.
[35]Yang Q.,Wang L.,Yang R.,Wang S.,Liao M.,and Nister,"Real-time global stereo matching using hierarchical belief propagation",In Proc.British Machine Vision Conference,Edinburgh,UK,2006
[36]Jiangbo Lu,Ke Zhang,Lafruit,G.,Catthoor,F.,"Real-time stereo matching:A cross-based local approach," Acoustics,Speech and Signal Processing,2009.ICASSP 2009.IEEE International Conference on vol.,no.,pp.733-736,19-24April 2009.
[37]Liang Wang,Miao Liao,Minglun Gong,Ruigang Yang,Nister,D.,"High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming,"3D Data Processing,Visualization,and Transmission,Third International Symposium on,vol.,no.,pp.798-805,14-16 June 2006.
[38]Kuk-Jin Yoon;In So Kweon,"Adaptive support-weight approach for correspondence search," Pattern Analysis and Machine Intelligence,IEEE Transactions on,vol.28,no.4,pp.650-656,April 2006.
[39]S.El-Etriby,A.Al-Hamadi,and B.Michaelis,"Dense stereo correspondence with slanted surface using phase-based algorithm",IEEE ISIE 2007.
[40]O.Veksler,"Stereo correspondence by dynamic programming on a tree",CVPR 2005.
[41]尹传历,刘冬梅,宋建中,“改进的基于图像分割的立体匹配算法”,计算机辅助设计与图形学学报,2008
[42]C.Strecha,R.Fransens,and L.Van Gool,"Combined depth and outlier estimation in multi-view stereo",CVPR 2006
[43]D.Scharstein and R.Szelisk,Middlebury Stereo Vision Research Page,http://vision.middlebury.edu/stereo/.
[44]C.Strecha,R.Fransens,and L.Van Gool.Combined depth and outlier estimation in multi-view stereo.CVPR 2006.
[45]DISTIMA.European RACE 2045 Project.
[46]Kai Luo,Dong-xiao Li,Ya-mei Feng,Ming Zhang,"Depth-aided inpainting for disocclusion restoration of multi-view images using Depth-Image-Based Rendering(DIBR)",Journal of Zhejiang University,Science-A.
[47]http://www.3d4you.eu/index.php.