摘要
随着影视、动漫与游戏行业的蓬勃发展,其对高真实感三维场景重建的需求越来越多。而在文物数字化等领域,对于三维模型重建要求更高,从三维重建的逼真度要求上升到了对三维形体准确度及表面色彩保真度的要求。基于多视图立体匹配的三维重建是实现上述需求的一种重要方法,可直接计算得到包含准确色彩纹理的三维模型结果。准确性、鲁棒性以及计算效率是评价基于多视图立体匹配三维重建的重要标准。图像的畸变、随机噪声、重复纹理以及物体间的遮挡等因素影响了多视图立体匹配算法的鲁棒性和重建结果的准确性。
本文主要从提升算法鲁棒性和重建结果的准确性两个方面来深入研究面向复杂场景的三维重建方法：一方面,研究高质量的深度图计算以及融合算法,通过对影响深度计算准确性的一些因素进行建模,提高计算结果的准确性。另一方面,研究基于连续优化的深度计算方法,利用连续优化计算鲁棒性高的特点,来提高三维重建算法的鲁棒性。
具体地,本文研究图像的径向畸变矫正、基于非凸连续优化的深度计算、基于凸连续优化的深度计算以及基于连续深度图融合的多视图立体匹配。主要工作与创新包括：
●提出了一种基于矩阵QR分解的图像径向畸变矫正算法,解决了现有三维重建管线中畸变参数计算不够鲁棒的问题,提升了多视图三维重建算法的鲁棒性和重建结果的准确性。通过将畸变参数计算转化成矩阵分解问题,简化了参数的计算过程。
●提出了一种基于对称连续优化的深度图计算方法,使能量泛函的解更趋于全局最优解,有效的提高了深度图的质量。通过将立体匹配问题转化成连续马尔科夫随机域的形式,建立了基于对称连续优化的深度计算模型。在模型的数据项中,引入颜色一致性约束和梯度一致性约束,提高了算法的准确性。设计了基于多层图像金字塔的迭代计算框架,有效地提高了计算出的深度图的质量。在匹配泛函模型的设计中,还引入了左右一致性约束,进一步提升了深度计算结果的准确性。
●提出了一种基于凸优化的深度图计算方法,有效地提高了深度计算过程的鲁棒性和计算结果的准确性。针对物体间的相互遮挡等原因导致深度并不是严格连续的问题,提出了分段连续假设条件下的深度图计算方法将深度计算问题转化成自由不连续泛函模型来实现深度的计算,同时在泛函模型中引入了图像分割的先验知识,有效地抑制深度图在图像低频区域的噪声。通过利用将泛函模型松懈成凸泛函的方法,确保了深度图的计算过程不依赖初始值,提升了算法的鲁棒性,提高了深度图的质量。
●提出了一种基于连续深度图融合的三维重建方法,提高了重建模型的准确性。通过利用左右一致性信息来控制深度图不同区域的更新速度,提高了深度图的质量。设计了一种利用近邻图像信息和深度信息进行深度图优化的机制,进一步提高了深度图的质量。综合利用左右一致性约束信息、点的法向量信息以及相机的视角信息有效解决了深度融合过程中的去噪问题。
With the development of film and game industry, digital preservation of cultural heritage,3D printing technology, the virtual3D reconstruction of the scene and object becomes more and more fascinating. Multi-view stereo based3D reconstruction is a significant technique for those applications. Accuracy, robustness, efficiency are the three key issue to consider when design a3D reconstruction algorithm. Because of image distortion, image noise, repetitive textures, occlusion and other reasons, design a3D reconstruction algorithm that both achieve accurate3D model and robustness is usually a hard work, which restrict its application.
In this dissertation, a series of algorithms are proposed to improve the accuracy and robustness of the image based3D reconstruction technique. On one hand, algorithm that can achieve accurate disparity and depth maps are designed, and depth merging algorithm is also proposed. On the other hand, continuous method is applied to improve the robustness of the3D reconstruction algorithm.
More specifically, in this thesis, algorithms that associated with image distortion rectification, non-convex continuous based disparity estimation, convex continuous based method based disparity estimation are proposed. Our contribution includes:
· We proposed a QR factorization based image radial distortion algorithm, improved the robustness of the radial parameter estimation of the multi-view3D reconstruction pipeline, which further improved the robustness of the3D reconstruction algorithm and the accuracy of the reconstructed3D model. The process is simplified by converting radial parameter estimation into matrix factorization.
· We proposed a symmetric continuous depth map estimation algorithm, improved the quality of the depth map. Through model the stereo matching as a continuous MRF(Markov Random Field) problem, we built a symmetric functional for depth map estimation. In the data term of the functional, we applied both the color consistency and gradient consistency constraint. We used a multi-scale scheme for depth estimation. We also apply the left-right consistency soft constraint in the functional to further improve the depth map.
· We proposed a convex optimization based depth estimation algorithm, improved the robustness of the algorithm and the accuracy of the depth map. We designed a functional for depth estimation with a hypothesis that the depth is piece-wise continuous, this assumption is more flexible than continuous assumption. We modeled the depth estimation problem as a free-discontinuity problem. We introduced the image segmentation prior into the functional to suppress the image noise. And we relaxed the proposed functional into a convex one, through which the estimated depth is independent of initial value, so it is more robust than the algorithms which depend on initial value.
· We proposed a multiple continuous depth maps merging based3D reconstruction algorithm, improve the accuracy of the estimated3D model. We applied left-right depth consistency information to estimate a distance map, by which to control the speed of depth map update in different area of the image, in this way the quality of the depth maps can be further improved. We also designed a scheme to use the neighbor depth maps and images to optimize the depth map. When merging the depth maps, left-right consistency, normal of the point cloud and view direction of the camera were applied to denoise the depth maps.
引文
[1]杨鑫.面向高性能图形绘制的加速结构设计[D].浙江大学,2012.
[2]张帆.激光扫描数据三维建模中高保真度纹理重建研究[D].武汉大学,2009.
[3]刁常宇.三维模型实拍纹理高精度重建研究[D].浙江大学,2007.
[4]田里.基于时间空间混合结构光编码的可移动式三维扫描技术研究[D].浙江大学,2010.
[5]D.Scharstein, R.Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms [J]. International Journal of Computer Vision,2002, 47(1-3):7-42.
[6]S.M. Seitz, B. Curless, J. Diebel, et al. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms [C]. CVPR,2006:519-528.
[7]C. Strecha, W.von Hanse, L. Van Gool, et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery [C]. CVPR,2008:1-8.
[8]K.Kolev, T. Brox, D. Cremers. Robust variational segmentation of 3d objects from multiple views [C] DAGM,2006:688-697.
[9]K.N. Kutulakos, S.M. Seitz. A theory of shape by space carving [C]. ICCV,1999:307-314.
[10]S.N. Sinha, P. Mordohai, M. Pollefeys. Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh [C]. ICCV,2007:1-8.
[11]A. Zaharescu, E. Boyer, R.P. Horaud. TransforMesh:a topology-adaptive mesh-based approach to surface evolution [C], ACCV,2007:166-175.
[12]J. Pons, R. Keriven, O. Faugeras. Modelling Dynamic Scenes by Registering Multi-View Image Sequences [C]. CVPR,2005:822-827.
[13]C.H Esteban, F. Schmitt. Silhouette and stereo fusion for 3D object modeling [J]. CVIU,2004,96(3):367-392.
[14]M. Lhuillier, L. Quan. A quasi-dense approach to surface reconstruction from uncalibrated images [J]. PAMI,2005,27(3):418-433.
[15]Y. Furukawa, J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis [C]. CVPR,2007:1-8.
[16]M. Habbecke, L. Kobbelt. A Surface-Growing Approach to Multi-View Stereo Reconstruction [C]. CVPR,2007:1-8.
[17]M Goesele, B. Curless, S.M. Seitz. Multi-View Stereo Revisited [C]. CVPR, 2006:2402-2409.
[18]D. Bradley, T. Boubekeur, W. Heidrich. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. CVPR,2008:1-8.
[19]Y. Liu, Q.Dai, W. Xu. A Point-Cloud-Based Multiview Stereo Algorithm for Free-Viewpoint Video [J]. TVCG,2010,16(3):407-418.
[20]Y.I. Abdel-Aziz, H.M. Karara. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry [c]. Proceedings of ASP/UI Symposium on Close-range photogrammetry,1971:1-18.
[21]R.Y Tsai. An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. CVPR,1986:364-374.
[22]Z.Zhang. A flexible new technique for camera calibration [J]. PAMI,2000, 22(11):1330-1334.
[23]孟晓桥,胡占义.一种新的基于圆环点的摄像机自标定方法[J].软件学报,2002,13(5)：957-965.
[24]F.Abad, Camahort, R. Vivo. Camera calibration using two concentric circles [C]. International Conference on Image Analysis and Recognition,2004:688-696.
[25]L.Wang, F.Wu, Z. Hu. Multi-Camera Calibration with One-Dimensional Object under General Motions [C]. ICCV,2007:1-7.
[26]R.Hartley, A. Zisserman, Multiple View Geometry in Computer Vision [M]. Cambridge University Press,2003.
[27]G.Zhang, X.Qin, W.Hua, et al.Robust Metric Reconstruction from Challenging Video Sequences [C]. CVPR,2007:1-8.
[28]J.Tardif, P.F. Sturm, S.B. Roy. Plane-based self-calibration of radial distortion [C]. ICCV,2007:1-8.
[29]Ma, L, Y.Q. Chen, K.L. Moore. Rational radial distortion models with analytical undistortion formulae[J]. CoRR 2003.
[30]A.Fitzgibbon. Simultaneous linear estimation of multiple view geometry and lens distortion [C]. CVPR.2001:125-132.
[31]A. Basu, S. Licardie. Alternative models for fish-eye lenses [J]. Pattern Recognition Letters,1995,16(4):433-441.
[32]R. Tsai. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses [J]. Journal of Robotics and Automation,1987,3(4):323-344.
[33]J.Barreto, K. Daniilidis, Fundamental Matrix for Cameras with Radial Distortion [C]. ICCV,2005:625-632.
[34]R.Steele, C. Jaynes. Overconstrained linear estimation of radial distortion and multi-view geometry [C]. ECCV,2006:253-264.
[35]M.T. Ahmed, A.A. Farag. Differential methods for nonmetric calibration of camera lens distortion [C]. CVPR,2001,477-482.
[36]R.Swaminathan, S.K. Nayar. Nonmetric Calibration of Wide-Angle Lenses and Polycameras [J]. PAMI,2000,22(10):1172-1178.
[37]F.Devernay, O. Faugeras. Straight lines have to be straight:automatic calibration and removal of distortion from scenes of structured enviroments [J]. Mach. Vision Appl,2001,13(1):14-24.
[38]P.Anandan. A computational framework and an algorithm for the measurement of visual motion [J]. IJCV,1989,2(3):283-310.
[39]L.Matthies, T. Kanade, R. Szeliski. Kalman Filter-based Algorithms for Estimating Depth from Image Sequences [J]. IJCV,1989,3(3):209-238.
[40]H.Kano. Development of a video-rate stereo machine. IROS,1995:95-100.
[41]T.Ryan, R.T. Gray, B.R. Hunt. Prediction of correlation errors in stereo-pair images [J]. Optical Engineering,1980,3(19):312-322.
[42]C.Cuadrado, A.Zuloaga, J.Martin, et al. Real-Time Stereo Vision Processing System in a FPGA [C]. IEEE Conference on Industrial Electronics. 2006:3455-3460.
[43]M.Okutomi, T. Kanade. A locally adaptive window for signal matching [J]. IJCV, 1992,7(2):143-162.
[44]T.Kanade, M. Okutomi. A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment [J]. PAMI,1994,16(9):920-932.
[45]O.Veksler. Stereo Correspondence with Compact Windows via Minimum Ratio Cycle [J]. PAMI,2002,24(12):1654-1660.
[46]Q.Yang. A Non-Local Cost Aggregation Method for Stereo Matching [C]. CVPR,2012:1402-1409.
[47]B.Lucas,T. Kanade. An iterative image registration technique with an application to stereo vision [C]. IJCAI,1981:674-679.
[48]Q.Tian, M.N. Huhns. Algorithms for subpixel registration [J]. Computer Vision, Graphics, and Image Processing,1986.35(2):220-233.
[49]E.S. Larsen, E.S, et al. Temporally Consistent Reconstruction from Multiple Video Streams Using Enhanced Belief Propagation [C]. ICCV,2007:1-8.
[50]V.Kolmogorov, R. Zabih. Multi-camera Scene Reconstruction via Graph Cuts [C]. ECCV,2002:82-96.
[51]V.Kolmogorov, R. Zabih, S. Gortler. Generalized Multi-camera Scene Reconstruction Using Graph Cuts [C]. EMMCVPR,2003:501-506.
[52]P.Felzenszwalb, D.P. Huttenlocher. Efficient Belief Propagation for Early Vision [J]. IJCV,2006,70(1):41-54.
[53]J.Sun, N. Zheng, H. Shum. Stereo Matching Using Belief Propagation [J]. PAMI, 2003,25(7):787-800.
[54]Q.Yang, L.Wang, R.Yang, et al. Real-time Global Stereo Matching Using Hierarchical Belief Propagation[C]. BMVC,2006:989-998.
[55]Q.Yang, L.Wang, R.Yang, et al. Stereo Matching with Color-Weighted Correlation, Hierachical Belief Propagation and Occlusion Handling [C]. CVPR,2006,2347-2354.
[56]V.Kolmogorov, R. Zabih. Computing visual correspondence with occlusions using graph cuts[C]. ICCV,2001:508-515.
[57]N.Papadakis, V. Caselles. Multi-label Depth Estimation for Graph Cuts Stereo Problems[J]. J. Math. Imaging Vis,2010,38(1):70-82.
[58]N.Slesareva, S. Bruhn, J. Weickert. Optic flow goes stereo:a variational method for estimating discontinuity-preserving dense disparity maps [C]. Proceedings of the 27th DAGM conference on Pattern Recognition,2005.
[59]R.Ben-Ari, N. Sochen. Variational Stereo Vision with Sharp Discontinuities and Occlusion Handling [C]. ICCV,2007:1-7.
[60]S.Kosov, T.Thormahlen, H.P. Seidel. Accurate Real-Time Disparity Estimation with Variational Methods [C]. ISVC,2009:796-807.
[61]V.Kolmogorov, R. Zabih. What Energy Functions Can Be Minimized via Graph Cuts?[C]. ECCV,2002:65-81.
[62]Y.Boykov, O.Veksler, R. Zabih, Fast Approximate Energy Minimization via Graph Cuts [J]. PAMI,2001,23(11):1222-1239.
[63]H.Ishikawa. Exact optimization for Markov random fields with convex priors [J]. PAMI,2003,25(10):1333-1336.
[64]Y.Boykov, V. Kolmogorov. An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision [J]. PAMI,2004,26(9):1124-1137.
[65]T.Yu, R.Lin, B.Super, et al. Efficient Message Representations for Belief Propagation [C]. ICCV,2007:1-7.
[66]O.Faugeras, R. Keriven. Variational principles, surface evolution, PDE's, level set methods and the stereo problem [J]. Trans. Img. Proc,2002:336-344.
[67]T.Brox, A.Bruhn, J.Weickert. High Accuracy Optical Flow Estimation Based on a Theory for Warping [C]. ECCV,2004:25-36.
[68]T.Brox, J. Malik. Large Displacement Optical Flow:Descriptor Matching in Variational Motion Estimation [J]. PAMI,2011,33(3):500-513.
[69]C.Zach, T. Pock, H. Bischof. A duality based approach for realtime TV-L1 optical flow [C]. Proceedings of the 29th DAGM conference on Pattern recognition,2007:214-223.
[70]Y. Liu, X. Cao, Q. Dai, et al. Continuous depth estimation for multi-view stereo [C].CVPR,2009:2121-2128.
[71]T.Pock, T.Schoenemann,G Graber, Horst Bischof, et al. A Convex Formulation of Continuous Multi-label Problems [C]. ECCV,2008:792-805.
[72]T.Pock, D.Cremers, H.Bischof, et al. Global Solutions of Variational Models with Convex Regularization [J]. SIAM J. Img. Sci,2010,3(4):1122-1145.
[73]F.Tombari, S. Mattoccia, L. Di Stefano. Segmentation-based adaptive support for accurate stereo correspondence [C]. PSIVT,2007:427-438.
[74]M.Bleyer, C. Rother, P. Kohli. Surface Stereo with Soft Segmentation [C]. CVPR,2010:1570-1577.
[75]L.Wang, R. Yang. Global stereo matching leveraged by sparse ground control points [C]. CVPR 2011:3033-3040.
[76]J.Franco, E. Boyer. Exact polyhedral visual hulls [C]. BMVC,2003:328-338.
[77]J.Li, Eric Li, Y. Chen, et al. Bundled depth-map merging for multi-view stereo [C]. CVPR,2010:2769-2776
[78]E.Tola, V. Lepetit, P. Fua. Daisy:an Efficient Dense Descriptor Applied to Wide Baseline Stereo [J]. PAMI,2010,32(5):815-830.
[79]G.Zhang, J.Jia, T.Wong, et al. Consistent Depth Maps Recovery from a Video Sequence [J]. PAMI,2009,31(6):974-988.
[80]G.Zhang, J.Jia, T. Wong, et al. Recovering consistent video depth maps via bundle optimization [C] CVPR,2008:1-8.
[81]N. Campbell, G.Vogiatzis, C.Hernandez, et al. Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo [C]. ECCV:766-779.
[82]C. Strecha, R. Fransens, L. Van Gool. Combined Depth and Outlier Estimation in Multi-View Stereo [C]. CVPR,2006:2394-2401.
[83]C. Strecha, T. Tuytelaars, L.V. Gool. Dense Matching of Multiple Wide-baseline Views [C]. ICCV,2003:1194-1201.
[84]R.Newcombe, A.J. Davison. Live Dense Reconstruction with a Single Moving Camera [C]. CVPR,2010:1498-1505.
[85]Jancosek, M, T. Pajdla. Segmentation based Multi-View Stereo. CVWW,2009.
[86]M.Goesele, N.Snavely, B.Curless, et al. Multi-View Stereo for Community Photo Collections. ICCV,2007:1-8.
[87]Y.Furukawa. Clustering views for multi-view stereo [EB/OL]. http://www.di.ens.fr/cmvs/
[88]Y.Furukawa, J. Ponce. Accurate camera calibration from multi-view stereo and bundle adjustment. CVPR,2008:1-8.
[89]Y.Furukawa, B.Curless, S.M. Steiz. Towards Internet-scale multi-view stereo. CVPR,2010:1434-1441.
[90]Multi-view Stereo[EB/OL]. http://vision.middlebury.edu/mview/
[91]C. Harris, M. Stephens. A Combined Corner and Edge Detection. Proceedings of The Fourth Alvey Vision Conference,1988:147-151.
[92]D. Low. Distinctive Image Features from Scale-Invariant Keypoints [J]. IJCV, 2004,60(2):91-110.
[93]Liu, S, D.B. Cooper. A complete statistical inverse ray tracing approach to multi-view stereo. CVPR.,2011:913-920.
[94]K.Kolev, M.Klodt, T.Brox, et al.Continuous global optimization in multiview 3D reconstruction. [C]. EMMCVPR,2007:441-452.
[95]K.Kolev, D. Cremers. Continuous ratio optimization via convex relaxation with applications to multiview 3D reconstruction. CVPR,2009,1858-1864.
[96]G.Vogiatzis, P.Torr, R. Cipolla. Multi-view Stereo via Volumetric Graph-cuts [C]. CVPR,2005:391-398.
[97]S.Tran, L. Davis.3D Surface Reconstruction Using Graph Cuts with Surface Constraints. ECCV,2006:219-231.
[98]A. Hornung, L. Kobbelt. Hierarchical Volumetric Multi-view Stereo Reconstruction of Manifold Surfaces based on Dual Graph Embedding. CVPR, 2006:503-510.
[99]J.Pons, R. Keriven, O. Faugeras. Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score [J]. IJCV,2007. 72(2):179-193.
[100]K.Kolev, T. Pock, D. Cremers. Anisotropic minimal surfaces integrating photoconsistency and normal information for multiview stereo [C]. ECCV,2010: 503-510.
[101]M.Klodt, et al. An Experimental Comparison of Discrete and Continuous Shape Optimization Methods[C]. ECCV,2008:332-345.
[102]P.Merrell, A.AkBarzadeh, L.Wang, et al. Real-Time Visibility-Based Fusion of Depth Maps[C]. CVPR,2007:1-8.
[103]Y.Furukawa, J. Ponce. Carved Visual Hulls for Image-Based Modeling [J]. IJCV, 2009,81(1):53-67.
[104]S.Osher, R. Fedkiw. Level Set Methods and Dynamic Implicit Surfaces [M]. Springer Press,2002.
[105]B.Curless, M. Levoy. A volumetric method for building complex models from range images[C]. SIGGRAP,1996:303-312.
[106]A.Hilton. Reliable Surface Reconstruction from Multiple Range Images. ECCV,1996:117-126.
[107]C.Zach, T. Pock, H. Bischof. A Globally Optimal Algorithm for Robust TV-L Range Image Integration [C]. ICCV,2007:1-8.
[108]J.Boissonnat, F.Cazals. Smooth surface reconstruction via natural neighbour interpolation of distance functions [C]. SCG,2000:223-232.
[109]J.Carr,R.Beatson,J.Cherrie, et al. Reconstruction and representation of 3D objects with radial basis functions [C]. SIGGRAPH,2001:67-76.
[110]M.Kazhdan, M. Bolitho, H. Hoppe. Poisson surface reconstruction [C]. SGP,2006:61-70.
[Ill]M.Bolitho. Parallel Poisson Surface Reconstruction [C]. ISVC,2009:678-689.
[112]W.Lorensen, H. Cline. Marching cubes:A high resolution 3D surface construction algorithm [C]. SIGGRAPH,1987:163-169.
[113]J.Bloomenthal. Polygonization of implicit surfaces[J]. Comput. Aided Geom. Des.1988,5(4):341-355.
[114]W.Press, S.Tevkolsky, W.Vetterling, et al. Numerical Recipes [M] Cambridge University Press,2007.
[115]G.Alberti, G.Bouchitte, G.Maso. The calibration method for the Mumford-Shah functional and free-discontinuity problems [J]. Calc.Var.Parital Differential Equations.2003.16(3):299-333.
1116] T.Pock, D.Cremers, H.Bischof, et al. An algorithm for minimizing the Mumford-Shah functional [C]. ICCV,2009:1133-1140.
[117]A.Fusiello, A.Trucco, A. Verri. A compact algorithm for rectification of stereo pairs. Mach. Vision Appl.2000,12(1):16-22.