小场景交互式稠密三维重建系统
|
摘要
近年来,随着GPU技术的深入发展和并行算法的日益成熟,使得实时三维重建成为可能。文中实现了一种针对小场景的交互式稠密三维重建系统,此系统借助先进的移动跟踪技术,可以准确地估计相机的即时位置。提出了一种改进的多视深度生成算法,在GPU加速下能够实时计算场景的深度。改进算法中的亚像素级的半全局匹配代价累积提高了多视立体匹配的精度,并结合全局优化的方法计算出了准确的场景深度信息。深度图被转换为距离场,使用全局优化的直方图压缩融合算法和并行的原始对偶算法实现了深度的实时融合。实验结果证明了重建系统的可行性和重建算法的正确性。
In recent years, it is possible to perform real-time 3 D reconstruction with the development of the GPU technology and maturity of parallel algorithms. This paper presents an interactive dense 3 D reconstruction system for small scenes, which can accurately estimate the real-time position of the camera by means of advanced mobile tracking technology. An improved multi-view depth generation algorithm is proposed, which can calculate scene depth in real-time under GPU acceleration. The cumulative cost of sub-pixel semi-global matching in the improved algorithm improves the accuracy of multi-view stereo matching, and combines the global optimization method to calculate the accurate scene depth information. Depth map is converted into distance field, and real-time depth fusion is realized by using globally optimized histogram compression fusion algorithm and parallel primal-dual algorithm. The experimental results prove the feasibility of the reconstruction system and the correctness of the reconstruction algorithm.
In recent years, it is possible to perform real-time 3 D reconstruction with the development of the GPU technology and maturity of parallel algorithms. This paper presents an interactive dense 3 D reconstruction system for small scenes, which can accurately estimate the real-time position of the camera by means of advanced mobile tracking technology. An improved multi-view depth generation algorithm is proposed, which can calculate scene depth in real-time under GPU acceleration. The cumulative cost of sub-pixel semi-global matching in the improved algorithm improves the accuracy of multi-view stereo matching, and combines the global optimization method to calculate the accurate scene depth information. Depth map is converted into distance field, and real-time depth fusion is realized by using globally optimized histogram compression fusion algorithm and parallel primal-dual algorithm. The experimental results prove the feasibility of the reconstruction system and the correctness of the reconstruction algorithm.
引文
[1]李白萍,韩新怡,吴冬梅.基于卷积神经网络的实时人群密度估计[J].图学学报,2018,39(4):728-734.
[2]王海瑶,唐娟,沈振辉.基于深度卷积神经网络的多任务细粒度车型识别[J].图学学报,2018,39(3):485-492.
[3]WU J J,ZHANG C K,ZHANG X M,et al.Learning shape priors for single-view 3D completion and reconstruction[C]//Proceedings of 2018 European Conference on Computer Vision.Heidelberg:Springer,2018:1-17.
[4]JACKSON A S,MANAFAS C,TZIMIROPOULOS G.3D human body reconstruction from a single image via volumetric regression[C]//Proceedings of 2018European Conference on Computer Vision Workshops.Heidelberg:Springer,2018:1-14.
[5]CARVALHO M,BERTRAND L S,PAULINE T P,et al.Deep depth form deforcus:How can defocus blur improve 3D estimation using dense neural network?[C]//Proceedings of 2018 European Conference on Computer Vision,3DRW Workshop.Heidelberg:Springer,2018:1-18.
[6]DELLAERT F,SEITZ S M,THORPE C E,et al.Structure from motion without correspondence[C]//2000 IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2000:557-564.
[7]POLLEFEYS M,NISTER J M,AKBARZADEH A,et al.Detailed real-time urban 3D reconstruction from video[J].International Journal of Computer Vision,2008,78(2):143-167.
[8]DAVISON A J.Real-time simultaneous localisation and mapping with a single camera[C]//Proceedings of the9th IEEE International Conference on Computer Vision.Washington,DC:IEEE Computer Society Press,2003:1403-1410.
[9]WHYTE D H,BAILEY T.Simultaneous localisation and mapping(slam):Part I the essential algorithms[J].IEEE Robotics and Automation Magazine,2006,13(2):99-110.
[10]KLEIN G,MURRAY D W.Parallel tracking and mapping for small AR workspaces[C]//2007 6th IEEEand ACM International Symposium on Mixed and Augmented Reality.New York:IEEE Press,2007:225-234.
[11]HENRY P,KRAININ M,HERBST E,et al.RGB-Dmapping:Using depth cameras for dense 3D modeling of indoor environments[C]//The 12th International Symposium Experimental Robotics.Heidelberg:Springer,2010:477-491.
[12]HORNUNG A,WURM K M,BENNEWITZ M,et al.OctoMap:An efficient probabilistic 3D mapping framework based on octrees[J].Autonomous Robots,2013,34(3):189-206.
[13]IZADI S,KIM D,HILLIGES O,et al.KinectFusion:Real-time 3D reconstruction and interaction using a moving depth camera[C]//Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology.New York:ACM Press,2011:559-568.
[14]NEWCOMBE R,IZADI S,HILLIGES O,et al.KinectFusion:Real-time dense surface mapping and tracking[C]//2011 10th IEEE International.Symposium on Mixed and Augmented Reality.New York:IEEEPress,2011:127-136.
[15]STUCKLER J,BEHNKE S.Robust real-time registration of RGB-D images using multi-resolution surfel representations[C]//Proceedings of the 7th German Conference on Robotics(ROBOTIK 2012).Munich:VDE Press,2012:1-4.
[16]GALLUP D,FRAHM J M,MORDOHAI P,et al.Real-time plane-sweeping stereo with multiple sweeping directions[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2007:1-8.
[17]NEWCOMBE R A,DAVISON A J.Live dense reconstruction with a single moving camera[C]//2010IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2010:1498-1505.
[18]GRABER G,POCK T,BISCHOF H.Online 3Dreconstruction using convex optimization[C]//2011IEEE International Conference on Computer Vision Workshops.New York:IEEE Press,2011:708-711.
[19]PRADEEP V,RHEMANN C,IZADI S,et al.MonoFusion:Real-time 3D reconstruction of small scenes with a single web camera[C]//2013 12th IEEEInternational Symposium on Mixed and Augmented Reality.New York:IEEE Press,2013:83-88.
[20]BLEYER M,RHEMANN C,ROTHER C.PatchMatch stereo-stereo matching with slanted support windows[EB/OL].[2018-05-05].http://www.brnva.org/bmvc/2011/proceedings/paper/4/index.html.
[21]ZACH C.Fast and high quality fusion of depth maps[EB/OL].[2018-05-05].http://inf.ethz.ch/personal/pomarc/pubs/paper196.pdf.
[22]ZHANG Z Y.A flexible new technique for camera calibration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(11):1330-1334.
[23]FISCHLER M A,BOLLES R C.Random sample consensus:A paradigm for model fitting with applications to image analysis and automated cartography[J].Communications of the ACM,1981,24(6):381-395.
[24]COLLINS R T.A space-sweep approach to true multi-image matching[C]//Proceedings of 1996 IEEEComputer Society Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,1996:358-363.
[25]HARTLEY R,ZISSERMAN A.Multiple view geometry in computer vision[M].New York:Cambridge University Press,2003:326-329.
[26]ZABIN R,WOODFILL J.Non-parametric local transforms for computing visual correspondence[C]//The 3rd European Conference on Computer Vision.Heidelberg:Springer,1994:151-158.
[27]NEWCOMBE R A,LOVEGROVE S J,DAVISON A J.DTAM:Dense tracking and mapping in real-time[C]//Proceedings of the 2011 International Conference on Computer Vision.Washington,DC:IEEE Computer Society Press,2011:2320-2327.
[28]CHAMBOLLE A,POCK T.A first-order primal dual algorithm for convex problems with applications to imaging[J].Journal of Mathe-matical Imaging and Vision,2011,40(1):120-145.
[29]Middlebury,Microsoft Research and NSF.The middlebury computer vision pages[EB/OL].[2018-09-10].http://vision.middlebury.edu/stereo/.
[30]HOPPE H,DEROSE T,DUCHAMP T,et al.Surface reconstruction from unorganized points[J].ACMSIGGRAPH Computer Graphics,1992,26(2):71-78.
[31]BOISSONNAT J D,CAZALS F.Smooth surface reconstruction via natural neighbour interpolation of distance functions[J].Computational Geometry,2002(22):185-203.
[32]CARR J C,BEATSON R K,CHERRIE J B,et al.Reconstruction and representation of 3D objects with radial basis functions[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM Press,2001:67-76.
[33]ZACH C,POCK T,BISCHOF H.A globally optimal algorithm for robust TV-L1 range image integration[C]//Proceedings of IEEE 11th International Conference on Computer Vision.New York:IEEE Press,2007:1-8.
[2]王海瑶,唐娟,沈振辉.基于深度卷积神经网络的多任务细粒度车型识别[J].图学学报,2018,39(3):485-492.
[3]WU J J,ZHANG C K,ZHANG X M,et al.Learning shape priors for single-view 3D completion and reconstruction[C]//Proceedings of 2018 European Conference on Computer Vision.Heidelberg:Springer,2018:1-17.
[4]JACKSON A S,MANAFAS C,TZIMIROPOULOS G.3D human body reconstruction from a single image via volumetric regression[C]//Proceedings of 2018European Conference on Computer Vision Workshops.Heidelberg:Springer,2018:1-14.
[5]CARVALHO M,BERTRAND L S,PAULINE T P,et al.Deep depth form deforcus:How can defocus blur improve 3D estimation using dense neural network?[C]//Proceedings of 2018 European Conference on Computer Vision,3DRW Workshop.Heidelberg:Springer,2018:1-18.
[6]DELLAERT F,SEITZ S M,THORPE C E,et al.Structure from motion without correspondence[C]//2000 IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2000:557-564.
[7]POLLEFEYS M,NISTER J M,AKBARZADEH A,et al.Detailed real-time urban 3D reconstruction from video[J].International Journal of Computer Vision,2008,78(2):143-167.
[8]DAVISON A J.Real-time simultaneous localisation and mapping with a single camera[C]//Proceedings of the9th IEEE International Conference on Computer Vision.Washington,DC:IEEE Computer Society Press,2003:1403-1410.
[9]WHYTE D H,BAILEY T.Simultaneous localisation and mapping(slam):Part I the essential algorithms[J].IEEE Robotics and Automation Magazine,2006,13(2):99-110.
[10]KLEIN G,MURRAY D W.Parallel tracking and mapping for small AR workspaces[C]//2007 6th IEEEand ACM International Symposium on Mixed and Augmented Reality.New York:IEEE Press,2007:225-234.
[11]HENRY P,KRAININ M,HERBST E,et al.RGB-Dmapping:Using depth cameras for dense 3D modeling of indoor environments[C]//The 12th International Symposium Experimental Robotics.Heidelberg:Springer,2010:477-491.
[12]HORNUNG A,WURM K M,BENNEWITZ M,et al.OctoMap:An efficient probabilistic 3D mapping framework based on octrees[J].Autonomous Robots,2013,34(3):189-206.
[13]IZADI S,KIM D,HILLIGES O,et al.KinectFusion:Real-time 3D reconstruction and interaction using a moving depth camera[C]//Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology.New York:ACM Press,2011:559-568.
[14]NEWCOMBE R,IZADI S,HILLIGES O,et al.KinectFusion:Real-time dense surface mapping and tracking[C]//2011 10th IEEE International.Symposium on Mixed and Augmented Reality.New York:IEEEPress,2011:127-136.
[15]STUCKLER J,BEHNKE S.Robust real-time registration of RGB-D images using multi-resolution surfel representations[C]//Proceedings of the 7th German Conference on Robotics(ROBOTIK 2012).Munich:VDE Press,2012:1-4.
[16]GALLUP D,FRAHM J M,MORDOHAI P,et al.Real-time plane-sweeping stereo with multiple sweeping directions[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2007:1-8.
[17]NEWCOMBE R A,DAVISON A J.Live dense reconstruction with a single moving camera[C]//2010IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2010:1498-1505.
[18]GRABER G,POCK T,BISCHOF H.Online 3Dreconstruction using convex optimization[C]//2011IEEE International Conference on Computer Vision Workshops.New York:IEEE Press,2011:708-711.
[19]PRADEEP V,RHEMANN C,IZADI S,et al.MonoFusion:Real-time 3D reconstruction of small scenes with a single web camera[C]//2013 12th IEEEInternational Symposium on Mixed and Augmented Reality.New York:IEEE Press,2013:83-88.
[20]BLEYER M,RHEMANN C,ROTHER C.PatchMatch stereo-stereo matching with slanted support windows[EB/OL].[2018-05-05].http://www.brnva.org/bmvc/2011/proceedings/paper/4/index.html.
[21]ZACH C.Fast and high quality fusion of depth maps[EB/OL].[2018-05-05].http://inf.ethz.ch/personal/pomarc/pubs/paper196.pdf.
[22]ZHANG Z Y.A flexible new technique for camera calibration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(11):1330-1334.
[23]FISCHLER M A,BOLLES R C.Random sample consensus:A paradigm for model fitting with applications to image analysis and automated cartography[J].Communications of the ACM,1981,24(6):381-395.
[24]COLLINS R T.A space-sweep approach to true multi-image matching[C]//Proceedings of 1996 IEEEComputer Society Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,1996:358-363.
[25]HARTLEY R,ZISSERMAN A.Multiple view geometry in computer vision[M].New York:Cambridge University Press,2003:326-329.
[26]ZABIN R,WOODFILL J.Non-parametric local transforms for computing visual correspondence[C]//The 3rd European Conference on Computer Vision.Heidelberg:Springer,1994:151-158.
[27]NEWCOMBE R A,LOVEGROVE S J,DAVISON A J.DTAM:Dense tracking and mapping in real-time[C]//Proceedings of the 2011 International Conference on Computer Vision.Washington,DC:IEEE Computer Society Press,2011:2320-2327.
[28]CHAMBOLLE A,POCK T.A first-order primal dual algorithm for convex problems with applications to imaging[J].Journal of Mathe-matical Imaging and Vision,2011,40(1):120-145.
[29]Middlebury,Microsoft Research and NSF.The middlebury computer vision pages[EB/OL].[2018-09-10].http://vision.middlebury.edu/stereo/.
[30]HOPPE H,DEROSE T,DUCHAMP T,et al.Surface reconstruction from unorganized points[J].ACMSIGGRAPH Computer Graphics,1992,26(2):71-78.
[31]BOISSONNAT J D,CAZALS F.Smooth surface reconstruction via natural neighbour interpolation of distance functions[J].Computational Geometry,2002(22):185-203.
[32]CARR J C,BEATSON R K,CHERRIE J B,et al.Reconstruction and representation of 3D objects with radial basis functions[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM Press,2001:67-76.
[33]ZACH C,POCK T,BISCHOF H.A globally optimal algorithm for robust TV-L1 range image integration[C]//Proceedings of IEEE 11th International Conference on Computer Vision.New York:IEEE Press,2007:1-8.