基于共同视域的自监督立体匹配算法
|
摘要
提出了一种基于共同视域的自监督立体匹配算法,该算法根据视差的左右一致性来确定双目图像的共同可视区域,从而抑制被遮挡区域产生的噪声,为网络模型的学习提供了更加准确的反馈信号。研究结果表明:在没有任何标签数据的前提下,所提算法的预测误差降低了11%～42%,且与有监督立体匹配算法的性能相当。
A self-supervised stereo matching algorithm is proposed based on common view. In this algorithm, the common visible region of the binocular images is determined according to the left-right consistency of disparity and thus the noise generated in the occluded region is suppressed, which provides more accurate feedback signals for the network model learning. The research results show that the prediction error of the proposed algorithm can be reduced by 11%-42% without any label data, and the performance of the proposed algorithm is comparable to that of the supervised stereo matching algorithm.
A self-supervised stereo matching algorithm is proposed based on common view. In this algorithm, the common visible region of the binocular images is determined according to the left-right consistency of disparity and thus the noise generated in the occluded region is suppressed, which provides more accurate feedback signals for the network model learning. The research results show that the prediction error of the proposed algorithm can be reduced by 11%-42% without any label data, and the performance of the proposed algorithm is comparable to that of the supervised stereo matching algorithm.
引文
[1] Liu C, Yuen J, Torralba A. SIFT flow: dense correspondence across scenes and its applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 978-994.
[2] Liu Y F, Cai Z J. Binocular stereo vision three-dimensional reconstruction algorithm based on ICP and SFM[J]. Laser and Optoelectronics Progress, 2018, 55(9): 091503. 刘一凡, 蔡振江. 基于ICP与SFM的双目立体视觉三维重构算法[J]. 激光与光电子学进展, 2018, 55(9): 091503.
[3] Sivaraman S, Trivedi M M. A review of recent developments in vision-based vehicle detection[C]. IEEE Intelligent Vehicles Symposium(IV), 2013: 310-315.
[4] Schmid K, Tomic T, Ruess F, et al. Stereo vision based indoor/outdoor navigation for flying robots[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013: 3955-3962.
[5] Scharstein D, Szeliski R, Zabih R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47(1): 7-42.
[6] Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[7] Dalal N, Triggs B. Histograms of oriented gradients for human detection[C]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2005: 886-893.
[8] Hirschmuller H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341.
[9] Liu D W, Han L, Han X Y. High spatial resolution remote sensing image classification based on deep learning[J]. Acta Optica Sinica, 2016, 36(4): 0428001. 刘大伟, 韩玲, 韩晓勇. 基于深度学习的高分辨率遥感影像分类研究[J]. 光学学报, 2016, 36(4): 0428001.
[10] Hou Y Q Y, Quan J C, Wei Y M. Valid aircraft detection system for remote sensing images based on cognitive models[J]. Acta Optica Sinica, 2018, 38(1): 0111005. 侯宇青阳, 全吉成, 魏湧明. 基于认知模型的遥感图像有效飞机检测系统[J]. 光学学报, 2018, 38(1): 0111005.
[11] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
[12] ?bontar J, LeCun Y. Computing the stereo matching cost with a convolutional neural network[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 1592-1599.
[13] ?bontar J, LeCun Y. Stereo matching by training a convolutional neural network to compare image patches[J]. Journal of Machine Learning Research, 2016, 17(1): 2287-2318.
[14] Xiao J S, Tian H, Zou W T, et al. Stereo matching based on convolutional neural network[J]. Acta Optica Sinica, 2018, 38(8): 0815017. 肖进胜, 田红, 邹文涛, 等. 基于深度卷积神经网络的双目立体视觉匹配算法[J]. 光学学报, 2018, 38(8): 0815017.
[15] Chen Z Y, Sun X, Wang L, et al. A deep visual correspondence embedding model for stereo matching costs[C]. IEEE International Conference on Computer Vision, 2015: 972-980.
[16] Park H, Lee K M. Look wider to match image patches with convolutional neural networks[J]. IEEE Signal Processing Letters, 2017, 24(12): 1788-1792.
[17] Güney F, Geiger A. Displets: resolving stereo ambiguities using object knowledge[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 4165-4175.
[18] Seki A, Pollefeys M. Patch based confidence prediction for dense disparity map[C]. British Machine Vision Conference (BMVC), 2016: 23.
[19] Mayer N, Ilg E, H?usser P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4040-4048.
[20] Pang J H, Sun W X, Ren J S, et al. Cascade residual learning: a two-stage convolutional neural network for stereo matching[C]. IEEE International Conference on Computer Vision Workshops, 2017: 887-895.
[21] Liang Z F, Feng Y L, Guo Y L, et al. Learning deep correspondence through prior and posterior feature constancy[EB/OL]. (2017-12-04)[2018-06-05]. http://cn.arxiv.org/abs/1712.01039.
[22] Kendall A, Martirosyan H, Dasgupta S, et al. End-to-end learning of geometry and context for deep stereo regression[C]. IEEE International Conference on Computer Vision, 2017: 66-75.
[23] Yu L D, Wang Y C, Wu Y W, et al. Deep stereo matching with explicit cost aggregation sub-architecture[EB/OL]. (2018-01-12)[2018-06-05]. http://cn.arxiv.org/abs/1801.04065.
[24] Chang J R, Chen Y S. Pyramid stereo matching network[EB/OL]. (2018-03-23)[2018-06-05]. http://cn.arxiv.org/abs/1803.08669.
[25] Xie J Y, Girshick R, Farhadi A. Deep 3D: fully automatic 2D-to-3D video conversion with deep convolutional neural networks[C]. European Conference on Computer Vision(ECCV), 2016: 842-857.
[26] GargR, Vijay K B G, Carneiro G, et al. Unsupervised CNN for single view depth estimation: Geometry to the rescue[C]. European Conference on Computer Vision (ECCV), 2016: 740-756.
[27] Godard C, Aodha O M, Brostow G J. Unsupervised monocular depth estimation with left-right consistency[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2017: 6602-6611.
[28] Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks[EB/OL]. (2015-06-05)[2018-06-05]. http://cn.arxiv.org/abs/1506.02025.
[29] Ye M L, Johns E, Handa A, et al. Self-supervised siamese learning on stereo image pairs for depth estimation in robotic surgery[EB/OL]. (2017-05-17)[2018-06-05]. http://cn.arxiv.org/abs/1705.08260.
[30] LiuX, Sinha A, Unberath M, et al. Self-supervised learning for dense depth estimation in monocular endoscopy[C]. European Conference on Computer Vision (ECCV), 2018, 11041: 128-138.
[31] Zhong Y R, Dai Y C, Li H D. Self-supervised learning for stereo matching with self-improving ability[EB/OL]. (2017-09-04)[2018-06-05]. http://cn.arxiv.org/abs/1709.00930.
[32] Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[33] Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2012: 3354-3361.
[34] Menze M, Geiger A. Object scene flow for autonomous vehicles[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2015: 3061-3070.
[35] wyf2017: Pytorch implementation of the several deep stereo matching network [EB/OL]. https://github.com/wyf2017/DSMnet.
[36] Kingma D P, Ba J. Adam: a method for stochastic optimization[EB/OL]. (2014-12-22)[2018-06-05]. http://cn.arxiv.org/abs/1412.6980.
[2] Liu Y F, Cai Z J. Binocular stereo vision three-dimensional reconstruction algorithm based on ICP and SFM[J]. Laser and Optoelectronics Progress, 2018, 55(9): 091503. 刘一凡, 蔡振江. 基于ICP与SFM的双目立体视觉三维重构算法[J]. 激光与光电子学进展, 2018, 55(9): 091503.
[3] Sivaraman S, Trivedi M M. A review of recent developments in vision-based vehicle detection[C]. IEEE Intelligent Vehicles Symposium(IV), 2013: 310-315.
[4] Schmid K, Tomic T, Ruess F, et al. Stereo vision based indoor/outdoor navigation for flying robots[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013: 3955-3962.
[5] Scharstein D, Szeliski R, Zabih R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47(1): 7-42.
[6] Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[7] Dalal N, Triggs B. Histograms of oriented gradients for human detection[C]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2005: 886-893.
[8] Hirschmuller H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341.
[9] Liu D W, Han L, Han X Y. High spatial resolution remote sensing image classification based on deep learning[J]. Acta Optica Sinica, 2016, 36(4): 0428001. 刘大伟, 韩玲, 韩晓勇. 基于深度学习的高分辨率遥感影像分类研究[J]. 光学学报, 2016, 36(4): 0428001.
[10] Hou Y Q Y, Quan J C, Wei Y M. Valid aircraft detection system for remote sensing images based on cognitive models[J]. Acta Optica Sinica, 2018, 38(1): 0111005. 侯宇青阳, 全吉成, 魏湧明. 基于认知模型的遥感图像有效飞机检测系统[J]. 光学学报, 2018, 38(1): 0111005.
[11] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
[12] ?bontar J, LeCun Y. Computing the stereo matching cost with a convolutional neural network[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 1592-1599.
[13] ?bontar J, LeCun Y. Stereo matching by training a convolutional neural network to compare image patches[J]. Journal of Machine Learning Research, 2016, 17(1): 2287-2318.
[14] Xiao J S, Tian H, Zou W T, et al. Stereo matching based on convolutional neural network[J]. Acta Optica Sinica, 2018, 38(8): 0815017. 肖进胜, 田红, 邹文涛, 等. 基于深度卷积神经网络的双目立体视觉匹配算法[J]. 光学学报, 2018, 38(8): 0815017.
[15] Chen Z Y, Sun X, Wang L, et al. A deep visual correspondence embedding model for stereo matching costs[C]. IEEE International Conference on Computer Vision, 2015: 972-980.
[16] Park H, Lee K M. Look wider to match image patches with convolutional neural networks[J]. IEEE Signal Processing Letters, 2017, 24(12): 1788-1792.
[17] Güney F, Geiger A. Displets: resolving stereo ambiguities using object knowledge[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 4165-4175.
[18] Seki A, Pollefeys M. Patch based confidence prediction for dense disparity map[C]. British Machine Vision Conference (BMVC), 2016: 23.
[19] Mayer N, Ilg E, H?usser P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4040-4048.
[20] Pang J H, Sun W X, Ren J S, et al. Cascade residual learning: a two-stage convolutional neural network for stereo matching[C]. IEEE International Conference on Computer Vision Workshops, 2017: 887-895.
[21] Liang Z F, Feng Y L, Guo Y L, et al. Learning deep correspondence through prior and posterior feature constancy[EB/OL]. (2017-12-04)[2018-06-05]. http://cn.arxiv.org/abs/1712.01039.
[22] Kendall A, Martirosyan H, Dasgupta S, et al. End-to-end learning of geometry and context for deep stereo regression[C]. IEEE International Conference on Computer Vision, 2017: 66-75.
[23] Yu L D, Wang Y C, Wu Y W, et al. Deep stereo matching with explicit cost aggregation sub-architecture[EB/OL]. (2018-01-12)[2018-06-05]. http://cn.arxiv.org/abs/1801.04065.
[24] Chang J R, Chen Y S. Pyramid stereo matching network[EB/OL]. (2018-03-23)[2018-06-05]. http://cn.arxiv.org/abs/1803.08669.
[25] Xie J Y, Girshick R, Farhadi A. Deep 3D: fully automatic 2D-to-3D video conversion with deep convolutional neural networks[C]. European Conference on Computer Vision(ECCV), 2016: 842-857.
[26] GargR, Vijay K B G, Carneiro G, et al. Unsupervised CNN for single view depth estimation: Geometry to the rescue[C]. European Conference on Computer Vision (ECCV), 2016: 740-756.
[27] Godard C, Aodha O M, Brostow G J. Unsupervised monocular depth estimation with left-right consistency[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2017: 6602-6611.
[28] Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks[EB/OL]. (2015-06-05)[2018-06-05]. http://cn.arxiv.org/abs/1506.02025.
[29] Ye M L, Johns E, Handa A, et al. Self-supervised siamese learning on stereo image pairs for depth estimation in robotic surgery[EB/OL]. (2017-05-17)[2018-06-05]. http://cn.arxiv.org/abs/1705.08260.
[30] LiuX, Sinha A, Unberath M, et al. Self-supervised learning for dense depth estimation in monocular endoscopy[C]. European Conference on Computer Vision (ECCV), 2018, 11041: 128-138.
[31] Zhong Y R, Dai Y C, Li H D. Self-supervised learning for stereo matching with self-improving ability[EB/OL]. (2017-09-04)[2018-06-05]. http://cn.arxiv.org/abs/1709.00930.
[32] Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[33] Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2012: 3354-3361.
[34] Menze M, Geiger A. Object scene flow for autonomous vehicles[C]. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2015: 3061-3070.
[35] wyf2017: Pytorch implementation of the several deep stereo matching network [EB/OL]. https://github.com/wyf2017/DSMnet.
[36] Kingma D P, Ba J. Adam: a method for stochastic optimization[EB/OL]. (2014-12-22)[2018-06-05]. http://cn.arxiv.org/abs/1412.6980.