非刚性稠密匹配大位移运动光流估计
|
摘要
光流场是目标检测,无人机定位等众多计算机视觉任务的重要基础.本文针对非刚性大位移运动等困难运动类型图像序列光流计算的准确性与鲁棒性问题,提出一种基于非刚性稠密匹配的TV-L1(Total Variational with L1 norm,TV-L1)大位移光流计算方法.首先,使用非刚性稠密块匹配计算图像序列初始最近邻域场,其次根据图像相邻块区域的相似性消除初始最近邻域场中的非一致性区域以得到准确的图像最近邻域场.然后,在图像金字塔分层计算框架下,将图像最近邻域场引入基于非局部约束的TV-L1光流估计模型,通过Quadratic Pseudo-Boolean Optimization (QPBO)融合算法在金字塔分层图像光流计算时对TV-L1模型光流估计进行大位移运动补偿.最后,采用标准测试图像序列对本文方法和当前代表性的变分方法LDOF(Large Displacement Optical Flow,LDOF)、Classic+NL、NNF(Nearest Neighbor Fields,NNF)以及深度学习方法FlowNet2.0进行对比分析.实验结果表明,本文方法能有效提高非刚性运动、大位移运动以及运动遮挡等困难运动类型光流估计的精度与鲁棒性.
Optical flow field is an important basis for many computer vision tasks such as target detection and unmanned aerial vehicle positioning.In order to develop the accuracy and robustness of optical flow estimation suffered from the difficult motion such as non-rigid movement and large displacement motion,this paper proposes a large displacement optical flow estimation approach based on non-rigid dense patch matching.Firstly,we utilize the non-rigid dense patch matching to compute the initial nearest neighbor field between the consecutive frames,and eliminate the inconsistent regions of the computed nearest neighbor field according to the consistency of the neighboring patches in the image to obtain an accurate image nearest neighbor field.Secondly,we merge the nearest neighbor field into the TV-L1(Total Variational with L1 norm,TV-L1) optical flow model,and employ the nearest neighbor field to compensate the large displacement optical flow of TV-L1 model by using the quadratic pseudo-Boolean optimization(QPBO) fusion algorithm during the coarse-to-fine computation scheme.Finally,we employ the standard test image sequences to evaluate the performance of our approach and some state-of-the-art methods including LDOF(Large Displacement Optical Flow,LDOF),Classic+NL,NNF(Nearest Neighbor Fields,NNF) and FlowNet2.0.The experimental results demonstrate that the proposed method has the higher accuracy and better robustness of optical flow estimation for difficult motion such as non-rigid movement,large displacement motion and motion occlusion.
Optical flow field is an important basis for many computer vision tasks such as target detection and unmanned aerial vehicle positioning.In order to develop the accuracy and robustness of optical flow estimation suffered from the difficult motion such as non-rigid movement and large displacement motion,this paper proposes a large displacement optical flow estimation approach based on non-rigid dense patch matching.Firstly,we utilize the non-rigid dense patch matching to compute the initial nearest neighbor field between the consecutive frames,and eliminate the inconsistent regions of the computed nearest neighbor field according to the consistency of the neighboring patches in the image to obtain an accurate image nearest neighbor field.Secondly,we merge the nearest neighbor field into the TV-L1(Total Variational with L1 norm,TV-L1) optical flow model,and employ the nearest neighbor field to compensate the large displacement optical flow of TV-L1 model by using the quadratic pseudo-Boolean optimization(QPBO) fusion algorithm during the coarse-to-fine computation scheme.Finally,we employ the standard test image sequences to evaluate the performance of our approach and some state-of-the-art methods including LDOF(Large Displacement Optical Flow,LDOF),Classic+NL,NNF(Nearest Neighbor Fields,NNF) and FlowNet2.0.The experimental results demonstrate that the proposed method has the higher accuracy and better robustness of optical flow estimation for difficult motion such as non-rigid movement,large displacement motion and motion occlusion.
引文
[1] Horn B K,Schunck B G.Determining optical flow[J].ArtificialIntelligence,1981,17(1):185-203.
[2] Wannenwetsch A S,Keuper M,Roth S.Probflow:Joint optical flow and uncertainty estimation[A].IEEE International Conference on Computer Vision[C].Providence:IEEE,2017.1182-1191.
[3] 陈震,张聪炫,晏文敬,等.基于图像局部结构的区域匹配变分光流算法[J].电子学报,2015,43(11):2200-2209.CHEN Zhen,ZHANG Cong-xuan,YAN Wen-jing,et al.Region matching variational optical flow algorithm based on image local structure[J].Acta Electronica Sinica,2015,43(11):2200-2209.(in Chinese).
[4] Yang J,Li H.Dense,accurate optical flow estimation with piecewise parametricmodel[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.1019-1027.
[5] Ranjan A,Black M J.Optical flow estimation using a spatial pyramid network[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.4161-4170.
[6] Brox T,Malik J.Large displacement optical flow:descriptor matching in variational motion estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(3):500-513.
[7] Sun D,Roth S,Black M J.Aquantitative analysis of current practices in optical flow estimation and the principles behind them[J].International Journal of Computer Vision,2014,106(2):115-137.
[8] Chen Z,Jin H,Lin Z,et al.Large displacement optical flow from nearest neighbor fields[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Portland:IEEE,2013.2443-2450.
[9] Zhang C,Chen Z,Wang M,et al.Robustnon-local TV-L1 optical flow estimation with occlusion detection[J].IEEE Transactions on Image Processing,2017,26(8):4055-4067.
[10] Ilg E,Mayer N,Saikia T,et al.FlowNet 2.0:evolution of optical flow estimation with deep networks[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.1647-1655.
[11] Boykov Y,Veksler O,Zabih R.Fast approximate energy minimization via graph cuts[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(11):1222-1239.
[12] Rother C,Kolmogorov V,Lempitsky V,et al.Optimizing binary MRFs via extended roof duality[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Minneapolis:IEEE,2007.1-8.
[13] Butler D J,Wulff J,Stanley G B,et al.A naturalistic open source movie for optical flow evaluation[A].European Conference on Computer Vision[C].Heidelberg:Springer,2012.611-625.
[14] Menze M,Geiger A.Object scene flow for autonomousvehicles[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.3061-3070.
[2] Wannenwetsch A S,Keuper M,Roth S.Probflow:Joint optical flow and uncertainty estimation[A].IEEE International Conference on Computer Vision[C].Providence:IEEE,2017.1182-1191.
[3] 陈震,张聪炫,晏文敬,等.基于图像局部结构的区域匹配变分光流算法[J].电子学报,2015,43(11):2200-2209.CHEN Zhen,ZHANG Cong-xuan,YAN Wen-jing,et al.Region matching variational optical flow algorithm based on image local structure[J].Acta Electronica Sinica,2015,43(11):2200-2209.(in Chinese).
[4] Yang J,Li H.Dense,accurate optical flow estimation with piecewise parametricmodel[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.1019-1027.
[5] Ranjan A,Black M J.Optical flow estimation using a spatial pyramid network[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.4161-4170.
[6] Brox T,Malik J.Large displacement optical flow:descriptor matching in variational motion estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(3):500-513.
[7] Sun D,Roth S,Black M J.Aquantitative analysis of current practices in optical flow estimation and the principles behind them[J].International Journal of Computer Vision,2014,106(2):115-137.
[8] Chen Z,Jin H,Lin Z,et al.Large displacement optical flow from nearest neighbor fields[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Portland:IEEE,2013.2443-2450.
[9] Zhang C,Chen Z,Wang M,et al.Robustnon-local TV-L1 optical flow estimation with occlusion detection[J].IEEE Transactions on Image Processing,2017,26(8):4055-4067.
[10] Ilg E,Mayer N,Saikia T,et al.FlowNet 2.0:evolution of optical flow estimation with deep networks[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.1647-1655.
[11] Boykov Y,Veksler O,Zabih R.Fast approximate energy minimization via graph cuts[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(11):1222-1239.
[12] Rother C,Kolmogorov V,Lempitsky V,et al.Optimizing binary MRFs via extended roof duality[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Minneapolis:IEEE,2007.1-8.
[13] Butler D J,Wulff J,Stanley G B,et al.A naturalistic open source movie for optical flow evaluation[A].European Conference on Computer Vision[C].Heidelberg:Springer,2012.611-625.
[14] Menze M,Geiger A.Object scene flow for autonomousvehicles[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.3061-3070.