梯度约束SLIC的快速视频目标提取方法
|
摘要
提出了一种基于梯度约束SLIC(simple linear iterative clustering)的快速视频目标提取方法,允许在关键视频帧上提供少量用户交互下,该方法能够快速并精确地提取复杂视频片段中的视频目标。首先,采用梯度约束的SLIC算法对视频片段进行预处理,有效降低待处理的视频数据量;其次,以预处理生成的超像素为结点构建三维无向图,在此基础上定义能量函数,并结合外观特征与运动特征建立鲁棒的相似外观度量机制;最后,采用最大流/最小割算法最小化能量函数以得到三维无向图的最优划分,从而最终实现视频目标提取。实验结果表明,该方法在处理包含复杂场景的视频片段时能够获得理想的视频目标提取结果,且时间效率相比现有视频目标提取方法明显提高。
This paper proposes a fast interactive video object extraction method based on gradient-constrained SLIC(simple linear iterative clustering) algorithm, which can quickly and accurately extract the video object in a complex video with less user interaction. Firstly, each video frame can be segmented into individual superpixels by using the gradient-constrained SLIC algorithm, which effectively reduces the complexity of the subsequent processing.Secondly, a three-dimensional undirected graph is constructed with the generated superpixels, and then a novel graph-cut based energy function is redefined with the robust measurement distance on appearance information and motion features. Finally, video object can be extracted when minimizing the energy function by using the maximum flow/min cut algorithm. The experiments show that this method can achieve high quality video object extraction even dealing with high-definition video fragments with complex scenes, and the time efficiency of video object extraction is obviously improved compared with the existing video object extraction methods.
This paper proposes a fast interactive video object extraction method based on gradient-constrained SLIC(simple linear iterative clustering) algorithm, which can quickly and accurately extract the video object in a complex video with less user interaction. Firstly, each video frame can be segmented into individual superpixels by using the gradient-constrained SLIC algorithm, which effectively reduces the complexity of the subsequent processing.Secondly, a three-dimensional undirected graph is constructed with the generated superpixels, and then a novel graph-cut based energy function is redefined with the robust measurement distance on appearance information and motion features. Finally, video object can be extracted when minimizing the energy function by using the maximum flow/min cut algorithm. The experiments show that this method can achieve high quality video object extraction even dealing with high-definition video fragments with complex scenes, and the time efficiency of video object extraction is obviously improved compared with the existing video object extraction methods.
引文
[1]Bai X,Sapiro G.A geodesic framework for fast interactive image and video segmentation and matting[J].International Journal of Computer Vision,2009,82(2):113-132.
[2]Bai X,Wang J,Simons D,et al.Video snapcut:robust video object cutout using localized classifiers[J].ACM Transactions on Graphics,2009,28(3):70.
[3]Zhong F,Qin X Y,Peng Q S,et al.Discontinuity-aware video object cutout[J].ACM Transactions on Graphics,2012,31(6):175.
[4]Fan Q N,Zhong F,Lischinski D,et al.Jump Cut:non-successive mask transfer and interpolation for video cutout[J].ACMTransactions on Graphics,2015,34(6):195.
[5]Li Y,Sun J,Shum H Y.Video object cut and paste[J].ACMTransactions on Graphics,2005,24(3):595-600.
[6]Wang J,Bhat P,Colburn R A,et al.Interactive video cutout[J].ACM Transactions on Graphics,2005,24(3):585-594.
[7]Huang Y C,Liu Q S,Metaxas D.Video object segmentation by hypergraph cut[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Miami,Jun 20-25,2005.Washington:IEEE Computer Society,2009:1738-1745.
[8]Tong R F,Zhang Y,Ding M.Video brush:a novel interface for efficient video cutout[J].Computer Graphics Forum,2011,30(7):2049-2057.
[9]Zhang H C,Zhang L,Huang H.Video cutout method based on spatio-temporal similarity neighboring graph[J].Journal of Graphics,2012,33(2):83-88.
[10]Wang T H,Han B,Collomosse J.Touchcut:fast image and video segmentation using single-touch interaction[J].Computer Vision and Image Understanding,2014,120:14-30.
[11]Vincent L,Soille P.Watersheds in digital spaces:an efficient algorithm based on immersion simulations[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1991,13(6):583-598.
[12]Comaniciu D,Meer P.Mean shift:a robust approach toward feature space analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2002,24(5):603-619.
[13]Ren X F,Malik J.Learning a classification model for segmentation[C]//Proceedings of the 9th IEEE International Conference on Computer Vision,Nice,Oct 13-16,2003.Washington:IEEE Computer Society,2003:10-17.
[14]Sundaram N,Brox T,Keutzer K.Dense point trajectories by GPU-accelerated large displacement optical flow[C]//LNCS 6311:Proceeding of the 11th European Conference on Computer Vision,Heraklion,Sep 5-11,2010.Berlin,Heidelberg:Springer,2010:438-451.
[15]Cour T,Bénézit F,Shi J B.Spectral segmentation with multiscale graph decomposition[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition,San Diego,Jun 20-25,2005.Washington:IEEE Computer Society,2005,2:1124-1131.
[16]Grundmann M,Kwatra V,Han M,et al.Efficient hierarchical graph-based video segmentation[C]//Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition,San Francisco,Jun 13-18,2010.Washington:IEEE Computer Society,2010:2141-2148.
[17]Papazoglou A,Ferrari V.Fast object segmentation in unconstrained video[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney,Dec 1-8,2013.Washington:IEEE Computer Society,2013:1777-1784.
[18]Levinshtein A,Stere A,Kutulakos K N,et al.Turbopixels:fast superpixels using geometric flows[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(12):2290-2297.
[19]Jain S D,Grauman K.Supervoxel-consistent foreground propagation in video[C]//LNCS 8692:Proceeding of the13th European Conference on Computer Vision,Zurich,Sep 5-12,2014.Berlin,Heidelberg:Springer,2014:656-671.
[20]Giordano D,Murabito F,Palazzo S,et al.Superpixel-based video object segmentation using perceptual organization and location prior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,Jun 7-12,2015.Washington:IEEE Computer Society,2015:4814-4822.
[21]Achanta R,Shaji A,Smith K,et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2012,34(11):2274-2282.
[22]Tsai Y H,Yang M H,Black M J.Video segmentation via object flow[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,Jun27-30,2016.Washington:IEEE Computer Society,2016:3899-3908.
[23]Kolmogorov V,Zabin R.What energy functions can be minimized via graph cuts?[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(2):147-159.
[24]Sochen N,Kimmel R,Malladi R.A general framework for low level vision[J].IEEE Transactions on Image Processing,1998,7(3):310-318.
[25]Rother C,Kolmogorov V,Blake A.Grabcut:interactive foreground extraction using iterated graph cuts[J].ACM Transactions on Graphics,2004,23(3):309-314.
[26]Manjunath B S,Ma W Y.Texture features for browsing and retrieval of image data[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1996,18(8):837-842.
[27]Liu M Y,Tuzel O,Ramalingam S,et al.Entropy rate superpixel segmentation[C]//Proceeding of the 24th IEEEConference on Computer Vision and Pattern Recognition,Colorado Springs,Jun 20-25,2011.Washington:IEEEComputer Society,2011:2097-2104.
[28]Tang D,Fu H Z,Cao X C.Topology preserved regular superpixel[C]//Proceedings of the IEEE International Conference on Multimedia and Expo,Melbourne,Jul 9-13,2012.Washington:IEEE Computer Society,2012:765-768.
[29]Li Z Q,Chen J S.Superpixel segmentation using linear spectral clustering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,Jun 7-12,2015.Washington:IEEE Computer Society,2015:1356-1363.
[2]Bai X,Wang J,Simons D,et al.Video snapcut:robust video object cutout using localized classifiers[J].ACM Transactions on Graphics,2009,28(3):70.
[3]Zhong F,Qin X Y,Peng Q S,et al.Discontinuity-aware video object cutout[J].ACM Transactions on Graphics,2012,31(6):175.
[4]Fan Q N,Zhong F,Lischinski D,et al.Jump Cut:non-successive mask transfer and interpolation for video cutout[J].ACMTransactions on Graphics,2015,34(6):195.
[5]Li Y,Sun J,Shum H Y.Video object cut and paste[J].ACMTransactions on Graphics,2005,24(3):595-600.
[6]Wang J,Bhat P,Colburn R A,et al.Interactive video cutout[J].ACM Transactions on Graphics,2005,24(3):585-594.
[7]Huang Y C,Liu Q S,Metaxas D.Video object segmentation by hypergraph cut[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Miami,Jun 20-25,2005.Washington:IEEE Computer Society,2009:1738-1745.
[8]Tong R F,Zhang Y,Ding M.Video brush:a novel interface for efficient video cutout[J].Computer Graphics Forum,2011,30(7):2049-2057.
[9]Zhang H C,Zhang L,Huang H.Video cutout method based on spatio-temporal similarity neighboring graph[J].Journal of Graphics,2012,33(2):83-88.
[10]Wang T H,Han B,Collomosse J.Touchcut:fast image and video segmentation using single-touch interaction[J].Computer Vision and Image Understanding,2014,120:14-30.
[11]Vincent L,Soille P.Watersheds in digital spaces:an efficient algorithm based on immersion simulations[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1991,13(6):583-598.
[12]Comaniciu D,Meer P.Mean shift:a robust approach toward feature space analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2002,24(5):603-619.
[13]Ren X F,Malik J.Learning a classification model for segmentation[C]//Proceedings of the 9th IEEE International Conference on Computer Vision,Nice,Oct 13-16,2003.Washington:IEEE Computer Society,2003:10-17.
[14]Sundaram N,Brox T,Keutzer K.Dense point trajectories by GPU-accelerated large displacement optical flow[C]//LNCS 6311:Proceeding of the 11th European Conference on Computer Vision,Heraklion,Sep 5-11,2010.Berlin,Heidelberg:Springer,2010:438-451.
[15]Cour T,Bénézit F,Shi J B.Spectral segmentation with multiscale graph decomposition[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition,San Diego,Jun 20-25,2005.Washington:IEEE Computer Society,2005,2:1124-1131.
[16]Grundmann M,Kwatra V,Han M,et al.Efficient hierarchical graph-based video segmentation[C]//Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition,San Francisco,Jun 13-18,2010.Washington:IEEE Computer Society,2010:2141-2148.
[17]Papazoglou A,Ferrari V.Fast object segmentation in unconstrained video[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney,Dec 1-8,2013.Washington:IEEE Computer Society,2013:1777-1784.
[18]Levinshtein A,Stere A,Kutulakos K N,et al.Turbopixels:fast superpixels using geometric flows[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(12):2290-2297.
[19]Jain S D,Grauman K.Supervoxel-consistent foreground propagation in video[C]//LNCS 8692:Proceeding of the13th European Conference on Computer Vision,Zurich,Sep 5-12,2014.Berlin,Heidelberg:Springer,2014:656-671.
[20]Giordano D,Murabito F,Palazzo S,et al.Superpixel-based video object segmentation using perceptual organization and location prior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,Jun 7-12,2015.Washington:IEEE Computer Society,2015:4814-4822.
[21]Achanta R,Shaji A,Smith K,et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2012,34(11):2274-2282.
[22]Tsai Y H,Yang M H,Black M J.Video segmentation via object flow[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,Jun27-30,2016.Washington:IEEE Computer Society,2016:3899-3908.
[23]Kolmogorov V,Zabin R.What energy functions can be minimized via graph cuts?[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(2):147-159.
[24]Sochen N,Kimmel R,Malladi R.A general framework for low level vision[J].IEEE Transactions on Image Processing,1998,7(3):310-318.
[25]Rother C,Kolmogorov V,Blake A.Grabcut:interactive foreground extraction using iterated graph cuts[J].ACM Transactions on Graphics,2004,23(3):309-314.
[26]Manjunath B S,Ma W Y.Texture features for browsing and retrieval of image data[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1996,18(8):837-842.
[27]Liu M Y,Tuzel O,Ramalingam S,et al.Entropy rate superpixel segmentation[C]//Proceeding of the 24th IEEEConference on Computer Vision and Pattern Recognition,Colorado Springs,Jun 20-25,2011.Washington:IEEEComputer Society,2011:2097-2104.
[28]Tang D,Fu H Z,Cao X C.Topology preserved regular superpixel[C]//Proceedings of the IEEE International Conference on Multimedia and Expo,Melbourne,Jul 9-13,2012.Washington:IEEE Computer Society,2012:765-768.
[29]Li Z Q,Chen J S.Superpixel segmentation using linear spectral clustering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,Jun 7-12,2015.Washington:IEEE Computer Society,2015:1356-1363.