基于SIFT算法的大场景视频拼接算法及优化
|
摘要
目前大量的由独立视频设备获取的小场景视频信息难以满足大场景下信息处理的要求,而通过多设备人工查阅的方式又存在效率低下、信息冗余和碎片化等问题。文中研究了大场景视频拼接技术,利用SIFT算法的尺度不变特性对关键点进行特征匹配,通过仿射矩阵变形完成对图像的拼接工作。在此过程中,对传统的SIFT拼接算法进行进一步的优化,主要是基于距离的优化算法来完善视频拼接的效果;对SIFT特征点匹配、加权优化算法、关键帧提取的技术等进行并行加速,以提高拼接效率。实验结果表明,提出的优化方法能更好地提取视频中的关键信息,以实现更好的视频拼接效果。在视觉效果上,所提方法得到的拼接结果中不存在传统方法出现的两幅图像的交接线。此外,在MATLAB环境下分别对关键点检测和拼接部分进行了加速优化,优化后的关键点检测效率提高了约20%,拼接部分的效率提高了将近57%。在C++环境下,关键点的检测效率提升了14%,拼接部分的检测效率提升了40%。
At present,a large number of video information obtained by independent video devices in small scenes cannot meet the requirements of information processing in large scenes,and the manual access by multiple devices is faced with such problems as low efficiency,information redundancy and fragmentation.This paper investigated large scene video stitching technology,carried out the feature matching of key points by using the scale-invariant feature of SIFT algorithm,and completed the image splicing by affine matrix deformation in this process.The traditional SIFT splicing algorithm is further optimized,mainly based on distance optimization algorithm to improve the effect of video splicing.In order to improve the efficiency of stitching,the techniques of key frame extraction based on SIFT feature point matching weighted optimization algorithm were accelerated in parallel.Experiments show that the proposed optimization method can extract essential information in the video more effectively,achieving better video stitching results.Visually,the intersection of the two images appeared in the results of the conventional method does not appear in the stitching results obtained by the proposed method.Moreover,the key point detection and splicing parts were accelerated and optimized respectively in the different environment.In MATLAB,the efficiency of the key point with optimized is improved by 20%,and that of splicing parts is increased by about 57%.In C++,the efficiency of the key point is improved by 14%,and that of splicing parts is increased by about 40%.
At present,a large number of video information obtained by independent video devices in small scenes cannot meet the requirements of information processing in large scenes,and the manual access by multiple devices is faced with such problems as low efficiency,information redundancy and fragmentation.This paper investigated large scene video stitching technology,carried out the feature matching of key points by using the scale-invariant feature of SIFT algorithm,and completed the image splicing by affine matrix deformation in this process.The traditional SIFT splicing algorithm is further optimized,mainly based on distance optimization algorithm to improve the effect of video splicing.In order to improve the efficiency of stitching,the techniques of key frame extraction based on SIFT feature point matching weighted optimization algorithm were accelerated in parallel.Experiments show that the proposed optimization method can extract essential information in the video more effectively,achieving better video stitching results.Visually,the intersection of the two images appeared in the results of the conventional method does not appear in the stitching results obtained by the proposed method.Moreover,the key point detection and splicing parts were accelerated and optimized respectively in the different environment.In MATLAB,the efficiency of the key point with optimized is improved by 20%,and that of splicing parts is increased by about 57%.In C++,the efficiency of the key point is improved by 14%,and that of splicing parts is increased by about 40%.
引文
[1] ZHANG J,CHEN G,JIA Z.An image stitching algorithm based on histogram matching and SIFT algorithm [J].International Journal of Pattern Recognition and Artificial Intelligence,2017,31(4):1754006.
[2] ZHENG G L,TANG B B.The stereo vision positioning system based on multiple fisheye cameras [J].Computer Simulation,2016,33(7):256-260.(in Chinese)郑贵林,唐贝贝.基于多鱼眼摄像头的立体视觉定位系统[J].计算机仿真,2016,33(7):256-260.
[3] KAUR S,ER S K.Analysis of Image Stitching for Noisy Images using SIFT [J].International Journal of Advanced Research in Computer Science,2017,8(5):2078-2082.
[4] WANG Y W,YU M,JIANG H,et al.An image stitching algorithm via adaptive quadtree segmentation[J].Journal of Ningbo University (Natural Science & Engineering),2018,31(4):52-58.(in Chinese)王元炜,郁梅,姜浩,等.一种自适应四叉树分块的图像拼接算法[J].宁波大学学报(理工版),2018,31(4):52-58.
[5] HE C,ZHOU J.Mesh-based image stitching algorithm with li- near structure protection[J].Journal of Image and Graphics,2018,23(7):0973-0983.(in Chinese)何川,周军.具有直线结构保护的网格化图像拼接[J].中国图象图形学报,2018,23(7):973-983.
[6] ZHOU X,CAO S,HE X J,et al.Image stitching based on the planar similarity among matching pairs of feature points [J].Journal of University Electronic Science and Technology of China,2017,46(6):877-882.(in Chinese)周雪,曹爽,何香静,等.基于特征点匹配对平面相似度的图像拼接[J].电子科技大学学报,2017,46(6):877-882.
[7] WANG F B,TU P,WU C,et al.Multi-image mosaic with SIFT and vision measurement for microscale structures processed by femtosecond laser [J].Optics and Lasers in Engineering,2018,100:124-130.
[8] FATHIMA A A,KARTHIK R,VAIDEHI V.Image stitching with combined moment invariants and sift features [J].Procedia Computer Science,2013,19:420-427.
[9] LIU W L,WANG Z Y,QING L B,et al.Application of LBP algorithm in rock slice image stitching[J].Computer & Digital Engineering,2016,44(2):326-330.(in Chinese)刘文亮,王正勇,卿粼波,等.LBP 算法在岩石薄片图像拼接中的应用[J].计算机与数字工程,2016,44(2):326-330.
[10] CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT feature vector image [J].Journal of Jilin University (Science Edition),2017,5(1):116-122.(in Chinese)陈月,赵岩,王世刚.基于 SIFT 特征矢量图的快速图像拼接方法[J].吉林大学学报(理学版),2017,55(1):116-122.
[11] LU J M,ZHU Z.Real-time 4K panoramic video stitching based on GPU acceleration [J].Computer Science,2017,44(8):18-21.(in Chinese)卢嘉铭,朱哲.基于 GPU 加速的实时 4K 全景视频拼接[J].计算机科学,2017,44(8):18-21.
[12] LU Y Y,ZHANG M.Improved Algorithm Based on SIFT Infrared Image Stitching Algorithm [J].Computer Systems & Applications,2015,24(8):165-170.(in Chinese)陆园园,张明.基于 SIFT 算法的红外图像拼接方法改进[J].计算机系统应用,2015,24(8):165-170.
[13] CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT with adaptive local image feature[J].Chinese Optics,2016,9(4):415-422.(in Chinese)陈月,赵岩,王世刚.图像局部特征自适应的快速SIFT图像拼接方法[J].中国光学,2016,9(4):415-422.
[14] ZHAO Y,CHEN Y,WANG S G.Corrected fast SIFT image stitching method by combining projection error [J].Optics and Precision Engineering,2017,25(6):1645-1651.(in Chinese)赵岩,陈月,王世刚.结合投影误差校正的快速 SIFT 图像拼接[J].光学精密工程,2017,25(6):1645-1651.
[15] XU W,MULLIGAN J.Performance evaluation of color correction approaches for automatic multi-view image and video stitching[C]//2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2010:263-270.
[16] DU C,YUAN J,DONG J,et al.GPU based Parallel Optimization for Real Time Panoramic Video Stitching [J].arXiv1810.03988,2018.
[17] JIANG W,GU J.Video stitching with spatial-temporal content-preserving warping[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.IEEE,2015:42-48.
[18] LIU J,LI S Y,LI R F.Multi-view Video Stitching of Outdoor Scenes [J].Computer Engineering,2016,42(4):259-265.(in Chinese)刘娟,李实英,李仁发.室外场景的多视角视频拼接[J].计算机工程,2016,42(4):259-265.
[19] LI Y,DU B X.Real-time video splicing technology based on small region fusion [J].Journal of Jilin University (Science Edition),2016,54(6):1367-1372.(in Chinese)李勇,杜丙新.基于小区域融合的实时视频拼接技术[J].吉林大学学报 (理学版),2016,54(6):1367-1372.
[20] YANG X P,HU Y,ZHANG K.Research on video mosaicking technology based on FPGA [J].Journal of Jilin University (Information Science Edition),2016,34(6):709-715.(in Chinese)杨晓萍,胡玉,张凯.基于 FPGA 的视频拼接技术研究[J].吉林大学学报(信息科学版),2016,34(6):709-715.
[21] CHANG J Y,QIN R,LI Q,et al.Image quality assessment of panoramic image[J].Computer Science,2014,41(6):278-281.(in Chinese)常嘉义,秦瑞,李庆,等.全景鸟瞰拼接图像的质量评价方法[J].计算机科学,2014,41(6):278-281.
[22] NOWOZIN S.Autopano-Sift,making panoramas fun[OL].ht- tp://user.cs.tu-berlin.De/nowozin/autopano-sift.
[23] PERBET F,JOHNSON S,PHAM M T,et al.Human Body Shape Estimation Using a Multi-resolution Manifold Forest[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Columbus,OH,USA,2014:668-675.
[24] WANG J,WANG J D,ZENG G,et al.Fast Neighborhood Graph Search Using Cartesian Concatenation[C]//2013 IEEE International Conference on Computer Vision (ICCV).Sydney,NSW,Australia,2013:2128-2135.
[2] ZHENG G L,TANG B B.The stereo vision positioning system based on multiple fisheye cameras [J].Computer Simulation,2016,33(7):256-260.(in Chinese)郑贵林,唐贝贝.基于多鱼眼摄像头的立体视觉定位系统[J].计算机仿真,2016,33(7):256-260.
[3] KAUR S,ER S K.Analysis of Image Stitching for Noisy Images using SIFT [J].International Journal of Advanced Research in Computer Science,2017,8(5):2078-2082.
[4] WANG Y W,YU M,JIANG H,et al.An image stitching algorithm via adaptive quadtree segmentation[J].Journal of Ningbo University (Natural Science & Engineering),2018,31(4):52-58.(in Chinese)王元炜,郁梅,姜浩,等.一种自适应四叉树分块的图像拼接算法[J].宁波大学学报(理工版),2018,31(4):52-58.
[5] HE C,ZHOU J.Mesh-based image stitching algorithm with li- near structure protection[J].Journal of Image and Graphics,2018,23(7):0973-0983.(in Chinese)何川,周军.具有直线结构保护的网格化图像拼接[J].中国图象图形学报,2018,23(7):973-983.
[6] ZHOU X,CAO S,HE X J,et al.Image stitching based on the planar similarity among matching pairs of feature points [J].Journal of University Electronic Science and Technology of China,2017,46(6):877-882.(in Chinese)周雪,曹爽,何香静,等.基于特征点匹配对平面相似度的图像拼接[J].电子科技大学学报,2017,46(6):877-882.
[7] WANG F B,TU P,WU C,et al.Multi-image mosaic with SIFT and vision measurement for microscale structures processed by femtosecond laser [J].Optics and Lasers in Engineering,2018,100:124-130.
[8] FATHIMA A A,KARTHIK R,VAIDEHI V.Image stitching with combined moment invariants and sift features [J].Procedia Computer Science,2013,19:420-427.
[9] LIU W L,WANG Z Y,QING L B,et al.Application of LBP algorithm in rock slice image stitching[J].Computer & Digital Engineering,2016,44(2):326-330.(in Chinese)刘文亮,王正勇,卿粼波,等.LBP 算法在岩石薄片图像拼接中的应用[J].计算机与数字工程,2016,44(2):326-330.
[10] CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT feature vector image [J].Journal of Jilin University (Science Edition),2017,5(1):116-122.(in Chinese)陈月,赵岩,王世刚.基于 SIFT 特征矢量图的快速图像拼接方法[J].吉林大学学报(理学版),2017,55(1):116-122.
[11] LU J M,ZHU Z.Real-time 4K panoramic video stitching based on GPU acceleration [J].Computer Science,2017,44(8):18-21.(in Chinese)卢嘉铭,朱哲.基于 GPU 加速的实时 4K 全景视频拼接[J].计算机科学,2017,44(8):18-21.
[12] LU Y Y,ZHANG M.Improved Algorithm Based on SIFT Infrared Image Stitching Algorithm [J].Computer Systems & Applications,2015,24(8):165-170.(in Chinese)陆园园,张明.基于 SIFT 算法的红外图像拼接方法改进[J].计算机系统应用,2015,24(8):165-170.
[13] CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT with adaptive local image feature[J].Chinese Optics,2016,9(4):415-422.(in Chinese)陈月,赵岩,王世刚.图像局部特征自适应的快速SIFT图像拼接方法[J].中国光学,2016,9(4):415-422.
[14] ZHAO Y,CHEN Y,WANG S G.Corrected fast SIFT image stitching method by combining projection error [J].Optics and Precision Engineering,2017,25(6):1645-1651.(in Chinese)赵岩,陈月,王世刚.结合投影误差校正的快速 SIFT 图像拼接[J].光学精密工程,2017,25(6):1645-1651.
[15] XU W,MULLIGAN J.Performance evaluation of color correction approaches for automatic multi-view image and video stitching[C]//2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2010:263-270.
[16] DU C,YUAN J,DONG J,et al.GPU based Parallel Optimization for Real Time Panoramic Video Stitching [J].arXiv1810.03988,2018.
[17] JIANG W,GU J.Video stitching with spatial-temporal content-preserving warping[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.IEEE,2015:42-48.
[18] LIU J,LI S Y,LI R F.Multi-view Video Stitching of Outdoor Scenes [J].Computer Engineering,2016,42(4):259-265.(in Chinese)刘娟,李实英,李仁发.室外场景的多视角视频拼接[J].计算机工程,2016,42(4):259-265.
[19] LI Y,DU B X.Real-time video splicing technology based on small region fusion [J].Journal of Jilin University (Science Edition),2016,54(6):1367-1372.(in Chinese)李勇,杜丙新.基于小区域融合的实时视频拼接技术[J].吉林大学学报 (理学版),2016,54(6):1367-1372.
[20] YANG X P,HU Y,ZHANG K.Research on video mosaicking technology based on FPGA [J].Journal of Jilin University (Information Science Edition),2016,34(6):709-715.(in Chinese)杨晓萍,胡玉,张凯.基于 FPGA 的视频拼接技术研究[J].吉林大学学报(信息科学版),2016,34(6):709-715.
[21] CHANG J Y,QIN R,LI Q,et al.Image quality assessment of panoramic image[J].Computer Science,2014,41(6):278-281.(in Chinese)常嘉义,秦瑞,李庆,等.全景鸟瞰拼接图像的质量评价方法[J].计算机科学,2014,41(6):278-281.
[22] NOWOZIN S.Autopano-Sift,making panoramas fun[OL].ht- tp://user.cs.tu-berlin.De/nowozin/autopano-sift.
[23] PERBET F,JOHNSON S,PHAM M T,et al.Human Body Shape Estimation Using a Multi-resolution Manifold Forest[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Columbus,OH,USA,2014:668-675.
[24] WANG J,WANG J D,ZENG G,et al.Fast Neighborhood Graph Search Using Cartesian Concatenation[C]//2013 IEEE International Conference on Computer Vision (ICCV).Sydney,NSW,Australia,2013:2128-2135.