基于视频的成像去抖动方法研究
|
摘要
电子稳像是新一代的稳像技术,它是利用计算机图像处理的方法实现图像序列稳定。因为具有稳像精度高、体积小、重量轻、功耗低以及成本低等优点,它在军事和民用领域得到了广泛的应用。电子稳像的关键技术是全局运动估计和运动补偿。如何提高全局运动估计的速度和精度,如何判断并分离摄像机的场景拍摄和随机抖动,使得运动补偿能够实时准确的稳定图像序列,是目前研究的热点。
视频序列的抖动包含平移、旋转和变焦等复杂运动,针对传统角点提取易过于集中在前景运动物体上,匹配步骤复杂导致速度过慢,且将所有角点直接代入计算全局运动参数,从而导致局部外点和误匹配点影响计算精度的问题,本文提出了基于图像分区角点匹配的电子稳像算法。首先利用Harris角点检测算子分区均匀提取参考帧中的角点,利用模板匹配的思想,以每一角点为中心,建立特征窗口。同时为了保证估计精度和速度,通过匹配验证技术,很好地解决了角点特征匹配过程中出现的误匹配问题,减少无效角点对的不利影响。再通过列文伯格-马夸尔特迭代法,计算得到全局运动参数。最后采用基于卡尔曼滤波的图像补偿方法,确定抖动参数并结合图像拼接技术来实现图像补偿,提高输出视频的稳定性和完整性。
实验结果表明,该算法对角点的预处理和迭代步骤,保证了运动检测的全局性,具有较高的精度,且补偿结果能够实时跟随摄像系统的扫描运动,输出稳定完整的视频序列。
Electronic image stabilization is a new technology to stabilize image sequences with computer image processing. It has a lot of advantages, such as high image stabilizing precision, compact size, lightweight, low power consumption and reasonable price. It has been widely used in military and civil fields. The key technologies of electronic image stabilization are global motion estimation and motion compensation. The research on improving efficiency and accuracy of global motion estimation and distinguishing camera scan from dithering to realize real-time and accurate compensation becomes a hot topic at present.
Dithering of the video sequence includes translation, rotation and zoom of complex movements. Traditional corner extraction method is easy to generate a lot of corners focused on moving objects in the scene. And the speed is low because of complex matching steps. If we use all corners to calculate the global motion parameters directly, it will result in accuracy problem because of the existence of local outliers and error matched points. So this paper presents an electronic image stabilization algorithm based on image partition and corner matching. First of all, the corners are extracted evenly from the reference frame by using Harris corner detection operator and then each feature window is built to find the corresponding corners in the current frame. At the same time in order to ensure accuracy and speed of estimation, by matching validation technique, we can solve the error matched problem arising in the course of matching and reduce the adverse effects of invalid corner pairs. Next, the global motion parameters can be obtained through Levenberg-Marquardt iterative method. Finally, the Kalman filter is applied to smooth the motion vectors to obtain the dithering vector and the image mosaic is used to compensate each current frame in order to improve the stability and integrity of the output video.
The experimental results show that the pre-processing and iterative steps to the corners of the algorithm ensure the overall of the motion detection with high precision. The compensation results can follow scanning movement of the camera system in real time. At last the stable and integral output video sequence can be obtained.
视频序列的抖动包含平移、旋转和变焦等复杂运动,针对传统角点提取易过于集中在前景运动物体上,匹配步骤复杂导致速度过慢,且将所有角点直接代入计算全局运动参数,从而导致局部外点和误匹配点影响计算精度的问题,本文提出了基于图像分区角点匹配的电子稳像算法。首先利用Harris角点检测算子分区均匀提取参考帧中的角点,利用模板匹配的思想,以每一角点为中心,建立特征窗口。同时为了保证估计精度和速度,通过匹配验证技术,很好地解决了角点特征匹配过程中出现的误匹配问题,减少无效角点对的不利影响。再通过列文伯格-马夸尔特迭代法,计算得到全局运动参数。最后采用基于卡尔曼滤波的图像补偿方法,确定抖动参数并结合图像拼接技术来实现图像补偿,提高输出视频的稳定性和完整性。
实验结果表明,该算法对角点的预处理和迭代步骤,保证了运动检测的全局性,具有较高的精度,且补偿结果能够实时跟随摄像系统的扫描运动,输出稳定完整的视频序列。
Electronic image stabilization is a new technology to stabilize image sequences with computer image processing. It has a lot of advantages, such as high image stabilizing precision, compact size, lightweight, low power consumption and reasonable price. It has been widely used in military and civil fields. The key technologies of electronic image stabilization are global motion estimation and motion compensation. The research on improving efficiency and accuracy of global motion estimation and distinguishing camera scan from dithering to realize real-time and accurate compensation becomes a hot topic at present.
Dithering of the video sequence includes translation, rotation and zoom of complex movements. Traditional corner extraction method is easy to generate a lot of corners focused on moving objects in the scene. And the speed is low because of complex matching steps. If we use all corners to calculate the global motion parameters directly, it will result in accuracy problem because of the existence of local outliers and error matched points. So this paper presents an electronic image stabilization algorithm based on image partition and corner matching. First of all, the corners are extracted evenly from the reference frame by using Harris corner detection operator and then each feature window is built to find the corresponding corners in the current frame. At the same time in order to ensure accuracy and speed of estimation, by matching validation technique, we can solve the error matched problem arising in the course of matching and reduce the adverse effects of invalid corner pairs. Next, the global motion parameters can be obtained through Levenberg-Marquardt iterative method. Finally, the Kalman filter is applied to smooth the motion vectors to obtain the dithering vector and the image mosaic is used to compensate each current frame in order to improve the stability and integrity of the output video.
The experimental results show that the pre-processing and iterative steps to the corners of the algorithm ensure the overall of the motion detection with high precision. The compensation results can follow scanning movement of the camera system in real time. At last the stable and integral output video sequence can be obtained.
引文
[1] Guestrin C.,Cozman F., Krotkov E. Image stabilization for feature tracking and generation of stable video overlays. Tech.Report CMU-RI-TR-97-42,Robotics Institute, Carnegie Mellon University, Nov.1997.
[2] Mitsuo Hashimoto, Tetsuya Kuno, Hiroaki Sugiura . A New Image-stabilization Method by Transferring Electric Charges. IEEE Transactions on Consumer Electronics,2007,53(4):1230~1236.
[3] Yeou-Min Yeh,Sheng-Jyh Wang. Digital camcorder image stabilizer based on gray coded bit plane block matching.Optical Engineering,2001,40(10):2172~2178.
[4]钟平.机载电子稳像技术研究,[博士学位论文].长春:中国科学院长春光学精密机械与物理研究所,2003.
[5]谷素梅,曲鲁杰.小型光惯稳跟仪器稳像精度测试方案探讨.光学技术, 1993,Vol.2:25~26.
[6]柴云峰.基于光纤陀螺的稳像系统研究,[硕士学位论文].哈尔滨:哈尔滨工业大学,2006.
[7] Sheng-Che Hsu, Sheng-Fu Liang, Chin-Teng Lin. A Robust Digital Image Stabilization Technique Based on Inverse Triangle Method and Background Detection. IEEE Transactions on Consumer Electronics,2005,51(2):335~345.
[8]郭雷,高世伟,杜亚琴,等.改进的基于LMA算法的电子稳像技术.微电子学与计算机,2008,25(8):76~79.
[9] Won-Ho Cho, Ki-Sang Hong .A Fast CIS Still Image Stabilization Method without Parallax and Moving Object Problems. IEEE Transactions on Consumer Electronics, 2008 ,54(2):197~205.
[10]黄亚博.一种基于特征匹配的鲁棒性稳像算法,[硕士学位论文].河南:河南大学,2004.
[11] Maheux J, Cruickshank J, Sevigny L. Video-rate image stabilization system. Proceedings of SPIE-The International Society for Optical Engineering,1998,3414:232~240.
[12]关升.几种电子稳像算法的初步研究,[硕士学位论文].哈尔滨:哈尔滨工业大学,2006.
[13]韩亚楠.基于灰度编码位平面匹配的视频图像稳定,[硕士学位论文].哈尔滨:哈尔滨工程大学,2005.
[14]钟平,于钱洋,王明佳,等.提高用于电子稳像的灰度投影算法精度的方法.光电子·激光,2003,14(2):182~186.
[15]赵红颖,晏磊,熊经武.一种稳定船载电视侦察图像序列的灰度投影拟合算法.光学技术,2004,30(1):83~88.
[16]杨雨东,徐光佑,朱志刚.2.5维数字图像序列稳定方法.清华大学学报(自然科学版),1997,37(9):78~81.
[17]孟龙,林行刚,王贵锦.视频抖动补偿系统的运动滤波.清华大学学报(自然科学版),2005,45(1):41~45.
[18]刘明,赵跃进,周渝斌.电子稳像中的运动补偿矢量处理方法的研究.光学技术,2005,31(3):457~459.
[19]罗诗途,张王己,王艳玲.一种基于特征匹配的实时电子稳像算法.国防科技大学学报,2005,27(3):46~48.
[20]陈硕.视频图像的数字稳像系统设计与目标识别,[硕士学位论文].西安:西安电子科技大学,2009.
[21]李慧娟.基于DSP的电子稳像系统的研究与实现,[硕士学位论文].西安:西安电子科技大学,2009.
[22]韩卿.基于特征跟踪的电子稳像技术研究,[硕士学位论文].西安:西安电子科技大学,2007.
[23] F.Dufaux, J.Konrad. Efficient. robust and fast global motion estimation for video coding. IEEE Transactions on Image Processing,2000,9(3):497~501.
[24] Litvin A, Konrad J, Karl W. Probabilistic video stabilization using Kalman filtering and mosaicking. Proceedings of SPIE-The International Society for Optical Engineering,2003, 5022:663~674.
[25] He Yu-Wen, Zhong Yu-Zhuo, Yang Shi-Qiang. A fast global motion compensation coding method. Acta Electronica Sinica,2001,29(2):175~177.
[26]马颂德,张正友.计算机视觉-计算理论与算法基础.北京:科学出版社,1997.
[27]邓再强.基于特征点匹配的电子稳像算法研究,[硕士学位论文].成都:电子科技大学,2008.
[28]孙平.基于改进特征点匹配的电子稳像算法研究,[硕士学位论文].苏州:苏州大学,2008.
[29] L-M Po, W-C Ma. A novel four-step search algorithm for fast block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology,1996,6(3):313~317.
[30] Hong-Bin Deng, Yun-De Jia, Yi-Hua Xu, Wei Liang. An Airborne Image Stabilization Method Based on Projection and the Gaussian Mixture Model. Proceedings of the Sixth International Conference on Machine Learning and Cybernetics,2007,19(22):345~349.
[31] Zhu Juan juan , Guo Bao long. Electronic image stabilization system based on global feature tracking. Journal of Systems Engineering and Electronics,2008,19(2):228~233.
[32]王志民,徐晓刚.基于单象限Fourier-Mellin域相位相关的快速稳像算法.中国图象图形学报,2009,14(9):1790~1795.
[33]朱克忠.基于光流法对移动目标的视频检测与应用研究,[硕士学位论文].合肥:合肥工业大学,2007.
[34]王新卫,周利莉,张帆.基于全局运动估计的视频对象分割算法.计算机工程与设计,2007,28(16):3937~3939.
[35]王云丽,张鑫,高超.航拍视频拼图中基于特征匹配的全局运动估计方法.航空学报,2008,29(5):1218~1225.
[36]郭丽.全局运动估计及其在视频编码中的应用,[硕士学位论文].苏州:苏州大学,2006
[37] H.P.Moravec. Visual mapping by a robot rover. International Joint Conference on artificial Intelligence,1979:598~600.
[38] S.M.Smith, J.M.Brady. SUSAN-a new approach to low level image processing .Int .Journal Computer Vision,1997,23(1):45~78.
[39] David G Lowe. Distinctive image features from scale-invariant key-points. International Journal of Computer Vision,2004,60(2):91~110.
[40] Wang Wei ,Tang Yi-ping ,Ren Juan-Li. Improved algorithm for Harris corner detection. Optics and Precision Engineering,2008,16(10):1995~2001.
[41] S.Zhu and K.K.Ma. A new diamond search algorithm for fast block-matching motion estimation. IEEE Transactions on Image Processing,2000,9(2):287~290.
[42] Press W H, Flannery B P, Teukolsky S A,etal. Numerical recipes in C:The art of scientific computing. U.K:Cambridge Univ.Press,1988.
[43] Hussein Alzoubi, W.David Pan. Fast and accurate global motion estimation algorithm using pixel subsampling. Information Sciences,2008,178:3415~3425.
[44]于露.基于图像匹配的电子稳像技术研究,[硕士学位论文].长春:长春理工大学,2009.
[45]杜丽娅,秦安,冯前进,等.基于B样条和L-M优化方法的快速弹性配准.计算机工程与应用,2009,45(14):188~190.
[46]柳宗浦,赵曙光,潘翔鹤,等.一种融合Kalman预测和Mean-shift搜索的视频运动目标跟踪新方法.光电子技术,2009,29(1):30~33.
[47] M.K.Gullu, E.Yaman, S.Erturk. Image sequence stabilization using fuzzy adaptive kalman filtering. Electronics Letters,2003,39(5):429~431.
[48] R.C.Jones, D.DeMenthon, D.S.Doermann. Building mosaics from video using MPEG motion vectors. Proceedings of the seventh ACM international conference on Multimedia (Part2),1999:29~32.
[49] H.Y.Shum, R.Szeliski. Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision,2000,16(1):63~84.
[50]刘艳.基于动态偏移场模型的视频稳定化技术研究,[博士学位论文].北京:中国科学院研究生院,2007.
[51]曹辉.船舶监控中的视频稳像与目标跟踪的研究,[博士学位论文].大连:大连海事大学,2008.
[52]马宏业.全景摄像系统电子稳像方法研究,[硕士学位论文].哈尔滨:哈尔滨工程大学,2009.
[53]史阳.基于特征窗跟踪的视频稳像和镜头检测算法研究,[硕士学位论文].西安:西安电子科技大学,2005.
[54] S.Erturk. Digital image stabilization with sub-image phase correlation based global motion estimation. IEEE Transactions on Consumer Electronics,2003,49(4):1320~1325.
[55]樊玉平.基于视频处理的视线稳定技术研究,[硕士学位论文].南京:南京航空航天大学,2009.
[56] Nakano K, Masatoshi O, Hasegawa Y. Camera calibration with precise extraction of feature points using projective transformation. IEEE proceedings of International Conference on Robotics and Automation,Vol.2002:2532~2538.
[57] Won-Ho Cho and Ki-Sang Hong. A Fast CIS Still Image Stabilization Method without Parallax and Moving Object Problems. IEEE Transactions on Consumer Electronics,2008,54(2):197~205.
[58] Joon K P, Yong C P, Dong W K .An adaptive motion decision system for digital image stabilizer based on edge pattern matching. IEEE Transactions on Consumer Electronics, 1992,38(3):607~616.
[2] Mitsuo Hashimoto, Tetsuya Kuno, Hiroaki Sugiura . A New Image-stabilization Method by Transferring Electric Charges. IEEE Transactions on Consumer Electronics,2007,53(4):1230~1236.
[3] Yeou-Min Yeh,Sheng-Jyh Wang. Digital camcorder image stabilizer based on gray coded bit plane block matching.Optical Engineering,2001,40(10):2172~2178.
[4]钟平.机载电子稳像技术研究,[博士学位论文].长春:中国科学院长春光学精密机械与物理研究所,2003.
[5]谷素梅,曲鲁杰.小型光惯稳跟仪器稳像精度测试方案探讨.光学技术, 1993,Vol.2:25~26.
[6]柴云峰.基于光纤陀螺的稳像系统研究,[硕士学位论文].哈尔滨:哈尔滨工业大学,2006.
[7] Sheng-Che Hsu, Sheng-Fu Liang, Chin-Teng Lin. A Robust Digital Image Stabilization Technique Based on Inverse Triangle Method and Background Detection. IEEE Transactions on Consumer Electronics,2005,51(2):335~345.
[8]郭雷,高世伟,杜亚琴,等.改进的基于LMA算法的电子稳像技术.微电子学与计算机,2008,25(8):76~79.
[9] Won-Ho Cho, Ki-Sang Hong .A Fast CIS Still Image Stabilization Method without Parallax and Moving Object Problems. IEEE Transactions on Consumer Electronics, 2008 ,54(2):197~205.
[10]黄亚博.一种基于特征匹配的鲁棒性稳像算法,[硕士学位论文].河南:河南大学,2004.
[11] Maheux J, Cruickshank J, Sevigny L. Video-rate image stabilization system. Proceedings of SPIE-The International Society for Optical Engineering,1998,3414:232~240.
[12]关升.几种电子稳像算法的初步研究,[硕士学位论文].哈尔滨:哈尔滨工业大学,2006.
[13]韩亚楠.基于灰度编码位平面匹配的视频图像稳定,[硕士学位论文].哈尔滨:哈尔滨工程大学,2005.
[14]钟平,于钱洋,王明佳,等.提高用于电子稳像的灰度投影算法精度的方法.光电子·激光,2003,14(2):182~186.
[15]赵红颖,晏磊,熊经武.一种稳定船载电视侦察图像序列的灰度投影拟合算法.光学技术,2004,30(1):83~88.
[16]杨雨东,徐光佑,朱志刚.2.5维数字图像序列稳定方法.清华大学学报(自然科学版),1997,37(9):78~81.
[17]孟龙,林行刚,王贵锦.视频抖动补偿系统的运动滤波.清华大学学报(自然科学版),2005,45(1):41~45.
[18]刘明,赵跃进,周渝斌.电子稳像中的运动补偿矢量处理方法的研究.光学技术,2005,31(3):457~459.
[19]罗诗途,张王己,王艳玲.一种基于特征匹配的实时电子稳像算法.国防科技大学学报,2005,27(3):46~48.
[20]陈硕.视频图像的数字稳像系统设计与目标识别,[硕士学位论文].西安:西安电子科技大学,2009.
[21]李慧娟.基于DSP的电子稳像系统的研究与实现,[硕士学位论文].西安:西安电子科技大学,2009.
[22]韩卿.基于特征跟踪的电子稳像技术研究,[硕士学位论文].西安:西安电子科技大学,2007.
[23] F.Dufaux, J.Konrad. Efficient. robust and fast global motion estimation for video coding. IEEE Transactions on Image Processing,2000,9(3):497~501.
[24] Litvin A, Konrad J, Karl W. Probabilistic video stabilization using Kalman filtering and mosaicking. Proceedings of SPIE-The International Society for Optical Engineering,2003, 5022:663~674.
[25] He Yu-Wen, Zhong Yu-Zhuo, Yang Shi-Qiang. A fast global motion compensation coding method. Acta Electronica Sinica,2001,29(2):175~177.
[26]马颂德,张正友.计算机视觉-计算理论与算法基础.北京:科学出版社,1997.
[27]邓再强.基于特征点匹配的电子稳像算法研究,[硕士学位论文].成都:电子科技大学,2008.
[28]孙平.基于改进特征点匹配的电子稳像算法研究,[硕士学位论文].苏州:苏州大学,2008.
[29] L-M Po, W-C Ma. A novel four-step search algorithm for fast block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology,1996,6(3):313~317.
[30] Hong-Bin Deng, Yun-De Jia, Yi-Hua Xu, Wei Liang. An Airborne Image Stabilization Method Based on Projection and the Gaussian Mixture Model. Proceedings of the Sixth International Conference on Machine Learning and Cybernetics,2007,19(22):345~349.
[31] Zhu Juan juan , Guo Bao long. Electronic image stabilization system based on global feature tracking. Journal of Systems Engineering and Electronics,2008,19(2):228~233.
[32]王志民,徐晓刚.基于单象限Fourier-Mellin域相位相关的快速稳像算法.中国图象图形学报,2009,14(9):1790~1795.
[33]朱克忠.基于光流法对移动目标的视频检测与应用研究,[硕士学位论文].合肥:合肥工业大学,2007.
[34]王新卫,周利莉,张帆.基于全局运动估计的视频对象分割算法.计算机工程与设计,2007,28(16):3937~3939.
[35]王云丽,张鑫,高超.航拍视频拼图中基于特征匹配的全局运动估计方法.航空学报,2008,29(5):1218~1225.
[36]郭丽.全局运动估计及其在视频编码中的应用,[硕士学位论文].苏州:苏州大学,2006
[37] H.P.Moravec. Visual mapping by a robot rover. International Joint Conference on artificial Intelligence,1979:598~600.
[38] S.M.Smith, J.M.Brady. SUSAN-a new approach to low level image processing .Int .Journal Computer Vision,1997,23(1):45~78.
[39] David G Lowe. Distinctive image features from scale-invariant key-points. International Journal of Computer Vision,2004,60(2):91~110.
[40] Wang Wei ,Tang Yi-ping ,Ren Juan-Li. Improved algorithm for Harris corner detection. Optics and Precision Engineering,2008,16(10):1995~2001.
[41] S.Zhu and K.K.Ma. A new diamond search algorithm for fast block-matching motion estimation. IEEE Transactions on Image Processing,2000,9(2):287~290.
[42] Press W H, Flannery B P, Teukolsky S A,etal. Numerical recipes in C:The art of scientific computing. U.K:Cambridge Univ.Press,1988.
[43] Hussein Alzoubi, W.David Pan. Fast and accurate global motion estimation algorithm using pixel subsampling. Information Sciences,2008,178:3415~3425.
[44]于露.基于图像匹配的电子稳像技术研究,[硕士学位论文].长春:长春理工大学,2009.
[45]杜丽娅,秦安,冯前进,等.基于B样条和L-M优化方法的快速弹性配准.计算机工程与应用,2009,45(14):188~190.
[46]柳宗浦,赵曙光,潘翔鹤,等.一种融合Kalman预测和Mean-shift搜索的视频运动目标跟踪新方法.光电子技术,2009,29(1):30~33.
[47] M.K.Gullu, E.Yaman, S.Erturk. Image sequence stabilization using fuzzy adaptive kalman filtering. Electronics Letters,2003,39(5):429~431.
[48] R.C.Jones, D.DeMenthon, D.S.Doermann. Building mosaics from video using MPEG motion vectors. Proceedings of the seventh ACM international conference on Multimedia (Part2),1999:29~32.
[49] H.Y.Shum, R.Szeliski. Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision,2000,16(1):63~84.
[50]刘艳.基于动态偏移场模型的视频稳定化技术研究,[博士学位论文].北京:中国科学院研究生院,2007.
[51]曹辉.船舶监控中的视频稳像与目标跟踪的研究,[博士学位论文].大连:大连海事大学,2008.
[52]马宏业.全景摄像系统电子稳像方法研究,[硕士学位论文].哈尔滨:哈尔滨工程大学,2009.
[53]史阳.基于特征窗跟踪的视频稳像和镜头检测算法研究,[硕士学位论文].西安:西安电子科技大学,2005.
[54] S.Erturk. Digital image stabilization with sub-image phase correlation based global motion estimation. IEEE Transactions on Consumer Electronics,2003,49(4):1320~1325.
[55]樊玉平.基于视频处理的视线稳定技术研究,[硕士学位论文].南京:南京航空航天大学,2009.
[56] Nakano K, Masatoshi O, Hasegawa Y. Camera calibration with precise extraction of feature points using projective transformation. IEEE proceedings of International Conference on Robotics and Automation,Vol.2002:2532~2538.
[57] Won-Ho Cho and Ki-Sang Hong. A Fast CIS Still Image Stabilization Method without Parallax and Moving Object Problems. IEEE Transactions on Consumer Electronics,2008,54(2):197~205.
[58] Joon K P, Yong C P, Dong W K .An adaptive motion decision system for digital image stabilizer based on edge pattern matching. IEEE Transactions on Consumer Electronics, 1992,38(3):607~616.