面向画质增强的去运动模糊技术研究
|
摘要
在数字摄影摄像过程中,常常由于摄影器材抖动或是场景快速变化造成图像或视频模糊不清晰。为了克服模糊效果带来的画质损失,去模糊技术近年来在计算机视觉领域和数字图像处理领域得到了广泛的研究。通常摄影过程中的模糊效果可分为失焦模糊和运动模糊两大类。失焦模糊是指摄影摄像过程中,目标物体所处位置不在光学镜头焦距附近,导致成像不清晰。摄像时聚焦不准或是景物深度超过设备景深是造成失焦模糊的两个主要原因。运动模糊是指景物在图像中的移动效果。造成运动模糊,主要是因为在光圈曝光时间内,摄影器材抖动或是景物快速移动造成图像拖影。本课题研究的去模糊,指的是去运动模糊。
运动模糊在数学模型上一般被解释为清晰图像卷积模糊核,叠加一定随机噪声后产生模糊图像。模糊核又被称作是点扩散方程,它描述了小范围内光的叠加的方式,包括密度和位置。去运动模糊按照模糊核是否可得分为盲卷和非盲卷去运动模糊。按照模糊核的单一性,又可以分为空间可变(多核)和空间不变(单核)去运动模糊。为了从模糊图像恢复出原始清晰图,去运动模糊技术常常遇到以下几个难题:(1)模糊核未知。基于单幅图像的去模糊技术通常都需要通过各种手段估算模糊核,因为模糊图像的信息干扰和损失问题,模糊核通常很难被精确估算。(2)反卷积的多解性。反卷运算的解是多个的,这导致了复原清晰图像时,可能会产生非清晰的结果。(3)去运动模糊中的水纹效应。这通常是由于反卷运算的采用的模型自身造成,例如有穷傅里叶级数等。(4)受噪音干扰较大。去模糊和反卷对噪音极其敏感,这加大了复原清晰图像的难度。
本课题的主要研究对象是基于单幅图像的去运动模糊算法,以三个主要方向为切入点研究面向画质增强的去运动模糊技术。首先,本课题尝试解决未知运动模糊核的估计,利用图像先验知识,滤波和梯度域算法等不同技术,获取图像抖动或景物移动信息,估算模糊核的大小,位置及密度。其次,本课题研究比较鲁棒的反卷积模型,该模型对随机噪声不敏感,效果稳定,且能够克服通常卷积过程中带来的水纹效应。第三,本文尝试研究空间可变的运动模糊模型。通常的运动模糊都是空间不变模型,即使用单一核描述图像上所有像素的运动。空间可变模型则使用多核模型模拟模糊过程,以求达到更加精确的结果。
本课题最终提出一种既能够复原锐利边缘,又能够抑制平滑区域噪声的,基于梯度域变换的核估计算法,并将该算法运用于单核或多核去运动模糊技术中。实验结果表明本算法能够从一般的模糊图像中准确估算模糊核并复原清晰图像。该去运动模糊技术将在摄影摄像领域中有良好的应用前景。
In photography, camera shaking or object relative movement may cause final images or videos blurry. In order to overcome the quality loss in blurry effect, image deblurring techniques have been widely studied in computer vision and digital image processing in recent years. Generally, image blurring can be classified into two basic categories: defocus blurring and motion blurring. When the position of object in image is out of focus or far from the optical focal length, result photos will have defocus blurring. Sometimes, depth of field is one of the photography techniques. In such case, people intentionally make the background out of focus. Motion blurring is the movement effect of object in scene. The main reason for motion blurring is the relative motion between lens and scene. The lens accumulates incoming light during the motion. This paper mainly researches on motion deblurring.
Mathematically, motion blurring is considered as a convolution process. The blurry image is made up by random noise and the convolution result of latent image and a blur kernel. The blur kernel is also called point-spread-function (PSF). It describes essential information about light accumulation in a small region, for example, light intensity density and position. According to the accessible of the blur kernel, there are two kinds of motion deblurring: blind deconvolution and non-blind deconvolution. According to the unicity of blur kernel, motion deblurring can be classified into motion variant deblur (multiple kernels) and motion invariant deblur (single kernel). In order to restore latent clear image as accurate as possible, there are many challenges in motion deblurring. (1) Unknown blur kernel. Single image based deblurring need to use several methods to accurately estimate blur kernel. However, the essential estimation information is loss in the blurry image. (2) Multiple solution of deconvolution. Because the solution of a deconvolution is no unique, result image may be still fuzzy. (3) Ringing effect. This is caused by the deconvolution model itself, for example finite Fourier series. (4) Noise. Normal deblurring method is sensitive to noise. This makes image restoration difficult.
This paper focus on single image based motion deblurring. It developed research on motion deblurring on image quality enhancement from three aspects. First, we try to estimate blur kernel using other techniques such as gradient prior, image filter and gradient domain method. We can estimate the size, position and density of the kernel. Second, we develop a robust deconvolution model, which is stable and not sensitive to noise. It can overcome the ringing effect in previous deblur method. Third, we try to tackle motion variant deblurring, which consider multiple blur kernel and can get more accurate result.
This paper proposed a novel edge-preserving gradient enhancement is used in kernel estimation, which can recover sharp edges in blurry image without intensifying the derivatives or noise in flat area. Experiment results show that our method can not only estimate blur kernel accurately but also restore clear image. This deblurring technique will have good application prospects in digital photography.
运动模糊在数学模型上一般被解释为清晰图像卷积模糊核,叠加一定随机噪声后产生模糊图像。模糊核又被称作是点扩散方程,它描述了小范围内光的叠加的方式,包括密度和位置。去运动模糊按照模糊核是否可得分为盲卷和非盲卷去运动模糊。按照模糊核的单一性,又可以分为空间可变(多核)和空间不变(单核)去运动模糊。为了从模糊图像恢复出原始清晰图,去运动模糊技术常常遇到以下几个难题:(1)模糊核未知。基于单幅图像的去模糊技术通常都需要通过各种手段估算模糊核,因为模糊图像的信息干扰和损失问题,模糊核通常很难被精确估算。(2)反卷积的多解性。反卷运算的解是多个的,这导致了复原清晰图像时,可能会产生非清晰的结果。(3)去运动模糊中的水纹效应。这通常是由于反卷运算的采用的模型自身造成,例如有穷傅里叶级数等。(4)受噪音干扰较大。去模糊和反卷对噪音极其敏感,这加大了复原清晰图像的难度。
本课题的主要研究对象是基于单幅图像的去运动模糊算法,以三个主要方向为切入点研究面向画质增强的去运动模糊技术。首先,本课题尝试解决未知运动模糊核的估计,利用图像先验知识,滤波和梯度域算法等不同技术,获取图像抖动或景物移动信息,估算模糊核的大小,位置及密度。其次,本课题研究比较鲁棒的反卷积模型,该模型对随机噪声不敏感,效果稳定,且能够克服通常卷积过程中带来的水纹效应。第三,本文尝试研究空间可变的运动模糊模型。通常的运动模糊都是空间不变模型,即使用单一核描述图像上所有像素的运动。空间可变模型则使用多核模型模拟模糊过程,以求达到更加精确的结果。
本课题最终提出一种既能够复原锐利边缘,又能够抑制平滑区域噪声的,基于梯度域变换的核估计算法,并将该算法运用于单核或多核去运动模糊技术中。实验结果表明本算法能够从一般的模糊图像中准确估算模糊核并复原清晰图像。该去运动模糊技术将在摄影摄像领域中有良好的应用前景。
In photography, camera shaking or object relative movement may cause final images or videos blurry. In order to overcome the quality loss in blurry effect, image deblurring techniques have been widely studied in computer vision and digital image processing in recent years. Generally, image blurring can be classified into two basic categories: defocus blurring and motion blurring. When the position of object in image is out of focus or far from the optical focal length, result photos will have defocus blurring. Sometimes, depth of field is one of the photography techniques. In such case, people intentionally make the background out of focus. Motion blurring is the movement effect of object in scene. The main reason for motion blurring is the relative motion between lens and scene. The lens accumulates incoming light during the motion. This paper mainly researches on motion deblurring.
Mathematically, motion blurring is considered as a convolution process. The blurry image is made up by random noise and the convolution result of latent image and a blur kernel. The blur kernel is also called point-spread-function (PSF). It describes essential information about light accumulation in a small region, for example, light intensity density and position. According to the accessible of the blur kernel, there are two kinds of motion deblurring: blind deconvolution and non-blind deconvolution. According to the unicity of blur kernel, motion deblurring can be classified into motion variant deblur (multiple kernels) and motion invariant deblur (single kernel). In order to restore latent clear image as accurate as possible, there are many challenges in motion deblurring. (1) Unknown blur kernel. Single image based deblurring need to use several methods to accurately estimate blur kernel. However, the essential estimation information is loss in the blurry image. (2) Multiple solution of deconvolution. Because the solution of a deconvolution is no unique, result image may be still fuzzy. (3) Ringing effect. This is caused by the deconvolution model itself, for example finite Fourier series. (4) Noise. Normal deblurring method is sensitive to noise. This makes image restoration difficult.
This paper focus on single image based motion deblurring. It developed research on motion deblurring on image quality enhancement from three aspects. First, we try to estimate blur kernel using other techniques such as gradient prior, image filter and gradient domain method. We can estimate the size, position and density of the kernel. Second, we develop a robust deconvolution model, which is stable and not sensitive to noise. It can overcome the ringing effect in previous deblur method. Third, we try to tackle motion variant deblurring, which consider multiple blur kernel and can get more accurate result.
This paper proposed a novel edge-preserving gradient enhancement is used in kernel estimation, which can recover sharp edges in blurry image without intensifying the derivatives or noise in flat area. Experiment results show that our method can not only estimate blur kernel accurately but also restore clear image. This deblurring technique will have good application prospects in digital photography.
引文
[1] Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. In: ACM SIGGRAPH 2008 papers.
[2] Likas, A.C., Galatsanos, N.P.: A variational approach for bayesian blind image deconvolution.
[3] Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. In: ACM SIGGRAPH 2006 papers.
[4] Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image deblurring with blurred/noisy image pairs. In: ACM SIGGRAPH 2007 papers.
[5] Joshi, N., Zitnick, L., Szeliski, R., Kriegman, D.: Image deblurring and denoising using color priors. In: Proceedings of IEEE CVPR '09.
[6] Levin, A.: Blind motion deblurring using image statistics. In: Advances in Neural Information Processing Systems.
[7] Jia, J.: Single image motion deblurring using transparency. In: Proceedings of IEEE CVPR '07.
[8] Joshi, N., Szeliski, R., Kriegman, D.J.: Psf estimation using sharp edge prediction. In: Proceedings of IEEE CVPR '08.
[9] Levin, A., Weiss, Y., Durand, F.: Understanding and evaluating blind deconvolution algorithms. In: Proceedings of IEEE CVPR '09.
[10] Tai, Y.W., Du, H., Brown, M., Lin, S.: Image/video deblurring using a hybrid camera. In: Proceedings of IEEE CVPR '08.
[11] Tai, Y.W., Kong, N., Lin, S., Shin, S.Y.: Coded exposure imaging for projective motion deblurring. In: Proceedings of IEEE CVPR '10.
[12] Dai, S., Wu, Y.: Motion from blur. In: Proceedings of IEEE CVPR '08.
[13] Shan, Q., Xiong, W., Jia, J.: Rotational motion deblurring of a rigid object from a single image. In: Proceedings of IEEE ICCV '07.
[14] Hirsch, M., Sra, S., Scholkopf, B., Harmeling, S.: Efficient filter flow for space variant multiframe blind deconvolution. In: Proceedings of IEEE CVPR '10.
[15] Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. In: Proceedings of IEEE CVPR '10.
[16] A. Gupta, L. Joshi, N.and Zitnick,M. Cohen, and B. Curless. Single image deblurring using motiondensity functions. In: Proceedings of European Conference on Computer Vision, 2010.
[17] Cho, S., Lee, S.: Fast motion deblurring. ACM Trans. Graph. 28
[18] Ji, H., Liu, C. 2008. Motion blur identification from image gradients. In: Proc. CVPR 2008, 1–8.
[19] Agrawal, A., Raskar, R. Resolving objects at higher resolution from a single motion-blurred image. In CVPR, 2007.
[20] Zhuo, S., Guo, D. and Sim, T. Robust Flash Deblurring, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010.
[21] Xu, L. Jia, J., Two-Phase Kernel Estimation for Robust Motion Deblurring. In: Proceedings of ECCV 2010.
[22] Carl, W.H. Image restoration by the method of least squares. Journal of the Optical Society of America, 1967.
[23]刘扬阳,金伟其,苏秉华.数字图像去模糊处理算法的对比研究[J].北京理工大学学报, 2004, 24(10).
[24] Richardson W H. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 1972, 62(1):55-59.
[25] Lucy L B. An iterative technique for the rectification of observed distribution[J]. The Astronomical Journal, 1974, 79(6):745-765.
[26]赵剡,张怡,许东.基于总变分规整化的湍流退化图像复原RL算法[J].中北大学学报:自然科学版, 2007, 28(1): 69-73.
[27] Biggs D S C, Andrews M. Iterative Blind Deconvolution of Extended Objects[C]//Proc. of International Conference on Image Processing. [S. l.]: IEEE Computer Society, 1997
[28] Minh N D, Vetterli M. The Contourlet Transform: An Efficient Directional Multisolution Image Representation [J]. IEEE Trans. on Image Processing, 2005, 14(12): 2091-2106.
[29]陈云龙,王平,王鹏.基于L-R非线性迭代的降质图像复原算法[J].计算机工程, 2010, 36(4): 202-204
[30] CHAN, T. F., AND WONG, C.-K. 1998. Total variation blind deconvolution. IEEE Trans. Image Processing 7, 3, 370–375.
[31] OSHER, S., AND RUDIN, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical
[32] GONZALEZ, R. C., AND WOODS, R. E. 2001. Digital Image Processing, 2nd ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
[33] ADOBE, SYSTEMS, I. 2010. Photoshop CS 5.
[34] TOMASI, C., AND MANDUCHI, R. 1998. Bilateral filtering for gray and color images. InProceedings of the Sixth International Conference on Computer Vision, IEEE Computer Society,Washington, DC, USA, ICCV’98, 839–846.
[35] HE, K., SUN, J., AND TANG, X. 2010. Guided image filtering. In Proceedings of the 11th European conference on Computer vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV’10, 1–14.
[36] FATTAL, R., LISCHINSKI, D., AND WERMAN, M. 2002. Gradient domain high dynamic range compression. ACM Trans. Graph.21 (July), 249–256.
[37] PEREZ, P., GANGNET, M., AND BLAKE, A. 2003. Poisson image editing. ACM Trans. Graph. 22 (July), 313–318.
[38] LEVIN, A., ZOMET, A., PELEG, S., AND WEISS, Y. 2003. Seamless image stitching in the gradient domain. In In Eighth European Conference on Computer Vision (ECCV 2004, Springer-Verlag, 377–389.
[39] LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2004. Colorization 479 using optimization. ACM Trans. Graph. 23 (August), 689–694.
[40] LISCHINSKI, D., FARBMAN, Z., UYTTENDAELE, M., AND SZELISKI, R. 2006. Interactive local adjustment of tonal values. ACM Trans. Graph. 25 (July), 646–653.
[41] ORZAN, A., BOUSSEAU, A., BARLA, P., AND THOLLOT, J. 2007. Structure-preserving manipulation of photographs. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering, ACM, New York, NY, USA, NPAR’07, 103–110.
[42] AGRAWAL, A., RASKAR, R., NAYAR, S. K., AND LI, Y. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24 (July), 828–835.
[43] AGRAWAL, A., RASKAR, R., AND CHELLAPPA, R. 2006. Edge suppression by gradient field transformation using cross-projection tensors. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2, IEEE Computer Society, Washington, DC, USA, CVPR’06, 2301–2308.
[44] HE, K., SUN, J., AND TANG, X. 2010. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 99, PrePrints.
[45] LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2008. A closed form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 228–242.
[2] Likas, A.C., Galatsanos, N.P.: A variational approach for bayesian blind image deconvolution.
[3] Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. In: ACM SIGGRAPH 2006 papers.
[4] Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image deblurring with blurred/noisy image pairs. In: ACM SIGGRAPH 2007 papers.
[5] Joshi, N., Zitnick, L., Szeliski, R., Kriegman, D.: Image deblurring and denoising using color priors. In: Proceedings of IEEE CVPR '09.
[6] Levin, A.: Blind motion deblurring using image statistics. In: Advances in Neural Information Processing Systems.
[7] Jia, J.: Single image motion deblurring using transparency. In: Proceedings of IEEE CVPR '07.
[8] Joshi, N., Szeliski, R., Kriegman, D.J.: Psf estimation using sharp edge prediction. In: Proceedings of IEEE CVPR '08.
[9] Levin, A., Weiss, Y., Durand, F.: Understanding and evaluating blind deconvolution algorithms. In: Proceedings of IEEE CVPR '09.
[10] Tai, Y.W., Du, H., Brown, M., Lin, S.: Image/video deblurring using a hybrid camera. In: Proceedings of IEEE CVPR '08.
[11] Tai, Y.W., Kong, N., Lin, S., Shin, S.Y.: Coded exposure imaging for projective motion deblurring. In: Proceedings of IEEE CVPR '10.
[12] Dai, S., Wu, Y.: Motion from blur. In: Proceedings of IEEE CVPR '08.
[13] Shan, Q., Xiong, W., Jia, J.: Rotational motion deblurring of a rigid object from a single image. In: Proceedings of IEEE ICCV '07.
[14] Hirsch, M., Sra, S., Scholkopf, B., Harmeling, S.: Efficient filter flow for space variant multiframe blind deconvolution. In: Proceedings of IEEE CVPR '10.
[15] Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. In: Proceedings of IEEE CVPR '10.
[16] A. Gupta, L. Joshi, N.and Zitnick,M. Cohen, and B. Curless. Single image deblurring using motiondensity functions. In: Proceedings of European Conference on Computer Vision, 2010.
[17] Cho, S., Lee, S.: Fast motion deblurring. ACM Trans. Graph. 28
[18] Ji, H., Liu, C. 2008. Motion blur identification from image gradients. In: Proc. CVPR 2008, 1–8.
[19] Agrawal, A., Raskar, R. Resolving objects at higher resolution from a single motion-blurred image. In CVPR, 2007.
[20] Zhuo, S., Guo, D. and Sim, T. Robust Flash Deblurring, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010.
[21] Xu, L. Jia, J., Two-Phase Kernel Estimation for Robust Motion Deblurring. In: Proceedings of ECCV 2010.
[22] Carl, W.H. Image restoration by the method of least squares. Journal of the Optical Society of America, 1967.
[23]刘扬阳,金伟其,苏秉华.数字图像去模糊处理算法的对比研究[J].北京理工大学学报, 2004, 24(10).
[24] Richardson W H. Bayesian-based iterative method of image restoration[J]. Journal of the Optical Society of America, 1972, 62(1):55-59.
[25] Lucy L B. An iterative technique for the rectification of observed distribution[J]. The Astronomical Journal, 1974, 79(6):745-765.
[26]赵剡,张怡,许东.基于总变分规整化的湍流退化图像复原RL算法[J].中北大学学报:自然科学版, 2007, 28(1): 69-73.
[27] Biggs D S C, Andrews M. Iterative Blind Deconvolution of Extended Objects[C]//Proc. of International Conference on Image Processing. [S. l.]: IEEE Computer Society, 1997
[28] Minh N D, Vetterli M. The Contourlet Transform: An Efficient Directional Multisolution Image Representation [J]. IEEE Trans. on Image Processing, 2005, 14(12): 2091-2106.
[29]陈云龙,王平,王鹏.基于L-R非线性迭代的降质图像复原算法[J].计算机工程, 2010, 36(4): 202-204
[30] CHAN, T. F., AND WONG, C.-K. 1998. Total variation blind deconvolution. IEEE Trans. Image Processing 7, 3, 370–375.
[31] OSHER, S., AND RUDIN, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical
[32] GONZALEZ, R. C., AND WOODS, R. E. 2001. Digital Image Processing, 2nd ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
[33] ADOBE, SYSTEMS, I. 2010. Photoshop CS 5.
[34] TOMASI, C., AND MANDUCHI, R. 1998. Bilateral filtering for gray and color images. InProceedings of the Sixth International Conference on Computer Vision, IEEE Computer Society,Washington, DC, USA, ICCV’98, 839–846.
[35] HE, K., SUN, J., AND TANG, X. 2010. Guided image filtering. In Proceedings of the 11th European conference on Computer vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV’10, 1–14.
[36] FATTAL, R., LISCHINSKI, D., AND WERMAN, M. 2002. Gradient domain high dynamic range compression. ACM Trans. Graph.21 (July), 249–256.
[37] PEREZ, P., GANGNET, M., AND BLAKE, A. 2003. Poisson image editing. ACM Trans. Graph. 22 (July), 313–318.
[38] LEVIN, A., ZOMET, A., PELEG, S., AND WEISS, Y. 2003. Seamless image stitching in the gradient domain. In In Eighth European Conference on Computer Vision (ECCV 2004, Springer-Verlag, 377–389.
[39] LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2004. Colorization 479 using optimization. ACM Trans. Graph. 23 (August), 689–694.
[40] LISCHINSKI, D., FARBMAN, Z., UYTTENDAELE, M., AND SZELISKI, R. 2006. Interactive local adjustment of tonal values. ACM Trans. Graph. 25 (July), 646–653.
[41] ORZAN, A., BOUSSEAU, A., BARLA, P., AND THOLLOT, J. 2007. Structure-preserving manipulation of photographs. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering, ACM, New York, NY, USA, NPAR’07, 103–110.
[42] AGRAWAL, A., RASKAR, R., NAYAR, S. K., AND LI, Y. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24 (July), 828–835.
[43] AGRAWAL, A., RASKAR, R., AND CHELLAPPA, R. 2006. Edge suppression by gradient field transformation using cross-projection tensors. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2, IEEE Computer Society, Washington, DC, USA, CVPR’06, 2301–2308.
[44] HE, K., SUN, J., AND TANG, X. 2010. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 99, PrePrints.
[45] LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2008. A closed form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 228–242.