数字图像修补模型、算法及其应用研究
|
摘要
数字图像修补，是利用受损区域周围的图像信息给受损区域填充信息的一门技术，本质上是一种图像插值问题。它属于图像复原的研究领域，是图像处理领域的一个分支。在多余物体的剔除(如视频图像和一些照片中删除部分人物、文字、小标题等)等方面得到了广泛的应用。除此之外，在博物馆中文物的保护方面(如对珍贵的书法与绘画作品进行的修复工作)也得到了很好的应用。正是图像修补技术的广泛应用，才使得这一技术成为了近几年的研究热点。
本文首先比较全面的综述了图像修补中的研究现状，讨论了图像修补的特殊性以及方法学。然后深入研究了现行的几种图像局部性修补的数学模型和修补算法。实现了基于偏微分方程的BSCB模型、基于贝叶斯理论的整体变分修补模型，并对比了不同模型的图像修补效果。
上述模型能够有效处理图像的几何特征，适应于非纹理图像的修补，但对富于纹理的图像修补效果不好。因此，本文给出了一种新的基于样本的图像修补模型，该模型通过计算样本填充的优先次序，将图像中的纹理与结构信息同时进行合理的填充，得到更令人满意的效果。
本文最后给出了两种兼顾图像受损结构和纹理区域的自适应修补算法，将受损纹理区域与结构区域有效区分，然后分别进行处理，得出良好的图像修补结果。
Digital image inpainting is to fill in image information on a blank domain based on the image information available outsidd. It is an interpolation problem, which belongs to the field of restoration and it's also a branch of image processing. Applications of image inpainting range from restoration of photographs, films and paintings, to removal of occlusions, such as large unwanted regions, superimposed text, subtitles, stamps from images. In addition, it is more significant to restore the precious calligraphies and paintings in the museum with image inpainting technique. The widely-used applications of the image inpainting make this technique a hot research in the image processing techniques recently.
Firstly,we summarize the recent work in this area ,discuss the particularity and methodology of inmage inpainting. Then, we are interested in the local inpainting methods, study several inpainting methods especially the Bertalmio- Sapiro-Caselles-B allester (BSCB) model and total variation (TV) model based on the Bayesian and variational principle with the comparision of results between two models.
It is more effective for two models mentioned above to deal with the geometrical character of image and adaptive to non-texture. Therefore, in this thesis, we present a new image inpainting method based on the existing exemplar-based image inpainting idea. This method can obtain preferable results in human vision by rational Confidence and Data computing method.
Finally, to overcome the disadvantage of those models mentioned above, we present two adaptive algorithms which combine image decomposition with local inpainting based on PDE and texture synthesis. Applying local inpainting and synthesis to non-texture and texture respectively gives satisfying results.
本文首先比较全面的综述了图像修补中的研究现状，讨论了图像修补的特殊性以及方法学。然后深入研究了现行的几种图像局部性修补的数学模型和修补算法。实现了基于偏微分方程的BSCB模型、基于贝叶斯理论的整体变分修补模型，并对比了不同模型的图像修补效果。
上述模型能够有效处理图像的几何特征，适应于非纹理图像的修补，但对富于纹理的图像修补效果不好。因此，本文给出了一种新的基于样本的图像修补模型，该模型通过计算样本填充的优先次序，将图像中的纹理与结构信息同时进行合理的填充，得到更令人满意的效果。
本文最后给出了两种兼顾图像受损结构和纹理区域的自适应修补算法，将受损纹理区域与结构区域有效区分，然后分别进行处理，得出良好的图像修补结果。
Digital image inpainting is to fill in image information on a blank domain based on the image information available outsidd. It is an interpolation problem, which belongs to the field of restoration and it's also a branch of image processing. Applications of image inpainting range from restoration of photographs, films and paintings, to removal of occlusions, such as large unwanted regions, superimposed text, subtitles, stamps from images. In addition, it is more significant to restore the precious calligraphies and paintings in the museum with image inpainting technique. The widely-used applications of the image inpainting make this technique a hot research in the image processing techniques recently.
Firstly,we summarize the recent work in this area ,discuss the particularity and methodology of inmage inpainting. Then, we are interested in the local inpainting methods, study several inpainting methods especially the Bertalmio- Sapiro-Caselles-B allester (BSCB) model and total variation (TV) model based on the Bayesian and variational principle with the comparision of results between two models.
It is more effective for two models mentioned above to deal with the geometrical character of image and adaptive to non-texture. Therefore, in this thesis, we present a new image inpainting method based on the existing exemplar-based image inpainting idea. This method can obtain preferable results in human vision by rational Confidence and Data computing method.
Finally, to overcome the disadvantage of those models mentioned above, we present two adaptive algorithms which combine image decomposition with local inpainting based on PDE and texture synthesis. Applying local inpainting and synthesis to non-texture and texture respectively gives satisfying results.
引文
[1] M. Bertalmio, G. Sapiro and V. Caselles et al,Image inpainting. Proc. ACM Conf. Comp. Graphics (SIGGRAPH2000), 2000: 417-424.
[2] Coloma Ballester,M.Bertalmio and V.Caselles et al,Filling_in by joint interpolation of vector fields and gray levels. IEEE Transactions on image processing, Vol. 10, NO. 8, August 2001.
[3] M.Bertalmio,B.A.L.ertozzi and G. Sapiro,Navier-stokes, fluid dynamics, and image and video inpainting. Proc. Conf. Comp. Vision Pattern Rec.2001.355-362.
[4] T. F. Chan and J. Shen,Non-texture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation,2003( Vol. 12, No.4): 436-449.
[5] Shantanu, D.Rane,Guillermo Sapiro and Marcelo Bertalmio ,Structure and t exture filling-in of missing image blocks in wireless transmission and
compression applications". IEEE Transactions on image processing, Vol. 12, NO. 3, March 2003.
[6] Celia A. Zorzo Barcelos, Marcos A.Batista, Adriana M. Martins and Antonio Carlos Nogueira ,Level Lines Continuation based Digital Inpainting. Proceedings of the XVII Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'04).
[7] Simon Masnou and Jean-Michel Morel,Level lines based disocclusion. Proceedings of International Conference on Image Processing( ICIP 98), 1998 Page(s):259 - 263 Vol.3.
[8] Celia A. Zorzo Barcelos, Marcos A. Batista,Image inpainting and denoising by nonlinear partial differential equations. Proceedings of the XVI Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'03).
[9] D.A.Karras and G.B.Mertzios,Discretization schemes and numerical approximations of PDE inpainting models and comparative evaluation on novel real world MRI reconstruction applications.IEEE IST'2004.
[10] Alexandru Telea,An image inpainting technique based on the fast marching method. Journal of graphics tools, 2004(Vol. 9,No. 1): 25-36.
[11] T. Chan and J. Shen,Mathematical Models for Local Deterministic Inpaintings. Technical Report CAM 00-11, Image Processing Research Group, UCLA, 2000.
[12] Simon Masnou,Disocclusion-A Variational Approach. IEEE Transactions on image processing, Vol. 11, NO. 2, 2002.
[13] J.P. Gu,S.L. Peng and X.L. wang ,Digital image inpainting using monte carlo method.2004 International conference on Image Processing(ICIP):961-964.
[14] C. Ballester, M. Bertalmio,V. Caselles,G.Sapiro,and J.Verdera,A Variational Model for Filling-In. Proceedings of Eighth IEEE International Conference on Computer Vision( ICCV 2001) 2001:10 -16 vol.1.
[15] Selim Esedoglu and Jianhong Shen,Digital inpainting based on the Mumford-Shah-Euler image model. Euro. J. Applied Mathematics (2002), vol. 13,353-370.
[16] A. Criminisi, P. Perez, and K. Toyama,Region filling and object removal by exemplar based inpainting. IEEE Transactions on image processing,2004(Vol.13,No.9): 1200-1212.
[17] A. Criminisi, P. Perez and K. Toyama,Object Removal by Exemplar-Based Inpainting. Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03).
[18] Iddo Drori,Daniel Cohen-Or and Hezy Yeshurun,Fragment-based image completion. Proceedingsof ACM SIGGRAPH 2003:303-312.
[19] H. Igehy and L. Pereira,Image Replacement through Texture Synthesis. In IEEE International conference on Image Processing(1997), vol. 3,186-189
[20] S. Borikar,K.K.Biswas and S.Pattanaik.Yunjun Zhang, Jiangjian Xiao and Mubarak Shah,Region Completion in a Single Image. EUROGRAPHICS 2004.
[21] S.Borikar,K.K.Biswas and S.Pattanaik,Fast Algorithm for Completion of Images with Natural Scenes. http://graphics.cs.ucf.edu/.
[22] H.Chong,GPU image inpainting via Texture synthesis http://www.people.fas.harvard.edu/～hchong/goodies.
[23] Jian Sun,Lu Yuan,Jiaya Jia and Heung-Yeung shum,Image Completion with Structure Propagation. ACM T GRAPHIC 24 (3): 861-868 JUL 2005.
[24] Hitoshi Yamauchi, Jorg Haber and Hans-Peter Seidel,Image Restoration using Multiresolution Texture Synthesis and Image Inpainting. ProceedingsComputer Graphics International, 2003: 120 - 125 .
[25] M. Bertalmio, L. Vese and G. Sapiro et al,Simultaneous structure and texture image inpainting, IEEE Transactions on Volume 12, Issue 8, Aug. 2003: 882 -889
[26] A.Levin, A.Zomet, and Y.Weiss, Learning how to inpaint form global image statistics. In Proceedings of Inte. Conf. on Comp. Vision, Ⅱ.305-313.
[27]Raymond Chan, Lixin Shen and Zuowei Shen,A framelet-based approach for image inpainting. http://www.math.cuhk.edu.hk/-rchan/paper/inpainting/.
[28]Vladimir Savchenko, Nikita Kojekine and Hiroshi Unno,A practical image retouching method. Proceedings of the First International Symposium on Cyber Worlds (CW.02).
[29]Timothy K. Shih., Rong-Chi Chang and Liang-Chen Lu et al,Adaptive Digital Image Inpainting. Proceedings of the 18th International Conference on Advanced Information Networking and Application (AINA'04).
[30]Anil N. Hirani and Takashi Totsuka,Combining Frequency and Spatial Domain Information for Fast Interactive. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 1996, p 269-276.
[31]Shih-Chang Hsia,An edge-oriented spatial interpolation for consecutive block error concealment. IEEE Signal Processing Letters, Vol. 11, NO. 6, June 2004.
[32]M. M. Oliveira, B. Bowen, and R.McKennaetc,Fast Digital Image Inpainting. Proceedings of the International Conf. on Visualization, Imaging and Image Processing (VIIP2001). 2001:261-266.
[33]Ping Tan, Stephen Lin, Long Quan and Heung-Yeung Shum,Highlight removal by illumination-constrained inpainting. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV'03).
[34]Timothy shih,Liang-chen Lu,Ying-Hong Wang and Rong-chi Chang,Multi-resolution image inpainting. Proceedings of International Conference on Multimedia and Expo, (ICME'03) Vol.1: 485-488.
[35]周廷方、汤锋、王进等“基于径向基函数的图像修复技术”，中国图象图形学报，Vol．9，No．10，2004．
[36]吴振宇、王朔中，“基于区域分割的损伤图像修复”，上海大学学报(自然科学版)，Vol．5，No．1，1999．
[37]A. - Efros and T. Leung,Texture Synthesis by Nonparametric Sampling. Proceedings of the Seventh IEEE International Conference on Computer Vision. 1999:1033-1038.
[38]D. J. Heeger and J. R. Bergen,Pyramid-Based Texture Analysis/Synthesis. In Computer Graphics (SIGGRAPH ' 95 Conf. Proc.), pages 229-238, Aug. 1995.
[39]Kedar A. Patwardhan, Guillermo Sapiro, and Marcelo Bertalmio,Video Inpainting of Occluding and Occluding Objects" .In IEEE International Conference on Image Processing (ICIP 2005). Vol.2 :69 - 72
[40] J. A. Sethian.A Fast Marching Level Set Method for Monotonically Advancing Fronts. Proceedings of the National Academy of Sciences 93:4. USA, 1996: 1591—1595.
[41] J. A. Sethian,Adaptive Fast Marching and Level Set. Acta Math. Univ. Comenianae Vol. LXVII, 1(1998), pp. 3-15.
[42] Alexei A. Efros and William T. Freeman,Image Quilting for Texture Synthesis and Transfer. Proceedings of ACM SIGGRAPH 2001:241-246.
[43] Jianhong (Jackie) Shen, Inpainting and the Fundamental Problem of Image Processing. from Slam News, Volume 36, Number 5, June 2003.
[44] D. Mumford, J. Shan,Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems.Comm.Pure Applied.Math,1989,X LH:577-685
[45] S. Geman, D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images.IEEE Trans.Pattern Anal.M achineI ntell.1984(6):721-741.
[46] D.C. Knil, W. Richards,Perception as Bayesian Inference.Cambridge U niv.Press, 1996.
[47] S. Masnou, J .M. Morel.Level-lines Based Disocclusion. Proceedings of 5th IEEE Int'l Conf .on Image Process.Chicago,1998(3):259-263.
[48] L.Y. Wei, d M. Levoy. Fast Texture Synthesis Using Tree-structured Vector Quantization. Proceedings of SIGGRAPH 2000.20 00(6):479-488.
[49] L. Rudin, S. Osher, E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms.Physical,1992(60):259-268.
[50] W. Gibbs,Elementary Principles of Statistical Mechanics.Yale University Press, 1902
[51] T.E. Chan, S.H.Kang,J.Shen,Euler's elastica and curvature based inpaintings.April 2001 .SIAM J.AppI.Math.2002,63(2):564-592.
[52] Xiaowei Shao, Zhengkai Liu and Houqiang Li, An image inpainting approach based on the Poisson equation, in Proceedings of the Second International Conference on Document Image Analysis for Libraries , Lyon, France, April 27-28,2006.
[2] Coloma Ballester,M.Bertalmio and V.Caselles et al,Filling_in by joint interpolation of vector fields and gray levels. IEEE Transactions on image processing, Vol. 10, NO. 8, August 2001.
[3] M.Bertalmio,B.A.L.ertozzi and G. Sapiro,Navier-stokes, fluid dynamics, and image and video inpainting. Proc. Conf. Comp. Vision Pattern Rec.2001.355-362.
[4] T. F. Chan and J. Shen,Non-texture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation,2003( Vol. 12, No.4): 436-449.
[5] Shantanu, D.Rane,Guillermo Sapiro and Marcelo Bertalmio ,Structure and t exture filling-in of missing image blocks in wireless transmission and
compression applications". IEEE Transactions on image processing, Vol. 12, NO. 3, March 2003.
[6] Celia A. Zorzo Barcelos, Marcos A.Batista, Adriana M. Martins and Antonio Carlos Nogueira ,Level Lines Continuation based Digital Inpainting. Proceedings of the XVII Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'04).
[7] Simon Masnou and Jean-Michel Morel,Level lines based disocclusion. Proceedings of International Conference on Image Processing( ICIP 98), 1998 Page(s):259 - 263 Vol.3.
[8] Celia A. Zorzo Barcelos, Marcos A. Batista,Image inpainting and denoising by nonlinear partial differential equations. Proceedings of the XVI Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'03).
[9] D.A.Karras and G.B.Mertzios,Discretization schemes and numerical approximations of PDE inpainting models and comparative evaluation on novel real world MRI reconstruction applications.IEEE IST'2004.
[10] Alexandru Telea,An image inpainting technique based on the fast marching method. Journal of graphics tools, 2004(Vol. 9,No. 1): 25-36.
[11] T. Chan and J. Shen,Mathematical Models for Local Deterministic Inpaintings. Technical Report CAM 00-11, Image Processing Research Group, UCLA, 2000.
[12] Simon Masnou,Disocclusion-A Variational Approach. IEEE Transactions on image processing, Vol. 11, NO. 2, 2002.
[13] J.P. Gu,S.L. Peng and X.L. wang ,Digital image inpainting using monte carlo method.2004 International conference on Image Processing(ICIP):961-964.
[14] C. Ballester, M. Bertalmio,V. Caselles,G.Sapiro,and J.Verdera,A Variational Model for Filling-In. Proceedings of Eighth IEEE International Conference on Computer Vision( ICCV 2001) 2001:10 -16 vol.1.
[15] Selim Esedoglu and Jianhong Shen,Digital inpainting based on the Mumford-Shah-Euler image model. Euro. J. Applied Mathematics (2002), vol. 13,353-370.
[16] A. Criminisi, P. Perez, and K. Toyama,Region filling and object removal by exemplar based inpainting. IEEE Transactions on image processing,2004(Vol.13,No.9): 1200-1212.
[17] A. Criminisi, P. Perez and K. Toyama,Object Removal by Exemplar-Based Inpainting. Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03).
[18] Iddo Drori,Daniel Cohen-Or and Hezy Yeshurun,Fragment-based image completion. Proceedingsof ACM SIGGRAPH 2003:303-312.
[19] H. Igehy and L. Pereira,Image Replacement through Texture Synthesis. In IEEE International conference on Image Processing(1997), vol. 3,186-189
[20] S. Borikar,K.K.Biswas and S.Pattanaik.Yunjun Zhang, Jiangjian Xiao and Mubarak Shah,Region Completion in a Single Image. EUROGRAPHICS 2004.
[21] S.Borikar,K.K.Biswas and S.Pattanaik,Fast Algorithm for Completion of Images with Natural Scenes. http://graphics.cs.ucf.edu/.
[22] H.Chong,GPU image inpainting via Texture synthesis http://www.people.fas.harvard.edu/～hchong/goodies.
[23] Jian Sun,Lu Yuan,Jiaya Jia and Heung-Yeung shum,Image Completion with Structure Propagation. ACM T GRAPHIC 24 (3): 861-868 JUL 2005.
[24] Hitoshi Yamauchi, Jorg Haber and Hans-Peter Seidel,Image Restoration using Multiresolution Texture Synthesis and Image Inpainting. ProceedingsComputer Graphics International, 2003: 120 - 125 .
[25] M. Bertalmio, L. Vese and G. Sapiro et al,Simultaneous structure and texture image inpainting, IEEE Transactions on Volume 12, Issue 8, Aug. 2003: 882 -889
[26] A.Levin, A.Zomet, and Y.Weiss, Learning how to inpaint form global image statistics. In Proceedings of Inte. Conf. on Comp. Vision, Ⅱ.305-313.
[27]Raymond Chan, Lixin Shen and Zuowei Shen,A framelet-based approach for image inpainting. http://www.math.cuhk.edu.hk/-rchan/paper/inpainting/.
[28]Vladimir Savchenko, Nikita Kojekine and Hiroshi Unno,A practical image retouching method. Proceedings of the First International Symposium on Cyber Worlds (CW.02).
[29]Timothy K. Shih., Rong-Chi Chang and Liang-Chen Lu et al,Adaptive Digital Image Inpainting. Proceedings of the 18th International Conference on Advanced Information Networking and Application (AINA'04).
[30]Anil N. Hirani and Takashi Totsuka,Combining Frequency and Spatial Domain Information for Fast Interactive. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 1996, p 269-276.
[31]Shih-Chang Hsia,An edge-oriented spatial interpolation for consecutive block error concealment. IEEE Signal Processing Letters, Vol. 11, NO. 6, June 2004.
[32]M. M. Oliveira, B. Bowen, and R.McKennaetc,Fast Digital Image Inpainting. Proceedings of the International Conf. on Visualization, Imaging and Image Processing (VIIP2001). 2001:261-266.
[33]Ping Tan, Stephen Lin, Long Quan and Heung-Yeung Shum,Highlight removal by illumination-constrained inpainting. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV'03).
[34]Timothy shih,Liang-chen Lu,Ying-Hong Wang and Rong-chi Chang,Multi-resolution image inpainting. Proceedings of International Conference on Multimedia and Expo, (ICME'03) Vol.1: 485-488.
[35]周廷方、汤锋、王进等“基于径向基函数的图像修复技术”，中国图象图形学报，Vol．9，No．10，2004．
[36]吴振宇、王朔中，“基于区域分割的损伤图像修复”，上海大学学报(自然科学版)，Vol．5，No．1，1999．
[37]A. - Efros and T. Leung,Texture Synthesis by Nonparametric Sampling. Proceedings of the Seventh IEEE International Conference on Computer Vision. 1999:1033-1038.
[38]D. J. Heeger and J. R. Bergen,Pyramid-Based Texture Analysis/Synthesis. In Computer Graphics (SIGGRAPH ' 95 Conf. Proc.), pages 229-238, Aug. 1995.
[39]Kedar A. Patwardhan, Guillermo Sapiro, and Marcelo Bertalmio,Video Inpainting of Occluding and Occluding Objects" .In IEEE International Conference on Image Processing (ICIP 2005). Vol.2 :69 - 72
[40] J. A. Sethian.A Fast Marching Level Set Method for Monotonically Advancing Fronts. Proceedings of the National Academy of Sciences 93:4. USA, 1996: 1591—1595.
[41] J. A. Sethian,Adaptive Fast Marching and Level Set. Acta Math. Univ. Comenianae Vol. LXVII, 1(1998), pp. 3-15.
[42] Alexei A. Efros and William T. Freeman,Image Quilting for Texture Synthesis and Transfer. Proceedings of ACM SIGGRAPH 2001:241-246.
[43] Jianhong (Jackie) Shen, Inpainting and the Fundamental Problem of Image Processing. from Slam News, Volume 36, Number 5, June 2003.
[44] D. Mumford, J. Shan,Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems.Comm.Pure Applied.Math,1989,X LH:577-685
[45] S. Geman, D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images.IEEE Trans.Pattern Anal.M achineI ntell.1984(6):721-741.
[46] D.C. Knil, W. Richards,Perception as Bayesian Inference.Cambridge U niv.Press, 1996.
[47] S. Masnou, J .M. Morel.Level-lines Based Disocclusion. Proceedings of 5th IEEE Int'l Conf .on Image Process.Chicago,1998(3):259-263.
[48] L.Y. Wei, d M. Levoy. Fast Texture Synthesis Using Tree-structured Vector Quantization. Proceedings of SIGGRAPH 2000.20 00(6):479-488.
[49] L. Rudin, S. Osher, E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms.Physical,1992(60):259-268.
[50] W. Gibbs,Elementary Principles of Statistical Mechanics.Yale University Press, 1902
[51] T.E. Chan, S.H.Kang,J.Shen,Euler's elastica and curvature based inpaintings.April 2001 .SIAM J.AppI.Math.2002,63(2):564-592.
[52] Xiaowei Shao, Zhengkai Liu and Houqiang Li, An image inpainting approach based on the Poisson equation, in Proceedings of the Second International Conference on Document Image Analysis for Libraries , Lyon, France, April 27-28,2006.