基于样例的图像画质增强
|
摘要
随着各种数字媒体技术日渐发展,各种先进的媒体采集设备得到普及。无论是专业的动画,影视制作企业,还是数量众多的非专业用户,人们都希望能够对自己拍摄或者制作的媒体素材进行各种进一步编辑处理。这些用户在从效果和效率上根据不同的情况,产生了更多,更具体的优化需求,这对各种后期处理工具无疑是一个大挑战,其中之一就是图像的画质增强。随着高清电视,高清投影仪等显示设备的普及带来了大量高像素的屏幕,一般分辨率的图像需要一定的调整,才能够较好地显示在此类设备上。在一些网络速度或者存储容量受到限制的情况下,图像不得不经过压缩再进行传输。如果能够增强这些质量有限的媒体信息,这样的解码技术也是非常有市场前景的。另外相关的应用方向还有卫星成像,电子地图的缩放,医疗的辅助,电影的后期特效等等。所有这些相关的应用前景都对图像以及视频的画质像素增强,也就是超分辨率技术的发展产生了极大的促进。
本文受到样例纹理合成相关技术的启发,即使用一系列样例来辅助画质调整的过程。这些具有代表性的样例能够提供待处理的源图像之外的附加信息,样例图像的选定可以由用户指定,也直接根据源图像自似性从其本身采集。接下来,为了提升原图像的画质,需要从样例图像中挑选吻合相似性并且清晰度更高的纹理元素来匹配并替换原图中的区域。另外,样例图像之间的关系也可以不是等比例的,如果提供合适的层次与拓扑关系的样例序列,用这些信息来改进,就可以获得更加效果更好的后期处理效果。
本文主要利用自身或者额外的样例图来提升媒体素材的质量。与传统的方法相比,关键在于引入额外的高频信息。充分利用这些具有一定层次结构的样例信息,就能够进一步提升放大倍数,优化图像放大后的效果。在算法中结合了上采样与纹理合成的优点,并通过加入随机扰动来获得更好的合成效果。此外,本文还使用GPU并行加速算法关键步骤的运算过程,使用户能在较短的时间内获得优化结果。
Owing to the rapid development of digital media technology, advanced media capture devices have become more and more popular in daily life. Whether it is for Professional animation, film and television production, or a large number of normal users, people want to handle the media material they shot or created for further editing. Due to different situation,these clients offer more specific, optimization needs on effects and efficiency. This is undoubtedly a big challenge for post product tools, one of which is the image quality enhancement. While high-definition television, high-definition projectors and other display devices are bringing large amount of high-resolution screen, most general resolution image requires some adjustments in order to be better displayed on these devices. As for some cases, the network speed or storage capacity is limited, so images need to be compressed before transmission. If one can provide a method for enhancing the quality of the limited media information, such as decoding technology, it will have a very promising market. Other related aspects in application include satellite imaging, electronic map zoom, medical assistance, post-film special effects and so on. All of these applications related to prospects for image and video quality enhancement, which is also a great promotion for the super-resolution technology.
Inspired of some example-based synthesis technologies, this paper brings up an implement using a series of examples to aid quality adjustment process. These representative examples can be processed to provide additional information beyond the source image. All these example images can be specified by the user, but also directly acquired from the source input image itself on its own self-similarity. After that, for the purpose of enhancing the original image quality, we need to select patches from the examples which best match the image similarity and higher-resolution texture elements. And then, use these patches to replace the original image in certain regions. What’s more, the relationship between exemplars can be set to different scales. And if appropriate level structure and topology of the example are provide, you can get better post-treatment effect.
In this paper, we use the input image itself or additional images as example in order to improve the quality of media material. Compared with traditional methods, the key is to bring in additional high-frequency information. Take advantage of these hierarchical example information, we can further enhance magnification, optimizing effect of the image enlarged. In this algorithm, we combine the advantages of upsampling and texture synthesis by adding random jitter to obtain better synthesis results. In addition, this paper also accelerated main step by using the GPU parallel computing algorithm so that users can get optimize results in a very short time.
本文受到样例纹理合成相关技术的启发,即使用一系列样例来辅助画质调整的过程。这些具有代表性的样例能够提供待处理的源图像之外的附加信息,样例图像的选定可以由用户指定,也直接根据源图像自似性从其本身采集。接下来,为了提升原图像的画质,需要从样例图像中挑选吻合相似性并且清晰度更高的纹理元素来匹配并替换原图中的区域。另外,样例图像之间的关系也可以不是等比例的,如果提供合适的层次与拓扑关系的样例序列,用这些信息来改进,就可以获得更加效果更好的后期处理效果。
本文主要利用自身或者额外的样例图来提升媒体素材的质量。与传统的方法相比,关键在于引入额外的高频信息。充分利用这些具有一定层次结构的样例信息,就能够进一步提升放大倍数,优化图像放大后的效果。在算法中结合了上采样与纹理合成的优点,并通过加入随机扰动来获得更好的合成效果。此外,本文还使用GPU并行加速算法关键步骤的运算过程,使用户能在较短的时间内获得优化结果。
Owing to the rapid development of digital media technology, advanced media capture devices have become more and more popular in daily life. Whether it is for Professional animation, film and television production, or a large number of normal users, people want to handle the media material they shot or created for further editing. Due to different situation,these clients offer more specific, optimization needs on effects and efficiency. This is undoubtedly a big challenge for post product tools, one of which is the image quality enhancement. While high-definition television, high-definition projectors and other display devices are bringing large amount of high-resolution screen, most general resolution image requires some adjustments in order to be better displayed on these devices. As for some cases, the network speed or storage capacity is limited, so images need to be compressed before transmission. If one can provide a method for enhancing the quality of the limited media information, such as decoding technology, it will have a very promising market. Other related aspects in application include satellite imaging, electronic map zoom, medical assistance, post-film special effects and so on. All of these applications related to prospects for image and video quality enhancement, which is also a great promotion for the super-resolution technology.
Inspired of some example-based synthesis technologies, this paper brings up an implement using a series of examples to aid quality adjustment process. These representative examples can be processed to provide additional information beyond the source image. All these example images can be specified by the user, but also directly acquired from the source input image itself on its own self-similarity. After that, for the purpose of enhancing the original image quality, we need to select patches from the examples which best match the image similarity and higher-resolution texture elements. And then, use these patches to replace the original image in certain regions. What’s more, the relationship between exemplars can be set to different scales. And if appropriate level structure and topology of the example are provide, you can get better post-treatment effect.
In this paper, we use the input image itself or additional images as example in order to improve the quality of media material. Compared with traditional methods, the key is to bring in additional high-frequency information. Take advantage of these hierarchical example information, we can further enhance magnification, optimizing effect of the image enlarged. In this algorithm, we combine the advantages of upsampling and texture synthesis by adding random jitter to obtain better synthesis results. In addition, this paper also accelerated main step by using the GPU parallel computing algorithm so that users can get optimize results in a very short time.
引文
[1] http://www.vreveal.com
[2] Rafael C. Gonzalez, Richard E. Woods,阮秋奇,阮宇智等.数字图像处理(第二版)[M].北京:电子工业出版社,2007
[3] Tsai R Y, Huang T S. Multiple frame image restoration and registration. In: Advances in computer vision and image Processing, Greenwich: JAI Press, 1984, 317–339
[4] Sun, J. and Xu, Z. and Shum, H.Y. Image super-resolution using gradient profile prior [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2008: 1—8.
[5] Fattal, R. Image upsampling via imposed edge statistics [J]. ACM Transactions on Graphics, 2007, 26(3): 95.
[6] Freeman, W.T. and Jones, T.R. and Pasztor, E.C. Example-based super-resolution [J]. Computer Graphics and Applications, IEEE, 2002, 22(2): 56—65.
[7]童佳奇.彩色图像超分辨率技术的研究.中国优秀硕士学位论文全文数据库, 2010
[8] Barbara Zitova , Jan Flusser. Image registration methods: a survey. Image and Vision Computing Volume 21, Issue 11, October 2003, Pages 977-1000
[9] Irani M;Peleg S. Improving Resolution by Image Registration Graphical Models& Image Processing. 1991
[10] Irani M;Peleg S .Motion Analysis for Image Enhancement Resolution,Occlusion,and Transparency. Elsevier Journal Visual Communication& Image Representation.1993
[11] Ben-Ezra, M.,Zomet, A.,Nayar, S.K. et al.Video super-resolution using controlled subpixel detector shifts[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2005,27(6):977-987.
[12] P. Thvenaz, T. Blu, and M. Unser. Image interpolation and resampling. In I. Bankman, editor, Handbook of Medical Imaging, Processing and Analysis, pages 393–420, San Diego CA, USA, 2000. Academic Press. 1, 2
[13] X. Li and M. T. Orchard. New edge-directed interpolation. IEEE Trans. Image Processing, 10(10):1521–1527, 2001. 2
[14] D. Su and P. Willis. Image interpolation by pixel-level data-dependent triangulation. Computer Graphics Forum,23(2):189–202, 2004. 2
[15] J. Sun, N.-N. Zheng, H. Tao, and H.-Y. Shum. Image hallucination with primal sketch priors. InProc. CVPR 2003,volume 2, pages II–729–36 vol.2, June 2003. 1, 2
[16] W. T. Freeman and E. C. Pasztor. Learning to estimate scenes from images. In Proc. of conf. on Advances in neural information processing systems II, pages 775–781, Cambridge, MA, USA, 1999. MIT Press. 3
[17] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Proc. SIGGRAPH’01, pages 327–340, New York, NY, USA, 2001. ACM. 3
[18] M. F. Tappen, B. C. Russell, and W. T. Freeman. Efficient graphical models for processing images. In Proc. CVPR 2004, pages 673–680, 2004. 3
[19] M. Ebrahimi and E. R. Vrscay. Solving the inverse problem of image zooming using self-examples. In Proc. ICIAR 2007, volume 4633 of Lecture Notes in Computer Science, pages 117–130. Springer, 2007. 2, 3, 6
[20] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proc. ICCV, 2009. 3, 6, 7
[21] S. Baker and T. Kanade. Hallucinating faces. In FG’00: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, page 83, Washington, DC, USA, 2000. IEEE Computer Society. 3
[22] C. Liu, H.-Y. Shum, and W. T. Freeman. Face hallucination: Theory and practice. Int. J. Comput. Vision, 75(1):115–134, 2007. 3
[23] L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk. State of the art in example-based texture synthesis. Eurographics 2009 State of The Art Report, April 2009. 2, 3
[24] D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. In Proc. SIGGRAPH’95, pages 229–238, New York, NY, USA, 1995. ACM. 3
[25] J. Portilla and E. P. Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision, 40(1):49–70, 2000. 3
[26] A. A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In Proc. ICCV’99, volume 2, pages 1033–1038, 1999. 3
[27] L.-Y. Wei and M. Levoy. Fast texture synthesis using treestructured vector quantization. In Proc. SIGGRAPH’00,pages 479–488, New York, NY, USA, 2000. 3
[28] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proc. SIGGRAPH’01, pages 341–346, New York, NY, USA, 2001. ACM. 3
[29] V. Kwatra, A. Sch¨odl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph., 22(3):277–286, 2003. 3
[30] V. Kwatra, I. Essa, A. Bobick, and N. Kwatra. Texture optimization for example-based synthesis. ACM Trans. Graph., 24(3):795–802, 2005. 2, 3, 5, 6, 7
[31] C. Han, E. Risser, R. Ramamoorthi, and E. Grinspun,―Multiscale texture synthesis,‖ACM Trans. Graph., vol. 27, pp. 51:1–51:8, August 2008.
[32] Y. HaCohen, R. Fattal and D. Lischinski,Image Upsampling via Texture Hallucination,IEEE International Conference on Computational Photography (ICCP 2010)
[33] David B. Kirk Wen-mei W. Hwu,陈曙晖,熊淑华.大规模并行处理器编程实战.北京:清华大学出版社.2010.9
[34] http://www.nvidia.com/object/cuda_home_new.html
[35] LIANG, L., LIU, C., XU, Y., GUO, B., AND SHUM, H.-Y. 2001. Real-time texture synthesis by patch-based sampling. ACM TOG 20(3), 127-150.
[36] S. Lefebvre and H. Hoppe,―Parallel controllable texture synthesis,‖ACM Transactions on Graphics (TOG), vol. 24, no. 3, pp. 777–786, 2005
[37] TONG, X., ZHANG, J., LIU, L., WANG, X., GUO, B., AND SHUM, H.-Y. 2002. Synthesis of bidirectional texture functions on arbitrary surfaces. In SIGGRAPH, 665–672.
[38] ZELINKA, S., AND GARLAND, M. 2002. Towards real-time texture synthesis with the jump map. In EGWR, 99–104.
[2] Rafael C. Gonzalez, Richard E. Woods,阮秋奇,阮宇智等.数字图像处理(第二版)[M].北京:电子工业出版社,2007
[3] Tsai R Y, Huang T S. Multiple frame image restoration and registration. In: Advances in computer vision and image Processing, Greenwich: JAI Press, 1984, 317–339
[4] Sun, J. and Xu, Z. and Shum, H.Y. Image super-resolution using gradient profile prior [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2008: 1—8.
[5] Fattal, R. Image upsampling via imposed edge statistics [J]. ACM Transactions on Graphics, 2007, 26(3): 95.
[6] Freeman, W.T. and Jones, T.R. and Pasztor, E.C. Example-based super-resolution [J]. Computer Graphics and Applications, IEEE, 2002, 22(2): 56—65.
[7]童佳奇.彩色图像超分辨率技术的研究.中国优秀硕士学位论文全文数据库, 2010
[8] Barbara Zitova , Jan Flusser. Image registration methods: a survey. Image and Vision Computing Volume 21, Issue 11, October 2003, Pages 977-1000
[9] Irani M;Peleg S. Improving Resolution by Image Registration Graphical Models& Image Processing. 1991
[10] Irani M;Peleg S .Motion Analysis for Image Enhancement Resolution,Occlusion,and Transparency. Elsevier Journal Visual Communication& Image Representation.1993
[11] Ben-Ezra, M.,Zomet, A.,Nayar, S.K. et al.Video super-resolution using controlled subpixel detector shifts[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2005,27(6):977-987.
[12] P. Thvenaz, T. Blu, and M. Unser. Image interpolation and resampling. In I. Bankman, editor, Handbook of Medical Imaging, Processing and Analysis, pages 393–420, San Diego CA, USA, 2000. Academic Press. 1, 2
[13] X. Li and M. T. Orchard. New edge-directed interpolation. IEEE Trans. Image Processing, 10(10):1521–1527, 2001. 2
[14] D. Su and P. Willis. Image interpolation by pixel-level data-dependent triangulation. Computer Graphics Forum,23(2):189–202, 2004. 2
[15] J. Sun, N.-N. Zheng, H. Tao, and H.-Y. Shum. Image hallucination with primal sketch priors. InProc. CVPR 2003,volume 2, pages II–729–36 vol.2, June 2003. 1, 2
[16] W. T. Freeman and E. C. Pasztor. Learning to estimate scenes from images. In Proc. of conf. on Advances in neural information processing systems II, pages 775–781, Cambridge, MA, USA, 1999. MIT Press. 3
[17] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Proc. SIGGRAPH’01, pages 327–340, New York, NY, USA, 2001. ACM. 3
[18] M. F. Tappen, B. C. Russell, and W. T. Freeman. Efficient graphical models for processing images. In Proc. CVPR 2004, pages 673–680, 2004. 3
[19] M. Ebrahimi and E. R. Vrscay. Solving the inverse problem of image zooming using self-examples. In Proc. ICIAR 2007, volume 4633 of Lecture Notes in Computer Science, pages 117–130. Springer, 2007. 2, 3, 6
[20] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proc. ICCV, 2009. 3, 6, 7
[21] S. Baker and T. Kanade. Hallucinating faces. In FG’00: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, page 83, Washington, DC, USA, 2000. IEEE Computer Society. 3
[22] C. Liu, H.-Y. Shum, and W. T. Freeman. Face hallucination: Theory and practice. Int. J. Comput. Vision, 75(1):115–134, 2007. 3
[23] L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk. State of the art in example-based texture synthesis. Eurographics 2009 State of The Art Report, April 2009. 2, 3
[24] D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. In Proc. SIGGRAPH’95, pages 229–238, New York, NY, USA, 1995. ACM. 3
[25] J. Portilla and E. P. Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision, 40(1):49–70, 2000. 3
[26] A. A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In Proc. ICCV’99, volume 2, pages 1033–1038, 1999. 3
[27] L.-Y. Wei and M. Levoy. Fast texture synthesis using treestructured vector quantization. In Proc. SIGGRAPH’00,pages 479–488, New York, NY, USA, 2000. 3
[28] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proc. SIGGRAPH’01, pages 341–346, New York, NY, USA, 2001. ACM. 3
[29] V. Kwatra, A. Sch¨odl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph., 22(3):277–286, 2003. 3
[30] V. Kwatra, I. Essa, A. Bobick, and N. Kwatra. Texture optimization for example-based synthesis. ACM Trans. Graph., 24(3):795–802, 2005. 2, 3, 5, 6, 7
[31] C. Han, E. Risser, R. Ramamoorthi, and E. Grinspun,―Multiscale texture synthesis,‖ACM Trans. Graph., vol. 27, pp. 51:1–51:8, August 2008.
[32] Y. HaCohen, R. Fattal and D. Lischinski,Image Upsampling via Texture Hallucination,IEEE International Conference on Computational Photography (ICCP 2010)
[33] David B. Kirk Wen-mei W. Hwu,陈曙晖,熊淑华.大规模并行处理器编程实战.北京:清华大学出版社.2010.9
[34] http://www.nvidia.com/object/cuda_home_new.html
[35] LIANG, L., LIU, C., XU, Y., GUO, B., AND SHUM, H.-Y. 2001. Real-time texture synthesis by patch-based sampling. ACM TOG 20(3), 127-150.
[36] S. Lefebvre and H. Hoppe,―Parallel controllable texture synthesis,‖ACM Transactions on Graphics (TOG), vol. 24, no. 3, pp. 777–786, 2005
[37] TONG, X., ZHANG, J., LIU, L., WANG, X., GUO, B., AND SHUM, H.-Y. 2002. Synthesis of bidirectional texture functions on arbitrary surfaces. In SIGGRAPH, 665–672.
[38] ZELINKA, S., AND GARLAND, M. 2002. Towards real-time texture synthesis with the jump map. In EGWR, 99–104.