视频图像艺术化处理的研究与实现
|
摘要
视频图像艺术化处理作为非真实感绘制的一个主要研究方面,有很大的研究前景与很丰富的研究内容。而近年来,视频图像的艺术化绘制的研究内容主要包括:1.提高视频图像艺术化的效果2.提高视频处理的实时性3.降低实现的算法复杂度等方面。
而本文主要实现了一个视频艺术化处理系统,该系统是为了能够减少艺术家们在艺术绘画领域,复杂而又繁重的工作量而提出的,现实意义比较明显。该系统是在个人PC机上完成的。其中,由于DirectShow在视频流媒体处理有很大的优势,因此我们的视频艺术化处理过程主要是利用DirectShow来对视频进行处理的,同时还结合数字图像处理的方法,共同来完成艺术化效果的。其优点主要在算法方面简单可行,艺术效果和实时性方面基本能满足要求。在视频处理的过程中,我们所做的工作主要有:视频帧的提取,对图像的噪声进行平滑,通过边缘检测实现图像的轮廓增强,利用区域颜色的均一化处理和填充实现卡通效果,利用各种卷积操作完成浮雕、铅笔画等相关处理,用指针和图像的存储信息实现马赛克的效果,利用DirectX中的纹理来实现帧图像的3D显示等。而在图像的卡通处理中,主要基于图像的特征空间转换、双线性滤波、边缘检测、图像的伪量化处理以及图像的边缘变形等一系列流程所实现其卡通的效果。最后把图像的卡通与视频的艺术化集成在一起,实现了一套基于视频播放、摄像头捕捉、视频与图像艺术化处理的系统,并完成了整体框架的设计、算法设计与代码实现等。
The processing of video artistic style, as a major research of a non-photorealistic rendering, has a great prospect and a very extensive study of research content. In recent years, Researching content of the artistic drawing of video and images include: 1. To improve the artistic effect of video and images. 2. To improve the real-timing in video processing 3. Reduce the algorithm complexity and so on.
The dissertation had accomplished a system of video artistic style, which is to reduce the artists complex and heavy workload in the artistic painting filed, and its practical significance is more obvious. The system has been done in a personal PC and the video artistic process makes use of DirectShow to process, which has a great advantage to deal with streaming media and it combined with digital image processing methods to achieve artistic effects together. The main advantage of the algorithm is simple and feasible, artistic effects and real-timing can satisfy the requirements. In the video processing, my works include: extraction of video frames, noise smoothing of the frame image, the outline of image enhancement by edge detection, making use of regional color uniformity processing and filling to achieve cartoon effect, making use of convolution operation is complete relief, pencil and so on, with the pointer and image storage information to achieve the effect of mosaic, using the texture in DirectX to achieve the 3D display of the frames. In the image cartoon processing, the thesis mainly achieve its cartoon effect by way of a series of processes which included image feature space conversion, bilinear filtering, edge detection, image pseudo-quantitative processing and image edge distortion and so on. Finally the dissertation has integrated of the image cartoon and video artistic processing to realize a video-playing, camera capturing, a processing system of video artistic style, and accomplished the design of the overall framework, algorithm design and code realization.
而本文主要实现了一个视频艺术化处理系统,该系统是为了能够减少艺术家们在艺术绘画领域,复杂而又繁重的工作量而提出的,现实意义比较明显。该系统是在个人PC机上完成的。其中,由于DirectShow在视频流媒体处理有很大的优势,因此我们的视频艺术化处理过程主要是利用DirectShow来对视频进行处理的,同时还结合数字图像处理的方法,共同来完成艺术化效果的。其优点主要在算法方面简单可行,艺术效果和实时性方面基本能满足要求。在视频处理的过程中,我们所做的工作主要有:视频帧的提取,对图像的噪声进行平滑,通过边缘检测实现图像的轮廓增强,利用区域颜色的均一化处理和填充实现卡通效果,利用各种卷积操作完成浮雕、铅笔画等相关处理,用指针和图像的存储信息实现马赛克的效果,利用DirectX中的纹理来实现帧图像的3D显示等。而在图像的卡通处理中,主要基于图像的特征空间转换、双线性滤波、边缘检测、图像的伪量化处理以及图像的边缘变形等一系列流程所实现其卡通的效果。最后把图像的卡通与视频的艺术化集成在一起,实现了一套基于视频播放、摄像头捕捉、视频与图像艺术化处理的系统,并完成了整体框架的设计、算法设计与代码实现等。
The processing of video artistic style, as a major research of a non-photorealistic rendering, has a great prospect and a very extensive study of research content. In recent years, Researching content of the artistic drawing of video and images include: 1. To improve the artistic effect of video and images. 2. To improve the real-timing in video processing 3. Reduce the algorithm complexity and so on.
The dissertation had accomplished a system of video artistic style, which is to reduce the artists complex and heavy workload in the artistic painting filed, and its practical significance is more obvious. The system has been done in a personal PC and the video artistic process makes use of DirectShow to process, which has a great advantage to deal with streaming media and it combined with digital image processing methods to achieve artistic effects together. The main advantage of the algorithm is simple and feasible, artistic effects and real-timing can satisfy the requirements. In the video processing, my works include: extraction of video frames, noise smoothing of the frame image, the outline of image enhancement by edge detection, making use of regional color uniformity processing and filling to achieve cartoon effect, making use of convolution operation is complete relief, pencil and so on, with the pointer and image storage information to achieve the effect of mosaic, using the texture in DirectX to achieve the 3D display of the frames. In the image cartoon processing, the thesis mainly achieve its cartoon effect by way of a series of processes which included image feature space conversion, bilinear filtering, edge detection, image pseudo-quantitative processing and image edge distortion and so on. Finally the dissertation has integrated of the image cartoon and video artistic processing to realize a video-playing, camera capturing, a processing system of video artistic style, and accomplished the design of the overall framework, algorithm design and code realization.
引文
[1]杨建平.数字视频技术及应用(高等院校信息与电子技术类规划教材).科学出版社.
[2]叶修梓,万华根,张引.非真实感图形学——造型、绘制与动画技术.电子工业出版社.
[3]何斌,马天予. Visual C++数字图像处理.人民邮电出版社.
[4]李盈,宋瀚涛,高利民.卡通渲染中边缘着墨技术原理研究.信息技术. 2006年第1期.
[5]陆其明. DirectShow开发指南.清华大学出版社.
[6]王洁宁.基于图形硬件加速的非真实感图形实时绘制技术.工程图学学报. 2005年第4期.
[7]钱小燕,肖亮,吴慧中.自适应画笔分布的非真实感绘制.工程图学学报. 2006年第2期.
[8]周亮.点画实现及其多样化----艺术风格图像渲染.大连理工大学硕士学位论文. 2004.3..
[9] Saito,Takahashi. Comprehensible rendering of 3D shapes. ACM SIGGRAPH 90, 197–206.
[10] Raskar. Nonphotorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM Trans. Graph. 23, 3, 679–688. 2004.
[11] Wang J, Xu Y., Cohen, M. F. Video tooning. ACM Trans. Graph. 23, 3, 574–583. 2004.
[12] Decarlo. D, Santella.A. Stylization and abstraction of photographs. ACM Trans::Graph. 21, 3, 769–776. 2002.
[13] Fischer.J, Bartz.D, Strasser.W. Stylized Augmented Reality for Improved Immersion. In Proc. of IEEE VR, 195–202. 2005.
[14] Gooch.B, Reinhard.E. Human facial illustrations:Creation and psychophysical evaluation. ACM Trans. Graph.23, 1, 27–44. 2004.
[15] P. Litwinowicz. Processing images and video for an impressionist effect. Proceedings of SIGGRAPH 97, pp407-414, 1997.
[16] A. Hertzmann. Painterly Rendering with Curved Brush Strokes of Multiple Sizes.
[17] A. Hertzmann, K. Perlin. Painterly rendering for video and interaction. In NPAR 2000, pp7-12, 2000.
[18] D. Decarlo, A. Santella. Stylization and abstraction of photographs. ACM Transactions on Graphics 21, pp769-776, 2002.
[19] A. Klein, P Sloan, A. Finkelstein, and M. Cohen. Stylized video cubes. ACM SIGGRAPH Symposium on Computer Animation, pp15-22, 2002.
[20] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, D.H. Salesin. Image analogies. Proceedings of SIGGRAPH 2001, pp327-340, 2001.
[21] A. Hertzmann. Fast paint texture. Proceedings of NPAR 2002, pp91-96, 2002.
[22]钱小燕.艺术风格图像的非真实感绘制理论与方法研究.南京理工大学博士学位论文.2007年1月.
[23]苏延辉,韦欢,费广正,石民勇.非真实感绘制技术研究.中国传煤大学学报自然科学版. 2006年6月第13卷第2期.
[24]闫文.基于GPU实时非真实感渲染的研究与实现.浙江大学计算机学院硕士论文. 2006.
[25]余守秋.基于GPU的实时视频艺术风格化系统.大连理工大学计算机学院硕士论文. 2005.
[26]王建中,姜昱明.基于Snake模型的视频对象分割和跟踪算法.计算机工程与设计. 2005年7月第26卷第7期.
[27]周心明,兰赛,徐燕.几种边缘检测的比较.现代电力. 2000年8月第17卷第3期.
[28]曹宁.基于跟踪算法的非真实感动画生成方法.计算机工程与应用. 2003.
[29] Palmer. E. Vision Science: Photons to Phenomenology. The MIT Press. 1999.
[30] Privitera. M., Stark. W. Algorithms for defining visual regions-of-interest: Comparison with eye fixations. IEEE Trans.on Pattern Analysis and Machine Intelligence 22, 9, 970–982. 2000.
[31] Itti.L, Koch.C. Computational modeling of visual attention.Nature Reviews Neuroscience 2, 3, 194–203. 2001.
[32] Tomasi.C, Manduchi.R. Bilateral filtering for gray and color images. In Proceedings of ICCV’98, 839. 1998.
[33] Wyszecki.G, Styles.W. Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley, New York, NY. 1982.
[34] Perona.P, Malik.J.. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 12, 7. 1991.
[35] Barash.D,Comaniciu.D. A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift.Image and Video Computing 22, 1, 73–81.2004.
[36] Marr.D, Hildreth.E. Theory of edge detection. Proc. Royal Soc. London, Bio. Sci. 207, 187–217.1980.
[37] Arad.N,Gotsman.C. Enhancement by image-dependent warping. IEEE Trans. on Image Processing 8, 9, 1063–1074. 1999.
[38] Loviscach.J. Scharfzeichner: Klare bilddetails durch verformung. Computer Technik 22, 236ff.1999.
[39]潘磊,吴小俊,尤媛媛.基于聚类的视频镜头分割和关键帧提取.红外与激光工程.2005年6月第34卷第3期.
[40]郭晓永,陈克林.基于图像的铅笔画绘制技术.重庆工商大学学报. 2009.6.
[41]郭纳.图像的卡通绘制技术.青岛大学研究生硕士学位论文. 2009.6.
[42] J. Hays, I. Essa, Image and video based painterly animation, Proceedings of NPAR 2004, pp113-120, 2004.
[43] Ashley M.Eden, Adam W. Bargteil, Tolga G. Goktekin, Sarah Beth Eisinger. A Method for Cartoon-Style Rendering of Liquid Animations. Graphics Interface 2007.
[44] Won-Il Hwang. CINEMA COMICS: CARTOON GENERATION FROM VIDEO STREAM .Dept. of Computer Science and Engineering, Pusan National University, San-30, Jangjeon-dong, Keumjeong-gu, PUSAN, 609-735, KOREA.
[45] Danijela Markovic, Margrit Gelautz. Comics-Like Motion Depiction from Stereo. Institute for Software Technology and Interactive Systems, Vienna University of Technology .Favoritenstrasse 9-11/188/2, A-1040 Vienna, Austria.
[46] Kalyan Sunkavalli, Wojciech Matusik, Hanspeter Pfister, Szymon Rusinkiewicz. Factored Time-Lapse Video.
[47] Byungmoon Kim, Irfan Essa. Video-based Nonphotorealistic and Expressive Illustration of Motion. GVU Center and College of Computing, Georgia Institute of Technology.
[48] Christian Richardt. Video abstraction & stylization. University of Cambridge computer lab.
[49] Aseem Agarwala. SnakeToonz : A Semi-Automatic Approach to Creating Cel Animation from Video. Starlab NV.
[50] Christoph Bregler. Turning to the Masters: Motion Capturing Cartoons. Stanford University.
[51] J. P. Collomosse, D. Rowntree, P. M. Hall. Cartoon-style Rendering of Motion from Video. Video and Graphics (2003) P. Hall, P. Willis (Editors).
[52] Simon M. Danner, Christoph J. Winklhofer. Cartoon Style Rendering.
[53] Doug DeCarlo, Anthony Santella. Stylization and Abstraction of Photographs. In SIGGRAPH 2002. Department of Computer Science & Center for Cognitive Science . Rutgers University.
[54] James Hays, Irfan Essa. Image and Video Based Painterly Animation.
[2]叶修梓,万华根,张引.非真实感图形学——造型、绘制与动画技术.电子工业出版社.
[3]何斌,马天予. Visual C++数字图像处理.人民邮电出版社.
[4]李盈,宋瀚涛,高利民.卡通渲染中边缘着墨技术原理研究.信息技术. 2006年第1期.
[5]陆其明. DirectShow开发指南.清华大学出版社.
[6]王洁宁.基于图形硬件加速的非真实感图形实时绘制技术.工程图学学报. 2005年第4期.
[7]钱小燕,肖亮,吴慧中.自适应画笔分布的非真实感绘制.工程图学学报. 2006年第2期.
[8]周亮.点画实现及其多样化----艺术风格图像渲染.大连理工大学硕士学位论文. 2004.3..
[9] Saito,Takahashi. Comprehensible rendering of 3D shapes. ACM SIGGRAPH 90, 197–206.
[10] Raskar. Nonphotorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM Trans. Graph. 23, 3, 679–688. 2004.
[11] Wang J, Xu Y., Cohen, M. F. Video tooning. ACM Trans. Graph. 23, 3, 574–583. 2004.
[12] Decarlo. D, Santella.A. Stylization and abstraction of photographs. ACM Trans::Graph. 21, 3, 769–776. 2002.
[13] Fischer.J, Bartz.D, Strasser.W. Stylized Augmented Reality for Improved Immersion. In Proc. of IEEE VR, 195–202. 2005.
[14] Gooch.B, Reinhard.E. Human facial illustrations:Creation and psychophysical evaluation. ACM Trans. Graph.23, 1, 27–44. 2004.
[15] P. Litwinowicz. Processing images and video for an impressionist effect. Proceedings of SIGGRAPH 97, pp407-414, 1997.
[16] A. Hertzmann. Painterly Rendering with Curved Brush Strokes of Multiple Sizes.
[17] A. Hertzmann, K. Perlin. Painterly rendering for video and interaction. In NPAR 2000, pp7-12, 2000.
[18] D. Decarlo, A. Santella. Stylization and abstraction of photographs. ACM Transactions on Graphics 21, pp769-776, 2002.
[19] A. Klein, P Sloan, A. Finkelstein, and M. Cohen. Stylized video cubes. ACM SIGGRAPH Symposium on Computer Animation, pp15-22, 2002.
[20] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, D.H. Salesin. Image analogies. Proceedings of SIGGRAPH 2001, pp327-340, 2001.
[21] A. Hertzmann. Fast paint texture. Proceedings of NPAR 2002, pp91-96, 2002.
[22]钱小燕.艺术风格图像的非真实感绘制理论与方法研究.南京理工大学博士学位论文.2007年1月.
[23]苏延辉,韦欢,费广正,石民勇.非真实感绘制技术研究.中国传煤大学学报自然科学版. 2006年6月第13卷第2期.
[24]闫文.基于GPU实时非真实感渲染的研究与实现.浙江大学计算机学院硕士论文. 2006.
[25]余守秋.基于GPU的实时视频艺术风格化系统.大连理工大学计算机学院硕士论文. 2005.
[26]王建中,姜昱明.基于Snake模型的视频对象分割和跟踪算法.计算机工程与设计. 2005年7月第26卷第7期.
[27]周心明,兰赛,徐燕.几种边缘检测的比较.现代电力. 2000年8月第17卷第3期.
[28]曹宁.基于跟踪算法的非真实感动画生成方法.计算机工程与应用. 2003.
[29] Palmer. E. Vision Science: Photons to Phenomenology. The MIT Press. 1999.
[30] Privitera. M., Stark. W. Algorithms for defining visual regions-of-interest: Comparison with eye fixations. IEEE Trans.on Pattern Analysis and Machine Intelligence 22, 9, 970–982. 2000.
[31] Itti.L, Koch.C. Computational modeling of visual attention.Nature Reviews Neuroscience 2, 3, 194–203. 2001.
[32] Tomasi.C, Manduchi.R. Bilateral filtering for gray and color images. In Proceedings of ICCV’98, 839. 1998.
[33] Wyszecki.G, Styles.W. Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley, New York, NY. 1982.
[34] Perona.P, Malik.J.. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 12, 7. 1991.
[35] Barash.D,Comaniciu.D. A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift.Image and Video Computing 22, 1, 73–81.2004.
[36] Marr.D, Hildreth.E. Theory of edge detection. Proc. Royal Soc. London, Bio. Sci. 207, 187–217.1980.
[37] Arad.N,Gotsman.C. Enhancement by image-dependent warping. IEEE Trans. on Image Processing 8, 9, 1063–1074. 1999.
[38] Loviscach.J. Scharfzeichner: Klare bilddetails durch verformung. Computer Technik 22, 236ff.1999.
[39]潘磊,吴小俊,尤媛媛.基于聚类的视频镜头分割和关键帧提取.红外与激光工程.2005年6月第34卷第3期.
[40]郭晓永,陈克林.基于图像的铅笔画绘制技术.重庆工商大学学报. 2009.6.
[41]郭纳.图像的卡通绘制技术.青岛大学研究生硕士学位论文. 2009.6.
[42] J. Hays, I. Essa, Image and video based painterly animation, Proceedings of NPAR 2004, pp113-120, 2004.
[43] Ashley M.Eden, Adam W. Bargteil, Tolga G. Goktekin, Sarah Beth Eisinger. A Method for Cartoon-Style Rendering of Liquid Animations. Graphics Interface 2007.
[44] Won-Il Hwang. CINEMA COMICS: CARTOON GENERATION FROM VIDEO STREAM .Dept. of Computer Science and Engineering, Pusan National University, San-30, Jangjeon-dong, Keumjeong-gu, PUSAN, 609-735, KOREA.
[45] Danijela Markovic, Margrit Gelautz. Comics-Like Motion Depiction from Stereo. Institute for Software Technology and Interactive Systems, Vienna University of Technology .Favoritenstrasse 9-11/188/2, A-1040 Vienna, Austria.
[46] Kalyan Sunkavalli, Wojciech Matusik, Hanspeter Pfister, Szymon Rusinkiewicz. Factored Time-Lapse Video.
[47] Byungmoon Kim, Irfan Essa. Video-based Nonphotorealistic and Expressive Illustration of Motion. GVU Center and College of Computing, Georgia Institute of Technology.
[48] Christian Richardt. Video abstraction & stylization. University of Cambridge computer lab.
[49] Aseem Agarwala. SnakeToonz : A Semi-Automatic Approach to Creating Cel Animation from Video. Starlab NV.
[50] Christoph Bregler. Turning to the Masters: Motion Capturing Cartoons. Stanford University.
[51] J. P. Collomosse, D. Rowntree, P. M. Hall. Cartoon-style Rendering of Motion from Video. Video and Graphics (2003) P. Hall, P. Willis (Editors).
[52] Simon M. Danner, Christoph J. Winklhofer. Cartoon Style Rendering.
[53] Doug DeCarlo, Anthony Santella. Stylization and Abstraction of Photographs. In SIGGRAPH 2002. Department of Computer Science & Center for Cognitive Science . Rutgers University.
[54] James Hays, Irfan Essa. Image and Video Based Painterly Animation.