基于控制点平滑的人脸变形算法及其在人脸动画中的应用
|
摘要
自计算机产生伊始,对计算机图形和动画的研究就没有停止过。随着影视业在人们生活中发挥着越来越重要的作用,计算机图形学和动画技术在过去的30个年头里得到了惊人的发展。与此同时,21世纪以来多媒体产业迅速兴起,数字娱乐方兴未艾。在手机、PC机上的计算机卡通动画娱乐深得人们的喜爱。其中,图像变形技术在计算机动画中发挥着举足轻重的作用。
本文基于面向数字娱乐的卡通人脸动画的课题背景,对人脸变形技术展开了研究,提出了一种快速、变形效果好的二维人脸变形算法,并将其应用在卡通人脸肖像生成和人脸动画的生成中,获得了很好的效果。
(1)提出了基于控制点平滑的二维人脸变形算法。它基于两种常用的图像变形算法:网格变形和薄板样条变形。其中网格变形由网格扭曲算法修改而来。仅采用网格变形方法达不到人脸局部变形的需求,而仅采用薄板样条方法在速度上也无法在实际中应用。在网格变形之前加入一个以薄板样条变形为原理的特征点平滑过程,使得图像变形的结果既远比网格变形的结果细腻自然又具有较快的速度。实验表明,平滑算法可以得到理想效果。
(2)针对卡通人脸变形中常用的局部变形,修改平滑算法,使其在局部变形上产生更加逼真的效果。将变形部位的控制点集合作为薄板样条变换部分的控制点子集,既能准确地达到局部的变形效果,又能反应出变形部位对周围区域的影响。
(3)将平滑变形应用与卡通人脸肖像生成和人脸动画的生成。在基于特征变换生成的卡通人脸肖像中,卡通肖像的形状确定好之后,使用平滑变形算法来进行纹理的变换。在文本和语音联合驱动的人脸动画中,平滑变形在关键帧插值得到中间帧的过程中对动画的逼真效果起到了决定性的作用。平滑算法也为带有表情的说话人头的实现奠定了基础。
The research on computer graphics and animation has never been halted. On contrary, as the show business plays a more and more important role in people’s daily life, the techniques of computer graphics and animation have been developed rapidly. At the same time, the multimedia techniques are rising, and the computer cartoon animation is popular around the world. Computer warping technique has played a very important role in this.
In our paper, we proposed a rapid and well-performed warping algorithm base the project of cartoon human face animation. The algorithm can be applied into the generation of cartoon portrait and the face animation.
(1) We proposed a new algorithm in 2D face warping based on the smoothness of the control points. The algorithm combined two common image warping methods with the characteristic of human face. Firstly it had the distribution of the feature points smoothed according to the principle of Thin-Plate Spline (TPS). Then the mesh warping was applied. The outcomes had highly encouraging effects and took little time. And the experiments results showed that it performed very well.
(2) We modified the algorithm to adopt the local warping. The control points of the warping parts were set as the subset of the control points in the Thin-Plate Spline transform. Then it not only make the warping parts transformed well, but also simulated the effects of surrounding areas of the warping parts.
(3) Applied the smoothness algorithm to the cartoon portraits generation and the face animation. In the cartoon portraits generation based on characteristic transform, the algorithm can be used to warp the texture as soon as the shape was determined. In the face animation, the key frames were interpolated to acquire the middle frames, and the smoothness warping algorithm worked here to make the animation vivid. It played a more important role in the emotional talking head.
本文基于面向数字娱乐的卡通人脸动画的课题背景,对人脸变形技术展开了研究,提出了一种快速、变形效果好的二维人脸变形算法,并将其应用在卡通人脸肖像生成和人脸动画的生成中,获得了很好的效果。
(1)提出了基于控制点平滑的二维人脸变形算法。它基于两种常用的图像变形算法:网格变形和薄板样条变形。其中网格变形由网格扭曲算法修改而来。仅采用网格变形方法达不到人脸局部变形的需求,而仅采用薄板样条方法在速度上也无法在实际中应用。在网格变形之前加入一个以薄板样条变形为原理的特征点平滑过程,使得图像变形的结果既远比网格变形的结果细腻自然又具有较快的速度。实验表明,平滑算法可以得到理想效果。
(2)针对卡通人脸变形中常用的局部变形,修改平滑算法,使其在局部变形上产生更加逼真的效果。将变形部位的控制点集合作为薄板样条变换部分的控制点子集,既能准确地达到局部的变形效果,又能反应出变形部位对周围区域的影响。
(3)将平滑变形应用与卡通人脸肖像生成和人脸动画的生成。在基于特征变换生成的卡通人脸肖像中,卡通肖像的形状确定好之后,使用平滑变形算法来进行纹理的变换。在文本和语音联合驱动的人脸动画中,平滑变形在关键帧插值得到中间帧的过程中对动画的逼真效果起到了决定性的作用。平滑算法也为带有表情的说话人头的实现奠定了基础。
The research on computer graphics and animation has never been halted. On contrary, as the show business plays a more and more important role in people’s daily life, the techniques of computer graphics and animation have been developed rapidly. At the same time, the multimedia techniques are rising, and the computer cartoon animation is popular around the world. Computer warping technique has played a very important role in this.
In our paper, we proposed a rapid and well-performed warping algorithm base the project of cartoon human face animation. The algorithm can be applied into the generation of cartoon portrait and the face animation.
(1) We proposed a new algorithm in 2D face warping based on the smoothness of the control points. The algorithm combined two common image warping methods with the characteristic of human face. Firstly it had the distribution of the feature points smoothed according to the principle of Thin-Plate Spline (TPS). Then the mesh warping was applied. The outcomes had highly encouraging effects and took little time. And the experiments results showed that it performed very well.
(2) We modified the algorithm to adopt the local warping. The control points of the warping parts were set as the subset of the control points in the Thin-Plate Spline transform. Then it not only make the warping parts transformed well, but also simulated the effects of surrounding areas of the warping parts.
(3) Applied the smoothness algorithm to the cartoon portraits generation and the face animation. In the cartoon portraits generation based on characteristic transform, the algorithm can be used to warp the texture as soon as the shape was determined. In the face animation, the key frames were interpolated to acquire the middle frames, and the smoothness warping algorithm worked here to make the animation vivid. It played a more important role in the emotional talking head.
引文
[1]付昀,人脸图像渐变算法研究[J],中国图像图形学报,Vol.8(A),2003。
[2]王章野,金小刚。图象变形综述[R]。2004。
[3] Wolberg G, Digital image warping [M]. IEEE Computer Society Press, Los Alam-itos, Calif, 1994.
[4] Beier T, Neely S. Feature-based image metamorphosis [J]. Computer Graphics (0097-8930), P35-42, 1992.
[5] Lee S, Wolberg G, Shin SY. Polymorph:morphing among multiple images [C]. IEEE Computer Graph Appl 18: P58-71,1998.
[6] Ruprecht D,Muller H. Image warping with scattered data interpolation [C]. IEEE Computer Graph Appl 15: P37-43,1995.
[7] Lee S, Chwa K-Y, Hahn J, Shin SY. Image morphing using deformable surfaces [C]. Proceedings of Computer Animation’94. IEEE Computer Society Press, Geneva, Switzerland, P31-39,1994.
[8] Litwinowicz P, Willam L.Animating images with drawings [J]. (Proceedings of SIGGRAPH’94) Computer Graph, P409-412,1994.
[9] Hassanien AE,Nakajima M. Image morphing of the facial image transformation based on Navier elastic body splines [C]. Proceedings of Computer Animation’98, Geneva, Switzerland, IEEE Computer Society Press, Los Alamitos, CA, P119-125.
[10] Arad N, Dyn N, Reisfeld D, Yeshurun Y. Image warping by radial basis functions: applications to facial expressions [J]. CVGIP:Graph Models Image Processing 56: P161-172,1994.
[11] Lee S,Chwa K-Y, Hahn J, Shin SY. Image morphing using deformation techniques [J]. The Journal of Visualization and Computer Animation (1049-8907), 1996.
[12] Lee S, Chawa K-Y, Shin SY, Wolberg G.Image metamorphosis using snakes and free-form deformations [J]. (Proceedings of SIGGRAPH’95)Computer Graph, P439-448,1995.
[13] Kass M, Witkin A, Terzopoulos D. Snakes:active contour models [J]. Int J Computer Vision I (4): P321-331,1988.
[14] Gao P,Sederberg TW. A work minimization approach to image morphing [J]. Visual Computer (this volume),1998.
[15] Serberg TW, Greenwood E. A physically based approach to 2D shape blending [J]. (Proceedings of SIGGRAPH’92) Comput Graph 26:P25-34,1992.
[16] Smythe DB. A two-pass mesh warping algorithm for object gransformation and image interpolation [R]. Technical report1 030, ILM Computer Graphics Department, Lucasfilm, San Rafael, Calif., 1990.
[17] Sederberg T W, Gao P S, Wang G J, Mu H. 2D shape blending: an intrinsic solution to thevertex path problem [J]. Computer Grphics, 27(3): P15~18, 1993.
[18] Zhang Yuefeng. A fuzzy approach to digital image warping [J]. IEEE Computer Graphics and Applications, 16(6): P34~41, 1996.
[19] Fant KM. A nonaliasing, real-time spatial transform technique [J]. IEEE Computer Graph Appl 6: P71-80, 1986.
[20] Yin-Chiao Tsai, Hong-Dun Lin, Yu-Chang Hu, Chin-Lung Yu, Kang-Ping Lin. Thin-Plate Spline Technique for Medical Image Deformation[J]. Journal of Medical and Biological Engineering (1609-0985), Vol 20, No 4, 2000.
[21] F. L. Bookstein and W. D. K. GreenA feature space for edges in image with landmarks [J]. Journal of Mathematical Imaging and Vision, 3:231-261, 1993.
[22] J. Meinguet. Surface spline interpolation: Basic theory and computational aspects [J]. Approximation Theory and Spline Functions. P127-142, Dordrecht, 1984.
[23]鲍虎军,金小刚,彭群生。计算机动画的算法基础[M]。浙江大学出版社。P275-368,2000。
[24]Prashant K. Oswal and Prashanth Y. Govindaraju. Image Morphing: A Comparative Study[Z].
[25]张静波,张桂林。基于特征网格的面部图像变形算法[J]。中国图象图形学报,Vol7(A), No.3,2002
[26]王奎武,董兰芳,王洵,陈意云。基于MPEG-4的人脸变形算法的研究[J]。计算机辅助设计与图形学学报,Vol.14, No.1,2002.
[27]柴秀娟,山世光,高文,陈熙霖。基于样例的面部特征自动标定算法[J]。软件学报,Vol.16, No.5, 2005。
[28] http://222.35.2.47/SohuWebSite/DTT.aspx? [Z]
[29] http://www.num5.com/[Z]
[30]周仁琴,周经野,陈益强,刘军发。基于特征发现的卡通人脸肖像生成[J]。计算机辅助设计与图形学报。Vol.18,No.9,2006.
[31] Vitaly Surazhsky, Craig Gotsman, Intrinsic morphing of compatible triangulations [A]. In Preceedings of 4th Bi-National Israel-Korea Conference on Geometric Modelling and Computer Graphics [C]. Tel Aviv, Israel, Feb: P45~50, 2003.
[32]王建宁,董峰,石教英。基于三角形对应关系的Morphing技术[J]。中国图象图形学报,3(8):P641～646, 1998.
[33] Kohonen T; Oja E, Simula O et al. Engineering application of the self-organizing map [J]. Proceedings of the IEEE. 84(10):1358~1383, 1996.
[34]Arad N, Reisfeld D. Image warping using few anchor points and radial basis functions [J]. Computer Graphics Forum, 14(1): P23~29, 1995.
[35] George Wolberg, Itzik Alfy. An energy-minimization framework for monotonic cubic spline interpolation [J]. Journal of Computational and Applied Mathematics; P145~18, 2002.
[36] Marcelo Bertalmio, Guillermo Sapiro, Gregory Randell. Morphing active contours [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):733~737, 2000.
[37] Paul Bao, Dan Xu. Multiresolution image morphing in wavelet domain [A]. In Pro. in International Conference on Information Visualisation(IV2000)[C]. London,England,P309~314, 2000.
[38]Brunelli R, Poggio T. Face recognition: features versus templates [J].IEEE Trans Patt Anal Machine Intell, 15: P1042~105,1993.
[39]Ohya J. Virtual metamorphosis systems [A]. In IEEE SMC’99 Conference Proceedings, 6: P64~69, 1999.
[40]Genc S, Yarman-Vural F T. Morphing as a tool for motion modeling, image analysis and processing[A]. In Proceedings of IEEE International Conference, P538~543, 1999.
[41] Junfa Liu, Yiqiang Chen, Wen Gao. Mapping Learning in Eigenspace for Harmonious Caricature Generation [C]. 14th ACM International Conference on Multimedia(ACM MM06), Santa Barbara, USA, P683-686, Oct.22-27, 2006.
[42]谢金晶,陈益强,刘军发。基于语音情感识别的多表情人脸动画方法[J]。计算机辅助设计与图形学学报。Vol.20, No.4, 2008.
[43] Cootes T F, Taylor C J, Cooper D, et al. Active shape models: their training and application[J], Computer Vision and Image Understanding, 61(1):38-59, 1995.
[44] Bichsel M. Automatic interpolation and recognition of face images by morphing [J]. IEEE Computer Society Press, Los Alamitos, CA, USA. P128-135, 1996.
[45] M. Covell, M. Withgott. Spanning the gap between motion estimation and morphing [C], Proc. ICASSP'94, 1994.
[46] Bregler C, Covell M, Slaney M. Video rewrite: Driving visual speech with audio [J]. Proceedings of Siggraph'97,Los Angels, P357～360, 1997.
[47] Tony Ezzat, Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation [J].ACM Transactions on Graphics(TOG), v.21 n.3, July 2002.
[48] Blanz V, Basso C, Poggio T. et al. Reanimating faces in images and video[A]. In: Proceedings of EUROGRAPH 2003, Granada, P641～650, 2003.
[49]涂欢.语音和文本联合驱动卡通人脸动画的方法及其面向移动娱乐应用[D].湘潭大学硕士学位论文,2007.
[2]王章野,金小刚。图象变形综述[R]。2004。
[3] Wolberg G, Digital image warping [M]. IEEE Computer Society Press, Los Alam-itos, Calif, 1994.
[4] Beier T, Neely S. Feature-based image metamorphosis [J]. Computer Graphics (0097-8930), P35-42, 1992.
[5] Lee S, Wolberg G, Shin SY. Polymorph:morphing among multiple images [C]. IEEE Computer Graph Appl 18: P58-71,1998.
[6] Ruprecht D,Muller H. Image warping with scattered data interpolation [C]. IEEE Computer Graph Appl 15: P37-43,1995.
[7] Lee S, Chwa K-Y, Hahn J, Shin SY. Image morphing using deformable surfaces [C]. Proceedings of Computer Animation’94. IEEE Computer Society Press, Geneva, Switzerland, P31-39,1994.
[8] Litwinowicz P, Willam L.Animating images with drawings [J]. (Proceedings of SIGGRAPH’94) Computer Graph, P409-412,1994.
[9] Hassanien AE,Nakajima M. Image morphing of the facial image transformation based on Navier elastic body splines [C]. Proceedings of Computer Animation’98, Geneva, Switzerland, IEEE Computer Society Press, Los Alamitos, CA, P119-125.
[10] Arad N, Dyn N, Reisfeld D, Yeshurun Y. Image warping by radial basis functions: applications to facial expressions [J]. CVGIP:Graph Models Image Processing 56: P161-172,1994.
[11] Lee S,Chwa K-Y, Hahn J, Shin SY. Image morphing using deformation techniques [J]. The Journal of Visualization and Computer Animation (1049-8907), 1996.
[12] Lee S, Chawa K-Y, Shin SY, Wolberg G.Image metamorphosis using snakes and free-form deformations [J]. (Proceedings of SIGGRAPH’95)Computer Graph, P439-448,1995.
[13] Kass M, Witkin A, Terzopoulos D. Snakes:active contour models [J]. Int J Computer Vision I (4): P321-331,1988.
[14] Gao P,Sederberg TW. A work minimization approach to image morphing [J]. Visual Computer (this volume),1998.
[15] Serberg TW, Greenwood E. A physically based approach to 2D shape blending [J]. (Proceedings of SIGGRAPH’92) Comput Graph 26:P25-34,1992.
[16] Smythe DB. A two-pass mesh warping algorithm for object gransformation and image interpolation [R]. Technical report1 030, ILM Computer Graphics Department, Lucasfilm, San Rafael, Calif., 1990.
[17] Sederberg T W, Gao P S, Wang G J, Mu H. 2D shape blending: an intrinsic solution to thevertex path problem [J]. Computer Grphics, 27(3): P15~18, 1993.
[18] Zhang Yuefeng. A fuzzy approach to digital image warping [J]. IEEE Computer Graphics and Applications, 16(6): P34~41, 1996.
[19] Fant KM. A nonaliasing, real-time spatial transform technique [J]. IEEE Computer Graph Appl 6: P71-80, 1986.
[20] Yin-Chiao Tsai, Hong-Dun Lin, Yu-Chang Hu, Chin-Lung Yu, Kang-Ping Lin. Thin-Plate Spline Technique for Medical Image Deformation[J]. Journal of Medical and Biological Engineering (1609-0985), Vol 20, No 4, 2000.
[21] F. L. Bookstein and W. D. K. GreenA feature space for edges in image with landmarks [J]. Journal of Mathematical Imaging and Vision, 3:231-261, 1993.
[22] J. Meinguet. Surface spline interpolation: Basic theory and computational aspects [J]. Approximation Theory and Spline Functions. P127-142, Dordrecht, 1984.
[23]鲍虎军,金小刚,彭群生。计算机动画的算法基础[M]。浙江大学出版社。P275-368,2000。
[24]Prashant K. Oswal and Prashanth Y. Govindaraju. Image Morphing: A Comparative Study[Z].
[25]张静波,张桂林。基于特征网格的面部图像变形算法[J]。中国图象图形学报,Vol7(A), No.3,2002
[26]王奎武,董兰芳,王洵,陈意云。基于MPEG-4的人脸变形算法的研究[J]。计算机辅助设计与图形学学报,Vol.14, No.1,2002.
[27]柴秀娟,山世光,高文,陈熙霖。基于样例的面部特征自动标定算法[J]。软件学报,Vol.16, No.5, 2005。
[28] http://222.35.2.47/SohuWebSite/DTT.aspx? [Z]
[29] http://www.num5.com/[Z]
[30]周仁琴,周经野,陈益强,刘军发。基于特征发现的卡通人脸肖像生成[J]。计算机辅助设计与图形学报。Vol.18,No.9,2006.
[31] Vitaly Surazhsky, Craig Gotsman, Intrinsic morphing of compatible triangulations [A]. In Preceedings of 4th Bi-National Israel-Korea Conference on Geometric Modelling and Computer Graphics [C]. Tel Aviv, Israel, Feb: P45~50, 2003.
[32]王建宁,董峰,石教英。基于三角形对应关系的Morphing技术[J]。中国图象图形学报,3(8):P641～646, 1998.
[33] Kohonen T; Oja E, Simula O et al. Engineering application of the self-organizing map [J]. Proceedings of the IEEE. 84(10):1358~1383, 1996.
[34]Arad N, Reisfeld D. Image warping using few anchor points and radial basis functions [J]. Computer Graphics Forum, 14(1): P23~29, 1995.
[35] George Wolberg, Itzik Alfy. An energy-minimization framework for monotonic cubic spline interpolation [J]. Journal of Computational and Applied Mathematics; P145~18, 2002.
[36] Marcelo Bertalmio, Guillermo Sapiro, Gregory Randell. Morphing active contours [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):733~737, 2000.
[37] Paul Bao, Dan Xu. Multiresolution image morphing in wavelet domain [A]. In Pro. in International Conference on Information Visualisation(IV2000)[C]. London,England,P309~314, 2000.
[38]Brunelli R, Poggio T. Face recognition: features versus templates [J].IEEE Trans Patt Anal Machine Intell, 15: P1042~105,1993.
[39]Ohya J. Virtual metamorphosis systems [A]. In IEEE SMC’99 Conference Proceedings, 6: P64~69, 1999.
[40]Genc S, Yarman-Vural F T. Morphing as a tool for motion modeling, image analysis and processing[A]. In Proceedings of IEEE International Conference, P538~543, 1999.
[41] Junfa Liu, Yiqiang Chen, Wen Gao. Mapping Learning in Eigenspace for Harmonious Caricature Generation [C]. 14th ACM International Conference on Multimedia(ACM MM06), Santa Barbara, USA, P683-686, Oct.22-27, 2006.
[42]谢金晶,陈益强,刘军发。基于语音情感识别的多表情人脸动画方法[J]。计算机辅助设计与图形学学报。Vol.20, No.4, 2008.
[43] Cootes T F, Taylor C J, Cooper D, et al. Active shape models: their training and application[J], Computer Vision and Image Understanding, 61(1):38-59, 1995.
[44] Bichsel M. Automatic interpolation and recognition of face images by morphing [J]. IEEE Computer Society Press, Los Alamitos, CA, USA. P128-135, 1996.
[45] M. Covell, M. Withgott. Spanning the gap between motion estimation and morphing [C], Proc. ICASSP'94, 1994.
[46] Bregler C, Covell M, Slaney M. Video rewrite: Driving visual speech with audio [J]. Proceedings of Siggraph'97,Los Angels, P357～360, 1997.
[47] Tony Ezzat, Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation [J].ACM Transactions on Graphics(TOG), v.21 n.3, July 2002.
[48] Blanz V, Basso C, Poggio T. et al. Reanimating faces in images and video[A]. In: Proceedings of EUROGRAPH 2003, Granada, P641～650, 2003.
[49]涂欢.语音和文本联合驱动卡通人脸动画的方法及其面向移动娱乐应用[D].湘潭大学硕士学位论文,2007.