人脸动画合成的研究与实现
|
摘要
人脸动画合成是计算机视觉及计算机图形学领域中的一个热门研究课题,它有着非常广泛的应用领域,如：影视制作、医学研究、虚拟会议、虚拟主持人等等。
由于不同类型的人脸在脸的大小、五官的位置、大小、比例等方面存在着差异,如果将运动参数不作调整直接作用于人脸模型,可能会得到与真实人脸不一致的表情,甚至怪异的表情。因此必须针对不同类型的人脸对运动参数做出适当的调整,从而得到比较自然的表情。具体而言,本文的研究工作如下：
1.对视频中人脸的特征点进行检测,得到人面部五官的运动参数。
2.本文对人脸模型按照一定的分区原则进行功能分区,然后手动获取人脸模型各个控制点的运动因子,利用已经获得的运动参数和运动因子得到每个功能分区的动画单元AU,为了消除人头部晃动带来的误差,对特征点的坐标进行修正,最终实现了视频驱动的人脸动画。
3.为了使不同人脸视频驱动模型获得较好的动画,本文采用了全局变换、局部变换、FAPU变换。实验结果表明,采用上述3种变换后,在每个实验者的视频驱动下,都基本得到了令人满意的动画效果。
Facial Animation Synthesis is a hot research topic in the fields of Computer Vision and Computer Graphics. It has a very wide range of applications, such as video production, medical research, virtual meetings, virtual host and so on.
Because the location, size and proportion of facial features vary greatly from person to person, if motion parameters are employed directly to drive the face model without adjustments, we may get expressions which are inconsistent with the real face, or even the strange ones. Therefore, it is necessary to make appropriate adjustments in the motion parameters of different types of face to obtain more natural expressions. The main research contents are as follows:
1. The feature points on the face in the video are detected at first. And then the facial motion parameters are obtained.
2. In this paper, the face model is divided into some function partitions in accordance with certain principles. Then the movement factors of the various control points on the face model are obtained manually. The animation units AU of every divided function are obtained by using the motion parameters and movement factors of the various control points. The coordinates of the features are modified in order to get rid of errors which are caused by shaking of head. Ultimately, the video-driven facial animation is achieved.
3. In order to obtain a better facial animation when different types of facial videos drive the face model, the global transformation, local transformation, and FAPU transformation are employed. Experimental results show that the satisfactory animation nearly has been obtained when all the experimenters'videos drive the standard face model after using the three kinds of transformation.
由于不同类型的人脸在脸的大小、五官的位置、大小、比例等方面存在着差异,如果将运动参数不作调整直接作用于人脸模型,可能会得到与真实人脸不一致的表情,甚至怪异的表情。因此必须针对不同类型的人脸对运动参数做出适当的调整,从而得到比较自然的表情。具体而言,本文的研究工作如下：
1.对视频中人脸的特征点进行检测,得到人面部五官的运动参数。
2.本文对人脸模型按照一定的分区原则进行功能分区,然后手动获取人脸模型各个控制点的运动因子,利用已经获得的运动参数和运动因子得到每个功能分区的动画单元AU,为了消除人头部晃动带来的误差,对特征点的坐标进行修正,最终实现了视频驱动的人脸动画。
3.为了使不同人脸视频驱动模型获得较好的动画,本文采用了全局变换、局部变换、FAPU变换。实验结果表明,采用上述3种变换后,在每个实验者的视频驱动下,都基本得到了令人满意的动画效果。
Facial Animation Synthesis is a hot research topic in the fields of Computer Vision and Computer Graphics. It has a very wide range of applications, such as video production, medical research, virtual meetings, virtual host and so on.
Because the location, size and proportion of facial features vary greatly from person to person, if motion parameters are employed directly to drive the face model without adjustments, we may get expressions which are inconsistent with the real face, or even the strange ones. Therefore, it is necessary to make appropriate adjustments in the motion parameters of different types of face to obtain more natural expressions. The main research contents are as follows:
1. The feature points on the face in the video are detected at first. And then the facial motion parameters are obtained.
2. In this paper, the face model is divided into some function partitions in accordance with certain principles. Then the movement factors of the various control points on the face model are obtained manually. The animation units AU of every divided function are obtained by using the motion parameters and movement factors of the various control points. The coordinates of the features are modified in order to get rid of errors which are caused by shaking of head. Ultimately, the video-driven facial animation is achieved.
3. In order to obtain a better facial animation when different types of facial videos drive the face model, the global transformation, local transformation, and FAPU transformation are employed. Experimental results show that the satisfactory animation nearly has been obtained when all the experimenters'videos drive the standard face model after using the three kinds of transformation.
引文
[1]S.C.W. Ong, S. Ranganath. Automatic sign language analysis:a survey and the future beyond lexical meaning. IEEE TPAMI,2005,27(6):873-891.
[2]Parke F I, Waters K. Computer Facial Animation. Wellesley, MA:A. K. Peters,1996:1-365.
[3]Parke F. Computer generated animation of faces. Technical Report, AD-762022,1972:1-87.
[4]Ip Horace HS, Chan CS. Script-Based facial gesture and speech animation using a NURBS based face model. Computer and Graphics.1996,20(6):881-891.
[5]Elson M. Displacement facial animation techniques. In:State of the Art in Facial Animation, ACM Siggraph'90 Course Notes,1990,26:21-42.
[6]徐琳,袁保宗,高文.真实感人脸建模研究的进展与展望.软件学报.2003,14(4).
[7]Platt SM, Badler NI. Animating facial expressions. Computer Graphics.1981,15(3):245-252.
[8]Koch R, Gross M H, Carls F et al. Simulating facial surgery using finite element methods. In: SIGGRAPH Conference Proceedings, New Orleans, USA,1996:421-428.
[9]Waters K. A muscle model for animating three-dimensional facial expression. Computer Graphics.1987,22(4):17-24.
[10]Thalmann NM, Primeau NE, Thalmann D. Abstract muscle actions procedures for human face animation. Visual Computer.1988,3(5):290-297.
[11]Thalmann NM, Minh H, Angelis M et al. Design, transformation and animation of human faces. Visual Computer.1989,5(1-2):32-39.
[12]Lee Y, Terzopoulos D, Waters K. Realistic modeling for facial animation. In:SIGGRAPH Conference Proceedings, Los Angeles, USA,1995:55-62.
[13]Akimoto T, Suenaga Y, Wallace R S. Automatic creation of 3D facial models. IEEE Computer Graphics and Applications.1993,13(5):16-22.
[14]F.Pighin, J.Hecker, D.Lischinski et al. Synthesizing realistic facial expressions from photographs. In:Computer Graphics, Annual Conference Series,1998,7:75-84.
[15]L.Malatesta, A.Raouzaiou, K.Karpouzis et al. MPEG-4 facial expression synthesis. Pers Ubiquit Comput.2009,13:77-83.
[16]Igor S.Pandzic. Facial Animation Framework for the Web and Mobile Platforms.3D technologies for the World Wide Web Proceedings of the 7th International Conference on 3D Web technology, Tempe, Arizona, USA,2002.2:27-34.
[17]晏洁,高文,尹宝才.具有真实感的三维虚拟特定人脸生成方法.计算机学报.1999,22(2)：147-153.
[18]晏洁.具有真实感的三维人脸合成方法的研究与实践：(博十学位论文).哈尔滨：哈尔滨工业大学,1999.
[19]王成章,尹宝才,白晓明等.一种兼容于MPEG-4的三维人脸动画.北京工业大学学报.2007,10.
[20]郑飞,陈梅.基于正面头像的人面三维造型.计算机学报.1998,21(11).
[21]山世光,高文,陈熙霖.基于纹理分布和变形模板的面部特征提取.软件学报.2001,12(4)：570-577.
[22]Bing-Yu Chen, Yutaka Ono, Tomoyuki Nishita. Character animation creation using hand-drawn sketches. Visual Computer.2005,21:551-558.
[23]梅丽.基于实拍图像的人脸真实感重建[J].计算机学报.2000：996-1002.
[24]Jie Qiu, Hock Soon Seah, Feng Tian, Zhongke Wu, Quan Chen. Feature-and region-based auto painting for 2D animation. Visual Computer.2005:21(11):928-944.
[25]J. Xiao, T. Kanade and J. Cohn. Robust full motion recovery of head by dynamic templates and registration techniques. In:Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition.2002:156-162.
[26]Li Huang, Congyong Su. Facial expression synthesis using manifold learning and belief propagation. Soft Computer.2006.10:1193-1200.
[27]NOH J, NEUMANN U. A survey of facial modeling and animation techniques[R]. California: University of Southern California,1998.
[28]Ken-ichi Anjyo, Shuhei Wemler, William Baxter. Tweakable Light and Shade for Cartoon Animation. Proceedings of the 4th International Symposium on Non-Photorealistic Animation and Rendering.2006:133-139.
[29]Debevec P, Hawkins T, Tchou C et al. Acquiring the reflectance field of a human face. In: Proceedings of ACM SIGGRAPH2000 Conference, New Orleans,2000:145-156.
[30]Z Liu et al. Expressive Expression Mapping with Ratio Image. Computer Graphics Proceedings, ACM SIGGRAPH,2001:217-276.
[31]Adam Lake, Carl Marshall, Mark Harris, Marc Blackstein. Stylized rendering techniques for scalable real-time 3D animation. Proceedings of the First International Symposium on Non-photorealistic Animation and Rendering,2000:13-20.
[32]J. Xiao and J. Chai. A linear solution to nonrigid structure from motion. In:Robotics Institute Technical Report, Carnegie Mellon University.2003. CMU-RI-TR-03-16.
[33]Daniel Sykora, Jan Burianek, Jiri Zara. Sketching Cartoons by Example. Eurographics Workshop on Sketch-Based Interfaces and Modeling.2005.
[34]E.Gladilin, S.Zachow, P.Deuflhard et al. Anatomy-and physics-based facial animation for craniofacial surgery simulations. Medical & Biological & Engineering & Computer.2004, 42:167-170.
[35]黄福珍,苏剑波.基于Level Set方法的人脸轮廓提取与跟踪.计算机学报.2003,26(4)：491-496.
[36]Holger Winnemoller, Sven C.Olsen, Bruce Gooch. Real-Time Video Abstraction. Proceedings of ACM SIGGRAPH.2006:1221-1226.
[37]Pighin F, Szeliski R, Salesin DH. Modeling and animation realistic faces from images. Proceedings International Journal of Computer Vision.2002,50(2):143-169.
[38]ZHANG Qing-shan, LIU Zi-cheng, GUO Bai-ning et al. Geometry-driven photorealistic facial expression synthesis [J]. IEEE Trans on Visualisation and Computer Graphics.2006,12(1):48-60.
[39]姜大龙.真实感三维人脸合成方法研究.中国科学院计算技术研究所.2005：37-60.
[40]Jin-xiang Chai, Jing Xiao, Jessica Hodgins. Vision-based Control of 3D Facial Animation. Eurographics/SIGGRAPH Symposium on Computer Animation.2003.
[41]H.Sidenbladh, M.J.Black and L.Sigal. Implicit probabilistic models of human motion for synthesis and tracking. In European Conference on Computer Vision. Apringer-Verlag, 2002:784-800.
[42]王琨,郑南宁.基于SFM算法的三维人脸模型重建.计算机学报.2005,28(6).
[43]胡永利,尹宝才,谷春亮等.基于形变模型的三维人脸重建方法及其改进.计算机学报.2005,10.
[44]宋明黎.人脸表情的识别、重建与合成：(博士学位论文).杭州：浙江大学,2005,10.
[45]Z.S.Bojkovic, D.A.Milovanovic. Audiovisual integration in multimedia communications based on MPEG-4 facial animation. CIRCUITS SYSTEMS SIGNAL PROCESSING.2001, 20(3):311-339.
[46]戴振龙,朱海一,张申等.基于MPEG-4的人脸表情图像变形研究.中国图象图形学报.2009.5,14(5).
[47]Jorgen Ahlberg. CANDIDE-3——AN UPDATED PARAMETERISED FACE [R]. SWEDEN: Image Coding Group. Dept. of Electrical Engineering, Linkoping University.2001:16.
[48]徐成华,王蕴红,谭铁牛.三维人脸建模与应用[J].中国图像图形学报.2004,9(8)：893-895.
[49]Brian Guenter, Cindy Grimm, Daniel Wood et al. Making Faces. In:Proc. International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH' 05),2005:55-66.
[50]Pengyu Hong, Zhen Wen and Thomas S. Huang. Real-Time Speech-Driven Face Animation with Expressions Using Neural Networks. IEEE Transactions on neural networks.2002.2, 13(1):100-111.
[51]WANG Yu-shun, ZHUANG Yue-ting, Wu Fei. Data-drive facial animation based on manifold Bayesian regression. Journal of Zhejiang University SCIENCE A.2006,7(4):556-563.
[52]P. Ekman and W. V. Friesen. Facial Action Coding System [N].Consulting Psychologist Press,1977.
[53]OpenGL Architecture Review Board, Dave Shreiner, Mason Woo et al. OpenGL Programming Guide, Sixth Edition. Beijing:China Machine Press,2009.2.
[54]Edward Angel著.段菲译.OpenGL:A Primer,3rd Edition.北京：清华大学出版社,2008.3.
[55]Chuan-Kai Yang · Wei-Ting Chiang. An interactive facial expression generation system. Multimedia Tools and Applications.2008,40:41-60.
[56]Olivier Martin, Irene Kotsia, Ioannis Pitas et al. SYNTHESIS OF EXPRESSIVE FACIAL ANIMATIONS:A MULTIMODAL CARICATURAL MIRROR. Journal on Multimodal User Interfaces.2007.3,1(1).
[57]Soo-Kyun Kim·Syung-Og An·Min Hong·Doo-Soon Park-Shin-Jin Kang. Decimation of human face model for real-time animation in intelligent multimedia systems. Multimedia Tools Applications.2010(47):147-162.
[58]Minhua Ma·Paul Mc Kevitt. Virtual human animation in natural language visualisation. Artif Intell Rev.2006,25:37-53.
[2]Parke F I, Waters K. Computer Facial Animation. Wellesley, MA:A. K. Peters,1996:1-365.
[3]Parke F. Computer generated animation of faces. Technical Report, AD-762022,1972:1-87.
[4]Ip Horace HS, Chan CS. Script-Based facial gesture and speech animation using a NURBS based face model. Computer and Graphics.1996,20(6):881-891.
[5]Elson M. Displacement facial animation techniques. In:State of the Art in Facial Animation, ACM Siggraph'90 Course Notes,1990,26:21-42.
[6]徐琳,袁保宗,高文.真实感人脸建模研究的进展与展望.软件学报.2003,14(4).
[7]Platt SM, Badler NI. Animating facial expressions. Computer Graphics.1981,15(3):245-252.
[8]Koch R, Gross M H, Carls F et al. Simulating facial surgery using finite element methods. In: SIGGRAPH Conference Proceedings, New Orleans, USA,1996:421-428.
[9]Waters K. A muscle model for animating three-dimensional facial expression. Computer Graphics.1987,22(4):17-24.
[10]Thalmann NM, Primeau NE, Thalmann D. Abstract muscle actions procedures for human face animation. Visual Computer.1988,3(5):290-297.
[11]Thalmann NM, Minh H, Angelis M et al. Design, transformation and animation of human faces. Visual Computer.1989,5(1-2):32-39.
[12]Lee Y, Terzopoulos D, Waters K. Realistic modeling for facial animation. In:SIGGRAPH Conference Proceedings, Los Angeles, USA,1995:55-62.
[13]Akimoto T, Suenaga Y, Wallace R S. Automatic creation of 3D facial models. IEEE Computer Graphics and Applications.1993,13(5):16-22.
[14]F.Pighin, J.Hecker, D.Lischinski et al. Synthesizing realistic facial expressions from photographs. In:Computer Graphics, Annual Conference Series,1998,7:75-84.
[15]L.Malatesta, A.Raouzaiou, K.Karpouzis et al. MPEG-4 facial expression synthesis. Pers Ubiquit Comput.2009,13:77-83.
[16]Igor S.Pandzic. Facial Animation Framework for the Web and Mobile Platforms.3D technologies for the World Wide Web Proceedings of the 7th International Conference on 3D Web technology, Tempe, Arizona, USA,2002.2:27-34.
[17]晏洁,高文,尹宝才.具有真实感的三维虚拟特定人脸生成方法.计算机学报.1999,22(2)：147-153.
[18]晏洁.具有真实感的三维人脸合成方法的研究与实践：(博十学位论文).哈尔滨：哈尔滨工业大学,1999.
[19]王成章,尹宝才,白晓明等.一种兼容于MPEG-4的三维人脸动画.北京工业大学学报.2007,10.
[20]郑飞,陈梅.基于正面头像的人面三维造型.计算机学报.1998,21(11).
[21]山世光,高文,陈熙霖.基于纹理分布和变形模板的面部特征提取.软件学报.2001,12(4)：570-577.
[22]Bing-Yu Chen, Yutaka Ono, Tomoyuki Nishita. Character animation creation using hand-drawn sketches. Visual Computer.2005,21:551-558.
[23]梅丽.基于实拍图像的人脸真实感重建[J].计算机学报.2000：996-1002.
[24]Jie Qiu, Hock Soon Seah, Feng Tian, Zhongke Wu, Quan Chen. Feature-and region-based auto painting for 2D animation. Visual Computer.2005:21(11):928-944.
[25]J. Xiao, T. Kanade and J. Cohn. Robust full motion recovery of head by dynamic templates and registration techniques. In:Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition.2002:156-162.
[26]Li Huang, Congyong Su. Facial expression synthesis using manifold learning and belief propagation. Soft Computer.2006.10:1193-1200.
[27]NOH J, NEUMANN U. A survey of facial modeling and animation techniques[R]. California: University of Southern California,1998.
[28]Ken-ichi Anjyo, Shuhei Wemler, William Baxter. Tweakable Light and Shade for Cartoon Animation. Proceedings of the 4th International Symposium on Non-Photorealistic Animation and Rendering.2006:133-139.
[29]Debevec P, Hawkins T, Tchou C et al. Acquiring the reflectance field of a human face. In: Proceedings of ACM SIGGRAPH2000 Conference, New Orleans,2000:145-156.
[30]Z Liu et al. Expressive Expression Mapping with Ratio Image. Computer Graphics Proceedings, ACM SIGGRAPH,2001:217-276.
[31]Adam Lake, Carl Marshall, Mark Harris, Marc Blackstein. Stylized rendering techniques for scalable real-time 3D animation. Proceedings of the First International Symposium on Non-photorealistic Animation and Rendering,2000:13-20.
[32]J. Xiao and J. Chai. A linear solution to nonrigid structure from motion. In:Robotics Institute Technical Report, Carnegie Mellon University.2003. CMU-RI-TR-03-16.
[33]Daniel Sykora, Jan Burianek, Jiri Zara. Sketching Cartoons by Example. Eurographics Workshop on Sketch-Based Interfaces and Modeling.2005.
[34]E.Gladilin, S.Zachow, P.Deuflhard et al. Anatomy-and physics-based facial animation for craniofacial surgery simulations. Medical & Biological & Engineering & Computer.2004, 42:167-170.
[35]黄福珍,苏剑波.基于Level Set方法的人脸轮廓提取与跟踪.计算机学报.2003,26(4)：491-496.
[36]Holger Winnemoller, Sven C.Olsen, Bruce Gooch. Real-Time Video Abstraction. Proceedings of ACM SIGGRAPH.2006:1221-1226.
[37]Pighin F, Szeliski R, Salesin DH. Modeling and animation realistic faces from images. Proceedings International Journal of Computer Vision.2002,50(2):143-169.
[38]ZHANG Qing-shan, LIU Zi-cheng, GUO Bai-ning et al. Geometry-driven photorealistic facial expression synthesis [J]. IEEE Trans on Visualisation and Computer Graphics.2006,12(1):48-60.
[39]姜大龙.真实感三维人脸合成方法研究.中国科学院计算技术研究所.2005：37-60.
[40]Jin-xiang Chai, Jing Xiao, Jessica Hodgins. Vision-based Control of 3D Facial Animation. Eurographics/SIGGRAPH Symposium on Computer Animation.2003.
[41]H.Sidenbladh, M.J.Black and L.Sigal. Implicit probabilistic models of human motion for synthesis and tracking. In European Conference on Computer Vision. Apringer-Verlag, 2002:784-800.
[42]王琨,郑南宁.基于SFM算法的三维人脸模型重建.计算机学报.2005,28(6).
[43]胡永利,尹宝才,谷春亮等.基于形变模型的三维人脸重建方法及其改进.计算机学报.2005,10.
[44]宋明黎.人脸表情的识别、重建与合成：(博士学位论文).杭州：浙江大学,2005,10.
[45]Z.S.Bojkovic, D.A.Milovanovic. Audiovisual integration in multimedia communications based on MPEG-4 facial animation. CIRCUITS SYSTEMS SIGNAL PROCESSING.2001, 20(3):311-339.
[46]戴振龙,朱海一,张申等.基于MPEG-4的人脸表情图像变形研究.中国图象图形学报.2009.5,14(5).
[47]Jorgen Ahlberg. CANDIDE-3——AN UPDATED PARAMETERISED FACE [R]. SWEDEN: Image Coding Group. Dept. of Electrical Engineering, Linkoping University.2001:16.
[48]徐成华,王蕴红,谭铁牛.三维人脸建模与应用[J].中国图像图形学报.2004,9(8)：893-895.
[49]Brian Guenter, Cindy Grimm, Daniel Wood et al. Making Faces. In:Proc. International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH' 05),2005:55-66.
[50]Pengyu Hong, Zhen Wen and Thomas S. Huang. Real-Time Speech-Driven Face Animation with Expressions Using Neural Networks. IEEE Transactions on neural networks.2002.2, 13(1):100-111.
[51]WANG Yu-shun, ZHUANG Yue-ting, Wu Fei. Data-drive facial animation based on manifold Bayesian regression. Journal of Zhejiang University SCIENCE A.2006,7(4):556-563.
[52]P. Ekman and W. V. Friesen. Facial Action Coding System [N].Consulting Psychologist Press,1977.
[53]OpenGL Architecture Review Board, Dave Shreiner, Mason Woo et al. OpenGL Programming Guide, Sixth Edition. Beijing:China Machine Press,2009.2.
[54]Edward Angel著.段菲译.OpenGL:A Primer,3rd Edition.北京：清华大学出版社,2008.3.
[55]Chuan-Kai Yang · Wei-Ting Chiang. An interactive facial expression generation system. Multimedia Tools and Applications.2008,40:41-60.
[56]Olivier Martin, Irene Kotsia, Ioannis Pitas et al. SYNTHESIS OF EXPRESSIVE FACIAL ANIMATIONS:A MULTIMODAL CARICATURAL MIRROR. Journal on Multimodal User Interfaces.2007.3,1(1).
[57]Soo-Kyun Kim·Syung-Og An·Min Hong·Doo-Soon Park-Shin-Jin Kang. Decimation of human face model for real-time animation in intelligent multimedia systems. Multimedia Tools Applications.2010(47):147-162.
[58]Minhua Ma·Paul Mc Kevitt. Virtual human animation in natural language visualisation. Artif Intell Rev.2006,25:37-53.