基于动态图像序列的人脸表情特征提取和识别算法的研究
|
摘要
人脸表情识别涉及到人工智能、模式识别、图像处理、心理学等诸多领域，是研究人机交互的一个重要方向。随着计算机运行速度的不断提高，人们也越发关注基于动态图像序列的人脸表情识别的研究。基于动态图像序列的人脸表情识别不同以往的基于静态图像的表情识别研究。基于动态图像序列提取出的表情特征较静态特征含有更多表征表情的静态和动态信息，静态信息主要体现在每幅图像中的表情特征提取，而动态信息则是多个图像集合而成的整体表情变化方向。本论文的研究内容概述如下：
1．首先针对基于Candide3三维人脸模型的头部姿态及面部表情跟踪算法进行了研究，设计了一种半自动的人脸模型匹配方法，用于跟踪算法中的模型参数初始化和形状无关纹理合时所需要的样本生成。在研究了原跟踪算法的基础上，根据存在的不足进行了跟踪算法流程的改进，分别引入了一种基于样例学习的模型匹配方法以匹配序列图像中的首幅人脸图像，并在跟踪过程中加入基于灰度归一化的光照抑制过程。实验表明改进后的跟踪过程较原跟踪过程更加准确和方便。
2．在基于Candide3人脸表情跟踪过程中，提取模型参数并集合成为动态特征。首先本文中提出了一种基于六模型参数的动态特征提取方法。后通过分析表情变化时所牵动的表情运动单元变化。在原动态特征提取方法的基础上又进一步提出了一种七模型参数的动态特征提取方法，并分别对两种方法进行了聚类分析，分析结果表明基于模型参数的动态特征提取方法就有良好的聚类效果，其中进一步改进后的七参数动态特征提取方法具有更好的效果。
3．在完成了动态特征提取之后，还需要通过设计分类器来最终完成表情的识别工作。分类器的设计是除特征提取以外另一个决定分类准确率的重要环节。研究中设计了多个分类器，包括K-NN分类器，SVM分类器，以及应用Adaboost算法级联贝叶斯分类器、线性判别分类器、K-NN分类器、SVM分类器等弱分类器。并通过实验，客观的给出了分类的准确率。
4．最后针对提出的非定长动态特征提取这一概念给出了说明，并尝试提出了一种基于相关性的图像序列分割方法，然后应用插值的方法归一化特征长度。
Face recognition involves expression of artificial intelligence, pattern classificationand image processing, psychology and so on many domains. It is an important directionabout the study of human interaction. Along with the computer running speedcontinuously improved, people pay more attention on the facial expression recognitionbased on dynamic image sequence
It is different between the facial expression recognition based on the dynamic imagesequence and the facial expression recognition based on static image. More information isinvolved in the dynamic features. This paper outlined the research contents are as follows:
1．In this part, we first studied the tracking theory based on Candide3facial model andaccording to the research we further developed the tracking process. A semi-automaticfacial model initialization method is put forward here. Then based on the semi-automaticfacial model initialization method, an example-based model initialization method isutilized. Last a normalization process is added in the whole tracking process to restrain theeffect of illumination.
2．Based on the research of facial tracking which I have mentioned, I put forward adynamic feature extraction method based on six parameters of facial model. Then in orderto enhance the recognition accuracy, we further studied the movement of facial action units during different expression process. According to this research a dynamic featureextraction method based on seven parameters of facial model was developed.
3． After the dynamic feature extraction, another important element of facialexpression recognition—classifier design—is considered. Various classifiers are designedhere and an objective judgment of recognition accuracy is stated.
4．An unfixed size dynamic feature extraction conception is put forward and explainedand according to the conception I made a preliminary study.
1．首先针对基于Candide3三维人脸模型的头部姿态及面部表情跟踪算法进行了研究，设计了一种半自动的人脸模型匹配方法，用于跟踪算法中的模型参数初始化和形状无关纹理合时所需要的样本生成。在研究了原跟踪算法的基础上，根据存在的不足进行了跟踪算法流程的改进，分别引入了一种基于样例学习的模型匹配方法以匹配序列图像中的首幅人脸图像，并在跟踪过程中加入基于灰度归一化的光照抑制过程。实验表明改进后的跟踪过程较原跟踪过程更加准确和方便。
2．在基于Candide3人脸表情跟踪过程中，提取模型参数并集合成为动态特征。首先本文中提出了一种基于六模型参数的动态特征提取方法。后通过分析表情变化时所牵动的表情运动单元变化。在原动态特征提取方法的基础上又进一步提出了一种七模型参数的动态特征提取方法，并分别对两种方法进行了聚类分析，分析结果表明基于模型参数的动态特征提取方法就有良好的聚类效果，其中进一步改进后的七参数动态特征提取方法具有更好的效果。
3．在完成了动态特征提取之后，还需要通过设计分类器来最终完成表情的识别工作。分类器的设计是除特征提取以外另一个决定分类准确率的重要环节。研究中设计了多个分类器，包括K-NN分类器，SVM分类器，以及应用Adaboost算法级联贝叶斯分类器、线性判别分类器、K-NN分类器、SVM分类器等弱分类器。并通过实验，客观的给出了分类的准确率。
4．最后针对提出的非定长动态特征提取这一概念给出了说明，并尝试提出了一种基于相关性的图像序列分割方法，然后应用插值的方法归一化特征长度。
Face recognition involves expression of artificial intelligence, pattern classificationand image processing, psychology and so on many domains. It is an important directionabout the study of human interaction. Along with the computer running speedcontinuously improved, people pay more attention on the facial expression recognitionbased on dynamic image sequence
It is different between the facial expression recognition based on the dynamic imagesequence and the facial expression recognition based on static image. More information isinvolved in the dynamic features. This paper outlined the research contents are as follows:
1．In this part, we first studied the tracking theory based on Candide3facial model andaccording to the research we further developed the tracking process. A semi-automaticfacial model initialization method is put forward here. Then based on the semi-automaticfacial model initialization method, an example-based model initialization method isutilized. Last a normalization process is added in the whole tracking process to restrain theeffect of illumination.
2．Based on the research of facial tracking which I have mentioned, I put forward adynamic feature extraction method based on six parameters of facial model. Then in orderto enhance the recognition accuracy, we further studied the movement of facial action units during different expression process. According to this research a dynamic featureextraction method based on seven parameters of facial model was developed.
3． After the dynamic feature extraction, another important element of facialexpression recognition—classifier design—is considered. Various classifiers are designedhere and an objective judgment of recognition accuracy is stated.
4．An unfixed size dynamic feature extraction conception is put forward and explainedand according to the conception I made a preliminary study.
引文
[1] F. Bourel, C.C. Chibelushi, A.A.Low. Robust Facial Expression Recognition Using a State-BasedModel of Spatially-Localised Facial Dynamics [C]. Proc. Fifth IEEE Int. Conf. Automatic Faceand Gesture Recognition,2002：106-111.
[2] Yumei Fan, Ning Cheng, Zhiliang Wang, Jiwei Liu, Changsheng Zhu. Real-Time Facial ExpressionRecognition system Based on HMM and Feature Point Localization [J]. Lecture Notes inComputer Science,2005:210-217.
[3] Y. Tian, T. Kanade, and J.F. Cohn. Facial Expression Analysis Handbook of Face Recognition [J].Springer,2003(10).
[4] Yu J G, Bhanu B. Evolutionary feature synthesis for facial expression recognition [J]. PatternRecognition Letters,2006,27(11):1289-1298
[5] Suwa M，Sugie N，Fujimora K．A preliminary note on pattern recognition of human emotionalexpression [C]． In： Proceedings of the Fourth International Joint Conference on PatternRecognition．Kyoto，Japan，1978：408-410．
[6]辛静.基于帧间灰度差的动态表情识别[D].天津：天津大学，2009.
[7] Mase K. Recognition of facial expression from optical flow [J]. IEICE transactions.1991：3474-3483.
[8] Ii-fan A．Essa，Alex Pentland．A vision system for observing and extracting facial action parameters[C]//Proceedings of the Computer Vision and Pattern Recognition Conference, IEEE ComputerSociety，1994：76-83
[9] Ming—Hsuan Yang，David J．Kriegman，and Narendra Ahuja，Detecting Faces in Images：A Survey[J]．IEEE. Transactions on Pattern Analysis and Machine Intelligence，2002(7)．
[10] Rana El Kaliouby and Peter Robinson. Generalization of a vision-based computational model ofmind-reading [C].1st Intl. Conf. on Affective Computing and Intelligent Interaction,2005.
[11] Jin Hui, Gao Wen, The human facial combined expression recognition system [J]. Chinese Journalof Computers,2000,23(6):602-608.
[12]薛雨丽，毛峡，郭叶，吕善伟.人机交互中的人脸表情识别研究进展[J].中国图像图形学报，2009(5):764-772
[13] Peng Yang, Qingshan Liu, Dimitris N. Metaxas. Boosting encoded dynamic features forexpression recognition [J]. Pattern Recognition Letters,2009:132-139.
[14]崔洁.面部表情识别方法研究.西北工业大学硕士学位论文[D]，2006
[15] Fadi Dornaika and Frank Davoine. On Appearance Based Face and Facial Action Tracking [J].IEEE Transactions on circuits and system for video technology,2006(9):566-571.
[16] J. Ahlberg, CANDIDE-3—An updated parameterized face. Dept. Elect. Eng., Linkoping Univ.,Sweden,2001, Rep. LiTH-ISY-R-2326
[17]刘晓旻，谭华春，章毓晋.人脸表情识别研究的新进展[J].中国图像图形学报，2006(10):1359-1368.
[18] J. Ahlberg. An active model for facial feature tracking [J]. DURASIP J. Appl. Signal Process.2002(6):566-571.
[19] J. Ahlberg. Model-based coding: Extraction, coding, and evaluation of face model parameters [D].Ph.D. dissertation, Linkoping Univ., Linkoping, Sweden,2002(9).
[20] F. Dornaika and J. Ahlberg. Fast and reliable active appearance model search for3-D face tracking[J].IEEE Trans. Syst. Man and Cybernetics,2004(8):1838-1853.
[21] F. Dornaika and F. Davoine, On appearance based face and facial action tracking [J], IEEE Trans.Circuits Syst. Video Technol.2006(9):1107-1124.
[22] G. J. Edwards, T.F. Cootes, and C. j. Taylor, Interpreting Face images using Active AppearanceModels [C]. Proc.3rd Int. Conf. on Automatic Face and Gesture Recognition, Nara, Japan,1998:300-305,.
[23] Allan D. Jepson, David J. Fleet, Thomas F. EI-Maraghi. Robust Online Appearance Models forVisual Tracking [C]. IEEE Conference on Computer Vision and Pattern Recognition, Kauai,2001:415-422.
[24] D. J. Fleet and A.D. Jepson. Stability of phase information [J]. IEEE Trans. PAMI,1993,15(12):1253-1268.
[25] Chai Xiujuan, Shan shiguang, Gao Wen et al. Example-based learning for automatic facealignment [J]. Journal of Software.2005,16(5):718-726.
[26]胡峰松，张茂军，邹北冀，马俊容.基于HMM的单样本可变光照、姿态人脸识别[J].计算机学报，2009(7):1424-1433.
[27] J. Strom, F. Davoine, J. Ahlberg, H. Li and R. Forchheimer. Very low bit rate facial texture coding
[C]. International Workshop on Synthetic-Natural Hybr.
[28] F. Bourel, el al. Robust Facial Expression Recognition Using a State-Based Model ofSpatially-Localised Facial Dynamics [C]. Proc Fifth IEEE Int.1Conf on Automatic Face andGesture Recognition (FGR-02),Washington D.C.2002:113-118.
[29] T. Kanade, J. Cohn, and Y. Tian. Comprehensive database for facial expression analysis [C]. inProc. IEEE int. Conf. Face and Gesture Recognition,2000(3):46-53.
[30] M. Pantic and L. J. M. Rothkrantz. Exper system for automatic analysis of facial expression [J].Image and Computing Journal,2000(8):881-905.
[31] Irene Kotsia and Ioannis Pitas. Facial Expression Recognition in Image Sequences UsingGeometric Deformation Features and Support Vector Machines [J]. IEEE Transactions on imageprocessing,2007(7):172-187.
[32] Richard O. Duda, Peter E. Hart, David G. Stork. Pattern Classification.
[33] Seung H. S., Lee D. D. The manifold ways of perception [J].Science,2000.
[34] Tenbaum J., Silva D. D., Langford J. A global geometric frame work for nonlinear dim ensionalityreduction [J]. Science,2000.
[35] Roweis S., Saul L. Nonlinear dimensionality reduction by locally linear embedding [J].Science,2000.
[36]章毓晋等.基于子空间的人脸识别[M].北京：清华大学出版社，2009.
[37] Cortes C., Vapnik V., Support-vector networks [J]. Machine Learning,1995:276-297.
[38]周宽久,张世荣.支持向量机分类算法研究[J].计算机工程与应用,2009(1):159-182.
[39] Schapire R. E. and Singer Y. Improved boosting algorithms using confidence-rated predictions [J].Machine Learning,1999(3).
[40]王宇博，艾海周，武勃，黄畅.人脸表情的实时分类.计算机辅助设计与图形学学报，2005(6):1296-1301.
[41] Peng Yang, Qingshan Liu, Xinyi Cui and Dimitris N.Metaxas. Facial Expression RecognitionUsing Encoded Dynamic Features [J]. Computer Vision and Pattern Recognition,2008:132-139.
[42] Xuchun Li, Lei Wang, Eric Sung. Adaboost with SVM-based component classifiers. EngineeringApplication of Artifical Inteligence,2008:785-795.
[43]王黎艳.基于Gabor小波变换和SVM的人脸表情识别[D].南京：南京理工大学.
[44] Ekman P., Friesen W. V., Facial Action Coding System: A Technique for the measurement of FacialMovement [M], Palo Alto Consulting Psychologists Press,1978.
[45] Cohn J. F., Zlochower A. J., Lien J. J., et al. Feature-point tracking by optical flow discriminationssubtle differences in facial expression [A]. Proceedings of the third IEEE InternationalConference on Automatic Face and Gesture Recognition [C], Nara, Japan,1998.
[46] Lucas D.Terissi, Juan C.Gomez.3D head pose and facial expression tracking using as singlecamera [J]. Journal of Universal Computer Science,2010(6):903-920.
[47] Fadi Dornaika and Bogdan Raducanu. Three-Dimensional Face Pose Dtection and Tracking UsingMonocular Videos: Tool and Application [J]. IEEE Transactions on systems,2009(8):935-944.
[48]王磊，邹北骥，彭小宁，潘丽丽.基于高斯过程的表情动作单元跟踪技术.电子学报[J],2007(11):2087-2091.
[49]王磊，邹北骥，彭小宁.针对表情动作单元跟踪的隧道隐变量法[J].自动化学报,2009(2):198-201.
[50] Wang, Xinzhu, Tian Yantao, Liu Shuaishi, Li Jinsong, Peng, Cheng. Face detection and trackingalgorithm in video images with complex background,2010[C].Tian Jin: Publisher ominous,2010.
[51]史于心.基于混合特征的面部表情识别算法研究[D].长春：吉林大学，2007.
[52]陆慧聪.面部表情识别系统中表情特征提取与识别算法的研究[D].南京：东南大学，2006.
[53]李晓雨.基于序列图像的表情特征提取和识别[D]大连：大连理工大学，2006.
[54]徐雪绒.基于单张正面照片的三维人脸建模及表情合成的研究[D].成都：西南交通大学，2011.
[55]王磊.人脸表情自动提取与跟踪技术研究[D].长沙：湖南大学，2007.
[56]朱涛.流形学习方法在图形处理中的应用研究[D].北京：北京交通大学，2009.
[57]刘学胜.基于PCA和SVM算法的人脸识别[D].计算机与数字工程，2011(7):124-126.
[2] Yumei Fan, Ning Cheng, Zhiliang Wang, Jiwei Liu, Changsheng Zhu. Real-Time Facial ExpressionRecognition system Based on HMM and Feature Point Localization [J]. Lecture Notes inComputer Science,2005:210-217.
[3] Y. Tian, T. Kanade, and J.F. Cohn. Facial Expression Analysis Handbook of Face Recognition [J].Springer,2003(10).
[4] Yu J G, Bhanu B. Evolutionary feature synthesis for facial expression recognition [J]. PatternRecognition Letters,2006,27(11):1289-1298
[5] Suwa M，Sugie N，Fujimora K．A preliminary note on pattern recognition of human emotionalexpression [C]． In： Proceedings of the Fourth International Joint Conference on PatternRecognition．Kyoto，Japan，1978：408-410．
[6]辛静.基于帧间灰度差的动态表情识别[D].天津：天津大学，2009.
[7] Mase K. Recognition of facial expression from optical flow [J]. IEICE transactions.1991：3474-3483.
[8] Ii-fan A．Essa，Alex Pentland．A vision system for observing and extracting facial action parameters[C]//Proceedings of the Computer Vision and Pattern Recognition Conference, IEEE ComputerSociety，1994：76-83
[9] Ming—Hsuan Yang，David J．Kriegman，and Narendra Ahuja，Detecting Faces in Images：A Survey[J]．IEEE. Transactions on Pattern Analysis and Machine Intelligence，2002(7)．
[10] Rana El Kaliouby and Peter Robinson. Generalization of a vision-based computational model ofmind-reading [C].1st Intl. Conf. on Affective Computing and Intelligent Interaction,2005.
[11] Jin Hui, Gao Wen, The human facial combined expression recognition system [J]. Chinese Journalof Computers,2000,23(6):602-608.
[12]薛雨丽，毛峡，郭叶，吕善伟.人机交互中的人脸表情识别研究进展[J].中国图像图形学报，2009(5):764-772
[13] Peng Yang, Qingshan Liu, Dimitris N. Metaxas. Boosting encoded dynamic features forexpression recognition [J]. Pattern Recognition Letters,2009:132-139.
[14]崔洁.面部表情识别方法研究.西北工业大学硕士学位论文[D]，2006
[15] Fadi Dornaika and Frank Davoine. On Appearance Based Face and Facial Action Tracking [J].IEEE Transactions on circuits and system for video technology,2006(9):566-571.
[16] J. Ahlberg, CANDIDE-3—An updated parameterized face. Dept. Elect. Eng., Linkoping Univ.,Sweden,2001, Rep. LiTH-ISY-R-2326
[17]刘晓旻，谭华春，章毓晋.人脸表情识别研究的新进展[J].中国图像图形学报，2006(10):1359-1368.
[18] J. Ahlberg. An active model for facial feature tracking [J]. DURASIP J. Appl. Signal Process.2002(6):566-571.
[19] J. Ahlberg. Model-based coding: Extraction, coding, and evaluation of face model parameters [D].Ph.D. dissertation, Linkoping Univ., Linkoping, Sweden,2002(9).
[20] F. Dornaika and J. Ahlberg. Fast and reliable active appearance model search for3-D face tracking[J].IEEE Trans. Syst. Man and Cybernetics,2004(8):1838-1853.
[21] F. Dornaika and F. Davoine, On appearance based face and facial action tracking [J], IEEE Trans.Circuits Syst. Video Technol.2006(9):1107-1124.
[22] G. J. Edwards, T.F. Cootes, and C. j. Taylor, Interpreting Face images using Active AppearanceModels [C]. Proc.3rd Int. Conf. on Automatic Face and Gesture Recognition, Nara, Japan,1998:300-305,.
[23] Allan D. Jepson, David J. Fleet, Thomas F. EI-Maraghi. Robust Online Appearance Models forVisual Tracking [C]. IEEE Conference on Computer Vision and Pattern Recognition, Kauai,2001:415-422.
[24] D. J. Fleet and A.D. Jepson. Stability of phase information [J]. IEEE Trans. PAMI,1993,15(12):1253-1268.
[25] Chai Xiujuan, Shan shiguang, Gao Wen et al. Example-based learning for automatic facealignment [J]. Journal of Software.2005,16(5):718-726.
[26]胡峰松，张茂军，邹北冀，马俊容.基于HMM的单样本可变光照、姿态人脸识别[J].计算机学报，2009(7):1424-1433.
[27] J. Strom, F. Davoine, J. Ahlberg, H. Li and R. Forchheimer. Very low bit rate facial texture coding
[C]. International Workshop on Synthetic-Natural Hybr.
[28] F. Bourel, el al. Robust Facial Expression Recognition Using a State-Based Model ofSpatially-Localised Facial Dynamics [C]. Proc Fifth IEEE Int.1Conf on Automatic Face andGesture Recognition (FGR-02),Washington D.C.2002:113-118.
[29] T. Kanade, J. Cohn, and Y. Tian. Comprehensive database for facial expression analysis [C]. inProc. IEEE int. Conf. Face and Gesture Recognition,2000(3):46-53.
[30] M. Pantic and L. J. M. Rothkrantz. Exper system for automatic analysis of facial expression [J].Image and Computing Journal,2000(8):881-905.
[31] Irene Kotsia and Ioannis Pitas. Facial Expression Recognition in Image Sequences UsingGeometric Deformation Features and Support Vector Machines [J]. IEEE Transactions on imageprocessing,2007(7):172-187.
[32] Richard O. Duda, Peter E. Hart, David G. Stork. Pattern Classification.
[33] Seung H. S., Lee D. D. The manifold ways of perception [J].Science,2000.
[34] Tenbaum J., Silva D. D., Langford J. A global geometric frame work for nonlinear dim ensionalityreduction [J]. Science,2000.
[35] Roweis S., Saul L. Nonlinear dimensionality reduction by locally linear embedding [J].Science,2000.
[36]章毓晋等.基于子空间的人脸识别[M].北京：清华大学出版社，2009.
[37] Cortes C., Vapnik V., Support-vector networks [J]. Machine Learning,1995:276-297.
[38]周宽久,张世荣.支持向量机分类算法研究[J].计算机工程与应用,2009(1):159-182.
[39] Schapire R. E. and Singer Y. Improved boosting algorithms using confidence-rated predictions [J].Machine Learning,1999(3).
[40]王宇博，艾海周，武勃，黄畅.人脸表情的实时分类.计算机辅助设计与图形学学报，2005(6):1296-1301.
[41] Peng Yang, Qingshan Liu, Xinyi Cui and Dimitris N.Metaxas. Facial Expression RecognitionUsing Encoded Dynamic Features [J]. Computer Vision and Pattern Recognition,2008:132-139.
[42] Xuchun Li, Lei Wang, Eric Sung. Adaboost with SVM-based component classifiers. EngineeringApplication of Artifical Inteligence,2008:785-795.
[43]王黎艳.基于Gabor小波变换和SVM的人脸表情识别[D].南京：南京理工大学.
[44] Ekman P., Friesen W. V., Facial Action Coding System: A Technique for the measurement of FacialMovement [M], Palo Alto Consulting Psychologists Press,1978.
[45] Cohn J. F., Zlochower A. J., Lien J. J., et al. Feature-point tracking by optical flow discriminationssubtle differences in facial expression [A]. Proceedings of the third IEEE InternationalConference on Automatic Face and Gesture Recognition [C], Nara, Japan,1998.
[46] Lucas D.Terissi, Juan C.Gomez.3D head pose and facial expression tracking using as singlecamera [J]. Journal of Universal Computer Science,2010(6):903-920.
[47] Fadi Dornaika and Bogdan Raducanu. Three-Dimensional Face Pose Dtection and Tracking UsingMonocular Videos: Tool and Application [J]. IEEE Transactions on systems,2009(8):935-944.
[48]王磊，邹北骥，彭小宁，潘丽丽.基于高斯过程的表情动作单元跟踪技术.电子学报[J],2007(11):2087-2091.
[49]王磊，邹北骥，彭小宁.针对表情动作单元跟踪的隧道隐变量法[J].自动化学报,2009(2):198-201.
[50] Wang, Xinzhu, Tian Yantao, Liu Shuaishi, Li Jinsong, Peng, Cheng. Face detection and trackingalgorithm in video images with complex background,2010[C].Tian Jin: Publisher ominous,2010.
[51]史于心.基于混合特征的面部表情识别算法研究[D].长春：吉林大学，2007.
[52]陆慧聪.面部表情识别系统中表情特征提取与识别算法的研究[D].南京：东南大学，2006.
[53]李晓雨.基于序列图像的表情特征提取和识别[D]大连：大连理工大学，2006.
[54]徐雪绒.基于单张正面照片的三维人脸建模及表情合成的研究[D].成都：西南交通大学，2011.
[55]王磊.人脸表情自动提取与跟踪技术研究[D].长沙：湖南大学，2007.
[56]朱涛.流形学习方法在图形处理中的应用研究[D].北京：北京交通大学，2009.
[57]刘学胜.基于PCA和SVM算法的人脸识别[D].计算机与数字工程，2011(7):124-126.