基于深度学习模型的自发学习表情识别方法研究
|
摘要
课堂教学环境中,面部表情自动识别是获取学习者情绪状态的重要方式。针对传统方法提取特征不全面、融合特征维度较高等问题,提出一种融合局部与全局特征的学习表情自动识别方法。该方法提取并融合表情图像的局部几何特征、KPCA降维的CLBP全局浅层纹理特征和CNN全局深度网络特征。此外,还构建一个全新的自发学习表情数据库,将课堂学习中的情绪分为困惑、快乐、疲倦、惊讶和中性等5种类型,用于CNN模型的训练。对比实验表明,该方法的识别正确率在CK+库、中国情绪图片系统和自发学习表情数据库中分别达到96.3%、86.7%和95.6%,高于传统的面部表情识别方法。该方法能够有效获取课堂中学生情绪变化,帮助教师准确全面地掌握班级学生的整体情况,促进课堂教学质量的提高。
Automatic facial expressions recognition is an important way to acquire the emotional state of the students in the classroom. The traditional methods often suffer from the problems of incomplete feature extraction and high dimension of fusion features. In this paper, we proposed an automatic expression recognition algorithm by extracting the local and global features of the facial expression images. The proposed algorithm extracted and fused the multi-features in facial expression images, such as the local geometric features, global shallow texture features of KPCA dimension reduced CLBP and CNN global depth network features. Moreover, we constructed a new spontaneous learning expression database. In the database, all the emotions images were classified into five types: confusion, happiness, fatigue, surprise and neutral. The developed database was utilized for the training of CNN models. Experimental results show that the recognition rates of this method in CK+ database, Chinese emotion image system and autonomous learning expression database are 96.3%, 86.7% and 95.6% respectively. It is higher than the traditional facial expression recognition methods. The proposed method can recognize the emotional changes of students in the classroom effectively. It is helpful for teachers to grasp the overall situation of classroom students accurately and comprehensively, and promote the improvement of classroom teaching quality.
Automatic facial expressions recognition is an important way to acquire the emotional state of the students in the classroom. The traditional methods often suffer from the problems of incomplete feature extraction and high dimension of fusion features. In this paper, we proposed an automatic expression recognition algorithm by extracting the local and global features of the facial expression images. The proposed algorithm extracted and fused the multi-features in facial expression images, such as the local geometric features, global shallow texture features of KPCA dimension reduced CLBP and CNN global depth network features. Moreover, we constructed a new spontaneous learning expression database. In the database, all the emotions images were classified into five types: confusion, happiness, fatigue, surprise and neutral. The developed database was utilized for the training of CNN models. Experimental results show that the recognition rates of this method in CK+ database, Chinese emotion image system and autonomous learning expression database are 96.3%, 86.7% and 95.6% respectively. It is higher than the traditional facial expression recognition methods. The proposed method can recognize the emotional changes of students in the classroom effectively. It is helpful for teachers to grasp the overall situation of classroom students accurately and comprehensively, and promote the improvement of classroom teaching quality.
引文
[1] Miňano P,Gilar R,Castejón J L.A structural model of cognitive-motivational variables as explanatory factors of academic achievement in Spanish Language and Mathematics[J].Anales De Psicología,2012,28(1):45-54.
[2] Frenzel A C, Pekrun R, Goetz T. Perceived learning environment and students’ emotional experiences: A multilevel analysis of mathematics classrooms[J]. Learning and Instruction, 2007, 17(5): 478-493.
[3] 梁迎丽,刘陈.人工智能教育应用的现状分析、典型特征与发展趋势[J].中国电化教育,2018(3):24-30.
[4] Mehrabian A,Russell J A.An approach to environmental psychology[M].MIT,1980.
[5] Kalsum T,Anwar S M,Majid M,et al.Emotion recognition from facial expressions using hybrid feature descriptors[J].IET Image Processing,2018,12(6):1004-1012.
[6] Kim D J.Facial expression recognition using ASM-based post-processing technique[J].Pattern Recognition and Image Analysis,2016,26(3):576-581.
[7] Li T,Du C,Naren T,et al.Using feature points and angles between them to recognize facial expressions by a neural network approach[J].IET Image Processing,2018,12(11).
[8] Ghimire D,Lee J,Li Z N,et al.Recognition of facial expressions based on salient geometric features and support vector machines[J].Multimedia Tools & Applications,2016,75(6):1-26.
[9] Zhang Z,Mu X,Gao L.Recognizing facial expressions based on Gabor filter selection[C]//International Congress on Image and Signal Processing.IEEE,2011:1544-1548.
[10] He J,Cai J F,Fang L Z,et al.A method of facial expression recognition based on LBP fusion of key expressions areas[C]//Control and Decision Conference.IEEE,2015:4200-4204.
[11] 周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.
[12] Sun W,Zhao H,Jin Z.A visual attention based roi detection method for facial expression recognition[J].Neurocomputing,2018,296.
[13] 罗翔云,周晓慧,付克博.基于深度学习的人脸表情识别[J].工业控制计算机,2017,30(5):92-93.
[14] Zhang S,He H,Kong L.Fusing multi-feature for video occlusion region detection based on graph cut[J].Acta Optica Sinica,2015,35(4):0415001.
[15] 兰兰,陈万忠,魏庭松.基于融合特征提取与LLE方法的表情识别[J].吉林大学学报(信息科学版),2017,35(4):384-391.
[16] Majumder A,Behera L,Subramanian V K.Automatic facial expression recognition system using deep network-based data fusion[J].IEEE Transactions on Cybernetics,2016,48(1):103-114.
[17] 付文博,孙涛,梁藉,等.深度学习原理及应用综述[J].计算机科学,2018,45(S1):11-15,40.
[18] Hijazi S,Kumar R,Rowen C,et al.Using convolutional neural networks for image recognition[OL].[2016-9-22].http://ip.cadence.com/uploads/901/cnn_wp-pdf.
[19] 徐新飞,刘惠义.基于卷积神经网络的人脸表情识别[J].国外电子测量技术,2018,37(1):106-110.
[20] Chen C.Remote sensing image scene classification using multi-scale completed local binary patterns and fisher vectors[J].Remote Sensing,2016,8(6):483.
[21] 袁宝华,王欢,任明武.基于完整LBP特征的人脸识别[J].计算机应用研究,2012,29(4):1557-1559.
[22] Wang Q.Kernel principal component analysis and its applications in face recognition and active shape models[EB].arXiv:1207.3538,2012.
[23] 白露,马慧,黄宇霞,等.中国情绪图片系统的编制——在46名中国大学生中的试用[J].中国心理卫生杂志,2005,19(11):719-722.
[24] 苏志铭,陈靓影.基于自回归模型的动态表情识别[J].计算机辅助设计与图形学学报,2017,29(6):1085-1092.
[25] 胡敏,江河,王晓华,等.基于几何和纹理特征的表情层级分类方法[J].电子学报,2017,45(1):164-172.
[26] 罗元,崔叶,王艳,等.基于融合DCT和LBP特征提取的面部表情识别[J].半导体光电,2014,35(2):330-333.
[2] Frenzel A C, Pekrun R, Goetz T. Perceived learning environment and students’ emotional experiences: A multilevel analysis of mathematics classrooms[J]. Learning and Instruction, 2007, 17(5): 478-493.
[3] 梁迎丽,刘陈.人工智能教育应用的现状分析、典型特征与发展趋势[J].中国电化教育,2018(3):24-30.
[4] Mehrabian A,Russell J A.An approach to environmental psychology[M].MIT,1980.
[5] Kalsum T,Anwar S M,Majid M,et al.Emotion recognition from facial expressions using hybrid feature descriptors[J].IET Image Processing,2018,12(6):1004-1012.
[6] Kim D J.Facial expression recognition using ASM-based post-processing technique[J].Pattern Recognition and Image Analysis,2016,26(3):576-581.
[7] Li T,Du C,Naren T,et al.Using feature points and angles between them to recognize facial expressions by a neural network approach[J].IET Image Processing,2018,12(11).
[8] Ghimire D,Lee J,Li Z N,et al.Recognition of facial expressions based on salient geometric features and support vector machines[J].Multimedia Tools & Applications,2016,75(6):1-26.
[9] Zhang Z,Mu X,Gao L.Recognizing facial expressions based on Gabor filter selection[C]//International Congress on Image and Signal Processing.IEEE,2011:1544-1548.
[10] He J,Cai J F,Fang L Z,et al.A method of facial expression recognition based on LBP fusion of key expressions areas[C]//Control and Decision Conference.IEEE,2015:4200-4204.
[11] 周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.
[12] Sun W,Zhao H,Jin Z.A visual attention based roi detection method for facial expression recognition[J].Neurocomputing,2018,296.
[13] 罗翔云,周晓慧,付克博.基于深度学习的人脸表情识别[J].工业控制计算机,2017,30(5):92-93.
[14] Zhang S,He H,Kong L.Fusing multi-feature for video occlusion region detection based on graph cut[J].Acta Optica Sinica,2015,35(4):0415001.
[15] 兰兰,陈万忠,魏庭松.基于融合特征提取与LLE方法的表情识别[J].吉林大学学报(信息科学版),2017,35(4):384-391.
[16] Majumder A,Behera L,Subramanian V K.Automatic facial expression recognition system using deep network-based data fusion[J].IEEE Transactions on Cybernetics,2016,48(1):103-114.
[17] 付文博,孙涛,梁藉,等.深度学习原理及应用综述[J].计算机科学,2018,45(S1):11-15,40.
[18] Hijazi S,Kumar R,Rowen C,et al.Using convolutional neural networks for image recognition[OL].[2016-9-22].http://ip.cadence.com/uploads/901/cnn_wp-pdf.
[19] 徐新飞,刘惠义.基于卷积神经网络的人脸表情识别[J].国外电子测量技术,2018,37(1):106-110.
[20] Chen C.Remote sensing image scene classification using multi-scale completed local binary patterns and fisher vectors[J].Remote Sensing,2016,8(6):483.
[21] 袁宝华,王欢,任明武.基于完整LBP特征的人脸识别[J].计算机应用研究,2012,29(4):1557-1559.
[22] Wang Q.Kernel principal component analysis and its applications in face recognition and active shape models[EB].arXiv:1207.3538,2012.
[23] 白露,马慧,黄宇霞,等.中国情绪图片系统的编制——在46名中国大学生中的试用[J].中国心理卫生杂志,2005,19(11):719-722.
[24] 苏志铭,陈靓影.基于自回归模型的动态表情识别[J].计算机辅助设计与图形学学报,2017,29(6):1085-1092.
[25] 胡敏,江河,王晓华,等.基于几何和纹理特征的表情层级分类方法[J].电子学报,2017,45(1):164-172.
[26] 罗元,崔叶,王艳,等.基于融合DCT和LBP特征提取的面部表情识别[J].半导体光电,2014,35(2):330-333.