基于约束性循环一致生成对抗网络的人脸表情识别方法
|
摘要
深度神经网络在表情分类识别中会由于样本数量不均衡,样本相似度高出现过拟合,导致识别结果不理想,而通过旋转变换、添加噪声等数据增强方法不能解决相似度较高的问题。为此,对循环一致生成对抗网络(Cycle-GAN)进行改进,通过引入类别约束性条件,构建一种基于约束性循环一致生成对抗网络(CCycle-GAN),实现了一对多的数据类别映射,减少了模型训练开销;同时,为了提高人脸表情识别效率,对Cycle-GAN判别器进行改进,用一个辅助表情分类器替换循环一致生成对抗网络的判别器,不仅可以判断输入图像的真假,而且可以对表情进行分类。在CK+和FER2013数据集上的实验结果表明,该方法可以有效地解决神经网络中的过拟合和样本不均问题,提高了表情样本生成质量,进而提高了人脸表情识别率。
Due to the similar features of samples and the imbalance of the sample category,over-fitting occurred in the expression classification and recognition of deep neural network,resulting in unsatisfactory experimental results. Data enhancement methods such as rotation transformation and noise addition cannot solve the problem of high similarity. To this end,this paper improves the cycleconsistent generative adversarial networks( Cycle-GAN) and constructs a constraint cycle-consistent generative adversarial Networks( CCycle-GAN) by introducing the class-constrained condition,which implements one-to-many data category mapping and reduces the costs of model training. At the same time,in order to improve the efficiency of facial expression recognition,the discriminator of CycleGAN is replaced by an auxiliary facial expression classifier. This improvement not only discriminates the authenticity of the input image,but also classifies the expression. The experimental results on the CK + and FER2013 datasets show that the proposed method can effectively solve the problem of over-fitting and sample imbalance,also improve the quality of generated images,thereby improve the recognition rate of facial expressions.
Due to the similar features of samples and the imbalance of the sample category,over-fitting occurred in the expression classification and recognition of deep neural network,resulting in unsatisfactory experimental results. Data enhancement methods such as rotation transformation and noise addition cannot solve the problem of high similarity. To this end,this paper improves the cycleconsistent generative adversarial networks( Cycle-GAN) and constructs a constraint cycle-consistent generative adversarial Networks( CCycle-GAN) by introducing the class-constrained condition,which implements one-to-many data category mapping and reduces the costs of model training. At the same time,in order to improve the efficiency of facial expression recognition,the discriminator of CycleGAN is replaced by an auxiliary facial expression classifier. This improvement not only discriminates the authenticity of the input image,but also classifies the expression. The experimental results on the CK + and FER2013 datasets show that the proposed method can effectively solve the problem of over-fitting and sample imbalance,also improve the quality of generated images,thereby improve the recognition rate of facial expressions.
引文
[1]LIAO C T,CHUANG H J,DUAN C H,et al.Learning spatial weighting for facial expression analysis via constrained quadratic programming[J].Pattern Recognition,2013,46(11):3103-3116.
[2]刘娟,胡敏,黄忠.Gabor多方向特征融合与分块统计的表情识别[J].电子测量与仪器学报,2015,29(11):1698-1705.LIU J,HU M,HUANG ZH.Expression recognition based on Gabor multi-orientation features fusion and block statistics[J].Journal of Electronic Measurement and Instrumentation,2015,29(11):1698-1705.
[3]齐梅,李艳秋.WGC特征描述的人脸表情识别[J].电子测量与仪器学报,2017,31(4):566-572.QI M,LI Y Q.Facial expression recognition of WGCfeature description[J].Journal of Electronic Measurement and Instrumentation,2017,31(4):566-572.
[4]王小玉,张亚洲,陈德运.基于多块局部二值模式特征和人眼定位的人脸检测[J].仪器仪表学报,2014,35(12):2739-2745.WANG X Y,ZHANG Y ZH,CHEN D Y.Face detection based on MB-LBP and eye tracking[J].Chinese Journal of Scientific Instrumentation,2014,35(12):2739-2745.
[5]DOGNERY SINALY SILUE.基于多特征融合的人脸识别方法[J].电子测量技术,2018,41(20):142-146.DOGNERY SINALY SILUE.The face recognition method based on the fusion of multi-features[J].Electronic Measurement Technology,2018,41(20):142-146.
[6]HAMESTER D,BARROS P,WERMTER S.Face expression recognitionwith a 2-channel convolutional neural network[C].International Joint Conference on Neural Networks(IJCNN),IEEE,2015.
[7]NGUYEN D,NGUYEN K,SRIDHARAN S,et al.Deep spatiotemporal features for multimodal emotionrecognition[C].IEEEWinter Conference on Applications of Computer Vision(WACV),2017:1215-1223.
[8]徐新飞,刘惠义.基于卷积神经网络的人脸表情识别[J].国外电子测量技术,2018,37(1):106-110.XU X F,LIU H Y.Facial expression recognition based on convolutional neural network[J].Foreign Electronic Measurement Technology,2018,37(1):106-110.
[9]FAN Y,LU X,LI D,et al.Video-based emotion recognition using cnn-rnn and c3d hybrid networks[C].18th ACM International Conference on Multimodal Interaction,2016:445-450.
[10]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].Computer Vision and Pattern Recognition,2014,ar Xiv:1409.1556.
[11]周先春,周杰,许瑞.多样本扩充的常规协同与逆线性回归结合的人脸识别算法[J].电子测量与仪器学报,2018,32(6):96-101.ZHOU X CH,ZHOU J,XU R.New algorithm for face recognition based on the combination of multi-sample conventional collaborative and inverse linear regression[J].Journal of Electronic Measurement and Instrumentation,2018,32(6):96-101.
[12]LOPES A T,DE AGUIAR E,OLIVEIRA-SANTOS T.Afacial expression recognition system using convolutional networks[C].28th SIBGRAPI Conference on Graphics,Patterns and Images(SIBGRAPI),IEEE,2015:273-280.
[13]孙晓,潘汀,任福继.基于ROI-KNN卷积神经网络的面部表情识别[J].自动化学报,2016,42(6):883-891.SUN X,PAN T,REN F J.Facial expression recognition using ROI-KNN deep convolutional neural networks[J].Acta Automatica Sinica,2016,42(6):883-891.
[14]GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C].International Conference on Neural Information Processing Systems,2014:2672-2680.
[15]ZHU J Y,PARK T,ISOLA P,et al.Unpaired imageto-image translation using cycle-consistent adversarial networks[J].Computer Vision and Pattern Recognition,2017,ar Xiv:1703.10593.
[16]GOODFELLOW I J,ERHAN D,CARRIER P L,et al.Challenges in representation learning:A report on three machine learning contests[C].International Conference on Neural Information Processing,2013:117-124.
[17]LUCEY P,COHN J F,KANADE T,et al.The extended cohn-kanade dataset(ck+):A complete dataset for action unit and emotion-specified expression[C].IEEEComputer Society Conference on Computer Vision and Pattern Recognition Workshops(CVPRW),2010:94-101.
[18]MENG Z,LIU P,CAI J,et al.Identity-aware convolutional neural network for facial expression recognition[C].12th IEEE International Conference on Automatic Face&Gesture Recognition(FG 2017),2017:558-565.
[19]李勇,林小竹,蒋梦莹.基于跨连接Le Net-5网络的面部表情识别[J].自动化学报,2018,44(1):176-182.LI Y,LIN X ZH,JIANG M Y.Facial expression recognition with cross-connect Le Net-5 network[J].Acta Automatica Sinica,2018,44(1):176-182.
[20]PITALOKA D A,WULANDARI A,BASARUDDIN T,et al.Enhancing CNN with preprocessing stage in automatic emotion recognition[J].Procedia Computer Science,2017(116):523-529.
[2]刘娟,胡敏,黄忠.Gabor多方向特征融合与分块统计的表情识别[J].电子测量与仪器学报,2015,29(11):1698-1705.LIU J,HU M,HUANG ZH.Expression recognition based on Gabor multi-orientation features fusion and block statistics[J].Journal of Electronic Measurement and Instrumentation,2015,29(11):1698-1705.
[3]齐梅,李艳秋.WGC特征描述的人脸表情识别[J].电子测量与仪器学报,2017,31(4):566-572.QI M,LI Y Q.Facial expression recognition of WGCfeature description[J].Journal of Electronic Measurement and Instrumentation,2017,31(4):566-572.
[4]王小玉,张亚洲,陈德运.基于多块局部二值模式特征和人眼定位的人脸检测[J].仪器仪表学报,2014,35(12):2739-2745.WANG X Y,ZHANG Y ZH,CHEN D Y.Face detection based on MB-LBP and eye tracking[J].Chinese Journal of Scientific Instrumentation,2014,35(12):2739-2745.
[5]DOGNERY SINALY SILUE.基于多特征融合的人脸识别方法[J].电子测量技术,2018,41(20):142-146.DOGNERY SINALY SILUE.The face recognition method based on the fusion of multi-features[J].Electronic Measurement Technology,2018,41(20):142-146.
[6]HAMESTER D,BARROS P,WERMTER S.Face expression recognitionwith a 2-channel convolutional neural network[C].International Joint Conference on Neural Networks(IJCNN),IEEE,2015.
[7]NGUYEN D,NGUYEN K,SRIDHARAN S,et al.Deep spatiotemporal features for multimodal emotionrecognition[C].IEEEWinter Conference on Applications of Computer Vision(WACV),2017:1215-1223.
[8]徐新飞,刘惠义.基于卷积神经网络的人脸表情识别[J].国外电子测量技术,2018,37(1):106-110.XU X F,LIU H Y.Facial expression recognition based on convolutional neural network[J].Foreign Electronic Measurement Technology,2018,37(1):106-110.
[9]FAN Y,LU X,LI D,et al.Video-based emotion recognition using cnn-rnn and c3d hybrid networks[C].18th ACM International Conference on Multimodal Interaction,2016:445-450.
[10]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].Computer Vision and Pattern Recognition,2014,ar Xiv:1409.1556.
[11]周先春,周杰,许瑞.多样本扩充的常规协同与逆线性回归结合的人脸识别算法[J].电子测量与仪器学报,2018,32(6):96-101.ZHOU X CH,ZHOU J,XU R.New algorithm for face recognition based on the combination of multi-sample conventional collaborative and inverse linear regression[J].Journal of Electronic Measurement and Instrumentation,2018,32(6):96-101.
[12]LOPES A T,DE AGUIAR E,OLIVEIRA-SANTOS T.Afacial expression recognition system using convolutional networks[C].28th SIBGRAPI Conference on Graphics,Patterns and Images(SIBGRAPI),IEEE,2015:273-280.
[13]孙晓,潘汀,任福继.基于ROI-KNN卷积神经网络的面部表情识别[J].自动化学报,2016,42(6):883-891.SUN X,PAN T,REN F J.Facial expression recognition using ROI-KNN deep convolutional neural networks[J].Acta Automatica Sinica,2016,42(6):883-891.
[14]GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C].International Conference on Neural Information Processing Systems,2014:2672-2680.
[15]ZHU J Y,PARK T,ISOLA P,et al.Unpaired imageto-image translation using cycle-consistent adversarial networks[J].Computer Vision and Pattern Recognition,2017,ar Xiv:1703.10593.
[16]GOODFELLOW I J,ERHAN D,CARRIER P L,et al.Challenges in representation learning:A report on three machine learning contests[C].International Conference on Neural Information Processing,2013:117-124.
[17]LUCEY P,COHN J F,KANADE T,et al.The extended cohn-kanade dataset(ck+):A complete dataset for action unit and emotion-specified expression[C].IEEEComputer Society Conference on Computer Vision and Pattern Recognition Workshops(CVPRW),2010:94-101.
[18]MENG Z,LIU P,CAI J,et al.Identity-aware convolutional neural network for facial expression recognition[C].12th IEEE International Conference on Automatic Face&Gesture Recognition(FG 2017),2017:558-565.
[19]李勇,林小竹,蒋梦莹.基于跨连接Le Net-5网络的面部表情识别[J].自动化学报,2018,44(1):176-182.LI Y,LIN X ZH,JIANG M Y.Facial expression recognition with cross-connect Le Net-5 network[J].Acta Automatica Sinica,2018,44(1):176-182.
[20]PITALOKA D A,WULANDARI A,BASARUDDIN T,et al.Enhancing CNN with preprocessing stage in automatic emotion recognition[J].Procedia Computer Science,2017(116):523-529.