基于编解码网络的多姿态人脸图像正面化方法
|
摘要
多姿态人脸图像正面化可以缓解头部姿态变化对人脸分析任务的影响.以往直接从多姿态人脸图像合成正面人脸图像的方法存在细节特征缺失的问题.针对这一问题,本文提出一种基于编解码网络的多姿态人脸图像正面化方法——多任务卷积编解码网络(MCEDN).该方法引入正面基础特征网络合成正面人脸基础特征,并在此基础上融合编码网络提取的多姿态人脸局部特征进行细节补偿,最终合成更加清晰的正面人脸图像.利用多任务学习机制建立端到端模型,统一局部特征提取、正面基础特征解析、正面图像合成3个模块,通过共享参数提升整个模型的效果.与已有方法对比, MCEDN在多个数据集上都可以合成结构稳定、细节清晰的正面人脸图像.我们直接使用合成的正面人脸图像进行人脸识别和表情识别,识别准确率达到先进水平,这表明MCEDN可以有效保留人脸细节特征,支持人脸分析任务.
Multi-pose face frontalization can alleviate the influence of pose variance on face analysis. The traditional method of synthesizing a frontal face image directly from a multi-pose face image presents a problem in missing face details. To overcome this problem, we propose a face frontalization method based on the encoderdecoder network, namely multitask convolutional encoder-decoder network(MCEDN). The MCEDN introduces a frontal raw feature network to synthesize the global raw features of the frontal face. Then, the network utilizes the decoder to synthesize a clearer frontal face image by fusing local features extracted by the encoder and global raw features. We use a multitask learning mechanism to build an end-to-end model. The method then integrates three modules, namely local feature extraction, global raw feature synthesis, and frontal image synthesis. The model performance was improved by sharing parameters. In comparison with existing methods, MCEDN can synthesize frontal face images with a stable structure and rich details on multiple datasets. Then, we use the synthesized frontal images for face recognition and face expression recognition, and the state-of-the-art results demonstrate that the MCEDN preserves a number of face details.
Multi-pose face frontalization can alleviate the influence of pose variance on face analysis. The traditional method of synthesizing a frontal face image directly from a multi-pose face image presents a problem in missing face details. To overcome this problem, we propose a face frontalization method based on the encoderdecoder network, namely multitask convolutional encoder-decoder network(MCEDN). The MCEDN introduces a frontal raw feature network to synthesize the global raw features of the frontal face. Then, the network utilizes the decoder to synthesize a clearer frontal face image by fusing local features extracted by the encoder and global raw features. We use a multitask learning mechanism to build an end-to-end model. The method then integrates three modules, namely local feature extraction, global raw feature synthesis, and frontal image synthesis. The model performance was improved by sharing parameters. In comparison with existing methods, MCEDN can synthesize frontal face images with a stable structure and rich details on multiple datasets. Then, we use the synthesized frontal images for face recognition and face expression recognition, and the state-of-the-art results demonstrate that the MCEDN preserves a number of face details.
引文
1 Zhu Z Y,Luo P,Wang X G,et al.Deep learning identity-preserving face space.In:Proceedings of the IEEEInternational Conference on Computer Vision,Sydney,2013.113-120
2 Zhu Z Y,Luo P,Wang X G,et al.Multi-view perceptron:a deep model for learning face identity and view representations.In:Proceedings of the Advances in Neural Information Processing Systems,Montreal,2014.217-225
3 Argyriou A,Evgeniou T,Pontil M.Multi-task feature learning.In:Proceedings of the 20th Annual Conference on Neural Information Processing Systems,Vancouver,2006.41-48
4 Zhu X Y,Lei Z,Yan J J,et al.High-fidelity pose and expression normalization for face recognition in the wild.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,2015.787-796
5 Asthana A,Marks T K,Jones M J,et al.Fully automatic pose-invariant face recognition via 3D pose normalization.In:Proceedings of the International Conference on Computer Vision,Barcelona,2011.937-944
6 Hassner T,Harel S,Paz E,et al.Effective face frontalization in unconstrained images.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,Boston,2015.4295-4304
7 Fang S Y,Zhou D K,Cao Y P,et al.Frontal face image synthesis based on pose estimation.Comput Eng,2015,41:240-244[方三勇,周大可,曹元鹏,等.基于姿态估计的正面人脸图像合成.计算机工程,2015,41:240-244]
8 Prince S J D,Warrell J,Elder J H,et al.Tied factor analysis for face recognition across large pose differences.IEEETrans Pattern Anal Mach Intel,2008,30:970-984
9 Chai X J,Shan S G,Chen X L,et al.Locally linear regression for pose-invariant face recognition.IEEE Trans Image Process,2007,16:1716-1725
10 Wang Y N,Su J B.Multipose face image recognition based on image synthesis.Pattern Recogn Artif Intel,2015,28:848-856[王亚南,苏剑波.基于图像合成的多姿态人脸图像识别方法.模式识别与人工智能,2015,28:848-856]
11 Li Y L,Feng J F.Multi-view face synthesis using minimum bending deformation.J Comput-Aided Design Comput Graph,2011,23:1085-1090[李月龙,封举富.基于最小扭曲变换的正面人脸图像合成.计算机辅助设计与图形学学报,2011,23:1085-1090]
12 Yi X B,Chen Y.Frontal face synthesizing based on poisson image fusion under piecewise affine warp.Comput Eng Appl,2016,52:172-177[仪晓斌,陈莹.分段仿射变换下基于泊松融合的正面人脸合成.计算机工程与应用,2016,52:172-177]
13 Kan M N,Shan S G,Chang H,et al.Stacked progressive auto-encoders(spae)for face recognition across poses.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Columbus,2014.1883-1890
14 Ouyang N,Ma Y T,Lin L P.Multi-pose face reconstruction and recognition based on multi-task learning.J Comput Appl,2016,37:896-900[欧阳宁,马玉涛,林乐平.基于多任务学习的多姿态人脸重建与识别.计算机应用,2016,37:896-900]
15 Yim J,Jung H,Yoo B,et al.Rotating your face using multi-task deep neural network.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,Boston,2015.676-684
16 Ghodrati A,Jia X,Pedersoli M,et al.Towards automatic image editing:learning to see another you.2015.ArXiv:151108446
17 Huang R,Zhang S,Li T Y,et al.Beyond face rotation:global and local perception gan for photorealistic and identity preserving frontal view synthesis.2017.ArXiv:170404086
18 Tran L,Yin X,Liu X M.Disentangled representation learning gan for pose-invariant face recognition.In:Proceedings of the Computer Vision and Pattern Recognition,Honolulu,2017.1283-1292
19 Theis L,Shi W,Cunningham A,et al.Lossy image compression with compressive autoencoders.2017.ArXiv:170300395
20 Goodfellow I,Bengio Y,Courville A,et al.Deep Learning.Cambridge:MIT Press,2016
21 Mayya V,Pai R M,Pai M M.Automatic facial expression recognition using DCNN.Procedia Comput Sci,2016,93:453-461
22 Nair V,Hinton G E.Rectified linear units improve restricted boltzmann machines.In:Proceedings of International Conference on Machine Learning,Haifa,2010.807-814
23 Gross R,Matthews I,Cohn J,et al.Multi-PIE.Image Vision Comput,2010,28:807-813
24 Gao W,Cao B,Shan S G,et al.The CAS-PEAL large-scale Chinese face database and baseline evaluations.IEEETrans Syst Man Cybern A,2008,38:149-161
25 Huang G B,Ramesh M,Berg T,et al.Labeled Faces in the Wild:A Database for Studying Face Recognition in Unconstrained Environments.Technical Report 07-49.2007
26 Liu Z,Luo P,Wang X G,et al.Deep learning face attributes in the wild.In:Proceedings of the IEEE International Conference on Computer Vision,Santiago,2015.3730-3738
27 Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:from error visibility to structural similarity.IEEETrans Image Process,2004,13:600-612
28 Ding H,Zhou S K,Chellappa R.Facenet2expnet:regularizing a deep face recognition net for expression recognition.In:Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition,Washington,2017.118-126
29 Wu X,He R,Sun Z A,et al.A light CNN for deep face representation with noisy labels.2015.ArXiv:151102683
2 Zhu Z Y,Luo P,Wang X G,et al.Multi-view perceptron:a deep model for learning face identity and view representations.In:Proceedings of the Advances in Neural Information Processing Systems,Montreal,2014.217-225
3 Argyriou A,Evgeniou T,Pontil M.Multi-task feature learning.In:Proceedings of the 20th Annual Conference on Neural Information Processing Systems,Vancouver,2006.41-48
4 Zhu X Y,Lei Z,Yan J J,et al.High-fidelity pose and expression normalization for face recognition in the wild.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston,2015.787-796
5 Asthana A,Marks T K,Jones M J,et al.Fully automatic pose-invariant face recognition via 3D pose normalization.In:Proceedings of the International Conference on Computer Vision,Barcelona,2011.937-944
6 Hassner T,Harel S,Paz E,et al.Effective face frontalization in unconstrained images.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,Boston,2015.4295-4304
7 Fang S Y,Zhou D K,Cao Y P,et al.Frontal face image synthesis based on pose estimation.Comput Eng,2015,41:240-244[方三勇,周大可,曹元鹏,等.基于姿态估计的正面人脸图像合成.计算机工程,2015,41:240-244]
8 Prince S J D,Warrell J,Elder J H,et al.Tied factor analysis for face recognition across large pose differences.IEEETrans Pattern Anal Mach Intel,2008,30:970-984
9 Chai X J,Shan S G,Chen X L,et al.Locally linear regression for pose-invariant face recognition.IEEE Trans Image Process,2007,16:1716-1725
10 Wang Y N,Su J B.Multipose face image recognition based on image synthesis.Pattern Recogn Artif Intel,2015,28:848-856[王亚南,苏剑波.基于图像合成的多姿态人脸图像识别方法.模式识别与人工智能,2015,28:848-856]
11 Li Y L,Feng J F.Multi-view face synthesis using minimum bending deformation.J Comput-Aided Design Comput Graph,2011,23:1085-1090[李月龙,封举富.基于最小扭曲变换的正面人脸图像合成.计算机辅助设计与图形学学报,2011,23:1085-1090]
12 Yi X B,Chen Y.Frontal face synthesizing based on poisson image fusion under piecewise affine warp.Comput Eng Appl,2016,52:172-177[仪晓斌,陈莹.分段仿射变换下基于泊松融合的正面人脸合成.计算机工程与应用,2016,52:172-177]
13 Kan M N,Shan S G,Chang H,et al.Stacked progressive auto-encoders(spae)for face recognition across poses.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Columbus,2014.1883-1890
14 Ouyang N,Ma Y T,Lin L P.Multi-pose face reconstruction and recognition based on multi-task learning.J Comput Appl,2016,37:896-900[欧阳宁,马玉涛,林乐平.基于多任务学习的多姿态人脸重建与识别.计算机应用,2016,37:896-900]
15 Yim J,Jung H,Yoo B,et al.Rotating your face using multi-task deep neural network.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,Boston,2015.676-684
16 Ghodrati A,Jia X,Pedersoli M,et al.Towards automatic image editing:learning to see another you.2015.ArXiv:151108446
17 Huang R,Zhang S,Li T Y,et al.Beyond face rotation:global and local perception gan for photorealistic and identity preserving frontal view synthesis.2017.ArXiv:170404086
18 Tran L,Yin X,Liu X M.Disentangled representation learning gan for pose-invariant face recognition.In:Proceedings of the Computer Vision and Pattern Recognition,Honolulu,2017.1283-1292
19 Theis L,Shi W,Cunningham A,et al.Lossy image compression with compressive autoencoders.2017.ArXiv:170300395
20 Goodfellow I,Bengio Y,Courville A,et al.Deep Learning.Cambridge:MIT Press,2016
21 Mayya V,Pai R M,Pai M M.Automatic facial expression recognition using DCNN.Procedia Comput Sci,2016,93:453-461
22 Nair V,Hinton G E.Rectified linear units improve restricted boltzmann machines.In:Proceedings of International Conference on Machine Learning,Haifa,2010.807-814
23 Gross R,Matthews I,Cohn J,et al.Multi-PIE.Image Vision Comput,2010,28:807-813
24 Gao W,Cao B,Shan S G,et al.The CAS-PEAL large-scale Chinese face database and baseline evaluations.IEEETrans Syst Man Cybern A,2008,38:149-161
25 Huang G B,Ramesh M,Berg T,et al.Labeled Faces in the Wild:A Database for Studying Face Recognition in Unconstrained Environments.Technical Report 07-49.2007
26 Liu Z,Luo P,Wang X G,et al.Deep learning face attributes in the wild.In:Proceedings of the IEEE International Conference on Computer Vision,Santiago,2015.3730-3738
27 Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:from error visibility to structural similarity.IEEETrans Image Process,2004,13:600-612
28 Ding H,Zhou S K,Chellappa R.Facenet2expnet:regularizing a deep face recognition net for expression recognition.In:Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition,Washington,2017.118-126
29 Wu X,He R,Sun Z A,et al.A light CNN for deep face representation with noisy labels.2015.ArXiv:151102683