基于多层次深度卷积神经网络的图像情感分类
|
摘要
由于图像的复杂性和人类情感的主观性,图像情感分类是一项非常具有挑战性的任务.针对深度学习方法没有充分考虑图像先验信息的问题,提出一个新的多层次深度卷积神经网络框架.该框架综合考虑全局和局部视角,引入显著主体、颜色和局部等先验信息,从多个层次学习图像的情感表达.实验结果表明,在公开的大数量级和小数量级情感图库上,该框架的分类准确率均高于现有的图像情感分类方法,其平均分类准确率比最优方法提高了2.8%,特别在情感类别"厌恶"上提高了15%.
Emotion classification of images is a challenging task regarding the complexity of various images and the subjectivity of human' s emotion perception. Most existing deep learning methods didn' t consider image prior information fully. A new multi-level deep convolutional neural network was proposed to predict the emotion based on the multi-level prior information from global and local view. Extensive experiments on both the large scale and small scale emotion datasets demonstrate the effectiveness of our method. The average classification accuracy of our method is 2. 8% higher than the state-of-art method,especially 15% higher in the category "disgust".
Emotion classification of images is a challenging task regarding the complexity of various images and the subjectivity of human' s emotion perception. Most existing deep learning methods didn' t consider image prior information fully. A new multi-level deep convolutional neural network was proposed to predict the emotion based on the multi-level prior information from global and local view. Extensive experiments on both the large scale and small scale emotion datasets demonstrate the effectiveness of our method. The average classification accuracy of our method is 2. 8% higher than the state-of-art method,especially 15% higher in the category "disgust".
引文
[1] JOSHI D,DATTA R,FEDOROVSKAYA E,et al.Aesthe-tics and emotions in images [J].IEEE Signal Processing Magazine,2011,28(5):94- 115.
[2] WANG W N,YU Y L,JIANG S M.Image retrieval by emotional semantics:A study of emotional space and feature extraction [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.Taipei:IEEE,2006,4:3534- 3539.
[3] MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory [C]//Proceedings of the 18th ACM International Conference on Multimedia.Italy:ACM,2010:83- 92.
[4] ZHAO S C,GAO Y,JIANG X L,et al.Exploring principles-of-art features for image emotion recognition [C]//Proceedings of the 22nd ACM International Conference on Multimedia.USA:ACM,2014:47- 56.
[5] YANULEYSKAYA V,VAN GEMERT J C,ROTH K,et al.Emotional valence categorization using holistic image features [C]//Proceedings of the International Conference on Image Processing.San Diego:IEEE,2008:101- 104.
[6] SOLLI M,LENZ R.Color based bags-of-emotions [C]//International Conference on Computer Analysis of Images and Patterns.Berlin:Springer-Verlag Berlin Heidelberg,2009:573- 580.
[7] YOU Q Z,LUO J,JIN H L,et al.Building a large scale dataset for image emotion recognition:the fine print and the benchmark [C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.Phoenix:AAAI,2016:308- 314.
[8] MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the international affective picture system [J].Behavior Research Methods,2005,37(4):626- 630.
[9] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Image-net classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.Nevada:MIT Press,2012:1097- 1105.
[10] RAO T R,XU M,XU D.Learning multi-level deep representations for image emotion classification [J].arXiv preprint arXiv:1611.07145,2016.
[11] WANG W N,CAI D,XU X M,et al.Visual saliency detection based on region descriptors and prior knowledge [J].Signal Processing:Image Communication,2014,29(3):424- 433.
[12] ZHENG H L,CHEN T L,YOU Q Z,et al.When saliency meets sentiment:understanding how Image content invokes emotion and sentiment [J].arXiv preprint arXiv:1611.04636,2016.
[13] ZHANG J M,SCLAROFF S,ZHE L,et al.Unconstrained salient object detection via proposal subset optimization [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Veg:IEEE Computer Society,2016:5733- 5742.
[14] ITTEN J.The art of color;the subjective experience and objective rationale of colour [M].[S.l.]:Reinhold,1962.
[15] NICOLAOU M A,GUNES H,PANTIC M.A multi-layer hybrid framework for dimensional emotion classification [C]//Proceedings of the 19th ACM International Conference on Multimedia.Scottsdale:ACM,2011:933- 936.
[16] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Re-cognition.Las Veg:IEEE Computer Society,2016:770- 778.
[17] RAO T R,XU M,LIU H Y,et al.Multi-scale blocks based image emotion classification using multiple instance learning [C]//The 23rd IEEE International Conference on Image Processing.Phoenix:IEEE,2016:634- 638.
[18] ANTONISSE H J.Image segmentation in pyramids [J].Computer Graphics and Image Processing,1982,19(4):367- 383.
[19] MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the international affective picture system [J].Behavior Research Methods,2005,37(4):626- 630.
[20] HUAN R,PAN Y.Decision fusion strategies for SAR ima-ge target recognition [J].IET Radar,Sonar & Navigation,2011,5(7):747- 755.
[2] WANG W N,YU Y L,JIANG S M.Image retrieval by emotional semantics:A study of emotional space and feature extraction [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.Taipei:IEEE,2006,4:3534- 3539.
[3] MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory [C]//Proceedings of the 18th ACM International Conference on Multimedia.Italy:ACM,2010:83- 92.
[4] ZHAO S C,GAO Y,JIANG X L,et al.Exploring principles-of-art features for image emotion recognition [C]//Proceedings of the 22nd ACM International Conference on Multimedia.USA:ACM,2014:47- 56.
[5] YANULEYSKAYA V,VAN GEMERT J C,ROTH K,et al.Emotional valence categorization using holistic image features [C]//Proceedings of the International Conference on Image Processing.San Diego:IEEE,2008:101- 104.
[6] SOLLI M,LENZ R.Color based bags-of-emotions [C]//International Conference on Computer Analysis of Images and Patterns.Berlin:Springer-Verlag Berlin Heidelberg,2009:573- 580.
[7] YOU Q Z,LUO J,JIN H L,et al.Building a large scale dataset for image emotion recognition:the fine print and the benchmark [C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.Phoenix:AAAI,2016:308- 314.
[8] MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the international affective picture system [J].Behavior Research Methods,2005,37(4):626- 630.
[9] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Image-net classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.Nevada:MIT Press,2012:1097- 1105.
[10] RAO T R,XU M,XU D.Learning multi-level deep representations for image emotion classification [J].arXiv preprint arXiv:1611.07145,2016.
[11] WANG W N,CAI D,XU X M,et al.Visual saliency detection based on region descriptors and prior knowledge [J].Signal Processing:Image Communication,2014,29(3):424- 433.
[12] ZHENG H L,CHEN T L,YOU Q Z,et al.When saliency meets sentiment:understanding how Image content invokes emotion and sentiment [J].arXiv preprint arXiv:1611.04636,2016.
[13] ZHANG J M,SCLAROFF S,ZHE L,et al.Unconstrained salient object detection via proposal subset optimization [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Veg:IEEE Computer Society,2016:5733- 5742.
[14] ITTEN J.The art of color;the subjective experience and objective rationale of colour [M].[S.l.]:Reinhold,1962.
[15] NICOLAOU M A,GUNES H,PANTIC M.A multi-layer hybrid framework for dimensional emotion classification [C]//Proceedings of the 19th ACM International Conference on Multimedia.Scottsdale:ACM,2011:933- 936.
[16] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Re-cognition.Las Veg:IEEE Computer Society,2016:770- 778.
[17] RAO T R,XU M,LIU H Y,et al.Multi-scale blocks based image emotion classification using multiple instance learning [C]//The 23rd IEEE International Conference on Image Processing.Phoenix:IEEE,2016:634- 638.
[18] ANTONISSE H J.Image segmentation in pyramids [J].Computer Graphics and Image Processing,1982,19(4):367- 383.
[19] MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the international affective picture system [J].Behavior Research Methods,2005,37(4):626- 630.
[20] HUAN R,PAN Y.Decision fusion strategies for SAR ima-ge target recognition [J].IET Radar,Sonar & Navigation,2011,5(7):747- 755.