基于词注意力卷积神经网络模型的情感分析研究
|
摘要
情感分类任务需要捕获文本中的情感特征,利用重要的局部特征构建文本的特征表示。卷积神经网络(convolutional neural networks,CNN)已经被证明拥有出色的特征学习能力,但是该模型无法判别输入文本中特征词与情感的相关性,卷积层缺乏对单一词特征的提取。基于目前运用非常成功的注意力模型,该文提出一种基于词注意力的卷积神经网络模型(word attention-based convolutional neural networks,WACNN)。相比于卷积神经网络,该模型以篇章的文本信息作为输入,首先在词嵌入层之后增加注意力机制层,获取重要的局部特征词,使模型有选择地进行特征提取;然后在卷积层中增加大小为1的卷积核,提取单一词的特征;最后该方法对输入文本进行适当的文本填充,保证每个词都存在上下文信息,使模型有效提取到每个词的n-grams局部特征,避免卷积处理过程中局部信息的丢失。该模型在MR5K和CR数据集上进行验证,较普通卷积神经网络和传统机器学习方法,在准确率上分别取得0.5%和2%的提升。
Sentiment classification task needs to capture the sentiment features from document and then combines them to construct the document representation.In this paper,we propose the model of Word Attention-based Convolution Neural Networks(WACNN).Compared with CNN,our model takes the document information as input.In detail,we put a word attention layer after the word embedding layer and before the CNN layer.Attention layer enables our model to focus on certain part of the input document and learn weights of each word.We also add a convolve filter with size of 1 in the convolution layer to extract the features of single word.To ensure the existance of context for each word,we pad the input of the convolution layer.This method can be used to extract the n-grams local features of each word effectively,avoiding information loss caused by convolution processing.Compared with traditional CNN and machine learning methods,the accuracy is improved by 0.5% and 2% on MR5 Kand CR datasets,respectively.
Sentiment classification task needs to capture the sentiment features from document and then combines them to construct the document representation.In this paper,we propose the model of Word Attention-based Convolution Neural Networks(WACNN).Compared with CNN,our model takes the document information as input.In detail,we put a word attention layer after the word embedding layer and before the CNN layer.Attention layer enables our model to focus on certain part of the input document and learn weights of each word.We also add a convolve filter with size of 1 in the convolution layer to extract the features of single word.To ensure the existance of context for each word,we pad the input of the convolution layer.This method can be used to extract the n-grams local features of each word effectively,avoiding information loss caused by convolution processing.Compared with traditional CNN and machine learning methods,the accuracy is improved by 0.5% and 2% on MR5 Kand CR datasets,respectively.
引文
[1]Kim Y.Convolutional neural networks for sentence classification[J].arXiv preprint arXiv:1408.5882,2014.
[2]Kalchbrenner N,Grefenstette E,Blunsom P.A convolutional neural network for modelling sentences[J].arXiv preprint arXiv:1404.2188,2014.
[3]Xu K,Ba J,Kiros R,et al.Show,attend and tell:Neural image caption generation with visual attention[C]//Proceedings of International Conference on Machine Learning,2015:2048-2057.
[4]Bahdanau D,Cho K,Bengio Y.Neural machine translation by jointly learning to align and translate[J].arXiv preprint arXiv:1409.0473,2014.
[5]Luong M T,Pham H,Manning C D.Effective approaches to attention-based neural machine translation[J].arXiv preprint arXiv:1508.04025,2015.
[6]Yin W,Schütze H,Xiang B,et al.ABCNN:Attentionbased convolutional neural network for modeling sentence pairs[J].arXiv:1512.05193v4,2015.
[7]Wang L,Cao Z,Melo G D,et al.Relation classification via multi-level attention CNNs[C]//Proceedings of Meeting of the Association for Computational Linguistics,2016:1298-1307.
[8]Liu B.Sentiment analysis and opinion mining[J].Synthesis Lectures on Human Language Technologies,2012,5(1):1-167.
[9]Pang B,Lee L,Vaithyanathan S.Thumbs up?:Sentiment classification using machine learning techniques[C]//Proceedings of the ACL-02Conference on Empirical Methods in Natural Language Processing-Volume 10.Association for Computational Linguistics,2002:79-86.
[10]Wang S,Manning C D.Baselines and bigrams:Simple,good sentiment and topic classification[C]//Proceedings of Meeting of the Association for Computational Linguistics:Short Papers.Association for Computational Linguistics,2012:90-94.
[11]Mikolov T,Chen K,Corrado G,et al.Efficient estimation of word representations in vector space[J].arXiv preprint arXiv:1301.3781,2013.
[12]Johnson R,Zhang T.Effective use of word order for text categorization with convolutional neural networks[J].arXiv preprint arXiv:1412.1058,2014.
[13]Socher R,Perelygin A,Wu J Y,et al.Recursive deep models for semantic compositionality over a sentiment treebank[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP),2013,1631:1642.
[14]Tang D,Qin B,Liu T.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing,2015:1422-1432.
[15]Zhou X,Wan X,Xiao J.Attention-based LSTM network for cross-lingual sentiment classification[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing,2016:247-256.
[16]Hu M,Liu B.Mining and summarizing customer reviews[C]//Proceedings of Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Seattle,Washington,Usa,August.DBLP,2004:168-177.
[17]Pang B,Lee L.Seeing stars:Exploiting class relationships for sentiment categorization with respect to rating scales[C]//Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics.Association for Computational Linguistics,2005:115-124.
[18]Zeiler M D.ADADELTA:An adaptive learning rate method[J].arXiv preprint arXiv:1212.5701,2012.
[19]Zhang Y,Wallace B.A sensitivity analysis of(and practitionersguide to)convolutional neural networks for sentence classification[J].arXiv preprint arXiv:1510.03820,2015.
(1)https://code.google.com/archive/p/word2vec/
[2]Kalchbrenner N,Grefenstette E,Blunsom P.A convolutional neural network for modelling sentences[J].arXiv preprint arXiv:1404.2188,2014.
[3]Xu K,Ba J,Kiros R,et al.Show,attend and tell:Neural image caption generation with visual attention[C]//Proceedings of International Conference on Machine Learning,2015:2048-2057.
[4]Bahdanau D,Cho K,Bengio Y.Neural machine translation by jointly learning to align and translate[J].arXiv preprint arXiv:1409.0473,2014.
[5]Luong M T,Pham H,Manning C D.Effective approaches to attention-based neural machine translation[J].arXiv preprint arXiv:1508.04025,2015.
[6]Yin W,Schütze H,Xiang B,et al.ABCNN:Attentionbased convolutional neural network for modeling sentence pairs[J].arXiv:1512.05193v4,2015.
[7]Wang L,Cao Z,Melo G D,et al.Relation classification via multi-level attention CNNs[C]//Proceedings of Meeting of the Association for Computational Linguistics,2016:1298-1307.
[8]Liu B.Sentiment analysis and opinion mining[J].Synthesis Lectures on Human Language Technologies,2012,5(1):1-167.
[9]Pang B,Lee L,Vaithyanathan S.Thumbs up?:Sentiment classification using machine learning techniques[C]//Proceedings of the ACL-02Conference on Empirical Methods in Natural Language Processing-Volume 10.Association for Computational Linguistics,2002:79-86.
[10]Wang S,Manning C D.Baselines and bigrams:Simple,good sentiment and topic classification[C]//Proceedings of Meeting of the Association for Computational Linguistics:Short Papers.Association for Computational Linguistics,2012:90-94.
[11]Mikolov T,Chen K,Corrado G,et al.Efficient estimation of word representations in vector space[J].arXiv preprint arXiv:1301.3781,2013.
[12]Johnson R,Zhang T.Effective use of word order for text categorization with convolutional neural networks[J].arXiv preprint arXiv:1412.1058,2014.
[13]Socher R,Perelygin A,Wu J Y,et al.Recursive deep models for semantic compositionality over a sentiment treebank[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP),2013,1631:1642.
[14]Tang D,Qin B,Liu T.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing,2015:1422-1432.
[15]Zhou X,Wan X,Xiao J.Attention-based LSTM network for cross-lingual sentiment classification[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing,2016:247-256.
[16]Hu M,Liu B.Mining and summarizing customer reviews[C]//Proceedings of Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Seattle,Washington,Usa,August.DBLP,2004:168-177.
[17]Pang B,Lee L.Seeing stars:Exploiting class relationships for sentiment categorization with respect to rating scales[C]//Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics.Association for Computational Linguistics,2005:115-124.
[18]Zeiler M D.ADADELTA:An adaptive learning rate method[J].arXiv preprint arXiv:1212.5701,2012.
[19]Zhang Y,Wallace B.A sensitivity analysis of(and practitionersguide to)convolutional neural networks for sentence classification[J].arXiv preprint arXiv:1510.03820,2015.
(1)https://code.google.com/archive/p/word2vec/