基于改进Sequence-to-Sequence模型的文本摘要生成方法
|
摘要
基于循环神经网络和注意力机制的Sequence-to-Sequence模型神经网络方法在信息抽取和自动摘要生成方面发挥了重要作用。然而,该方法不能充分利用文本的语言特征信息,且生成结果中存在未登录词问题,从而影响文本摘要的准确性和可读性。为此,利用文本语言特征改善输入的特性,同时引入拷贝机制缓解摘要生成过程未登录词问题。在此基础上,提出基于Sequence-to-Sequence模型的新方法 Copy-Generator模型,以提升文本摘要生成效果。采用中文摘要数据集LCSTS为数据源进行实验,结果表明所提方法能够有效地提高生成摘要的准确率,可应用于自动文本摘要提取任务。
The neural network method based on Sequence-to-Sequence model with Recurrent Neural Networks(RNN)and attention mechanism plays an important role in information extraction and automatic summary generation. However, this method cannot take full advantage of the linguistic features of text, and has the problem of out-of-vocabulary in the generated summarization, which influences the accuracy and readability of text summarization. To address the above problems,using text linguistics features to improve the input features, and introducing copy mechanism to alleviate the out-of-vocabulary problem in the process of summarization generation, this paper proposes a new method named Copy-Generator model based on Sequence-to-Sequence model to promote the generated summarization result. Taking the Chinese summarization dataset LCSTS as data source, the experimental results show that the proposed method can improve the accuracy of generated summarization, and can be applied to large-scale automatic text summarization task.
The neural network method based on Sequence-to-Sequence model with Recurrent Neural Networks(RNN)and attention mechanism plays an important role in information extraction and automatic summary generation. However, this method cannot take full advantage of the linguistic features of text, and has the problem of out-of-vocabulary in the generated summarization, which influences the accuracy and readability of text summarization. To address the above problems,using text linguistics features to improve the input features, and introducing copy mechanism to alleviate the out-of-vocabulary problem in the process of summarization generation, this paper proposes a new method named Copy-Generator model based on Sequence-to-Sequence model to promote the generated summarization result. Taking the Chinese summarization dataset LCSTS as data source, the experimental results show that the proposed method can improve the accuracy of generated summarization, and can be applied to large-scale automatic text summarization task.
引文
[1] Das D,Martins A F T.A survey on automatic text summarization[J].Literature Survey for the Language and Statistics II Course at CMU,2007,4:192-195.
[2]胡侠,林晔,王灿,等.自动文本摘要技术综述[J].情报杂志,2010,29(8):144-147.
[3]王振超,孙锐,姬东鸿.基于事件指导的多文档生成式摘要方法[J].计算机应用研究,2017,34(2):343-346.
[4] Jean S,Firat O,Cho K,et al.Montreal neural machine translation systems for WMT’15[C]//Proceedings of the Tenth Workshop on Statistical Machine Translation,Lisboa,2015:134-140.
[5] Bahdanau D,Chorowski J,Serdyuk D,et al.End-to-end attention-based large vocabulary speech recognition[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing,Shanghai,China,2016:4945-4949.
[6] Kaikhah K.Automatic text summarization with neural networks[C]//Proceedings of the 2nd International IEEE Intelligent Systems Conference,Varna,2004:40-44.
[7]饶高琦,于东,荀恩东,等.基于自然标注信息和隐含主题模型的无监督文本特征抽取[J].中文信息学报,2015,29(6):141-149.
[8]邓崇崴.词性特征提取及其在文本分析中的应用[D].天津:天津大学,2014.
[9]韩艳,林煜熙,姚建民.基于统计信息的未登录词的扩展识别方法[J].中文信息学报,2009,23(3):24-30.
[10] Rush A M,Chopra S,Weston J.A neural attention model for abstractive sentence summarization[C]//Proceeedings of EMNLP 2015,2015.
[11] Takase S,Suzuki J,Okazaki N,et al.Neural headline generation on abstract meaning representation[C]//Proceedings of the 2016 Conference on Empirical Methods for Natural Language Processing,Austin,2016:1054-1059.
[12] Sutskever I,Vinyals O,Le Q V.Sequence to sequence learning with neural networks[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems,Montreal,2014:3104-3112.
[13] Chopra S,Auli M,Rush A M.Abstractive sentence summarization with attentive recurrent neural networks[C]//Proceeedings of Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,San Diego,2016:93-98.
[14] Hu B,Chen Q,Zhu F.LCSTS:A large scale Chinese short text summarization dataset[C]//Proceeedings of Conference on Empirical Methods in Natural Language Processing,Lisbon,2015:1967-1972.
[15] Shi Y,Wiggers P,Jonker C M.Towards recurrent neural networks language models with linguistic and contextual features[C]//Proceeedings of 13th Annual Conference of the International Speech Communication Association,Portland,2012:1664-1667.
[16] Padmalahari E,Kumar D V N S,Prasad S.Automatic text summarization with statistical and linguistic features using successive thresholds[C]//Proceeedings of International Conference on Advanced Communication Control and Computing Technologies,2014:1519-1524.
[17] Nallapati R,Zhou B,Gulcehre C,et al.Abstractive text summarization using sequence-to-sequence rnns and beyond[C]//Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning,Berlin,2016:280-290.
[18] Gulcehre C,Ahn S,Nallapati R,et al.Pointing the unknown words[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,Berlin,2016:140-149.
[19] Gu J,Lu Z,Li H,et al.Incorporating copying mechanism in sequence-to-sequence learning[C]//Proceedings of the54th Annual Meeting of the Association for Computational Linguistics,Berlin,2016:1631-1640.
[20] Vinyals O,Fortunato M,Jaitly N.Pointer networks[C]//Advances in Neural Information Processing Systems,Montreal,2015:2692-2700.
[21] See A,Liu P J,Manning C D.Get to the point:summarization with pointer-generator networks[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,Vancouver,2017:1073-1083.
[22] Sennrich R,Haddow B.Linguistic input features improve neural machine translation[C]//Proceedings of the First Conference on Machine Translation,Berlin,2016:83-91.
[23] Mikolov T,Sutskever I,Chen K,et al.Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems,Lake Tahoe,2013:3111-3119.
[24] Memisevic R,Zach C,Hinton G E,et al.Gated softmax classification[C]//Proceedings of the 23th International Conference on Neural Information Processing Systems,Vancouver,2010:1603-1611.
[25] Mnih V,Heess N,Graves A.Recurrent models of visual attention[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems,Montreal,2014:2204-2212.
[26] Bahdanau D,Cho K,Bengio Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of ICLR 2015,2015.
[27] Lin C Y,Hovy E.Automatic evaluation of summaries using N-gram co-occurrence statistics[C]//Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology,Edmonton,2003:71-78.
[2]胡侠,林晔,王灿,等.自动文本摘要技术综述[J].情报杂志,2010,29(8):144-147.
[3]王振超,孙锐,姬东鸿.基于事件指导的多文档生成式摘要方法[J].计算机应用研究,2017,34(2):343-346.
[4] Jean S,Firat O,Cho K,et al.Montreal neural machine translation systems for WMT’15[C]//Proceedings of the Tenth Workshop on Statistical Machine Translation,Lisboa,2015:134-140.
[5] Bahdanau D,Chorowski J,Serdyuk D,et al.End-to-end attention-based large vocabulary speech recognition[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing,Shanghai,China,2016:4945-4949.
[6] Kaikhah K.Automatic text summarization with neural networks[C]//Proceedings of the 2nd International IEEE Intelligent Systems Conference,Varna,2004:40-44.
[7]饶高琦,于东,荀恩东,等.基于自然标注信息和隐含主题模型的无监督文本特征抽取[J].中文信息学报,2015,29(6):141-149.
[8]邓崇崴.词性特征提取及其在文本分析中的应用[D].天津:天津大学,2014.
[9]韩艳,林煜熙,姚建民.基于统计信息的未登录词的扩展识别方法[J].中文信息学报,2009,23(3):24-30.
[10] Rush A M,Chopra S,Weston J.A neural attention model for abstractive sentence summarization[C]//Proceeedings of EMNLP 2015,2015.
[11] Takase S,Suzuki J,Okazaki N,et al.Neural headline generation on abstract meaning representation[C]//Proceedings of the 2016 Conference on Empirical Methods for Natural Language Processing,Austin,2016:1054-1059.
[12] Sutskever I,Vinyals O,Le Q V.Sequence to sequence learning with neural networks[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems,Montreal,2014:3104-3112.
[13] Chopra S,Auli M,Rush A M.Abstractive sentence summarization with attentive recurrent neural networks[C]//Proceeedings of Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,San Diego,2016:93-98.
[14] Hu B,Chen Q,Zhu F.LCSTS:A large scale Chinese short text summarization dataset[C]//Proceeedings of Conference on Empirical Methods in Natural Language Processing,Lisbon,2015:1967-1972.
[15] Shi Y,Wiggers P,Jonker C M.Towards recurrent neural networks language models with linguistic and contextual features[C]//Proceeedings of 13th Annual Conference of the International Speech Communication Association,Portland,2012:1664-1667.
[16] Padmalahari E,Kumar D V N S,Prasad S.Automatic text summarization with statistical and linguistic features using successive thresholds[C]//Proceeedings of International Conference on Advanced Communication Control and Computing Technologies,2014:1519-1524.
[17] Nallapati R,Zhou B,Gulcehre C,et al.Abstractive text summarization using sequence-to-sequence rnns and beyond[C]//Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning,Berlin,2016:280-290.
[18] Gulcehre C,Ahn S,Nallapati R,et al.Pointing the unknown words[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,Berlin,2016:140-149.
[19] Gu J,Lu Z,Li H,et al.Incorporating copying mechanism in sequence-to-sequence learning[C]//Proceedings of the54th Annual Meeting of the Association for Computational Linguistics,Berlin,2016:1631-1640.
[20] Vinyals O,Fortunato M,Jaitly N.Pointer networks[C]//Advances in Neural Information Processing Systems,Montreal,2015:2692-2700.
[21] See A,Liu P J,Manning C D.Get to the point:summarization with pointer-generator networks[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,Vancouver,2017:1073-1083.
[22] Sennrich R,Haddow B.Linguistic input features improve neural machine translation[C]//Proceedings of the First Conference on Machine Translation,Berlin,2016:83-91.
[23] Mikolov T,Sutskever I,Chen K,et al.Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems,Lake Tahoe,2013:3111-3119.
[24] Memisevic R,Zach C,Hinton G E,et al.Gated softmax classification[C]//Proceedings of the 23th International Conference on Neural Information Processing Systems,Vancouver,2010:1603-1611.
[25] Mnih V,Heess N,Graves A.Recurrent models of visual attention[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems,Montreal,2014:2204-2212.
[26] Bahdanau D,Cho K,Bengio Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of ICLR 2015,2015.
[27] Lin C Y,Hovy E.Automatic evaluation of summaries using N-gram co-occurrence statistics[C]//Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology,Edmonton,2003:71-78.