面向非任务型对话系统的人工标注中文数据集
|
摘要
该文针对非任务导向型对话的回复质量构建了一个大规模的人工标注中文数据集,该数据集包含了从社交媒体收集到的超过27 000个对话问题以及超过82 000个对话问题的回复①。为了产生高质量的标注数据,邀请了专业人员根据对话回复的相关性、连贯性、信息性、趣味性,以及是否潜在地具有让对话继续延续的特性进行标注,在标注中定义了一个五级评分方法,分别是:极差的、较差的、一般的、较好的、极好的。为了测试标注产生的数据集是否具有有效性和实用性,以对话回复选择为任务,在标注数据集上测试了多种无监督和有监督模型。实验结果表明,该数据集对于提升对话回复选择的质量有显著效果。
This paper presents a large-scale corpus for non-task-oriented dialogue systems,which contains over 27 K distinct prompts with more than 82 Kresponses collected from social media.To annotate this corpus,we define a 5-grade rating scheme(bad,mediocre,acceptable,good,and excellent)with respect to the relevance,coherence,informativeness,interestingness,and the potential to move a conversation forward.To test the validity and usefulness of the produced corpus,we compare various unsupervised and supervised models for response selection.Experimental results confirm that the proposed corpus is helpful in training response selection models.
This paper presents a large-scale corpus for non-task-oriented dialogue systems,which contains over 27 K distinct prompts with more than 82 Kresponses collected from social media.To annotate this corpus,we define a 5-grade rating scheme(bad,mediocre,acceptable,good,and excellent)with respect to the relevance,coherence,informativeness,interestingness,and the potential to move a conversation forward.To test the validity and usefulness of the produced corpus,we compare various unsupervised and supervised models for response selection.Experimental results confirm that the proposed corpus is helpful in training response selection models.
引文
[1]Alan M Turing.Computing machinery and intelligence[J].Mind,1950,59(236):433-460.
[2]https://www.apple.com/ios/siri/[DB/OL].
[3]Anjuli Kannan,et al.Smart Reply:Automated response suggestion for email[C]//Proceedings of the22nd ACM SIGKDD Infernational Conference,2016:955-964.
[4]John Markoff,Paul Mozur.For sympathetic ear,more Chinese turn to smartphone program[C]//Proceedings of the NY Times,2015.
[5]Steve J Young,et al.POMDP-based statistical spoken dialog systems:A review[C]//Proceedings of the IEEE,2013,101(5):1160-1179.
[6]Diana Perez Marin.Conversational agents and natural language interaction:Techniques and Effective practices[C]//Proceedings of the IGI Global,2011.
[7]David S Pallett,et al.DARPA ATIS Test Results June 1990[C]//Proceedings of the HLT,1990.
[8]李萍,郑建明.智慧图书馆中智能交互系统的研究和应用[J].图书馆学研究,2016(11):34-38.
[9]Weizenbaum J.ELIZA:A computer program for the study of natural laguage communication between man and machine[J].Communications of the ACM,1966,9(1):36-45.
[10]Colby K M,Weber S,Hilf F D.Artificial paranoia[J].Artificial Intelligence,1971,2(1):1-25.
[11]Wallace R S.The anatomy of A.L.I.C.E.[EB/OL],A.L.I.C.E.Artificial Intelligence Foundation Inc,2004.
[12]I Serban,et al.A survey of available corpora for building data-driven dialogue systems[J].arXiv preprint arXiv:1512.05742,2015.
[13]Hao Wang,et al.A dataset for research on shorttext conversations[C]//Proceedings of EMNLP,2013:935-945.
[14]Jiwei Li,et al.A diversity-promoting objective function for neural conversation models[C]//Proceedings of HLT-NAACL 2016:110-119.
[15]Yuanlong Shao,et al.Generating high-quality and informative conversation Responses with sequence-tosequence models[C]//Proceedings of EMNLP 2017:2210-2219.
[16]Ryan Lowe,et al.Towards an automatic turing Test:Learning to evaluate dialogue responses[C]//Proceedings of ACL(1)2017:1116-1126.
[17]张伟男,张杨子,刘挺.对话系统评价方法综述[J].中国科学:信息科学,2017(47):953-966.
[18]Tanveer J Siddiqui,Uma Shanker Tiwary.Integrating relation and keyword matching in information retrieval[J].KES(4),2005:64-73.
[19]Glen Pink.Slot filling[D].University of Sydney,Australia,2017.
[20]Suket Arora,Kamaljeet Batra,Sarabjit Singh.Dialogue system:A brief review[DB/OL].CoRR abs/1306.4134,2013.
[21]代六玲,等.中文文本分类中特征抽取方法的比较研究[J].中文信息学报,2003,18(1):26-32.
[22]Adam Coates,Honglak Lee,Andrew Y Ng.Analysis of single-layer networks in unsupervised feature learning[C]//Proceedings of the 14th International Conference on Artificial Intelligence and Statistics(AISTATS),2011:215-223.
[23]Lifeng Shang,Zhengdong Lu,Hang Li.Neural Responding machine for short-text conversation[C]//Proceedings of ACL,2015:1577-1586.
[24]Ryan Lowe,et al.Towards an automatic turing test:Learning to evaluate dialogue responses[C]//Proceedings of ACL,2017:1116-1126.
[25]Ryan Lowe,et al.The Ubuntu dialogue corpus:Alarge dataset for research in unstructured multi-turn dialogue systems[C]//Proceedings of SIGDIAL Conference,2015:285-294.
[26]Kilem L Gwet.Handbook of inter-rater reliability:The definitive guide to measuring the extent of agreement among raters[C]//Proceedings of the Advanced Analytics,LLC,2014.
[27]IvanKopecek.Modeling of the information retrieval dialogue systems[C]//Proceedings of the TSD,1999:302-307.
[28]Christopher D Manning,Prabhakar Raghavan,Hinrich Schütze.Introduction to information retrieval[M].Cambridge University Press,2008.
[29]Thorsten Joachims.Optimizing search engines using clickthrough data[C]//Proceedings of KDD,2002:133-142.
[30]Jerome H Friedman.Greedy function approximation:A gradient boosting machine[J].annals of statistics,2001:1189-1232.
[31]Ming Tan,Bing Xiang,Bowen Zhou.LSTM-based deep learning models for non-factoid answer selection[J].arXiv preprint arXiv:abs/1511.04108,2015.
[32]Aliaksei Severyn,Alessandro Moschitti.Learning to rank short text pairs with convolutional deep neural networks[C]//Proceedings of SIGIR,2015:373-382.
[33]Mean Squared Error.Encyclopedia of machine learning and data mining,2017:808.
[34]Rich Caruana,Steve Lawrence,C Lee Giles.Overfitting in neural nets:Backpropagation,conjugate gradient,and early stopping[C]//Proceedings of NIPS,2000:402-408.
(1)本文提出的数据集在此处公布:http://ai.tencent.com/ailab/upload/PapersUploads/A_Manually_Annotated_Chinese_Corpus_for_Non-task-oriented_Dialogue_System
(1)在社交媒体等场景下,用户对其他公开用户发表的某些状态或者评论等进行相应的回复,以此产生的文本我们称之为交互文本。更一般地,任何用户相互之间进行交流产生的文本都可以被认为是交互文本。
(2)https://weibo.com
(3)由于社交媒体上通用回复的普遍性,以往通过社交媒体语料训练的聊天机器人,往往倾向于生成类似的“万能回复”,妨碍聊天的正常进行。因此,通用回复与更高质量的回复需要被有效地区分。
(1)https://tieba.baidu.com
(2)https://zhidao.baidu.com
(3)https://www.douban.com
(4)我们从这些不同网站上抽取的主题列表具有比较高的相似性。
(5)https://jsoup.org/
(1)https://github.com/fxsjy/jieba
(2)https://en.wikipedia.org/wiki/Cosine_similarity
[2]https://www.apple.com/ios/siri/[DB/OL].
[3]Anjuli Kannan,et al.Smart Reply:Automated response suggestion for email[C]//Proceedings of the22nd ACM SIGKDD Infernational Conference,2016:955-964.
[4]John Markoff,Paul Mozur.For sympathetic ear,more Chinese turn to smartphone program[C]//Proceedings of the NY Times,2015.
[5]Steve J Young,et al.POMDP-based statistical spoken dialog systems:A review[C]//Proceedings of the IEEE,2013,101(5):1160-1179.
[6]Diana Perez Marin.Conversational agents and natural language interaction:Techniques and Effective practices[C]//Proceedings of the IGI Global,2011.
[7]David S Pallett,et al.DARPA ATIS Test Results June 1990[C]//Proceedings of the HLT,1990.
[8]李萍,郑建明.智慧图书馆中智能交互系统的研究和应用[J].图书馆学研究,2016(11):34-38.
[9]Weizenbaum J.ELIZA:A computer program for the study of natural laguage communication between man and machine[J].Communications of the ACM,1966,9(1):36-45.
[10]Colby K M,Weber S,Hilf F D.Artificial paranoia[J].Artificial Intelligence,1971,2(1):1-25.
[11]Wallace R S.The anatomy of A.L.I.C.E.[EB/OL],A.L.I.C.E.Artificial Intelligence Foundation Inc,2004.
[12]I Serban,et al.A survey of available corpora for building data-driven dialogue systems[J].arXiv preprint arXiv:1512.05742,2015.
[13]Hao Wang,et al.A dataset for research on shorttext conversations[C]//Proceedings of EMNLP,2013:935-945.
[14]Jiwei Li,et al.A diversity-promoting objective function for neural conversation models[C]//Proceedings of HLT-NAACL 2016:110-119.
[15]Yuanlong Shao,et al.Generating high-quality and informative conversation Responses with sequence-tosequence models[C]//Proceedings of EMNLP 2017:2210-2219.
[16]Ryan Lowe,et al.Towards an automatic turing Test:Learning to evaluate dialogue responses[C]//Proceedings of ACL(1)2017:1116-1126.
[17]张伟男,张杨子,刘挺.对话系统评价方法综述[J].中国科学:信息科学,2017(47):953-966.
[18]Tanveer J Siddiqui,Uma Shanker Tiwary.Integrating relation and keyword matching in information retrieval[J].KES(4),2005:64-73.
[19]Glen Pink.Slot filling[D].University of Sydney,Australia,2017.
[20]Suket Arora,Kamaljeet Batra,Sarabjit Singh.Dialogue system:A brief review[DB/OL].CoRR abs/1306.4134,2013.
[21]代六玲,等.中文文本分类中特征抽取方法的比较研究[J].中文信息学报,2003,18(1):26-32.
[22]Adam Coates,Honglak Lee,Andrew Y Ng.Analysis of single-layer networks in unsupervised feature learning[C]//Proceedings of the 14th International Conference on Artificial Intelligence and Statistics(AISTATS),2011:215-223.
[23]Lifeng Shang,Zhengdong Lu,Hang Li.Neural Responding machine for short-text conversation[C]//Proceedings of ACL,2015:1577-1586.
[24]Ryan Lowe,et al.Towards an automatic turing test:Learning to evaluate dialogue responses[C]//Proceedings of ACL,2017:1116-1126.
[25]Ryan Lowe,et al.The Ubuntu dialogue corpus:Alarge dataset for research in unstructured multi-turn dialogue systems[C]//Proceedings of SIGDIAL Conference,2015:285-294.
[26]Kilem L Gwet.Handbook of inter-rater reliability:The definitive guide to measuring the extent of agreement among raters[C]//Proceedings of the Advanced Analytics,LLC,2014.
[27]IvanKopecek.Modeling of the information retrieval dialogue systems[C]//Proceedings of the TSD,1999:302-307.
[28]Christopher D Manning,Prabhakar Raghavan,Hinrich Schütze.Introduction to information retrieval[M].Cambridge University Press,2008.
[29]Thorsten Joachims.Optimizing search engines using clickthrough data[C]//Proceedings of KDD,2002:133-142.
[30]Jerome H Friedman.Greedy function approximation:A gradient boosting machine[J].annals of statistics,2001:1189-1232.
[31]Ming Tan,Bing Xiang,Bowen Zhou.LSTM-based deep learning models for non-factoid answer selection[J].arXiv preprint arXiv:abs/1511.04108,2015.
[32]Aliaksei Severyn,Alessandro Moschitti.Learning to rank short text pairs with convolutional deep neural networks[C]//Proceedings of SIGIR,2015:373-382.
[33]Mean Squared Error.Encyclopedia of machine learning and data mining,2017:808.
[34]Rich Caruana,Steve Lawrence,C Lee Giles.Overfitting in neural nets:Backpropagation,conjugate gradient,and early stopping[C]//Proceedings of NIPS,2000:402-408.
(1)本文提出的数据集在此处公布:http://ai.tencent.com/ailab/upload/PapersUploads/A_Manually_Annotated_Chinese_Corpus_for_Non-task-oriented_Dialogue_System
(1)在社交媒体等场景下,用户对其他公开用户发表的某些状态或者评论等进行相应的回复,以此产生的文本我们称之为交互文本。更一般地,任何用户相互之间进行交流产生的文本都可以被认为是交互文本。
(2)https://weibo.com
(3)由于社交媒体上通用回复的普遍性,以往通过社交媒体语料训练的聊天机器人,往往倾向于生成类似的“万能回复”,妨碍聊天的正常进行。因此,通用回复与更高质量的回复需要被有效地区分。
(1)https://tieba.baidu.com
(2)https://zhidao.baidu.com
(3)https://www.douban.com
(4)我们从这些不同网站上抽取的主题列表具有比较高的相似性。
(5)https://jsoup.org/
(1)https://github.com/fxsjy/jieba
(2)https://en.wikipedia.org/wiki/Cosine_similarity
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |