基于标签分解的口语理解模型
|
摘要
在双向长短时记忆网络的基础上,提出一种用于口语理解的标签拆分策略,并构建一个联合模型。通过将1次127种标签分类转换成3次独立的分类,平衡ATIS数据集的标签。针对ATIS数据集资源较少的问题,引入外部词向量以提升模型的分类性能。实验结果表明,与循环神经网络及其变体结构相比,该模型的F1值有显著提升,最高可达95.63%。
Based on the Bi-Long Short Term Memory(BiLSTM),this paper proposes a label splitting strategy for Spoken Language Understanding(SLU)and constructs a joint model.The model convert a classification of 127 labels into 3 independent classifications to balance the labels in the ATIS database.Due to the scarcity of ATIS data,this paper introduces external word embedding to improve the classification performance of the model.Experimental results show that compared with the traditional recurrent neural network and its variants,the proposed joint model obtains significantly improvement in F1 value,which can be up to 95.63%.
Based on the Bi-Long Short Term Memory(BiLSTM),this paper proposes a label splitting strategy for Spoken Language Understanding(SLU)and constructs a joint model.The model convert a classification of 127 labels into 3 independent classifications to balance the labels in the ATIS database.Due to the scarcity of ATIS data,this paper introduces external word embedding to improve the classification performance of the model.Experimental results show that compared with the traditional recurrent neural network and its variants,the proposed joint model obtains significantly improvement in F1 value,which can be up to 95.63%.
引文
[1] MORI R D,FREDERIC B,HAKKANI-TUR D,et al.Spoken language understanding[J].IEEE Signal Processing Magazine,2008,25(3):50-58.
[2] HAFFNER P,TUR G,WRIGHT J H.Optimizing SVMs for complex call classification[C]//Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing.Washington D.C.,USA:IEEE Press,2003:632-635.
[3] SARIKAYA R,HINTON G E,RAMABHADRAN B.Deep belief nets for natural language call-routing[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2011:5680-5683.
[4] MCCALLUM A,FREITAG D,PEREIRA F.Maximum entropy Markov models for information extraction and segmentation[C]//Proceedings of the 17th International Conference on Machine Learning.San Francisco,USA:Morgan Kaufmann Publishers Inc.,2000:591-598.
[5] RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of International Speech Communication Association.Antwerp,Belgium:[s.n.],2007:1605-1608.
[6] YAO Kaisheng,PENG Baolin,ZHANG Yu,et al.Spoken language understanding using long short-term memory neural networks[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2015:189-194.
[7] MESNIL G,DAUPHIN Y,YAO Kaisheng,et al.Using recurrent neural networks for slot filling in spoken language understanding[J].IEEE/ACM Transactions on Audio Speech and Language Processing,2015,23(3):530-539.
[8] LIU Bing,LANE I.Recurrent neural network structured output prediction for spoken language understanding[C]//Proceedings of NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions.Montreal,Canada:[s.n.],2015:1-9.
[9] GUO D,TUR G,YIH W T,et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2015:554-559.
[10] XU Puyang,SARIKAYA R.Convolutional neural network based triangular CRF for joint intent detection and slot filling[C]//Proceedings of Automatic Speech Recognition and Understanding.Washington D.C.,USA:IEEE Press,2014:78-83.
[11] MINKER W,BENNACEF S,GAUVAIN J.A stochastic case frame approach for natural language understanding[C]//Proceedings of International Conference on Spoken Language.Washington D.C.,USA:IEEE Press,1996:1013-1016.
[12] RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of International Speech Communication Association.Antwerp,Belgium:[s.n.],2007:1605-1608.
[13] LAFFERTY J D,MCCALLUM A,PEREIRA F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning.San Francisco,USA:Morgan Kaufmann Publishers Inc.,2001:282-289.
[14] MESNIL G,HE Xiaodong,DENG Li,et al.Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding[C]//Proceedings of International Speech Communication Association.Lyon,France:[s.n.],2013:3771-3775.
[15] KOLBOEK M,TAN Zhenghua,JENSEN J.Speech enhancement using long short-term memory based recurrent neural networks for noise robust speaker verification[C]//Proceedings of IEEE Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2016:305-311.
[16] BENGIO Y,DUCHARME R,VINCENT P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2006,3(6):1137-1155.
[17] 吴旭康,杨旭光,陈园园,等.主题联合词向量模型[J].计算机工程,2018,44(2):233-237.
[18] 余冲,李晶,孙旭东,等.基于词嵌入与概率主题模型的社会媒体话题识别[J].计算机工程,2017,43(12):184-191.
[19] HEMPHILL C T,GODFREY J J,DODDINGTON G R.The ATIS spoken language systems pilot corpus[C]//Proceedings of the Darpa Speech and Natural Language Workshop.Hidden Valley,USA:[s.n.],1990:96-101.
[20] TUR G,HAKKANI-TUR D,HECK L.What is left to be understood in ATIS?[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2011:19-24.
[21] RAMSHAW L A,MARCUS M P.Text chunking using transformation-based learning[J].Text Speech and Language Technology,1995,11:82-94.
[22] ELMAN J.Finding structure in time[J].Cognitive Science,1990,14 (2):179-211.
[23] PENG Baolin,YAO Kaisheng,JING Li,et al.Recurrent neural networks with external memory for spoken language understanding[C]//Proceeding of Natural Language Processing and Chinese Computing.Berlin,Germany:Springer,2015:25-35.
[24] JORDAN M I.Serial order:A parallel distributed processing approach[J].Advances in Psychology,1997,121:471-495.
[2] HAFFNER P,TUR G,WRIGHT J H.Optimizing SVMs for complex call classification[C]//Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing.Washington D.C.,USA:IEEE Press,2003:632-635.
[3] SARIKAYA R,HINTON G E,RAMABHADRAN B.Deep belief nets for natural language call-routing[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2011:5680-5683.
[4] MCCALLUM A,FREITAG D,PEREIRA F.Maximum entropy Markov models for information extraction and segmentation[C]//Proceedings of the 17th International Conference on Machine Learning.San Francisco,USA:Morgan Kaufmann Publishers Inc.,2000:591-598.
[5] RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of International Speech Communication Association.Antwerp,Belgium:[s.n.],2007:1605-1608.
[6] YAO Kaisheng,PENG Baolin,ZHANG Yu,et al.Spoken language understanding using long short-term memory neural networks[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2015:189-194.
[7] MESNIL G,DAUPHIN Y,YAO Kaisheng,et al.Using recurrent neural networks for slot filling in spoken language understanding[J].IEEE/ACM Transactions on Audio Speech and Language Processing,2015,23(3):530-539.
[8] LIU Bing,LANE I.Recurrent neural network structured output prediction for spoken language understanding[C]//Proceedings of NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions.Montreal,Canada:[s.n.],2015:1-9.
[9] GUO D,TUR G,YIH W T,et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2015:554-559.
[10] XU Puyang,SARIKAYA R.Convolutional neural network based triangular CRF for joint intent detection and slot filling[C]//Proceedings of Automatic Speech Recognition and Understanding.Washington D.C.,USA:IEEE Press,2014:78-83.
[11] MINKER W,BENNACEF S,GAUVAIN J.A stochastic case frame approach for natural language understanding[C]//Proceedings of International Conference on Spoken Language.Washington D.C.,USA:IEEE Press,1996:1013-1016.
[12] RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of International Speech Communication Association.Antwerp,Belgium:[s.n.],2007:1605-1608.
[13] LAFFERTY J D,MCCALLUM A,PEREIRA F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning.San Francisco,USA:Morgan Kaufmann Publishers Inc.,2001:282-289.
[14] MESNIL G,HE Xiaodong,DENG Li,et al.Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding[C]//Proceedings of International Speech Communication Association.Lyon,France:[s.n.],2013:3771-3775.
[15] KOLBOEK M,TAN Zhenghua,JENSEN J.Speech enhancement using long short-term memory based recurrent neural networks for noise robust speaker verification[C]//Proceedings of IEEE Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2016:305-311.
[16] BENGIO Y,DUCHARME R,VINCENT P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2006,3(6):1137-1155.
[17] 吴旭康,杨旭光,陈园园,等.主题联合词向量模型[J].计算机工程,2018,44(2):233-237.
[18] 余冲,李晶,孙旭东,等.基于词嵌入与概率主题模型的社会媒体话题识别[J].计算机工程,2017,43(12):184-191.
[19] HEMPHILL C T,GODFREY J J,DODDINGTON G R.The ATIS spoken language systems pilot corpus[C]//Proceedings of the Darpa Speech and Natural Language Workshop.Hidden Valley,USA:[s.n.],1990:96-101.
[20] TUR G,HAKKANI-TUR D,HECK L.What is left to be understood in ATIS?[C]//Proceedings of Spoken Language Technology Workshop.Washington D.C.,USA:IEEE Press,2011:19-24.
[21] RAMSHAW L A,MARCUS M P.Text chunking using transformation-based learning[J].Text Speech and Language Technology,1995,11:82-94.
[22] ELMAN J.Finding structure in time[J].Cognitive Science,1990,14 (2):179-211.
[23] PENG Baolin,YAO Kaisheng,JING Li,et al.Recurrent neural networks with external memory for spoken language understanding[C]//Proceeding of Natural Language Processing and Chinese Computing.Berlin,Germany:Springer,2015:25-35.
[24] JORDAN M I.Serial order:A parallel distributed processing approach[J].Advances in Psychology,1997,121:471-495.