引入外部记忆的循环神经网络的口语理解
|
摘要
循环神经网络(RNN)越来越在口语理解(Spoken Language Understanding,SLU)任务中显示出优势。然而,由于梯度消失和梯度爆炸问题,简单循环神经网络的存储容量受到限制。提出一种使用外部存储器来提高记忆能力的循环神经网络。并在ATIS数据集上进行了实验,并与其他公开报道的模型进行比较。结果说明,在口语理解任务上,提出的引入外部记忆的循环神经网络在准确性、召回率和F1值都有较明显提高,优于传统循环神经网络及其变体结构。
Recurrent Neural Network(RNN)has increasingly shown its advantages in the Spoken Language Understanding(SLU)task. However, because of the problem of gradient disappearance and gradient explosion, the storage capacity of simple recurrent neural network is limited. A RNN that uses external memory is proposed to improve memory. Experiments are carried out on the ATIS data set and compared with other publicly reported models. The results show that, in oral comprehension tasks, the RNN introduced external memory has significantly improved accuracy, recall rate and F1-score, which is superior to traditional recurrent neural network and its variant structure.
Recurrent Neural Network(RNN)has increasingly shown its advantages in the Spoken Language Understanding(SLU)task. However, because of the problem of gradient disappearance and gradient explosion, the storage capacity of simple recurrent neural network is limited. A RNN that uses external memory is proposed to improve memory. Experiments are carried out on the ATIS data set and compared with other publicly reported models. The results show that, in oral comprehension tasks, the RNN introduced external memory has significantly improved accuracy, recall rate and F1-score, which is superior to traditional recurrent neural network and its variant structure.
引文
[1]Mctear M,Callejas Z,Griol D.Spoken language understanding[M].[S.l.]:Springer International Publishing,2016.
[2]Wang Y Y,Yu D,Ju Y C,et al.An introduction to voice search[J].IEEE Signal Processing Magazine,2008,25(3):28-38.
[3]Minker W,Bennacef S,Gauvain J.A stochastic case frame approach for natural language understanding[C]//Proceedings of the International Conference on Spoken Language,1996.
[4]Raymond C,Riccardi G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of the INTERSPEECH,2007.
[5]Lafferty J D,Mccallum A,Pereira F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the Eighteenth International Conference on Machine Learning,2001.
[6]Bengio Y,Ducharme R,Vincent P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2003,3(6):1137-1155.
[7]Mikolov T,Karafiát M,Burget L,et al.Recurrent neural network based language model[C]//Proceedings of the INTERSPEECH 2010,Conference of the International Speech Communication Association,Makuhari,Chiba,Japan,September 2010.
[8]Mesnil G,He X,Deng L,et al.Investigation of recurrentneural-network architectures and learning methods for spoken language understanding[C]//Proceedings of the INTERSPEECH,2013.
[9]Yao K,Zweig G,Hwang M Y,et al.Recurrent neural networks for language understanding[C]//Proceedings of the INTERSPEECH,2013.
[10]Mesnil G,Dauphin Y,Yao K,et al.Using recurrent neural networks for slot filling in spoken language understanding[J].IEEE/ACM Transactions on Audio Speech&Language Processing,2015,23(3):530-539.
[11]Jaech A,Heck L P,Ostendorf M.Domain adaptation of recurrent neural networks for natural language understanding[J].arXiv:1604.00117,2016.
[12]Jozefowicz R,Zaremba W,Sutskever I.An empirical exploration of recurrent network architectures[C]//Proceedings of the International Conference on International Conference on Machine Learning,2015.
[13]Hochreiter S,Schmidhuber J.Long short-term memory[J].Neural Computation,2013,9(8):1735-1780.
[14]Cho K,Merrienboer B V,Gulcehre C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].Computer Science,2014.
[15]Graves A,Wayne G,Danihelka I.Neural turing machines[J].Computer Science,2014.
[16]Weston J,Chopra S,Bordes A.Memory networks[J].Eprint Arxiv,2014.
[17]Zhao K,Li Y,Zhang C,et al.PRNN:recurrent neural network with persistent memory[J].ar Xiv:1801.08094v2,2018.
[18]Dahl D A,Bates M,Brown M,et al.Expanding the scope of the ATIS task:the ATIS-3 corpus[C]//Proceedings of the Workshop on Human Language Technology,1994.
[19]Tur G,Hakkanitur D,Heck L.What is left to be understood in ATIS?[C]//2010 IEEE Spoken Langnage Technology Workshop,2010.
[20]Ramshaw L A,Marcus M P.Text chunking using transformation-based learning[J].Text Speech&Language Technology,1995,11:82-94.
[21]Pascanu R,Mikolov T,Bengio Y.On the difficulty of training Recurrent Neural Networks[J].ar Xiv:1211.5063v2,2012.
[22]Bergstra J,Breuleux O,Bastien F,et al.Theano:a CPUand GPU math compiler in Python[C]//Proceedings of the 9th Python in Science Conference,2010:3-10.
[2]Wang Y Y,Yu D,Ju Y C,et al.An introduction to voice search[J].IEEE Signal Processing Magazine,2008,25(3):28-38.
[3]Minker W,Bennacef S,Gauvain J.A stochastic case frame approach for natural language understanding[C]//Proceedings of the International Conference on Spoken Language,1996.
[4]Raymond C,Riccardi G.Generative and discriminative algorithms for spoken language understanding[C]//Proceedings of the INTERSPEECH,2007.
[5]Lafferty J D,Mccallum A,Pereira F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the Eighteenth International Conference on Machine Learning,2001.
[6]Bengio Y,Ducharme R,Vincent P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2003,3(6):1137-1155.
[7]Mikolov T,Karafiát M,Burget L,et al.Recurrent neural network based language model[C]//Proceedings of the INTERSPEECH 2010,Conference of the International Speech Communication Association,Makuhari,Chiba,Japan,September 2010.
[8]Mesnil G,He X,Deng L,et al.Investigation of recurrentneural-network architectures and learning methods for spoken language understanding[C]//Proceedings of the INTERSPEECH,2013.
[9]Yao K,Zweig G,Hwang M Y,et al.Recurrent neural networks for language understanding[C]//Proceedings of the INTERSPEECH,2013.
[10]Mesnil G,Dauphin Y,Yao K,et al.Using recurrent neural networks for slot filling in spoken language understanding[J].IEEE/ACM Transactions on Audio Speech&Language Processing,2015,23(3):530-539.
[11]Jaech A,Heck L P,Ostendorf M.Domain adaptation of recurrent neural networks for natural language understanding[J].arXiv:1604.00117,2016.
[12]Jozefowicz R,Zaremba W,Sutskever I.An empirical exploration of recurrent network architectures[C]//Proceedings of the International Conference on International Conference on Machine Learning,2015.
[13]Hochreiter S,Schmidhuber J.Long short-term memory[J].Neural Computation,2013,9(8):1735-1780.
[14]Cho K,Merrienboer B V,Gulcehre C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].Computer Science,2014.
[15]Graves A,Wayne G,Danihelka I.Neural turing machines[J].Computer Science,2014.
[16]Weston J,Chopra S,Bordes A.Memory networks[J].Eprint Arxiv,2014.
[17]Zhao K,Li Y,Zhang C,et al.PRNN:recurrent neural network with persistent memory[J].ar Xiv:1801.08094v2,2018.
[18]Dahl D A,Bates M,Brown M,et al.Expanding the scope of the ATIS task:the ATIS-3 corpus[C]//Proceedings of the Workshop on Human Language Technology,1994.
[19]Tur G,Hakkanitur D,Heck L.What is left to be understood in ATIS?[C]//2010 IEEE Spoken Langnage Technology Workshop,2010.
[20]Ramshaw L A,Marcus M P.Text chunking using transformation-based learning[J].Text Speech&Language Technology,1995,11:82-94.
[21]Pascanu R,Mikolov T,Bengio Y.On the difficulty of training Recurrent Neural Networks[J].ar Xiv:1211.5063v2,2012.
[22]Bergstra J,Breuleux O,Bastien F,et al.Theano:a CPUand GPU math compiler in Python[C]//Proceedings of the 9th Python in Science Conference,2010:3-10.