On the Gradient-Based Sequential Tuning of the Echo State Network Reservoir Parameters

详细信息    查看全文

• 刊名：Lecture Notes in Computer Science
• 出版年：2016
• 出版时间：2016
• 年：2016
• 卷：9810
• 期：1
• 页码：651-660
• 全文大小：222 KB
• 参考文献：1.Haykin, S.S.: Adaptive Filter Theory, 4th edn. Pearson Education India, New Delhi (2005)MATH
  2.Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. Technical report GMD Report 148, German National Research Center for Information Technology (2001)
  3.Jaeger, H.: Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach. GMD-Forschungszentrum Informationstechnik (2002)
  4.Jaeger, H.: Adaptive nonlinear system identification with echo state networks. Networks 8, 9 (2003)
  5.Jaeger, H.: Reservoir riddle: suggestions for echo state network research. In: Proceedings of International Joint Conference on Neural Networks, pp. 1460–1462 (2005)
  6.Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)CrossRef
  7.Jiang, F., Berry, H., Schoenauer, M.: Supervised and evolutionary learning of echo state networks. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 215–224. Springer, Heidelberg (2008)CrossRef
  8.Küçükemre, A.U.: Echo state networks for adaptive filtering. Ph.D. thesis, University of Applied Sciences (2006)
  9.Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012)
  10.Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)CrossRef MATH
  11.Petersen, K.B., Pedersen, M.S., et al.: The matrix cookbook. Technical University of Denmark, vol. 7, p. 15 (2008)
  12.Schrauwen, B., Verstraeten, D., Van Campenhout, J.: An overview of reservoir computing: theory, applications and implementations. In: Proceedings of the 15th European Symposium on Artificial Neural Networks, pp. 471–482 (2007)
  13.Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Comput. 1(2), 270–280 (1989)CrossRef
  14.Xia, Y., Jelfs, B., Van Hulle, M.M., Príncipe, J.C., Mandic, D.P.: An augmented echo state network for nonlinear adaptive filtering of complex noncircular signals. IEEE Trans. Neural Netw. 22(1), 74–83 (2011)CrossRef
  15.Yuenyong, S.: Fast and effective tuning of echo state network reservoir parameters using evolutionary algorithms and template matrices. In: 19th International Computer Science and Engineering Conference (ICSEC), November 2015
  16.Yuenyong, S., Nishihara, A.: Evolutionary pre-training for CRJ-type reservoir of echo state networks. Neurocomputing 149, 1324–1329 (2015)CrossRef
  
• 作者单位：Sumeth Yuenyong (15)
  
  15. School of Information Technology, Shinawatra University, 99 Moo 10 Bang Toey, Sam Khok, 12160, Pathum Thani, Thailand
  
• 丛书名：PRICAI 2016: Trends in Artificial Intelligence
• ISBN：978-3-319-42911-3
• 刊物类别：Computer Science
• 刊物主题：Artificial Intelligence and Robotics
  Computer Communication Networks
  Software Engineering
  Data Encryption
  Database Management
  Computation by Abstract Devices
  Algorithm Analysis and Problem Complexity
  
• 出版者：Springer Berlin / Heidelberg
• ISSN：1611-3349
• 卷排序：9810

文摘

In this paper, the derivative of the input scaling and spectral radius parameters of Echo State Network reservoir are derived. This was achieved by re-writing the reservoir state update equation in terms of template matrices whose eigenvalues can be pre-calculated, so the two parameters appear in the state update equation in the form of simple multiplication which is differentiable. After that the paper derives the derivatives and then discusses why direct application of these two derivatives in gradient descent to optimize reservoirs in a sequential manner would be ineffective due to the nature of the error surface and the problem of large eigenvalue spread on the reservoir state matrix. Finally it is suggested how to apply the derivatives obtained here for joint-optimizing the reservoir and readout at the same time.