基于卡尔曼滤波器算法的径向基神经网络训练算法研究
|
摘要
卡尔曼滤波算法是工业中常用的优化算法之一，广泛应用于去噪、滤波、优化等。由于其优越的数学特性，所以很多的文献中已经将它用于例如前向神经网络以及递归神经网络等一些神经网络的训练。本文首先对各种滤波算法应用于(RBFN)的训练进行仿真研究，找出其优缺点，在此基础上提出了采用无先导卡尔曼滤波算法(Unscented Kalman Filter-UKF)来训练径向基神经网络(RBFN)的新方法。
扩展卡尔曼滤波器(EKF)已经被广泛的应用于神经网络的训练。但是本文通过仿真、研究，发现EKF的缺点是当训练集很大的时候，这种算法的计算量将会非常的大而复杂以至于不能完成训练任务，尤其对于RBFN。原因是因为EKF的状态向量包含了所有神经网络参数，这其中包括网络中心点、权值等等内容，运算量非常大。针对这些问题，之后本文尝试运用双重卡尔曼滤波器算法(DEKF)，目的是将作为卡尔曼滤波器状态变量的RBFN参数进行降维，改为由两个并行处理的滤波器进行优化计算，最终结果虽然有一定的改善，但是并没有从根本上解决上述问题。
在大量的理论分析以及实际仿真的基础上，我们提出了一种新的用于RBFN训练的算法一无先导卡尔曼滤波器(UKF)算法。针对EKF和DEKF的对函数的一阶近似，该算法中对非线性函数采用二阶近似展开。最重要的一点是UKF不用求取系统的雅克比矩阵，所以大大减小的计算量。仿真结果证明了该方法在时间序列预测、函数逼近以及分类问题上的有效性和运算速度。
Kalman Filter has been widely used in modern industry such as noise-reducing, filtering, optimizing, and so on. It has been involved in training feedforward neural networks and recurrent neural networks because of its excellent mathematic characteristic in many researches. In this thesis, RBFN was trained using several kinds of Kalman filter, their disadvantages and merits were studied, and eventually, a method of applying unscented Kalman Filter(UKF) for training of RBF neural network was proposed .
Extended Kalman Filter has been successfully used for training neural networks. In the study, simulation results show that EKF can't complete the training mission when the training set is too large, especially for RBFN. The reason is thatthe state vector of EKF for training RBF neural network including all the parameters of the network, such as kernel points, weights of thelayers and so on, so the calculational complexity is significantly large. Aim at the point, the dual Extended Kalman filter (DEKF) was tested for reducing the dimensions of the EKF's state vector. Though it improves calculational complexity at a certain extent, DEKF can't change essential disadvantage.
A new method for training RBFN named "Unscented Kalman Filter" (UKF) through a mass of academic analysis based on the optimization was proposed instead. Different from EKF and DEKF which execute first order approximation, UKF uses second order approximation to extend nonlinear function. And the most important is: UKF doesn't need to calculate system Jacobian matrix so the calculational complexity of training process can be reduced signaflcantly. Simulation results show its validity and speediness in function approximation, chaotic time series prediction and classification problems.
扩展卡尔曼滤波器(EKF)已经被广泛的应用于神经网络的训练。但是本文通过仿真、研究，发现EKF的缺点是当训练集很大的时候，这种算法的计算量将会非常的大而复杂以至于不能完成训练任务，尤其对于RBFN。原因是因为EKF的状态向量包含了所有神经网络参数，这其中包括网络中心点、权值等等内容，运算量非常大。针对这些问题，之后本文尝试运用双重卡尔曼滤波器算法(DEKF)，目的是将作为卡尔曼滤波器状态变量的RBFN参数进行降维，改为由两个并行处理的滤波器进行优化计算，最终结果虽然有一定的改善，但是并没有从根本上解决上述问题。
在大量的理论分析以及实际仿真的基础上，我们提出了一种新的用于RBFN训练的算法一无先导卡尔曼滤波器(UKF)算法。针对EKF和DEKF的对函数的一阶近似，该算法中对非线性函数采用二阶近似展开。最重要的一点是UKF不用求取系统的雅克比矩阵，所以大大减小的计算量。仿真结果证明了该方法在时间序列预测、函数逼近以及分类问题上的有效性和运算速度。
Kalman Filter has been widely used in modern industry such as noise-reducing, filtering, optimizing, and so on. It has been involved in training feedforward neural networks and recurrent neural networks because of its excellent mathematic characteristic in many researches. In this thesis, RBFN was trained using several kinds of Kalman filter, their disadvantages and merits were studied, and eventually, a method of applying unscented Kalman Filter(UKF) for training of RBF neural network was proposed .
Extended Kalman Filter has been successfully used for training neural networks. In the study, simulation results show that EKF can't complete the training mission when the training set is too large, especially for RBFN. The reason is thatthe state vector of EKF for training RBF neural network including all the parameters of the network, such as kernel points, weights of thelayers and so on, so the calculational complexity is significantly large. Aim at the point, the dual Extended Kalman filter (DEKF) was tested for reducing the dimensions of the EKF's state vector. Though it improves calculational complexity at a certain extent, DEKF can't change essential disadvantage.
A new method for training RBFN named "Unscented Kalman Filter" (UKF) through a mass of academic analysis based on the optimization was proposed instead. Different from EKF and DEKF which execute first order approximation, UKF uses second order approximation to extend nonlinear function. And the most important is: UKF doesn't need to calculate system Jacobian matrix so the calculational complexity of training process can be reduced signaflcantly. Simulation results show its validity and speediness in function approximation, chaotic time series prediction and classification problems.
引文
[1] 李慧，段培永．基于HCMAC神经网络的控制研究[J]．山东建筑工程学院学报，2003，45(2)：55-60
[2] 朱大奇．人工神经网络研究现状及其展望[J]．江南大学学报，2004，3(2)：103-110
[3] 胡守仁，余少波．神经网络导论[M]．长沙：国防科技大学出版社，1992．11
[4] 吴成茂 范九伦．确定RBFN隐含层节点数的最大矩阵元法[J]．计算机工程与应用，2004，20(2)：62-67
[5] 朱海军，高大启．一种自适应RBF分类神经网络模型的构造方法[J]．计算机工程与应用，2004，21(3)：31-35
[6] 高大启，杨根兴．改进的RBF神经网络模式分类方法理论研究．华东理工大学学报(自然科学版)，2001，27(6)：667—684
[7] 朱明星，张德龙．RBF网络基函数中心选取算法的研究．安徽大学学报(自然科学版)，2000，24(1)：72-79
[8] 李江红，胡照文，郑哲文．RBF神经网络的一种新的学习算．长沙电力学院学报(自然科学版)，2000，15(1)：39-42
[9] Schwenker F, Kestler C, Hans A.; Palm G. Three learning phases for radial-basis-function networks[J]. Neural Networks, 2001, 14:439-458
[10] Stankiewicz B. The effect of the RBF intralayer[J]. Thin Solid Films, 1996, 280:178-182
[11] Donald K, Krzysztof J. Time series forecasting by combining RBF networks, certainty factors, and the Box-Jenkins model[J]. Neurocomputing, 1996, 10:149-168
[12] Adya M. Corrections to rule-based forecasting: findings from a replication[J]. International Journal of Forecasting, 2000, 16:125-127
[13] Silipo R, Bortolan G., Marchesi C. Design of hybrid architectures based on neural classifier and RBF pre-processing for ECG analysis[J]. International Journal of Approximate Reasoning, 1999, 21:177-196
[14] Fasshauer G. Dual bases and discrete reproducing kernels: a unified framework for RBF and MLS approximation[J]. Engineering Analysis with Boundary Elements, 2005, 29:313-325
[15] Larsson, E, Fornberg B. A numerical study of some radial basis function based solution methods for elliptic PDEs[J]. Computers and Mathematics with Applications, 2003, 46: 891-902
[16] Bugmann G.Normalized Gaussian Radial Basis Function networks[J]. Neurocomputing, 1998, 20:97-110
[17] Acosta F, Miguel A. Radial basis function and related models: An overview[J]. Signal Processing, 1995, 45:37-58
[18] Yu W, Yu, DL. A comparison study on a chemical reactor modelling with a physical model and PLRBF networks[J]. Engineering Applications of Artificial Intelligence, 2003, 16, 629-645
[19] Arnott R. Diversity combining for digital mobile radio using radial basis function networks[J]. Signal Processing, 1997, 63: 1-16
[20] Xu L. RBF nets mixture experts and Bayesian Ying-Yang learning[J]. Neurocomputing, 1998, 19:223-257
[21] Fomberg B, Wright G.. Stable computation of multiquadric interpolants for all values of the shape parameter[J]. Computers and Mathematics with Applications, 2004, 48:853-867
[22] Kuncheva L Initializing of an RBF network by a genetic algorithm[J]. Neurocomputing, 1997, 14:273-288
[23] Sanchez A. Second derivative dependent placement of RBF centers[J]. Neurocomputing, 1995, 7:311-317
[24] Chen W, Tanaka M. A meshless integration-free and boundary-only RBF technique[J]. Computers and Mathematics with Applications, 2002, 43:379-391
[25] Tan K, Tang K. Taguchi-tuned radial basis function with application to high precision motion control[J]. Artificial Intelligence in Engineering, 2001, 15:25-36
[26] Xin L. On simultaneous approximations by radial basis function neural networks[J]. Applied Mathematics and Computation, 1998, 95:75-89
[27] Sheta F, Jong K. Time-series forecasting using GA-tuned radial basis functions[J]. Information Sciences, 2001, 133:221-228
[28] 岳彩青，常青美，庞学民．基于聚类分析的RBF网络建模方法及应用的研究[J]．计算机仿真，2003，23(1)：120-123
[29] Zhong Q, Huang C. A mended hybrid learning algorithm for radial basis function neural networks to improve generalization capability[J]. Applied Mathematical Modelling, 2007, 31:1271-1281
[30] 冀雅生．投影数据矩阵序列的伪逆递推公式[J]．成都大学学报(自然科学版)，2001，20(3)：22-24
[31] Delin Chu, Bart De Moor. On a variational formulation of the QSVD and the RSVD[J]. Linear Algebra and its Applications: 2000, 311 (15): 61-78
[32] 苏小红，侯秋香，马培，王亚东．RBF神经网络的混合学习算法[J]．哈尔滨工业大学学报(自然科学版)，2006，38(9)：1446-1449
[33] 叶世伟，史忠植．神经网络原理[M]．北京：机械工业出版社，2004．1
[34] Michael E. Tipping, David Lowe._Shadow targets: A novel algorithm for topographic projections by radial basis functions[J]. Neurocomputing, 1998, 19(3): 211-222
[35] Huijuan Wu, Yumei Wen, Ping Li. Dynamic discrimination of convergence of the LMS time delay estimation in complicated noisy environments[J]. Applied Acoustics: 2007, 68(6): 628-641
[36] Qin Yan, Saeed Vaseghi, Esfandiar Zavarehei. Formant tracking linear prediction model using HMMs and Kalman filters for noisy speech processing[J]. Computer Speech & Language: 2007, 21 (3): 543-561
[37] E.J.Godolphin, Kostas Triantafyllopoulos. Decomposition of time series models in state-space form[J]. Computational Statistics & Data Analysis: 2006, 50(9): 2232-2246
[38] S.A. Soliman, R. A. Alammari. Harmonic modeling of linear and nonlinear loads based on Kalman filtering algorithm[J]. Electric Power Systems Research: 2004, 72(2): 147-155
[39] I. Yaesh, U. Shaked. Min-max Kalman filtering[J]. Systems & Control Letters: 2004, 53(3): 217-228
[40] Alberto Corigliano, Stefano Mariani. Parameter identification in explicit structural dynamics: performance of the extended Kalman filter[J]. Computer Methods in Applied Mechanics and Engineering: 2004, 193(36-38): 3807-3835
[41] Sergiy A. Vorobyov, Andrzej Cichocki. Hyper Radial Basis Function Neural Networks for Interference Cancellation with Nonlinear Processing of Reference Signal [J]. Digital Signal Processing: 2001, 11(3): 204-221
[42] M. Marinaro, S. Scarpetta. On-line learning in RBF neural networks: a stochastic approach. Neural Networks: 2000,13(7): 719-729
[43] V. David Sanchez A. Robustization of a learning method for RBF networks[J]. Neurocomputing:1995, 9(1): 85-94
[44] J. L. Valdes, R. Biscay, J. C. Jimenez. Geometric selection of centers for radial basis function approximations involved in intensive computer simulations[J]. Mathematics and Computers in Simulation: 1999,48(3): 295-306
[45] Konrad Reif, Frank Sonnemann, Rolf Unbehauen. An EKF-Based Nonlinear Observer with a Prescribed Degree of Stability[J]. Automatica: 1998, 34(9): 1119-1123
[46] Jamshaid Ali, Jiancheng Fang. SINS/ANS integration for augmented performance navigation solution using unscented Kalman filtering[J]. Aerospace Science and Technology: 2006,10(3): 233-238
[47] Pramod Vachhani, Shankar Narasimhan, Raghunathan Rengaswamy. Robust and reliable estimation via Unscented Recursive Nonlinear Dynamic Data Reconciliation[J]. Journal of Process Control: 2006,16(10): 1075-1086
[48] Jose A. Romagnoli, Rafiqul Gani. Studies of distributed parameter systems: Decoupling the state-parameter estimation problem[J]. Chemical Engineering Science: 1983, 38(11): 1831-1843
[49] V. David Sanchez A. Searching for a solution to the automatic RBF network design problem[J]. Neurocomputing: 2002,42(1-4): 147-170
[50] John P. Owens, Douglas G Steigerwald. Noise reduced realized volatility: a kalman filter approach[J]. Advances in Econometrics: 2006, 20(1): 211-227
[51] G. Bolzon, R. Fedele, G Maier. Parameter identification of a cohesive crack model by Kalman filter[J]. Computer Methods in Applied Mechanics and Engineering: 2002, 191(25-26): 2847-2871
[2] 朱大奇．人工神经网络研究现状及其展望[J]．江南大学学报，2004，3(2)：103-110
[3] 胡守仁，余少波．神经网络导论[M]．长沙：国防科技大学出版社，1992．11
[4] 吴成茂 范九伦．确定RBFN隐含层节点数的最大矩阵元法[J]．计算机工程与应用，2004，20(2)：62-67
[5] 朱海军，高大启．一种自适应RBF分类神经网络模型的构造方法[J]．计算机工程与应用，2004，21(3)：31-35
[6] 高大启，杨根兴．改进的RBF神经网络模式分类方法理论研究．华东理工大学学报(自然科学版)，2001，27(6)：667—684
[7] 朱明星，张德龙．RBF网络基函数中心选取算法的研究．安徽大学学报(自然科学版)，2000，24(1)：72-79
[8] 李江红，胡照文，郑哲文．RBF神经网络的一种新的学习算．长沙电力学院学报(自然科学版)，2000，15(1)：39-42
[9] Schwenker F, Kestler C, Hans A.; Palm G. Three learning phases for radial-basis-function networks[J]. Neural Networks, 2001, 14:439-458
[10] Stankiewicz B. The effect of the RBF intralayer[J]. Thin Solid Films, 1996, 280:178-182
[11] Donald K, Krzysztof J. Time series forecasting by combining RBF networks, certainty factors, and the Box-Jenkins model[J]. Neurocomputing, 1996, 10:149-168
[12] Adya M. Corrections to rule-based forecasting: findings from a replication[J]. International Journal of Forecasting, 2000, 16:125-127
[13] Silipo R, Bortolan G., Marchesi C. Design of hybrid architectures based on neural classifier and RBF pre-processing for ECG analysis[J]. International Journal of Approximate Reasoning, 1999, 21:177-196
[14] Fasshauer G. Dual bases and discrete reproducing kernels: a unified framework for RBF and MLS approximation[J]. Engineering Analysis with Boundary Elements, 2005, 29:313-325
[15] Larsson, E, Fornberg B. A numerical study of some radial basis function based solution methods for elliptic PDEs[J]. Computers and Mathematics with Applications, 2003, 46: 891-902
[16] Bugmann G.Normalized Gaussian Radial Basis Function networks[J]. Neurocomputing, 1998, 20:97-110
[17] Acosta F, Miguel A. Radial basis function and related models: An overview[J]. Signal Processing, 1995, 45:37-58
[18] Yu W, Yu, DL. A comparison study on a chemical reactor modelling with a physical model and PLRBF networks[J]. Engineering Applications of Artificial Intelligence, 2003, 16, 629-645
[19] Arnott R. Diversity combining for digital mobile radio using radial basis function networks[J]. Signal Processing, 1997, 63: 1-16
[20] Xu L. RBF nets mixture experts and Bayesian Ying-Yang learning[J]. Neurocomputing, 1998, 19:223-257
[21] Fomberg B, Wright G.. Stable computation of multiquadric interpolants for all values of the shape parameter[J]. Computers and Mathematics with Applications, 2004, 48:853-867
[22] Kuncheva L Initializing of an RBF network by a genetic algorithm[J]. Neurocomputing, 1997, 14:273-288
[23] Sanchez A. Second derivative dependent placement of RBF centers[J]. Neurocomputing, 1995, 7:311-317
[24] Chen W, Tanaka M. A meshless integration-free and boundary-only RBF technique[J]. Computers and Mathematics with Applications, 2002, 43:379-391
[25] Tan K, Tang K. Taguchi-tuned radial basis function with application to high precision motion control[J]. Artificial Intelligence in Engineering, 2001, 15:25-36
[26] Xin L. On simultaneous approximations by radial basis function neural networks[J]. Applied Mathematics and Computation, 1998, 95:75-89
[27] Sheta F, Jong K. Time-series forecasting using GA-tuned radial basis functions[J]. Information Sciences, 2001, 133:221-228
[28] 岳彩青，常青美，庞学民．基于聚类分析的RBF网络建模方法及应用的研究[J]．计算机仿真，2003，23(1)：120-123
[29] Zhong Q, Huang C. A mended hybrid learning algorithm for radial basis function neural networks to improve generalization capability[J]. Applied Mathematical Modelling, 2007, 31:1271-1281
[30] 冀雅生．投影数据矩阵序列的伪逆递推公式[J]．成都大学学报(自然科学版)，2001，20(3)：22-24
[31] Delin Chu, Bart De Moor. On a variational formulation of the QSVD and the RSVD[J]. Linear Algebra and its Applications: 2000, 311 (15): 61-78
[32] 苏小红，侯秋香，马培，王亚东．RBF神经网络的混合学习算法[J]．哈尔滨工业大学学报(自然科学版)，2006，38(9)：1446-1449
[33] 叶世伟，史忠植．神经网络原理[M]．北京：机械工业出版社，2004．1
[34] Michael E. Tipping, David Lowe._Shadow targets: A novel algorithm for topographic projections by radial basis functions[J]. Neurocomputing, 1998, 19(3): 211-222
[35] Huijuan Wu, Yumei Wen, Ping Li. Dynamic discrimination of convergence of the LMS time delay estimation in complicated noisy environments[J]. Applied Acoustics: 2007, 68(6): 628-641
[36] Qin Yan, Saeed Vaseghi, Esfandiar Zavarehei. Formant tracking linear prediction model using HMMs and Kalman filters for noisy speech processing[J]. Computer Speech & Language: 2007, 21 (3): 543-561
[37] E.J.Godolphin, Kostas Triantafyllopoulos. Decomposition of time series models in state-space form[J]. Computational Statistics & Data Analysis: 2006, 50(9): 2232-2246
[38] S.A. Soliman, R. A. Alammari. Harmonic modeling of linear and nonlinear loads based on Kalman filtering algorithm[J]. Electric Power Systems Research: 2004, 72(2): 147-155
[39] I. Yaesh, U. Shaked. Min-max Kalman filtering[J]. Systems & Control Letters: 2004, 53(3): 217-228
[40] Alberto Corigliano, Stefano Mariani. Parameter identification in explicit structural dynamics: performance of the extended Kalman filter[J]. Computer Methods in Applied Mechanics and Engineering: 2004, 193(36-38): 3807-3835
[41] Sergiy A. Vorobyov, Andrzej Cichocki. Hyper Radial Basis Function Neural Networks for Interference Cancellation with Nonlinear Processing of Reference Signal [J]. Digital Signal Processing: 2001, 11(3): 204-221
[42] M. Marinaro, S. Scarpetta. On-line learning in RBF neural networks: a stochastic approach. Neural Networks: 2000,13(7): 719-729
[43] V. David Sanchez A. Robustization of a learning method for RBF networks[J]. Neurocomputing:1995, 9(1): 85-94
[44] J. L. Valdes, R. Biscay, J. C. Jimenez. Geometric selection of centers for radial basis function approximations involved in intensive computer simulations[J]. Mathematics and Computers in Simulation: 1999,48(3): 295-306
[45] Konrad Reif, Frank Sonnemann, Rolf Unbehauen. An EKF-Based Nonlinear Observer with a Prescribed Degree of Stability[J]. Automatica: 1998, 34(9): 1119-1123
[46] Jamshaid Ali, Jiancheng Fang. SINS/ANS integration for augmented performance navigation solution using unscented Kalman filtering[J]. Aerospace Science and Technology: 2006,10(3): 233-238
[47] Pramod Vachhani, Shankar Narasimhan, Raghunathan Rengaswamy. Robust and reliable estimation via Unscented Recursive Nonlinear Dynamic Data Reconciliation[J]. Journal of Process Control: 2006,16(10): 1075-1086
[48] Jose A. Romagnoli, Rafiqul Gani. Studies of distributed parameter systems: Decoupling the state-parameter estimation problem[J]. Chemical Engineering Science: 1983, 38(11): 1831-1843
[49] V. David Sanchez A. Searching for a solution to the automatic RBF network design problem[J]. Neurocomputing: 2002,42(1-4): 147-170
[50] John P. Owens, Douglas G Steigerwald. Noise reduced realized volatility: a kalman filter approach[J]. Advances in Econometrics: 2006, 20(1): 211-227
[51] G. Bolzon, R. Fedele, G Maier. Parameter identification of a cohesive crack model by Kalman filter[J]. Computer Methods in Applied Mechanics and Engineering: 2002, 191(25-26): 2847-2871