对神经网络学习算法的研究
|
摘要
神经网络的学习算法一直是人工神经网络研究和应用领域中的一个重要问题，尤其是对前向神经网络学习算法(设计)的研究。对此，至今没有一个十分理想的解决办法。本论文在参考了大量国内外有关科技文献的基础上，对神经网络学习算法作了深入的研究，给出了切实可行的算法，主要解决了非线性样本的分类问题。
本文由两部分组成：第一部分侧重对径向基神经网络的研究，实现了一种基于RBF网络的新的学习算法；第二部分侧重对前向神经网络的规划学习算法的研究，根据支持向量机理论与神经网络的规划算法的关系，给出了一种新的神经网络基于规划学习算法。
第一部分从RBF网络出发，通过递归分割将输入空间划分为两部分，从而将输入空间变成一个用超矩形构成的回归树(二叉树)。回归树的结点可以很容易的转换为径向基函数，通过对回归树结点的访问，可以选择出使网络达到最优的基函数集，形成最终的网络，此算法可以很好的应用到函数逼近、图像处理等方面。
第二部分从神经网络的几何意义出发，根据支持向量机理论与神经网络规划算法的关系，给出了一种新的构造型学习算法。实验表明该学习算法解决了线性不可分样本的分类问题，而且大大的降低了学习复杂度，并能应用到大样本分类问题中。
The learning algorithm of neural network has always been an important problem in both research and application fields of artificial neural networks, especially to the study of the learning (design) of feedforward neural networks. Up to now, there's no practical way good enough to solve it. In this paper, we profoundly research on the learning algorithm of neural networks after referring to lots of domestic and foreign scientific literature and give a practical classification algorithm under the non-linear separability condition.
There are two parts in the contents of this thesis. The first part mainly introduces the study of RBF neural networks, realized a new learning algorithm based on RBF neural networks. The second part mainly introduces the study of feedforward neural networks, and presents a new programming based learning algorithms in neural networks under the equivalent between SVM and programming based learning algorithms.
The first part of this thesis describes the theory of RBF neural networks. The input space is thus divided into hyperrectangles organized into a regression tree (binary tree) by recursively partition the input space in two. It's easy to translate the node of regression tree into radial basis function. After the nodes of regression tree are visited, we can generate a set of radial basis functions from which the final network can be selected. This algorithm fits to the application of function approximation, image procession and so on.
The second part introduces the geometrical representation of neural networks and presents a new constructive learning approach based on the relationship between SVM based algorithms and programming based learning algorithms in neural networks. Experimental results show that the new algorithm can solve classification problems of non-linear separability samples. It also can greatly reduce the learning complexity and can be applied to real classification problems with a vast amount of data.
本文由两部分组成：第一部分侧重对径向基神经网络的研究，实现了一种基于RBF网络的新的学习算法；第二部分侧重对前向神经网络的规划学习算法的研究，根据支持向量机理论与神经网络的规划算法的关系，给出了一种新的神经网络基于规划学习算法。
第一部分从RBF网络出发，通过递归分割将输入空间划分为两部分，从而将输入空间变成一个用超矩形构成的回归树(二叉树)。回归树的结点可以很容易的转换为径向基函数，通过对回归树结点的访问，可以选择出使网络达到最优的基函数集，形成最终的网络，此算法可以很好的应用到函数逼近、图像处理等方面。
第二部分从神经网络的几何意义出发，根据支持向量机理论与神经网络规划算法的关系，给出了一种新的构造型学习算法。实验表明该学习算法解决了线性不可分样本的分类问题，而且大大的降低了学习复杂度，并能应用到大样本分类问题中。
The learning algorithm of neural network has always been an important problem in both research and application fields of artificial neural networks, especially to the study of the learning (design) of feedforward neural networks. Up to now, there's no practical way good enough to solve it. In this paper, we profoundly research on the learning algorithm of neural networks after referring to lots of domestic and foreign scientific literature and give a practical classification algorithm under the non-linear separability condition.
There are two parts in the contents of this thesis. The first part mainly introduces the study of RBF neural networks, realized a new learning algorithm based on RBF neural networks. The second part mainly introduces the study of feedforward neural networks, and presents a new programming based learning algorithms in neural networks under the equivalent between SVM and programming based learning algorithms.
The first part of this thesis describes the theory of RBF neural networks. The input space is thus divided into hyperrectangles organized into a regression tree (binary tree) by recursively partition the input space in two. It's easy to translate the node of regression tree into radial basis function. After the nodes of regression tree are visited, we can generate a set of radial basis functions from which the final network can be selected. This algorithm fits to the application of function approximation, image procession and so on.
The second part introduces the geometrical representation of neural networks and presents a new constructive learning approach based on the relationship between SVM based algorithms and programming based learning algorithms in neural networks. Experimental results show that the new algorithm can solve classification problems of non-linear separability samples. It also can greatly reduce the learning complexity and can be applied to real classification problems with a vast amount of data.
引文
[1] 张德富 殷正坤．人工神经网络的发展及其哲理．科学技术与辩证法，2000／8，Vol．4：17～20
[2] Zhu Mingxing. Zhang Delong. An algorithm study with the center selection of RBFN basis functions. Journal of Anhui University, 2000, Vol 24: 72～78.
[3] 周俊武，孙传尧，王福利．径向基函数(RBF)网络的研究及实现．矿冶，2001，12(10)：71～75
[4] T Poggio, F Girosi. Nerworks for Approximation and Learning. Proc. IEEE, 1990, 78(9): 1481～1497
[5] G Cybenko. Approxirnation by Superposition of a Sigmoidal Functin. Mathematics of Control, Signal and Systems, 1989, 2:303～314
[6] J Moody, C J Darken. Fast Learning in Networks of Locally-tuned Processing Units. Neural Computation, 1989, 1:281～294
[7] 张铃 张钹，多层前馈神经网络的学习和综合算法．软件学报，1995，7
[8] Chen Q C et.al. Generating - Shrinking Algorithm for Learning Arbitrary Classification, Neural Networks, 1994, 7:1477～1489
[9] Powell M I D. Radial basis function for multivariable interpolation, Review IMA Conf. on algorithms for the Approximation of Functions and Data, RMCS Shrivanham, 1985
[10] Broomhead D S, Lowe D. Multivariable function interpolation and adaptive network, Complex Systems, 1988(2)
[11] L. Breiman, J. Friedman, J.Olsen, and C.Stone. Classification and Regression Trees. Wadsworth, Belmont, CA, 1984.
[12] Jackon I R H. Convergence properties of radial basis function, Constr. Approx., 1988(4): 243～264
[13] M. Kubat and I.Ivanova. Initialize of RBF networks with decision trees. In Prof. of the 5th Belgian-Dutch Conf. Machine Learning, 1995:61～70
[14] M. Kubat. Decision trees can initialize radial-basis function networks. IEEE Transactions on Neural Networks, 1998.9(5), p813～821
[15] Mark J. L. Orr, Recent advances in radial basis function networks. Technical report, Institute for Adaptive and Neural Computation, 1999.6, p17～21
[16] Mark J. L. Orr, Introduction to radial basis function networks. Institute for Adaptive and Neural Computation, 1996.4, p19～35
[17] 张铃．支持向量机理论与基于规划的神经网络学习算法．计算机学报，2001，2(24)：113～118
[18] Poggio T. and Girosi F. Regularization Algorithms for Learning that are Eguivalent to Multilayer Networks. Science, 1990,247:982～987
[19] Y. Lee, Handwritten digit recognition using K-nearest-neighbor, radial basis function and backpropagation neural networks, Neural Computation, 1991,vol.3:440～449
[20] D. Lowe, Adaptive radial basis function nonlinearities and the problem of generalization,in 1st. Conf. Artificial Neural Networks. London U.K., 1989: 171～175
[21] S. Chen, C. F. N. Cowan and P. M. Grant, Orthogonal least square learning algorithm for radial basis
function networks. IEEE Tran. Neural Networks, 1991, vol.2:302～309
[22] Y. H. Cheng and C. S. Lin, A learning algorithm for radial basis function networks with the capability of adding and pruning neuron. Proc. IEEE, 1994: 797～801
[23] J. E. Moody and C. J. Darken, Fast algorithms in networks of locally tuned processing units, Neural Computation, 1989,vo1.1:282～294
[24] M. T. Musavi, W. Ahmed, K. H. Chan, K. B. Faris and D. M. Hummek, On the training of radial basis function classifiers, Neural Networks, 1992,vol.5: 595～603
[25] D. Lowe, Adaptive radial basis function nonlinearities and the problem of generalization, in 1th 1st Int. Conf. Artificial Neural Networks, London, U.K., 1989: 171～175.
[26] A. Saha and J. D. Keeler, "Algorithms for better representation and faster learning in radial basis function networks," in Advances in Neural Information Processing Systems 2, Ed. D. S. Touretzky. San Mateo,CA, Morgan Kaufmann, 1990, pp. 482～489.
[27] S. Haykin, A Comprehensive Foundation, Neural Networks, New York: Maxmillan, 1994
[28] J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo: Morgan Kaufmann, 1993.
[29] 于秀丽，沈雪勤．RBF神经网络的一种新的学习算法．南京大学学报，2002(38)：28—32
[30] Vapnik V. The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995
[31] Cortes C, Vapnik V. Support Vector Networks. Machine Learning, 1995, 20:273～297
[32] Osuna E, Freund R, Girosi F, Support Vector Machines: Training and Applications. AI Memo 1602, MIT Al Lab, 1997
[33] Barzilay O. Brailovsky V. L. On domain knowledge and feature selection using a support vector machine. Pattern reconition Letters, 1999, 20:475～484
[34] Amari S. Wu S. Improving support vector machine classifier by modifying kernel functions. Neural Networks, 1999,12:783～789
[35] Herbrich R, Weston J. Adaptive margin vector machines for classification. Artificial Neural Networks, Conference Publication No. 470, IEEE 1999:880～885
[36] Mattera D. Palmieri F. Haykin S. Simple and robust methods for support vector machine classifiers. IEEE Trans Neural Networks, 1999,10(5):1038～1047
[37] Courant R. Hibert B. Methods of mathematical physics. Vol. 1. New York: Widely-interscience, 1953
[38] Scholkopf B. Mika S. Burges C J C et al. Input space versus feature space in kernel-based methods. IEEE Trans Neural Networks, 1999,10(5): 1000～1017
[39] Sholkopf B. Smola A. Muller K R. Nonliear component analysis as a kernel eigenvalue problem. Neural Computers, 1998,10:1229～1319
[40] 于秀丽，沈雪勤．支持向量机与神经网络的单纯形迭代算法．河北省科学院学报，2002(19)：197～199
[41] 张铃，张钹．前向神经网络设计问题的回顾与探索．计算机工程与科学，1998，11(20)：1～10
[42] Zhang Ling, The Theory of SVM and Programming Based Learning Algorithms in Neural Networks, Chinese J. Computers, 2001, 2(2): 113～118
[43] Vapnik V N. An Overview of Statistical Learning Theory. IEEE Trans Neural Networks, 1999,10(5): 988～999
[44] McCulloch W. S. , Pitts W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysies, 1943, 5: 115～133
[45] Rujan P., Marchand M. A geometric approach to learning in neural networks. In: Proceedings of the Inernational Joint Conference on Neural Networks'89 ,Vol Ⅱ. Washington , DC: IEEE TAB Neural Network Committee, 1989: 105～110
[46] Ramacher U., Wesseling M. A geometrical approach to neural network design. In: Proceedings of the International Joint Conference on Neural Networks'89 ,Vol Ⅱ. Washington , DC: IEEE TAB Neural Network Committee, 1989:147～154
[47] 张铃，张钹．多层前馈神经网络的综合和学习算法．软件学报，1997，8(4)：252～258
[48] Baum E. B., Lang K. J. Constracting hidden units using examples and queries. In: Lippman R P et al eds. Neural Information Processing. San Mateo, CA: Morgan Kaufmann Publishers. Inc., 1991:904～910
[49] Chen Q. C. et al. Generating-shrinking algorithm for learning arbitrary classification. Neural Networks, 1994,5(7): 1477～1489
[50] Fahlman S. E. , Lebiere C. The cascade-correlation learning archtecture. In: Tourdtzhy D S ed. Advances in Neural Information-processing System. San Mateo, CA: Morgan Kaufmann Publishers, Inc., 1990: 524～532
[2] Zhu Mingxing. Zhang Delong. An algorithm study with the center selection of RBFN basis functions. Journal of Anhui University, 2000, Vol 24: 72～78.
[3] 周俊武，孙传尧，王福利．径向基函数(RBF)网络的研究及实现．矿冶，2001，12(10)：71～75
[4] T Poggio, F Girosi. Nerworks for Approximation and Learning. Proc. IEEE, 1990, 78(9): 1481～1497
[5] G Cybenko. Approxirnation by Superposition of a Sigmoidal Functin. Mathematics of Control, Signal and Systems, 1989, 2:303～314
[6] J Moody, C J Darken. Fast Learning in Networks of Locally-tuned Processing Units. Neural Computation, 1989, 1:281～294
[7] 张铃 张钹，多层前馈神经网络的学习和综合算法．软件学报，1995，7
[8] Chen Q C et.al. Generating - Shrinking Algorithm for Learning Arbitrary Classification, Neural Networks, 1994, 7:1477～1489
[9] Powell M I D. Radial basis function for multivariable interpolation, Review IMA Conf. on algorithms for the Approximation of Functions and Data, RMCS Shrivanham, 1985
[10] Broomhead D S, Lowe D. Multivariable function interpolation and adaptive network, Complex Systems, 1988(2)
[11] L. Breiman, J. Friedman, J.Olsen, and C.Stone. Classification and Regression Trees. Wadsworth, Belmont, CA, 1984.
[12] Jackon I R H. Convergence properties of radial basis function, Constr. Approx., 1988(4): 243～264
[13] M. Kubat and I.Ivanova. Initialize of RBF networks with decision trees. In Prof. of the 5th Belgian-Dutch Conf. Machine Learning, 1995:61～70
[14] M. Kubat. Decision trees can initialize radial-basis function networks. IEEE Transactions on Neural Networks, 1998.9(5), p813～821
[15] Mark J. L. Orr, Recent advances in radial basis function networks. Technical report, Institute for Adaptive and Neural Computation, 1999.6, p17～21
[16] Mark J. L. Orr, Introduction to radial basis function networks. Institute for Adaptive and Neural Computation, 1996.4, p19～35
[17] 张铃．支持向量机理论与基于规划的神经网络学习算法．计算机学报，2001，2(24)：113～118
[18] Poggio T. and Girosi F. Regularization Algorithms for Learning that are Eguivalent to Multilayer Networks. Science, 1990,247:982～987
[19] Y. Lee, Handwritten digit recognition using K-nearest-neighbor, radial basis function and backpropagation neural networks, Neural Computation, 1991,vol.3:440～449
[20] D. Lowe, Adaptive radial basis function nonlinearities and the problem of generalization,in 1st. Conf. Artificial Neural Networks. London U.K., 1989: 171～175
[21] S. Chen, C. F. N. Cowan and P. M. Grant, Orthogonal least square learning algorithm for radial basis
function networks. IEEE Tran. Neural Networks, 1991, vol.2:302～309
[22] Y. H. Cheng and C. S. Lin, A learning algorithm for radial basis function networks with the capability of adding and pruning neuron. Proc. IEEE, 1994: 797～801
[23] J. E. Moody and C. J. Darken, Fast algorithms in networks of locally tuned processing units, Neural Computation, 1989,vo1.1:282～294
[24] M. T. Musavi, W. Ahmed, K. H. Chan, K. B. Faris and D. M. Hummek, On the training of radial basis function classifiers, Neural Networks, 1992,vol.5: 595～603
[25] D. Lowe, Adaptive radial basis function nonlinearities and the problem of generalization, in 1th 1st Int. Conf. Artificial Neural Networks, London, U.K., 1989: 171～175.
[26] A. Saha and J. D. Keeler, "Algorithms for better representation and faster learning in radial basis function networks," in Advances in Neural Information Processing Systems 2, Ed. D. S. Touretzky. San Mateo,CA, Morgan Kaufmann, 1990, pp. 482～489.
[27] S. Haykin, A Comprehensive Foundation, Neural Networks, New York: Maxmillan, 1994
[28] J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo: Morgan Kaufmann, 1993.
[29] 于秀丽，沈雪勤．RBF神经网络的一种新的学习算法．南京大学学报，2002(38)：28—32
[30] Vapnik V. The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995
[31] Cortes C, Vapnik V. Support Vector Networks. Machine Learning, 1995, 20:273～297
[32] Osuna E, Freund R, Girosi F, Support Vector Machines: Training and Applications. AI Memo 1602, MIT Al Lab, 1997
[33] Barzilay O. Brailovsky V. L. On domain knowledge and feature selection using a support vector machine. Pattern reconition Letters, 1999, 20:475～484
[34] Amari S. Wu S. Improving support vector machine classifier by modifying kernel functions. Neural Networks, 1999,12:783～789
[35] Herbrich R, Weston J. Adaptive margin vector machines for classification. Artificial Neural Networks, Conference Publication No. 470, IEEE 1999:880～885
[36] Mattera D. Palmieri F. Haykin S. Simple and robust methods for support vector machine classifiers. IEEE Trans Neural Networks, 1999,10(5):1038～1047
[37] Courant R. Hibert B. Methods of mathematical physics. Vol. 1. New York: Widely-interscience, 1953
[38] Scholkopf B. Mika S. Burges C J C et al. Input space versus feature space in kernel-based methods. IEEE Trans Neural Networks, 1999,10(5): 1000～1017
[39] Sholkopf B. Smola A. Muller K R. Nonliear component analysis as a kernel eigenvalue problem. Neural Computers, 1998,10:1229～1319
[40] 于秀丽，沈雪勤．支持向量机与神经网络的单纯形迭代算法．河北省科学院学报，2002(19)：197～199
[41] 张铃，张钹．前向神经网络设计问题的回顾与探索．计算机工程与科学，1998，11(20)：1～10
[42] Zhang Ling, The Theory of SVM and Programming Based Learning Algorithms in Neural Networks, Chinese J. Computers, 2001, 2(2): 113～118
[43] Vapnik V N. An Overview of Statistical Learning Theory. IEEE Trans Neural Networks, 1999,10(5): 988～999
[44] McCulloch W. S. , Pitts W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysies, 1943, 5: 115～133
[45] Rujan P., Marchand M. A geometric approach to learning in neural networks. In: Proceedings of the Inernational Joint Conference on Neural Networks'89 ,Vol Ⅱ. Washington , DC: IEEE TAB Neural Network Committee, 1989: 105～110
[46] Ramacher U., Wesseling M. A geometrical approach to neural network design. In: Proceedings of the International Joint Conference on Neural Networks'89 ,Vol Ⅱ. Washington , DC: IEEE TAB Neural Network Committee, 1989:147～154
[47] 张铃，张钹．多层前馈神经网络的综合和学习算法．软件学报，1997，8(4)：252～258
[48] Baum E. B., Lang K. J. Constracting hidden units using examples and queries. In: Lippman R P et al eds. Neural Information Processing. San Mateo, CA: Morgan Kaufmann Publishers. Inc., 1991:904～910
[49] Chen Q. C. et al. Generating-shrinking algorithm for learning arbitrary classification. Neural Networks, 1994,5(7): 1477～1489
[50] Fahlman S. E. , Lebiere C. The cascade-correlation learning archtecture. In: Tourdtzhy D S ed. Advances in Neural Information-processing System. San Mateo, CA: Morgan Kaufmann Publishers, Inc., 1990: 524～532