一种新的支持向量回归预测模型
|
摘要
支持向量机(SVM)是九十年代中叶Vapnik教授领导的研究小组提出的一种新的智能机器,源于七十年代迅速发展起来的统计学习理论,特别是体现了结构风险最小化的思想和方法。由于具有较完备的理论基础和较好的学习性能,能很好地解决小样本、非线性、高维数和局部极小点等实际问题,因而成为机器学习理论的研究热点。
SVM在很多领域都得到了成功的应用,如模式识别、回归估计、函数逼近等。但是作为一种新兴技术,SVM在很多领域的研究还有待于探索和完善,如何设计快速有效的回归估计算法就是SVM实际应用中的问题之一。
本文主要研究了支持向量机回归算法,并对它进行了改进,主要工作如下:首先概要介绍了支持向量机的基本原理,并对最小二乘回归估计和支持向量机回归估计算法进行了研究和比较。而后,在此基础上通过理论推导,提出一种改进的支持向量机回归估计算法,我们称之为SVR-LS方法。接着,用SVR-LS方法对Sinc函数进行回归逼近,并与LSE方法和SVR方法的实验结果进行了对比,我们发现新方法在拟合逼近方面也有着不错的效果。最后,我们将SVR-LS算法和标准的支持向量机回归估计算法应用于两个不同的数据集进行回归估计,从学习速度和精度两个方面对两种算法进行了实验分析比较,结果表明SVR-LS算法优于标准的支持向量机回归估计算法。
Support vector machines (SVM) is a new kind of intelligent machine presented by Vapnik and his study group in the middle of the 1990Th. SVM is based on statistical learning theory developed in the 1970Th. It embodies the theory of structure risk minimization (SRM). Because it has quite perfect theoretical properties and good learning performance, and can solve some practical problems with low sample size, non-linearity, high-dimension of feature space and local minimization, SVM becomes a hot spot of machine learning theory.
SVM has successful applications in many fields, such as pattern recognition, regression estimation, function approaching and so on. However, as a new technique, SVM still has many problems that need to be studied and improved, and researches in regression estimation based on SVM need to be enhanced. How to design fast and efficient SVM algorithms applied to regression estimation becomes a great challenge in practical applications of support vector machines.
In this paper, we have learned some regress algorithms and proposed several developed Support Vector Regression (SVR). The main works are as follows:
First of all, the principles of SVM are reviewed and the relationship between Least Square Estimation (LSE) and regression estimation algorithms of SVM (SVR) are compared and analyzed. Secondly, an improved regression estimation algorithm named SVR-LS is presented by theoretical deduction. And then, the proposed SVR-LS algorithm is applied to approaching the function of Sinc. Comparing to experimental results of the SVR and LES algorithm, we find that the new algorithm has good effect in function approaching. Finally, the proposed SVR-LS algorithm and normal regression estimation algorithm of SVM are applied to regression estimation of two different data sets, and learning speed and learning precision in regression are compared between the two algorithms. The experimental results show that the proposed SVR-LS algorithm is better than the normal regression estimation algorithm of SVM.
SVM在很多领域都得到了成功的应用,如模式识别、回归估计、函数逼近等。但是作为一种新兴技术,SVM在很多领域的研究还有待于探索和完善,如何设计快速有效的回归估计算法就是SVM实际应用中的问题之一。
本文主要研究了支持向量机回归算法,并对它进行了改进,主要工作如下:首先概要介绍了支持向量机的基本原理,并对最小二乘回归估计和支持向量机回归估计算法进行了研究和比较。而后,在此基础上通过理论推导,提出一种改进的支持向量机回归估计算法,我们称之为SVR-LS方法。接着,用SVR-LS方法对Sinc函数进行回归逼近,并与LSE方法和SVR方法的实验结果进行了对比,我们发现新方法在拟合逼近方面也有着不错的效果。最后,我们将SVR-LS算法和标准的支持向量机回归估计算法应用于两个不同的数据集进行回归估计,从学习速度和精度两个方面对两种算法进行了实验分析比较,结果表明SVR-LS算法优于标准的支持向量机回归估计算法。
Support vector machines (SVM) is a new kind of intelligent machine presented by Vapnik and his study group in the middle of the 1990Th. SVM is based on statistical learning theory developed in the 1970Th. It embodies the theory of structure risk minimization (SRM). Because it has quite perfect theoretical properties and good learning performance, and can solve some practical problems with low sample size, non-linearity, high-dimension of feature space and local minimization, SVM becomes a hot spot of machine learning theory.
SVM has successful applications in many fields, such as pattern recognition, regression estimation, function approaching and so on. However, as a new technique, SVM still has many problems that need to be studied and improved, and researches in regression estimation based on SVM need to be enhanced. How to design fast and efficient SVM algorithms applied to regression estimation becomes a great challenge in practical applications of support vector machines.
In this paper, we have learned some regress algorithms and proposed several developed Support Vector Regression (SVR). The main works are as follows:
First of all, the principles of SVM are reviewed and the relationship between Least Square Estimation (LSE) and regression estimation algorithms of SVM (SVR) are compared and analyzed. Secondly, an improved regression estimation algorithm named SVR-LS is presented by theoretical deduction. And then, the proposed SVR-LS algorithm is applied to approaching the function of Sinc. Comparing to experimental results of the SVR and LES algorithm, we find that the new algorithm has good effect in function approaching. Finally, the proposed SVR-LS algorithm and normal regression estimation algorithm of SVM are applied to regression estimation of two different data sets, and learning speed and learning precision in regression are compared between the two algorithms. The experimental results show that the proposed SVR-LS algorithm is better than the normal regression estimation algorithm of SVM.
引文
[1] V N Vapnik著,张学工译.统计学习理论的本质[M].北京:清华大学出版社,2000:1-155.
[2] V N Vapnik著,许建华,张学工译.统计学习理论[M].北京:电子工业出版社,2004.
[3]边肇祺,张学工.模式识别[M].北京:清华大学出版社,2000.
[4]傅京孙.模式识别及其应用[M].北京:科学出版社,1983.
[5]何晓群,刘文卿.应用回归分析.北京:中国人民大学出版社,2001
[6] V N Vapnik. The Nature of Statistical Learning Theory [M]. New York: Springer, 1995.
[7] V N Vapnik. Statistical learning theory [M]. New York, 1998
[8]邓乃扬,田英杰.数据挖掘中的新方法-支持向量机[M].北京:科学出版社,2004.
[9]阎辉,张学工,李衍达.支持向量机与最小二乘法的关系研究[J].清华大学学报2001,41(9):77-80.
[10]张学工.关于统计学习理论与支持向量机[J].自动化学报.2000,26(1):32-42.
[11] Vapnik V N. An overview of statistical learning theory [J]. IEEE Trans on Neural Networks. 1999,10(s):Support988-999
[12] Comes C, Vapnik V N.vector networks [J]. Machine Learning. 199s20: 273-297.
[13] Vapnik V N, Levin E, Le Cun Y. Measuring the VC dimension of a learning machine [J]. Neural Computation. 1994, 6: 8s1-876.
[14]安金龙,王下欧.一种适合于增量学习的支持向量机的快速循环算法[J].计算机应用.2003, 23(10): 12-14.
[15]刘江华,程君实,陈佳品.支持向量机训练算法综述[J].信息与控制.2002,31(1):45-50.
[16]崔伟东,周志华,李星.支持向量机研究[J].计算机工程与应用.2001, 1:58-61.
[17]柳回春,马树元.支持向量机的研究现状[J].中国图象图形学报(A版),2002,7(6): 618-623.
[18]王晓月,王积勤.支持向量机研究与应用明.空军工程大学学报(自然科学版),2004,5(3):49-55.
[19] Osuna E, Freund R, Girosi F. An improved training algorithm for support vector machines [A]. Proc. the 1997 IEEE workshop on neural netwoks for signal processing[C]. Amelea Island, FL, 1997: 276-285.
[20] Osuna E,Freund R, Girosi F. Training support vector machines: an application to face detection[C]. In: Proc. of CVPR'97, Puerto Rico, 1997.
[21] Joachims T. Making large-scale SVM learning practical[C]. In: Scholkopf B, Burges C J C, Smola A eds. Advances in kernel methods: support vector learning, Cambridge, MA:MIT Press, 1998: 169-184
[22] Suykens J A K, Branbanter J K, Lukas L, et al. Weighted least squares support vector machines: robustness and spare approximation [J). Neurocomputing. 2002, 48(1):85-105
[23]朱家元,陈开陶,张恒喜.最小二乘支持向量机算法研究[J].计算机科学. 2003, 30(7): 157-159.
[24]杜树新,吴铁军.用于回归估计的支持向量机方法[J].系统仿真学报.2003, 15(11):1580-1585.
[25]杜树新,吴铁军.回归型加权支持向量机方法及其应用[J].浙江大学学报.2004,38(3): 302-306.
[26] Zhang X. Using class-center vectors to build support vector machines [C]. In:Proceedings of NNSP'99, 1999.
[27]张铃.支持向量机理论与基于规划的神经网络学习算法[J],计算机学报2001,24(2): 113-118.
[28]王定成.支持向量机回归与控制的研究[D]博士毕业论文.中国科学技术大学,2003.
[29]陶亮.基于人脸识别的身份认证方法研究[D].博士毕业论文.中国科学技术大学,2003.
[30]丁蕾,朱德权,陶亮.支持向量机在物理试验中的应用[J],安庆师范学院学报(自然科学版).2005,11(2):64-65.
[31] V N Vapnik., Golowich S, Smola A. Support vector method for function approximation, regression estimation, and signal processing [A]. In: Mozer M, Jordan M, Petsche T(eds). Neural Information Processing Systems [M], Cambridge, MA:MIT Press. 1997: 281-287.
[32]祁亨年,支持向量机及其应用研究综述[J].计算机工程,2004,30(10): 6-9
[33] Osuna E, Freund R. Training Support Vector Machines: an Application to Face Detection [A]. Proc. of Computer Vision and Pattern Recognition [C]. San Juan, Detection [A]. Proc. of Computer Vision and Pattern Recognition [C]. San Juan,Puerto Rico, IEEE Computer Soc, 1997: 130-136.
[34] Deniz O, Casterillon M,Hernandez M. Face Recognition Using Independent Component Analysis and Support Vector Machines[J].Pattern Recognition Letters,2003, 24: 2153-2157
[35] Chen Datong, Jean-Marc Odobez. Comparision of Support Vector Machine and Neural Network for Text Texture Verification, IDIAP-RR 02-19, 2002.
[36] Bottou L, Comes C, Dcnker et aI. Comparison of classifier methods: A case study in handwriting digit recognition [C]. In: International Conference on Patten Recognition, IEEE Computer Society Press, 1994: 77-87.
[37] Scholkopf B, et al. Comparing support vector machines with gaussian kernels to radial basis function classifiers [J]. IEEE Trans. on Signal Processing. 1997, 45:2758-2765.
[38]张庆. MATLAB语言在非线性最小二乘中的应用[J],测绘与地理空间信息.2004,27(3):35-38.
[39]许启发. MATLAB中方程组的解与回归模型的最小二乘估计[J],统计与决策.2005,3:21-23.
[40] Blake C L, Merz C J. UCI Repository of Machine Learning Databases. (http://www.ics.uci.edu/~mlearn/MLRepository.html), 2005.
[41] Burges C J C. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, Vol.2, No.2, 1998.
[42] Burges C J C. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, Vol.2, No.2, 1998.
[43] A. J. Smola, B. Scholkopf. On a Kemel-based method for pattern, recognition, regression, approximation and operator inversion. Algnrithmica.Vol.22, 1998:211-231
[44] Dick Kai Tik Chow ,Tong Lee. Image Approximation and Smoothing by Support Vector Regression. IEEE Proceedings of the International Joint Conference on Neural Networks v4 2001:2427-2432A
[45] J Smola,B Scholkopf. A Tutorial on Support Vector Regression. Technical Report NC2-TR-1998-030. NeuroColt2 Technical Renort Series. 1998:1-20
[46]邓先礼.《最优化技术》.重庆大学出版社,1998 : 38-46
[47]周恕义等.用最小二乘法提高CCD干涉计量的精度[J].仪器仪表学报.1999,20(1): 100-102.
[48]张志涌.等编《MATLAB教程-基于6.x版本》.北京航空航天大学出版社,2001
[49] Peter J. Brockwell , Richard A. Davis著,田铮译.时间序列的理论和方法[M].北京:高等教育出版社,施普林格出版社,2001
[2] V N Vapnik著,许建华,张学工译.统计学习理论[M].北京:电子工业出版社,2004.
[3]边肇祺,张学工.模式识别[M].北京:清华大学出版社,2000.
[4]傅京孙.模式识别及其应用[M].北京:科学出版社,1983.
[5]何晓群,刘文卿.应用回归分析.北京:中国人民大学出版社,2001
[6] V N Vapnik. The Nature of Statistical Learning Theory [M]. New York: Springer, 1995.
[7] V N Vapnik. Statistical learning theory [M]. New York, 1998
[8]邓乃扬,田英杰.数据挖掘中的新方法-支持向量机[M].北京:科学出版社,2004.
[9]阎辉,张学工,李衍达.支持向量机与最小二乘法的关系研究[J].清华大学学报2001,41(9):77-80.
[10]张学工.关于统计学习理论与支持向量机[J].自动化学报.2000,26(1):32-42.
[11] Vapnik V N. An overview of statistical learning theory [J]. IEEE Trans on Neural Networks. 1999,10(s):Support988-999
[12] Comes C, Vapnik V N.vector networks [J]. Machine Learning. 199s20: 273-297.
[13] Vapnik V N, Levin E, Le Cun Y. Measuring the VC dimension of a learning machine [J]. Neural Computation. 1994, 6: 8s1-876.
[14]安金龙,王下欧.一种适合于增量学习的支持向量机的快速循环算法[J].计算机应用.2003, 23(10): 12-14.
[15]刘江华,程君实,陈佳品.支持向量机训练算法综述[J].信息与控制.2002,31(1):45-50.
[16]崔伟东,周志华,李星.支持向量机研究[J].计算机工程与应用.2001, 1:58-61.
[17]柳回春,马树元.支持向量机的研究现状[J].中国图象图形学报(A版),2002,7(6): 618-623.
[18]王晓月,王积勤.支持向量机研究与应用明.空军工程大学学报(自然科学版),2004,5(3):49-55.
[19] Osuna E, Freund R, Girosi F. An improved training algorithm for support vector machines [A]. Proc. the 1997 IEEE workshop on neural netwoks for signal processing[C]. Amelea Island, FL, 1997: 276-285.
[20] Osuna E,Freund R, Girosi F. Training support vector machines: an application to face detection[C]. In: Proc. of CVPR'97, Puerto Rico, 1997.
[21] Joachims T. Making large-scale SVM learning practical[C]. In: Scholkopf B, Burges C J C, Smola A eds. Advances in kernel methods: support vector learning, Cambridge, MA:MIT Press, 1998: 169-184
[22] Suykens J A K, Branbanter J K, Lukas L, et al. Weighted least squares support vector machines: robustness and spare approximation [J). Neurocomputing. 2002, 48(1):85-105
[23]朱家元,陈开陶,张恒喜.最小二乘支持向量机算法研究[J].计算机科学. 2003, 30(7): 157-159.
[24]杜树新,吴铁军.用于回归估计的支持向量机方法[J].系统仿真学报.2003, 15(11):1580-1585.
[25]杜树新,吴铁军.回归型加权支持向量机方法及其应用[J].浙江大学学报.2004,38(3): 302-306.
[26] Zhang X. Using class-center vectors to build support vector machines [C]. In:Proceedings of NNSP'99, 1999.
[27]张铃.支持向量机理论与基于规划的神经网络学习算法[J],计算机学报2001,24(2): 113-118.
[28]王定成.支持向量机回归与控制的研究[D]博士毕业论文.中国科学技术大学,2003.
[29]陶亮.基于人脸识别的身份认证方法研究[D].博士毕业论文.中国科学技术大学,2003.
[30]丁蕾,朱德权,陶亮.支持向量机在物理试验中的应用[J],安庆师范学院学报(自然科学版).2005,11(2):64-65.
[31] V N Vapnik., Golowich S, Smola A. Support vector method for function approximation, regression estimation, and signal processing [A]. In: Mozer M, Jordan M, Petsche T(eds). Neural Information Processing Systems [M], Cambridge, MA:MIT Press. 1997: 281-287.
[32]祁亨年,支持向量机及其应用研究综述[J].计算机工程,2004,30(10): 6-9
[33] Osuna E, Freund R. Training Support Vector Machines: an Application to Face Detection [A]. Proc. of Computer Vision and Pattern Recognition [C]. San Juan, Detection [A]. Proc. of Computer Vision and Pattern Recognition [C]. San Juan,Puerto Rico, IEEE Computer Soc, 1997: 130-136.
[34] Deniz O, Casterillon M,Hernandez M. Face Recognition Using Independent Component Analysis and Support Vector Machines[J].Pattern Recognition Letters,2003, 24: 2153-2157
[35] Chen Datong, Jean-Marc Odobez. Comparision of Support Vector Machine and Neural Network for Text Texture Verification, IDIAP-RR 02-19, 2002.
[36] Bottou L, Comes C, Dcnker et aI. Comparison of classifier methods: A case study in handwriting digit recognition [C]. In: International Conference on Patten Recognition, IEEE Computer Society Press, 1994: 77-87.
[37] Scholkopf B, et al. Comparing support vector machines with gaussian kernels to radial basis function classifiers [J]. IEEE Trans. on Signal Processing. 1997, 45:2758-2765.
[38]张庆. MATLAB语言在非线性最小二乘中的应用[J],测绘与地理空间信息.2004,27(3):35-38.
[39]许启发. MATLAB中方程组的解与回归模型的最小二乘估计[J],统计与决策.2005,3:21-23.
[40] Blake C L, Merz C J. UCI Repository of Machine Learning Databases. (http://www.ics.uci.edu/~mlearn/MLRepository.html), 2005.
[41] Burges C J C. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, Vol.2, No.2, 1998.
[42] Burges C J C. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, Vol.2, No.2, 1998.
[43] A. J. Smola, B. Scholkopf. On a Kemel-based method for pattern, recognition, regression, approximation and operator inversion. Algnrithmica.Vol.22, 1998:211-231
[44] Dick Kai Tik Chow ,Tong Lee. Image Approximation and Smoothing by Support Vector Regression. IEEE Proceedings of the International Joint Conference on Neural Networks v4 2001:2427-2432A
[45] J Smola,B Scholkopf. A Tutorial on Support Vector Regression. Technical Report NC2-TR-1998-030. NeuroColt2 Technical Renort Series. 1998:1-20
[46]邓先礼.《最优化技术》.重庆大学出版社,1998 : 38-46
[47]周恕义等.用最小二乘法提高CCD干涉计量的精度[J].仪器仪表学报.1999,20(1): 100-102.
[48]张志涌.等编《MATLAB教程-基于6.x版本》.北京航空航天大学出版社,2001
[49] Peter J. Brockwell , Richard A. Davis著,田铮译.时间序列的理论和方法[M].北京:高等教育出版社,施普林格出版社,2001