Regularized learning in Banach spaces as an optimization problem: representer theorems

详细信息    查看全文

• 作者：Haizhang Zhang (12) haizhang@umich.edu
  Jun Zhang (1) junz@umich.edu
• 关键词：Reproducing kernel Banach spaces – ; Semi ; inner products – ; Representer theorems – ; Regularization networks – ; Support vector machine classification
• 刊名：Journal of Global Optimization
• 出版年：2012
• 出版时间：October 2012
• 年：2012
• 卷：54
• 期：2
• 页码：235-250
• 全文大小：242.9 KB
• 参考文献：1. Argyriou, A., Micchelli, C.A., Pontil, M.: When is there a representer theorem? Vector versus matrix regularizers. Preprint, arXiv:0809.1590v1 (2008)
  2. Aronszajn N.: Theory of reproducing kernels. Trans. Amer. Math. Soc. 68, 337–404 (1950)
  3. Becker T., Weispfenning V.: Gr枚bner Bases: A Computational Approach to Commutative Algebra. Springer-Verlag, New York (1993)
  4. Bennett, K., Bredensteiner, E.: Duality and geometry in SVM classifier. In: Langley, P. (ed.) Proceeding of the Seventeenth International Conference on Machine Learning, pp. 57–64. Morgan Kaufmann, San Francisco (2000)
  5. Berlinet A., Thomas-Agnan C.: Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, Norwell, MA (2004)
  6. Boege W., Gebauer R., Kredel H.: Some examples for solving systems of algebraic equations by calculating Gr枚bner bases. J. Symb. Comput. 1, 83–98 (1986)
  7. Canu, S., Mary, X., Rakotomamonjy, A.: Functional learning through kernel. In: Suykens, J., Horvath, G., Basu, S., Micchelli, C.A., Vandewalle, J. (eds.) Advances in Learning Theory: Methods, Models and Applications. NATO Science Series III: Computer and Systems Sciences, vol. 190, pp. 89–110. IOS Press, Amsterdam (2003)
  8. Conway J.B.: A Course in Functional Analysis, 2nd edn. Springer-Verlag, New York (1990)
  9. Cox D., O’Sullivan F.: Asymptotic analysis of penalized likelihood and related estimators. Ann. Statist. 18, 1676–1695 (1990)
  10. Cucker F., Smale S.: On the mathematical foundations of learning. Bull. Amer. Math. Soc. 39, 1–49 (2002)
  11. Cudia D.F.: On the localization and directionalization of uniform convexity. Bull. Amer. Math. Soc. 69, 265–267 (1963)
  12. Der, R., Lee, D.: Large-margin classification in Banach spaces. JMLR Workshop and Conference Proceedings 2: AISTATS, 91–98 (2007)
  13. Evgeniou T., Pontil M., Poggio T.: Regularization networks and support vector machines. Adv. Comput. Math. 13, 1–50 (2000)
  14. Fabian M. et al.: Functional Analysis and Infinite-Dimensional Geometry. Springer, New York (2001)
  15. Gentile C.: A new approximate maximal margin classification algorithm. J. Mach. Learn. Res. 2, 213–242 (2001)
  16. Giles J.R.: Classes of semi-inner-product spaces. Trans. Amer. Math. Soc. 129, 436–446 (1967)
  17. Hein M., Bousquet O., Sch枚lkopf B.: Maximal margin classification for metric spaces. J. Comput. System Sci. 71, 333–359 (2005)
  18. Kimber D., Long P.M.: On-line learning of smooth functions of a single variable. Theoret. Comput. Sci. 148, 141–156 (1995)
  19. Kimeldorf G., Wahba G.: Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33, 82–95 (1971)
  20. Lumer G.: Semi-inner-product spaces. Trans. Amer. Math. Soc. 100, 29–43 (1961)
  21. Megginson R.E.: An Introduction to Banach Space Theory. Springer, New York (1998)
  22. Mercer J.: Functions of positive and negative type and their connection with the theorey of integral equations. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 209, 415–446 (1909)
  23. Micchelli, C.A., Pontil, M.: A function representation for learning in Banach spaces. In: Learning Theory, pp. 255–269. Lecture Notes in Computer Science, 3120, Springer, Berlin (2004)
  24. Micchelli C.A., Pontil M.: Feature space perspectives for learning the kernel. Machine Learning 66, 297–319 (2007)
  25. Moore E.H.: On properly positive Hermitian matrices. Bull. Amer. Math. Soc. 23, 59 (1916)
  26. Moore, E.H.: General Analysis. Memoirs of the American Philosophical Society, Part I (1935), Part II (1939)
  27. Pardalos, P.M., Hansen, P. (eds.): Data Mining and Mathematical Programming. Papers from the workshop held in Montreal, QC, October 10–13, 2006. CRM Proceedings & Lecture Notes 45. American Mathematical Society, Providence, RI (2008)
  28. Sch枚lkopf, B., Herbrich, R., Smola, A.J.: A generalized representer theorem. Proceeding of the Fourteenth Annual Conference on Computational Learning Theory and the Fifth European Conference on Computational Learning Theory, pp. 416–426. Springer-Verlag, London, UK (2001)
  29. Sch枚lkopf B., Smola A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, Mass (2002)
  30. Shawe-Taylor J., Cristianini N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)
  31. Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-posed Problems. V. H. Winston & Sons (distributed by Wiley), New York (1977)
  32. Tropp J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 52, 1030–1051 (2006)
  33. Vapnik V.N.: Statistical Learning Theory. Wiley, New York (1998)
  34. von Luxburg U., Bousquet O.: Distance-based classification with Lipschitz functions. J. Mach. Learn. Res. 5, 669–695 (2004)
  35. Wahba G.: Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV. In: Sch枚lkopf, B., Burge, C., Smola, A.J. (eds) Advances in Kernel Methods–Support Vector Learning., pp. 69–88. MIT Press, Cambridge, Mass (1999)
  36. Xu Y., Zhang H.: Refinable kernels. J. Mach. Learn. Res. 8, 2083–2120 (2007)
  37. Xu Y., Zhang H.: Refinement of reproducing kernels. J. Mach. Learn. Res. 10, 107–140 (2009)
  38. Young R.M.: An Introduction to Nonharmonic Fourier Series. Academic Press, New York (1980)
  39. Zhang H., Xu Y., Zhang J.: Reproducing kernel Banach spaces for machine learning. J. Mach. Learn. Res. 10, 2741–2775 (2009)
  40. Zhang, H., Zhang, J.: Generalized semi-inner products with applications to regularized learning. J. Math. Anal. Appl., accepted
  41. Zhang T.: On the dual formulation of regularized linear systems with convex risks. Machine Learning 46, 91–129 (2002)
  42. Zhou, D., Xiao, B., Zhou, H., Dai, R.: Global geometry of SVM classifiers. Technical Report 30-5-02. Institute of Automation, Chinese Academy of Sciences (2002)
• 作者单位：1. University of Michigan, Ann Arbor, MI 48109, USA2. School of Mathematics and Computational Science, Sun Yat-sen University, Guangzhou, 510275 China
• ISSN：1573-2916

文摘

We view regularized learning of a function in a Banach space from its finite samples as an optimization problem. Within the framework of reproducing kernel Banach spaces, we prove the representer theorem for the minimizer of regularized learning schemes with a general loss function and a nondecreasing regularizer. When the loss function and the regularizer are differentiable, a characterization equation for the minimizer is also established.