基于角度的分类方法综述（英文）
|
摘要
统计分类问题经常出现在很多应用中,如人脸识别、欺诈检测和手写字符识别等。对有监督分类问题的统计方法进行综述。特别地,介绍基于角度的分类结构,将二分类与多分类问题纳入一个统一的框架中。讨论基于角度分类器的若干新变体,如稳健学习和加权学习。此外,还指出这些分类器关于Fisher相合性的若干理论结果。
Statistical classification problems are widely encountered in many applications, e.g., face recognition, fraud detection, and hand-written character recognition. In this article we make a comprehensive analysis on statistical methods for supervised classification problems. Specifically, we introduce the angle-based classification structure, which combines binary and multicategory problems in a unified framework. Several new variants of the angle-based classifiers are also discussed, such as robust learning and weighted learning. Furthermore, we show some theoretical results about Fisher consistency for these angle-based classifiers.
Statistical classification problems are widely encountered in many applications, e.g., face recognition, fraud detection, and hand-written character recognition. In this article we make a comprehensive analysis on statistical methods for supervised classification problems. Specifically, we introduce the angle-based classification structure, which combines binary and multicategory problems in a unified framework. Several new variants of the angle-based classifiers are also discussed, such as robust learning and weighted learning. Furthermore, we show some theoretical results about Fisher consistency for these angle-based classifiers.
引文
[1] Mitchell T M.Machine learning[M].New York:McGraw-Hill Inc,1997.
[2] Duda R O,Hart P E,Stork D G.Pattern classification[M].2nd ed.New York:Wiley-Interscience,2000.
[3] Bishop C M.Pattern recognition and machine learning[M].New York:Springer,2006.
[4] Hastie T,Tibshirani R,Friedman J.The elements of statistical learning:data mining,inference and prediction[M].2nd ed.New York:Springer,2009.
[5] Murphy K P.Machine learning:a probabilistic perspective[M].Cambridge,MA:MIT press,2012.
[6] Wahba G.Support vector machines,reproducing kernel Hilbert spaces and the randomized GACV[M]//Advances in Kernel Methods-Support Vector Learning:vol 6.Cambridge,MA:MIT Press,1999:69-88.
[7] Boser B E,Guyon I M,Vapnik V N.A training algorithm for optimal margin classifiers[C]//Proceedings of the fifth annual workshop on computational learning theory.New York:ACM,1992:144-152.
[8] Cortes C,Vapnik V.Support-vector networks[J].Machine Learning,1995,20 (3):273-297.
[9] Rosenblatt F.The perceptron,a perceiving and recognizing automaton project para[M].Buffalo,NY:Cornell Aeronautical Laboratory,1957.
[10] Cox D R.The regression analysis of binary sequences[J].Journal of the Royal Statistical Society.Series B,1958,20 (2):215-242.
[11] Walker S H,Duncan D B.Estimation of the probability of an event as a function of several independent variables[J].Biometrika,1967,54 (1):167-179.
[12] Freund Y,Schapire R E.A decision-theoretic generalization of on-line learning and an application to boosting[J].Journal of Computer and System Sciences,1997,55 (1):119-139.
[13] Suykens J A,Vandewalle J.Least squares support vector machine classifiers[J].Neural Processing Letters,1999,9 (3):293-300.
[14] Chang K W,Hsieh C J,Lin C J.Coordinate descent method for large-scale l2-loss linear support vector machines[J].Journal of Machine Learning Research,2008,9 (Jul):1 369-1 398.
[15] Liu Y,Zhang H H,Wu Y.Hard or soft classification Large-margin unified machines[J].Journal of the American Statistical Association,2011,106 (493):166-177.
[16] Marron J S,Todd M J,Ahn J.Distance-weighted discrimination[J].Journal of the American Statistical Association,2007,102 (480):1 267-1 271.
[17] Marron J S.Distance-weighted discrimination[J].Wiley Interdisciplinary Reviews:Computational Statistics,2015,7 (2):109-114.
[18] Tibshirani R.Regression shrinkage and selection via the lasso[J].Journal of the Royal Statistical Society.Series B,1996,58 (1):267-288.
[19] Liu Y,Zhang H H,Park C,et al.Support vector machines with adaptive Lq penalty[J].Computational Statistics & Data Analysis,2007,51 (12):6 380-6 394.
[20] Zou H,Hastie T.Regularization and variable selection via the elastic net[J].Journal of the Royal Statistical Society.Series B,2005,67 (2):301-320.
[21] Yuan M,Lin Y.Model selection and estimation in regression with grouped variables[J].Journal of the Royal Statistical Society.Series B,2006,68 (1):49-67.
[22] Fan J,Li R.Variable selection via nonconcave penalized likelihood and its oracle properties[J].Journal of the American Statistical Association,2001,96 (456):1 348-1 360.
[23] Zhang C H.Nearly unbiased variable selection under minimax concave penalty[J].The Annals of Statistics,2010,38 (2):894-942.
[24] Boyd S,Vandenberghe L.Convex optimization[M].New York:Cambridge University Press,2004.
[25] Dietterich T G,Bakiri G.Solving multiclass learning problems via error-correcting output codes[J].Journal of Artificial Intelligence Research,1995,2:263-286.
[26] Allwein E L,Schapire R E,Singer Y.Reducing multiclass to binary:a unifying approach for margin classifiers[J].Journal of Machine Learning Research,2001,1 (2):113-141.
[27] Lee Y,Lin Y,Wahba G.Multicategory support vector machines:Theory and application to the classification of microarray data and satellite radiance data[J].Journal of the American Statistical Association,2004,99 (465):67-81.
[28] Liu Y,Yuan M.Reinforced multicategory support vector machines[J].Journal of Computational and Graphical Statistics,2011,20 (4):901-919.
[29] Weston J,Watkins C.Support vector machines for multi-class pattern recognition[C]//Proc European Symposium on Artificial Neural Networks:vol 99.Bruges,Belgium:D-Facto public,1999:219-224.
[30] Crammer K,Singer Y.On the algorithmic implementation of multiclass kernel-based vector machines[J].Journal of Machine Learning Research,2002,2 (2):265-292.
[31] Tang Y,Zhang H H.Multiclass proximal support vector machines[J].Journal of Computational and Graphical Statistics,2006,15 (2):339-355.
[32] Park S Y,Liu Y,Liu D,et al.Multicategory composite least squares classifiers[J].Statistical Analysis & Data Mining,2010,3 (4):272-286.
[33] Zhang C,Liu Y.Multicategory large-margin unified machines[J].Journal of Machine Learning Research,2013,14 (1):1 349-1 386.
[34] Shen X,Tseng G C,Zhang X,et al.On ψ-learning[J].Journal of the American Statistical Association,2003,98 (463):724-734.
[35] Liu Y,Shen X.Multicategory ψ-learning[J].Journal of the American Statistical Association,2006,101 (474):500-509.
[36] Wu Y,Liu Y.On multicategory truncated hinge loss support vector machines[M]//Prediction and Discovery:AMS-IMS-SIAM Joint Summer Research Conference,Machine and Statistical Learning:volume 443.Snowbird,Utah:American Mathematical Society,2006:49-58.
[37] Wu Y,Liu Y.Robust truncated hinge loss support vector machines[J].Journal of the American Statistical Association,2007,102 (479):974-983.
[38] Wu Y,Liu Y.Adaptively weighted large margin classifiers[J].Journal of Computational and Graphical Statistics,2013,22 (2):416-432.
[39] Zhang C,Liu Y.Multicategory angle-based large-margin classification[J].Biometrika,2014,101 (3):625-640.
[40] Hill S I,Doucet A.A framework for kernel-based multi-category classification[J].Journal of Artificial Intelligence Research,2007,30:525-564.
[41] Lange K,Wu T.An MM algorithm for multicategory vertex discriminant analysis[J].Journal of Computational and Graphical Statistics,2008,17 (3):527-544.
[42] Saberian M J,Vasconcelos N.Multiclass boosting:theory and algorithms[C]//Advances in Neural Information Processing Systems:vol 24.Granada,Spain:Curran Associates,Inc,2011:2 124-2 132.
[43] Mroueh Y,Poggio T,Rosasco L,et al.Multiclass learning with simplex coding[C]//Advances in Neural Information Processing Systems:vol 25.Lake Tahoe,Nevada:Curran Associates,Inc,2012:2 789-2 797.
[44] Zhang C,Liu Y,Wang J,et al.Reinforced angle-based multicategory support vector machines[J].Journal of Computational and Graphical Statistics,2016,25 (3):806-825.
[45] Fu S,Zhang S,Liu Y.Adaptively weighted large-margin angle-based classifiers[J].Journal of Multivariate Analysis,2018,166:282 - 299.
[46] Zhang C,Pham M,Fu S,et al.Robust multicategory support vector machines using difference convex algorithm[J].Mathematical Programming,2017.doi:10.1007/s10107-017-1209-5.
[47] Vapnik V N.Statistical learning theory[M].New York:Wiley,1998.
[48] Zhang T.Statistical behavior and consistency of classification methods based on convex risk minimization[J].The Annals of Statistics,2004,32 (1):56-85.
[49] Zhang T.Statistical analysis of some multi-category large margin classification methods[J].Journal of Machine Learning Research,2004,5 (Oct):1 225-1 251.
[50] Bartlett P L,Jordan M I,McAuliffe J D.Convexity,classification,and risk bounds[J].Journal of the American Statistical Association,2006,101 (473):138-156.
[51] Liu Y.Fisher consistency of multicategory support vector machines[C]//Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics:volume 2.San Juan,Puerto Rico:PMLR,2007:291-298.
[52] Zou H,Zhu J,Hastie T.New multicategory boosting algorithms based on multicategory Fisher-consistent losses[J].The Annals of Applied Statistics,2008,2 (4):1 290.
[53] Zhang C,Lu X,Zhu Z,et al.REC:fast sparse regression-based multicategory classification[J].Statistics and Its Interface,2017,10 (2):175-185.
[54] Sun H,Craig B A,Zhang L.Angle-based multicategory distance-weighted SVM[J].Journal of Machine Learning Research,2017,18 (1):2 981-3 001.
① This part is from Fu’s recent paper “robust outcome weighted learning for optimal individualized treatment rules”.
② This part is from Zhang’s recent paper “targeted local angle-based multi-category support vector machine”.
[2] Duda R O,Hart P E,Stork D G.Pattern classification[M].2nd ed.New York:Wiley-Interscience,2000.
[3] Bishop C M.Pattern recognition and machine learning[M].New York:Springer,2006.
[4] Hastie T,Tibshirani R,Friedman J.The elements of statistical learning:data mining,inference and prediction[M].2nd ed.New York:Springer,2009.
[5] Murphy K P.Machine learning:a probabilistic perspective[M].Cambridge,MA:MIT press,2012.
[6] Wahba G.Support vector machines,reproducing kernel Hilbert spaces and the randomized GACV[M]//Advances in Kernel Methods-Support Vector Learning:vol 6.Cambridge,MA:MIT Press,1999:69-88.
[7] Boser B E,Guyon I M,Vapnik V N.A training algorithm for optimal margin classifiers[C]//Proceedings of the fifth annual workshop on computational learning theory.New York:ACM,1992:144-152.
[8] Cortes C,Vapnik V.Support-vector networks[J].Machine Learning,1995,20 (3):273-297.
[9] Rosenblatt F.The perceptron,a perceiving and recognizing automaton project para[M].Buffalo,NY:Cornell Aeronautical Laboratory,1957.
[10] Cox D R.The regression analysis of binary sequences[J].Journal of the Royal Statistical Society.Series B,1958,20 (2):215-242.
[11] Walker S H,Duncan D B.Estimation of the probability of an event as a function of several independent variables[J].Biometrika,1967,54 (1):167-179.
[12] Freund Y,Schapire R E.A decision-theoretic generalization of on-line learning and an application to boosting[J].Journal of Computer and System Sciences,1997,55 (1):119-139.
[13] Suykens J A,Vandewalle J.Least squares support vector machine classifiers[J].Neural Processing Letters,1999,9 (3):293-300.
[14] Chang K W,Hsieh C J,Lin C J.Coordinate descent method for large-scale l2-loss linear support vector machines[J].Journal of Machine Learning Research,2008,9 (Jul):1 369-1 398.
[15] Liu Y,Zhang H H,Wu Y.Hard or soft classification Large-margin unified machines[J].Journal of the American Statistical Association,2011,106 (493):166-177.
[16] Marron J S,Todd M J,Ahn J.Distance-weighted discrimination[J].Journal of the American Statistical Association,2007,102 (480):1 267-1 271.
[17] Marron J S.Distance-weighted discrimination[J].Wiley Interdisciplinary Reviews:Computational Statistics,2015,7 (2):109-114.
[18] Tibshirani R.Regression shrinkage and selection via the lasso[J].Journal of the Royal Statistical Society.Series B,1996,58 (1):267-288.
[19] Liu Y,Zhang H H,Park C,et al.Support vector machines with adaptive Lq penalty[J].Computational Statistics & Data Analysis,2007,51 (12):6 380-6 394.
[20] Zou H,Hastie T.Regularization and variable selection via the elastic net[J].Journal of the Royal Statistical Society.Series B,2005,67 (2):301-320.
[21] Yuan M,Lin Y.Model selection and estimation in regression with grouped variables[J].Journal of the Royal Statistical Society.Series B,2006,68 (1):49-67.
[22] Fan J,Li R.Variable selection via nonconcave penalized likelihood and its oracle properties[J].Journal of the American Statistical Association,2001,96 (456):1 348-1 360.
[23] Zhang C H.Nearly unbiased variable selection under minimax concave penalty[J].The Annals of Statistics,2010,38 (2):894-942.
[24] Boyd S,Vandenberghe L.Convex optimization[M].New York:Cambridge University Press,2004.
[25] Dietterich T G,Bakiri G.Solving multiclass learning problems via error-correcting output codes[J].Journal of Artificial Intelligence Research,1995,2:263-286.
[26] Allwein E L,Schapire R E,Singer Y.Reducing multiclass to binary:a unifying approach for margin classifiers[J].Journal of Machine Learning Research,2001,1 (2):113-141.
[27] Lee Y,Lin Y,Wahba G.Multicategory support vector machines:Theory and application to the classification of microarray data and satellite radiance data[J].Journal of the American Statistical Association,2004,99 (465):67-81.
[28] Liu Y,Yuan M.Reinforced multicategory support vector machines[J].Journal of Computational and Graphical Statistics,2011,20 (4):901-919.
[29] Weston J,Watkins C.Support vector machines for multi-class pattern recognition[C]//Proc European Symposium on Artificial Neural Networks:vol 99.Bruges,Belgium:D-Facto public,1999:219-224.
[30] Crammer K,Singer Y.On the algorithmic implementation of multiclass kernel-based vector machines[J].Journal of Machine Learning Research,2002,2 (2):265-292.
[31] Tang Y,Zhang H H.Multiclass proximal support vector machines[J].Journal of Computational and Graphical Statistics,2006,15 (2):339-355.
[32] Park S Y,Liu Y,Liu D,et al.Multicategory composite least squares classifiers[J].Statistical Analysis & Data Mining,2010,3 (4):272-286.
[33] Zhang C,Liu Y.Multicategory large-margin unified machines[J].Journal of Machine Learning Research,2013,14 (1):1 349-1 386.
[34] Shen X,Tseng G C,Zhang X,et al.On ψ-learning[J].Journal of the American Statistical Association,2003,98 (463):724-734.
[35] Liu Y,Shen X.Multicategory ψ-learning[J].Journal of the American Statistical Association,2006,101 (474):500-509.
[36] Wu Y,Liu Y.On multicategory truncated hinge loss support vector machines[M]//Prediction and Discovery:AMS-IMS-SIAM Joint Summer Research Conference,Machine and Statistical Learning:volume 443.Snowbird,Utah:American Mathematical Society,2006:49-58.
[37] Wu Y,Liu Y.Robust truncated hinge loss support vector machines[J].Journal of the American Statistical Association,2007,102 (479):974-983.
[38] Wu Y,Liu Y.Adaptively weighted large margin classifiers[J].Journal of Computational and Graphical Statistics,2013,22 (2):416-432.
[39] Zhang C,Liu Y.Multicategory angle-based large-margin classification[J].Biometrika,2014,101 (3):625-640.
[40] Hill S I,Doucet A.A framework for kernel-based multi-category classification[J].Journal of Artificial Intelligence Research,2007,30:525-564.
[41] Lange K,Wu T.An MM algorithm for multicategory vertex discriminant analysis[J].Journal of Computational and Graphical Statistics,2008,17 (3):527-544.
[42] Saberian M J,Vasconcelos N.Multiclass boosting:theory and algorithms[C]//Advances in Neural Information Processing Systems:vol 24.Granada,Spain:Curran Associates,Inc,2011:2 124-2 132.
[43] Mroueh Y,Poggio T,Rosasco L,et al.Multiclass learning with simplex coding[C]//Advances in Neural Information Processing Systems:vol 25.Lake Tahoe,Nevada:Curran Associates,Inc,2012:2 789-2 797.
[44] Zhang C,Liu Y,Wang J,et al.Reinforced angle-based multicategory support vector machines[J].Journal of Computational and Graphical Statistics,2016,25 (3):806-825.
[45] Fu S,Zhang S,Liu Y.Adaptively weighted large-margin angle-based classifiers[J].Journal of Multivariate Analysis,2018,166:282 - 299.
[46] Zhang C,Pham M,Fu S,et al.Robust multicategory support vector machines using difference convex algorithm[J].Mathematical Programming,2017.doi:10.1007/s10107-017-1209-5.
[47] Vapnik V N.Statistical learning theory[M].New York:Wiley,1998.
[48] Zhang T.Statistical behavior and consistency of classification methods based on convex risk minimization[J].The Annals of Statistics,2004,32 (1):56-85.
[49] Zhang T.Statistical analysis of some multi-category large margin classification methods[J].Journal of Machine Learning Research,2004,5 (Oct):1 225-1 251.
[50] Bartlett P L,Jordan M I,McAuliffe J D.Convexity,classification,and risk bounds[J].Journal of the American Statistical Association,2006,101 (473):138-156.
[51] Liu Y.Fisher consistency of multicategory support vector machines[C]//Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics:volume 2.San Juan,Puerto Rico:PMLR,2007:291-298.
[52] Zou H,Zhu J,Hastie T.New multicategory boosting algorithms based on multicategory Fisher-consistent losses[J].The Annals of Applied Statistics,2008,2 (4):1 290.
[53] Zhang C,Lu X,Zhu Z,et al.REC:fast sparse regression-based multicategory classification[J].Statistics and Its Interface,2017,10 (2):175-185.
[54] Sun H,Craig B A,Zhang L.Angle-based multicategory distance-weighted SVM[J].Journal of Machine Learning Research,2017,18 (1):2 981-3 001.
① This part is from Fu’s recent paper “robust outcome weighted learning for optimal individualized treatment rules”.
② This part is from Zhang’s recent paper “targeted local angle-based multi-category support vector machine”.