逐层数据再表达的前后端融合学习的理论及其模型和算法
|
摘要
基于学习的两个主要研究内容,本文提出了学习的二元分层模式,给出了前端学习、后端学习、前后端组合学习和前后端融合学习的概念,构建了前后端融合学习的理论框架与最优化模型;针对前端学习,模拟大脑的分级工作机制,提出了数据与模型混合驱动的逐层数据再表达的方法;最后,以视觉(图像)学习为例,本文给出了一种数据与模型混合驱动的逐层数据再表达的具体方法.
Based on two research contents of machine learning, a two-element layered model of machine learning is proposed. In addition, the concepts of front-end learning, back-end learning, a combination of front-end and back-end learning, and the fusion of front-end and back-end leaning are presented. Specifically, a framework and optimization model for the fusion of front-end and back-end learning is constructed. For front-end learning, which is a simulated hierarchical working mechanism of the brain, we present a layer-by-layer data re-representation method, which is driven by both data and a model. In addition, we propose a specific implementation of the data re-representation method for visual learning.
Based on two research contents of machine learning, a two-element layered model of machine learning is proposed. In addition, the concepts of front-end learning, back-end learning, a combination of front-end and back-end learning, and the fusion of front-end and back-end leaning are presented. Specifically, a framework and optimization model for the fusion of front-end and back-end learning is constructed. For front-end learning, which is a simulated hierarchical working mechanism of the brain, we present a layer-by-layer data re-representation method, which is driven by both data and a model. In addition, we propose a specific implementation of the data re-representation method for visual learning.
引文
1 Mitchell T M.Machine Learning.New York:McGraw-Hill Science,1997
2 Sabour S,Frosst N,Hinton G E.Dynamic routing between capsules.In:Proceedings of the 31st Conference on Neural Information Processing System,Long Beac,2017
3 Xie J B,Xing J L,Zhang L N,et al.20 Lectures on Visual Machine Learning.Beijing:Tsinghua University Press,2015[谢剑斌,兴军亮,张立宁,等.视觉机器学习20讲.北京:清华大学出版社,2015]
4 Wu W,Yang J.L1/2regularization methods for weights sparsification of neural networks.Sci Sin Math,2015,45:1487-1504[吴微,杨洁.用于神经网络权值稀疏化的L1/2正则化方法.中国科学数学,2015,45:1487-1504]
5 Liu J W,Liu Y,Luo X L.Semi-supervised learning methods.Chinese J Comput,2015,38:1592-1617[刘建伟,刘媛,罗雄麟.半监督学习方法.计算机学报,2015,38:1592-1617]
6 Ma L R,Song D D,Liao L J,et al.PSVM:a preference-enhanced SVM model using preference data for classification.Sci China Inf Sci,2017,60:122103
7 Deng C W,Huang G B,Xu J,et al.Extreme learning machines:new trends and applications.Sci China Inf Sci,2015,58:020301
8 Feng X D,He X M.Robust low-rank data matrix approximations.Sci China Math,2017,60:189-200
9 Hinton G E,Salakhutdinov R R.Reducing the dimensionality of data with neural networks.Science,2006,313:504-507
10 Qu W,Wang D L,Feng S,et al.A novel cross-modal hashing algorithm based on multimodal deep learning.Sci China Inf Sci,2017,60:092104
11 Gao W,Zhou Z H.Dropout rademacher complexity of deep neural networks.Sci China Inf Sci,2016,59:072104
12 Shao G Q,Wu Y P,Yong A,et al.Fingerprint compression based on sparse representation.IEEE Trans Image Process,2014,23:489-501
13 Shao G Q,Han C Y,Guo T D,et al.An NMF-based method for the fingerprint orientation field estimation.In:Proceedings of Computer and Information Science,Warsaw,2012.93-104
14 Li M Q,Han C Y,Guo T D.New gradient algorithms for optimization problems constrained by a cartesian product of unit balls.Acta Math Appl Sin,2018,41:43-54[李明强,韩丛英,郭田德.新的梯度算法求解单位球笛卡尔积约束优化问题.应用数学学报,2018,41:43-54]
15 Li M Q,Han C Y,Wang R X,et al.Shrinking gradient descent algorithms for total variation regularized image denoising.Comput Opt Appl,2017,68:643-660
16 Donoho D L.Compressed sensing.IEEE Trans Inf Theory,2006,52:1289-1306
17 Candes E J,Romberg J,Tao T.Robust uncertainty principles:exact signal reconstruction from highly incomplete frequency information.IEEE Trans Inf Theory,2006,52:489-509
18 Olshausen B A,Field D J.Emergence of simple-cell receptive field properties by learning a sparse code for natural images.Nature,1996,381:607-609
19 Olshausen B A,Field D J.Sparse coding with an overcomplete basis set:a strategy employed by V1?Vision Res,1997,37:3311-3325
20 Olshausen B A,Field D J.Natural image statistics and efficient coding.Netw-Comput Neural Syst,1996,7:333-339
21 Li M Q.Optimization theory and algorithms for image denoising and representation layer by layer.Dissertation for Ph.D.Degree.Beijing:University of Chinese Academy of Sciences,2017[李明强.图像去噪与逐层表达的优化理论与算法研究.博士学位论文.北京:中国科学院大学,2017]
22 Conte D,Foggia P,Sansone C,et al.Thirty years of graph matching in pattern recognition.Int J Pattern Recogn Artif Intel,2004,18:265-298
23 Bourgeois F,Lassalle J C.An extension of the munkres algorithm for the assignment problem to rectangular matrices.Commun ACM,1971,14:802-804
24 Birgin E G,Mario M J.Large-scale active-set box-constrained optimization method with spectral projected gradients.Comput Opt Appl,2002,23:101-125
25 Beck A,Teboulle M.Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.IEEE Trans Image Process,2009,18:2419-2434
1) 本文用列向量来表示灰度图像.
2) 本文中使用的矩阵零范数与 1 范数均表示将矩阵拉成向量对应的定义.
3) http://redwood.berkeley.edu/bruno/sparsepyr/.
2 Sabour S,Frosst N,Hinton G E.Dynamic routing between capsules.In:Proceedings of the 31st Conference on Neural Information Processing System,Long Beac,2017
3 Xie J B,Xing J L,Zhang L N,et al.20 Lectures on Visual Machine Learning.Beijing:Tsinghua University Press,2015[谢剑斌,兴军亮,张立宁,等.视觉机器学习20讲.北京:清华大学出版社,2015]
4 Wu W,Yang J.L1/2regularization methods for weights sparsification of neural networks.Sci Sin Math,2015,45:1487-1504[吴微,杨洁.用于神经网络权值稀疏化的L1/2正则化方法.中国科学数学,2015,45:1487-1504]
5 Liu J W,Liu Y,Luo X L.Semi-supervised learning methods.Chinese J Comput,2015,38:1592-1617[刘建伟,刘媛,罗雄麟.半监督学习方法.计算机学报,2015,38:1592-1617]
6 Ma L R,Song D D,Liao L J,et al.PSVM:a preference-enhanced SVM model using preference data for classification.Sci China Inf Sci,2017,60:122103
7 Deng C W,Huang G B,Xu J,et al.Extreme learning machines:new trends and applications.Sci China Inf Sci,2015,58:020301
8 Feng X D,He X M.Robust low-rank data matrix approximations.Sci China Math,2017,60:189-200
9 Hinton G E,Salakhutdinov R R.Reducing the dimensionality of data with neural networks.Science,2006,313:504-507
10 Qu W,Wang D L,Feng S,et al.A novel cross-modal hashing algorithm based on multimodal deep learning.Sci China Inf Sci,2017,60:092104
11 Gao W,Zhou Z H.Dropout rademacher complexity of deep neural networks.Sci China Inf Sci,2016,59:072104
12 Shao G Q,Wu Y P,Yong A,et al.Fingerprint compression based on sparse representation.IEEE Trans Image Process,2014,23:489-501
13 Shao G Q,Han C Y,Guo T D,et al.An NMF-based method for the fingerprint orientation field estimation.In:Proceedings of Computer and Information Science,Warsaw,2012.93-104
14 Li M Q,Han C Y,Guo T D.New gradient algorithms for optimization problems constrained by a cartesian product of unit balls.Acta Math Appl Sin,2018,41:43-54[李明强,韩丛英,郭田德.新的梯度算法求解单位球笛卡尔积约束优化问题.应用数学学报,2018,41:43-54]
15 Li M Q,Han C Y,Wang R X,et al.Shrinking gradient descent algorithms for total variation regularized image denoising.Comput Opt Appl,2017,68:643-660
16 Donoho D L.Compressed sensing.IEEE Trans Inf Theory,2006,52:1289-1306
17 Candes E J,Romberg J,Tao T.Robust uncertainty principles:exact signal reconstruction from highly incomplete frequency information.IEEE Trans Inf Theory,2006,52:489-509
18 Olshausen B A,Field D J.Emergence of simple-cell receptive field properties by learning a sparse code for natural images.Nature,1996,381:607-609
19 Olshausen B A,Field D J.Sparse coding with an overcomplete basis set:a strategy employed by V1?Vision Res,1997,37:3311-3325
20 Olshausen B A,Field D J.Natural image statistics and efficient coding.Netw-Comput Neural Syst,1996,7:333-339
21 Li M Q.Optimization theory and algorithms for image denoising and representation layer by layer.Dissertation for Ph.D.Degree.Beijing:University of Chinese Academy of Sciences,2017[李明强.图像去噪与逐层表达的优化理论与算法研究.博士学位论文.北京:中国科学院大学,2017]
22 Conte D,Foggia P,Sansone C,et al.Thirty years of graph matching in pattern recognition.Int J Pattern Recogn Artif Intel,2004,18:265-298
23 Bourgeois F,Lassalle J C.An extension of the munkres algorithm for the assignment problem to rectangular matrices.Commun ACM,1971,14:802-804
24 Birgin E G,Mario M J.Large-scale active-set box-constrained optimization method with spectral projected gradients.Comput Opt Appl,2002,23:101-125
25 Beck A,Teboulle M.Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.IEEE Trans Image Process,2009,18:2419-2434
1) 本文用列向量来表示灰度图像.
2) 本文中使用的矩阵零范数与 1 范数均表示将矩阵拉成向量对应的定义.
3) http://redwood.berkeley.edu/bruno/sparsepyr/.