基于深度高斯过程的多元类别数据分布估计
|
摘要
多元类别数据的可能取值会随向量长度的增长呈指数级增长,从而造成数据稀疏性问题。通过将观察数据嵌入到连续空间中训练识别数据之间的相似性,构建多元类别数据的线性高斯隐变量模型和类别隐高斯过程(CLGP)。在CLGP模型基础上,建立小样本多元类别数据分布估计的多元类别深度隐高斯过程模型,并结合蒙特卡洛采样的变分推断方法对模型进行参数优化。实验结果表明,与CLGP模型相比,该模型分布估计精确度有所提升。
The possible value of multivariate categorical data increases exponentially with the length of the vector,resulting in data sparsity. The similarity between the identified data is trained by embedding the observation data into the continuous space,and the linear Gaussian hidden variable model and the Categorical Latent Gaussian Process(CLGP) of the multi-category data are constructed. Based on the CLGP model,a multi-class deep hidden Gaussian process model for small sample multi-class data distribution estimation is proposed,and the parameters are optimized by Monte Carlo sampling. Experimental results show that compared with the CLGP model,this model distribution estimation accuracy has improved.
The possible value of multivariate categorical data increases exponentially with the length of the vector,resulting in data sparsity. The similarity between the identified data is trained by embedding the observation data into the continuous space,and the linear Gaussian hidden variable model and the Categorical Latent Gaussian Process(CLGP) of the multi-category data are constructed. Based on the CLGP model,a multi-class deep hidden Gaussian process model for small sample multi-class data distribution estimation is proposed,and the parameters are optimized by Monte Carlo sampling. Experimental results show that compared with the CLGP model,this model distribution estimation accuracy has improved.
引文
[1]王盛玉,曾碧卿,胡翩翩.基于卷积神经网络参数优化的中文情感分析[J].计算机工程,2017,43(8):200-207,214.
[2]BENGIO Y,SCHWENK H,SENCAL J S,et al.Neural probabilistic language models[M].In Innovations in Machine Learning.Berlin,Germany:Springer,2006:137-186.
[3]COLLOBERT R,WESTON J.A unified architecture for natural language processing:deep neural networks with multitask learning[C]//Proceedings of the 25th International Conference on Machine learning.New York,USA:ACMPress,2008:160-167.
[4]GAL Y,CHEN Y,ZOUBIN G.Latent Gaussian process for distribution estimation of multivariate categorical data[EB/OL].[2017-11-18].https://arxiv.org/pdf/1503.02182.pdf
[5]KHAN M E,MOHAMED S,MARLIN B R,et al.A stickbreaking likelihood for categorical data analysis with latent Gaussian models[EB/OL].[2017-11-18].https://www.shakirm.com/papers/catLGM-AIstats 20 12.pdf.
[6]何志昆,刘光斌,赵曦晶,等.高斯过程回归方法综述[J].控制与决策,2013,28(8):1121-1129,1137.
[7]RASMUSSEN C E,WILLIAM K I.Gaussian process for machine learning[EB/OL].[2017-11-18].http://www.gaussianprocess.org/gpml/.
[8]KO J,FOX D.GP-Bayes filters:Bayesian filtering using Gaussian process prediction and observation models[C]//Proceedings of IEEE/RSJ Intelligent Robots and Systems.Washington D.C.,USA:IEEE Press,2008:3471-3476.
[9]DEISENROTH M P,RASMUSSEN C E.PILCO:a modelbased and data-efficient approach to policy search[C]//Proceedings of the 28th International Conference on Machine Learning.[S.l.]:Omnipress,2011:465-472.
[10]CRESSIE N,WIKLE K.Statistics for spatio-temporal data[EB/OL].[2017-11-18].https://www.wiley.com/en-us/Statistics+for+Spatio+Temporal+Datap-9780471692744.
[11]王鑫,李红丽.台风最大风速预测的高斯过程回归模型[J].计算机应用研究,2015,32(1):59-62.
[12]BRIOL F X,OATES C J,GIROLAMI M,et al.Probabilistic integration:a role for statisticians in numerical analysis?[EB/OL].[2017-11-18].https://arxiv.org/pdf/1512.00933v5.pdf.
[13]GUESTRIN C,KRAUSE A,SINGH A P.Near-optimal sensor placements in Gaussian processes[C]//Proceedings of the 22nd International Conference on Machine Learning.New York,USA:ACM Press,2005:265-272.
[14]孙晓燕,陈姗姗,巩敦卫,等.基于区间适应值交互式遗传算法的加权多输出高斯过程代理模型[J].自动化学报,2014,40(2):172-184.
[15]SNOEK J,LAROCHELLE H,ADAMS R P.Practical bayesian optimization of machine learning algorithms[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems[S.l.]:Curran Associates Inc.,2012:2951-2959.
[16]REZENDE D J,MOHAMED S,WIERSTRA D.Stochastic back propagation and approximate inference in deep generative models[J].Pattern Recognition and Machine Learning,2014,32(2):1278-1286.
[17]GHAHRAMANI Z.Probabilistic machine learning and artificial intelligence[J].Nature,2015,521:452-459.
[18]DAMIANOU A C,LAWRENCE N D.Deep Gaussian processes[EB/OL].[2017-11-18].https://core.ac.uk/download/pdf/46564399.pdf.
[19]WILSON A G,HU Z,SALAKHUTDINOV R,et al.Deep kernel learning[EB/OL].[2017-11-18].https://arxiv.org/pdf/1511.02222.pdf.
[20]DURRANDE N,GINSBOURGER D,ROUSTANT O.Additive kernels for Gaussian process modeling[EB/OL].[2017-11-18].https://arxiv.org/pdf/1103.4023.pdf.
[21]DAVID D,JAMES R L,ROGER G,et al.Structure discovery in nonparametric regression through compositional kernel search[EB/OL].[2017-11-18].http://www.cs.toronto.edu/~rgrosse/icml2013-gp.pdf.
[22]HENSMAN J,LAWRENCE N D.Nested variational compression in deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1412.1370.pdf.
[23]VAFA K.Training deep Gaussian processes with sampling[EB/OL].[2017-11-18].http://approxi mateinference.org/accepted/Vafa2016.pdf.
[24]WANG Y,BRUBAKER M,CHAIB-DRAA B,et al.Sequential inference for deep Gaussian process[EB/OL].[2017-11-18].http://proceedings.mlr.press/v51/wang16c.pdf.
[25]BUI T D,HERNNDEZ-LOBATO D,LI Y,et al.Deep Gaussian processes for regression using approximate expectation propagation[EB/OL].[2017-11-18].http://proceedings.mlr.press/v48/bui16.pdf.
[26]DAI Z,DAMIANOU A,GONZLEZ J,et al.Variational autoencoded deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1511.06455.pdf.
[27]DAMIANOU A D,TITSIAS M K,LAWRENCE N D.Variational Gaussian process dynamical systems[EB/OL].[2017-11-18].http://papers.nips.cc/paper/4330-variationalgaussianprocess-dynamical-systems.pdf.
[28]CUTAJAR K,BONILLA E V,MICHIARDI P,et al.Practical learning of deep Gaussian processes via random fourier features[EB/OL].[2017-11-18].https://pdfs.semanticscholar.org/bafa/7e2d586e7bfe77d9a55ac1cff4eb2f6ff292.pdf.
[29]HUGH S,MARC D.Doubly stochastic variational inference for deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1705.08933.pdf.
[30]TITSIAS M K.Variational learning of inducing variables in sparse Gaussian processes[EB/OL].[2017-11-18].http://proceedings.mlr.press/v5/titsias09a/titsias09a.pdf.
[31]BLEI D M,KUCUKELBIR A,MCAULIFFE J D.Variational inference:a review for statisticians[EB/OL].[2017-11-18].https://arxiv.org/pdf/1601.00670.pdf.
[32]JOHN D,ELAD H,YORAM S.Adaptive subgradient methods for online learning and stochastic optimization[J].Journal of Machine Learning Research,2011,12:2121-2159.
[33]TIELEMAN T,HINTON G.Lecture 6.5-rmsprop,COURSERA:neural networks for machine learning[EB/OL].[2017-11-18].http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
[34]TOM S,IOANNIS A,DAVID S.Unit tests for stochastic optimization[EB/OL].[2017-11-18].https://arxiv.org/pdf/1312.6055.pdf.
[35]Perplexity[G/OL].[2017-11-18].https://en.wiki pedia.org/wiki/Perplexity.
[36]Tensorflow[EB/OL].[2017-11-18].https://tenso rflow.google.cn/.
[37]MATTHEWS A G D G,MARK V D W,NICKSON T,et al.GPflow:a gaussian process library using tensorflow[EB/OL].[2017-11-18].http://adsabs.harvard.edu/abs/2016arXiv161008733M.
[38]Handwritten digits[DB/OL].[2017-11-18].https://cs.nyu.edu/~roweis/data.html.
[39]MATJAZ Z,MILAN S.Breast cancer data set[DB/OL].[2017-11-18].http://archive.ics.uci.edu/ml/datasets/Breast+Cancer.
[2]BENGIO Y,SCHWENK H,SENCAL J S,et al.Neural probabilistic language models[M].In Innovations in Machine Learning.Berlin,Germany:Springer,2006:137-186.
[3]COLLOBERT R,WESTON J.A unified architecture for natural language processing:deep neural networks with multitask learning[C]//Proceedings of the 25th International Conference on Machine learning.New York,USA:ACMPress,2008:160-167.
[4]GAL Y,CHEN Y,ZOUBIN G.Latent Gaussian process for distribution estimation of multivariate categorical data[EB/OL].[2017-11-18].https://arxiv.org/pdf/1503.02182.pdf
[5]KHAN M E,MOHAMED S,MARLIN B R,et al.A stickbreaking likelihood for categorical data analysis with latent Gaussian models[EB/OL].[2017-11-18].https://www.shakirm.com/papers/catLGM-AIstats 20 12.pdf.
[6]何志昆,刘光斌,赵曦晶,等.高斯过程回归方法综述[J].控制与决策,2013,28(8):1121-1129,1137.
[7]RASMUSSEN C E,WILLIAM K I.Gaussian process for machine learning[EB/OL].[2017-11-18].http://www.gaussianprocess.org/gpml/.
[8]KO J,FOX D.GP-Bayes filters:Bayesian filtering using Gaussian process prediction and observation models[C]//Proceedings of IEEE/RSJ Intelligent Robots and Systems.Washington D.C.,USA:IEEE Press,2008:3471-3476.
[9]DEISENROTH M P,RASMUSSEN C E.PILCO:a modelbased and data-efficient approach to policy search[C]//Proceedings of the 28th International Conference on Machine Learning.[S.l.]:Omnipress,2011:465-472.
[10]CRESSIE N,WIKLE K.Statistics for spatio-temporal data[EB/OL].[2017-11-18].https://www.wiley.com/en-us/Statistics+for+Spatio+Temporal+Datap-9780471692744.
[11]王鑫,李红丽.台风最大风速预测的高斯过程回归模型[J].计算机应用研究,2015,32(1):59-62.
[12]BRIOL F X,OATES C J,GIROLAMI M,et al.Probabilistic integration:a role for statisticians in numerical analysis?[EB/OL].[2017-11-18].https://arxiv.org/pdf/1512.00933v5.pdf.
[13]GUESTRIN C,KRAUSE A,SINGH A P.Near-optimal sensor placements in Gaussian processes[C]//Proceedings of the 22nd International Conference on Machine Learning.New York,USA:ACM Press,2005:265-272.
[14]孙晓燕,陈姗姗,巩敦卫,等.基于区间适应值交互式遗传算法的加权多输出高斯过程代理模型[J].自动化学报,2014,40(2):172-184.
[15]SNOEK J,LAROCHELLE H,ADAMS R P.Practical bayesian optimization of machine learning algorithms[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems[S.l.]:Curran Associates Inc.,2012:2951-2959.
[16]REZENDE D J,MOHAMED S,WIERSTRA D.Stochastic back propagation and approximate inference in deep generative models[J].Pattern Recognition and Machine Learning,2014,32(2):1278-1286.
[17]GHAHRAMANI Z.Probabilistic machine learning and artificial intelligence[J].Nature,2015,521:452-459.
[18]DAMIANOU A C,LAWRENCE N D.Deep Gaussian processes[EB/OL].[2017-11-18].https://core.ac.uk/download/pdf/46564399.pdf.
[19]WILSON A G,HU Z,SALAKHUTDINOV R,et al.Deep kernel learning[EB/OL].[2017-11-18].https://arxiv.org/pdf/1511.02222.pdf.
[20]DURRANDE N,GINSBOURGER D,ROUSTANT O.Additive kernels for Gaussian process modeling[EB/OL].[2017-11-18].https://arxiv.org/pdf/1103.4023.pdf.
[21]DAVID D,JAMES R L,ROGER G,et al.Structure discovery in nonparametric regression through compositional kernel search[EB/OL].[2017-11-18].http://www.cs.toronto.edu/~rgrosse/icml2013-gp.pdf.
[22]HENSMAN J,LAWRENCE N D.Nested variational compression in deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1412.1370.pdf.
[23]VAFA K.Training deep Gaussian processes with sampling[EB/OL].[2017-11-18].http://approxi mateinference.org/accepted/Vafa2016.pdf.
[24]WANG Y,BRUBAKER M,CHAIB-DRAA B,et al.Sequential inference for deep Gaussian process[EB/OL].[2017-11-18].http://proceedings.mlr.press/v51/wang16c.pdf.
[25]BUI T D,HERNNDEZ-LOBATO D,LI Y,et al.Deep Gaussian processes for regression using approximate expectation propagation[EB/OL].[2017-11-18].http://proceedings.mlr.press/v48/bui16.pdf.
[26]DAI Z,DAMIANOU A,GONZLEZ J,et al.Variational autoencoded deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1511.06455.pdf.
[27]DAMIANOU A D,TITSIAS M K,LAWRENCE N D.Variational Gaussian process dynamical systems[EB/OL].[2017-11-18].http://papers.nips.cc/paper/4330-variationalgaussianprocess-dynamical-systems.pdf.
[28]CUTAJAR K,BONILLA E V,MICHIARDI P,et al.Practical learning of deep Gaussian processes via random fourier features[EB/OL].[2017-11-18].https://pdfs.semanticscholar.org/bafa/7e2d586e7bfe77d9a55ac1cff4eb2f6ff292.pdf.
[29]HUGH S,MARC D.Doubly stochastic variational inference for deep Gaussian processes[EB/OL].[2017-11-18].https://arxiv.org/pdf/1705.08933.pdf.
[30]TITSIAS M K.Variational learning of inducing variables in sparse Gaussian processes[EB/OL].[2017-11-18].http://proceedings.mlr.press/v5/titsias09a/titsias09a.pdf.
[31]BLEI D M,KUCUKELBIR A,MCAULIFFE J D.Variational inference:a review for statisticians[EB/OL].[2017-11-18].https://arxiv.org/pdf/1601.00670.pdf.
[32]JOHN D,ELAD H,YORAM S.Adaptive subgradient methods for online learning and stochastic optimization[J].Journal of Machine Learning Research,2011,12:2121-2159.
[33]TIELEMAN T,HINTON G.Lecture 6.5-rmsprop,COURSERA:neural networks for machine learning[EB/OL].[2017-11-18].http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
[34]TOM S,IOANNIS A,DAVID S.Unit tests for stochastic optimization[EB/OL].[2017-11-18].https://arxiv.org/pdf/1312.6055.pdf.
[35]Perplexity[G/OL].[2017-11-18].https://en.wiki pedia.org/wiki/Perplexity.
[36]Tensorflow[EB/OL].[2017-11-18].https://tenso rflow.google.cn/.
[37]MATTHEWS A G D G,MARK V D W,NICKSON T,et al.GPflow:a gaussian process library using tensorflow[EB/OL].[2017-11-18].http://adsabs.harvard.edu/abs/2016arXiv161008733M.
[38]Handwritten digits[DB/OL].[2017-11-18].https://cs.nyu.edu/~roweis/data.html.
[39]MATJAZ Z,MILAN S.Breast cancer data set[DB/OL].[2017-11-18].http://archive.ics.uci.edu/ml/datasets/Breast+Cancer.