用于域适应的多边缘降噪自动编码器
|
摘要
神经网络模型被广泛用于跨领域分类学习。边缘堆叠降噪自动编码器(marginalized stacked denoising autoencoders,mSDA)作为一种神经网络模型,通过对源领域和目标领域数据进行边缘化加噪损坏,学习一个公共的、健壮的特征表示空间,从而解决领域适应问题。然而,mSDA对所有的特征都采取相同的边缘化加噪处理方式,没有考虑到不同特征对分类结果的影响不同。为此,对特征进行区分性的噪音系数干扰,提出多边缘降噪自动编码器(multi-marginalized denoising autoencoders,M-MDA)。首先,利用改进的权重似然率(weighted log-likelihood ratio update,WLLRU)区分出领域间的共享和特有特征;然后,通过计算特征在两个领域的距离,对共享特征和特有特征进行不同方式的边缘化降噪处理,并基于单层边缘降噪自动编码器(marginalized denoising autoencoders,MDA)学习获取更健壮的特征;最后,对新的特征空间进行二次损坏以强化共享特征的比例。实验结果表明,该方法在跨领域情感分类方面优于基线算法。
Neural network models are used to address domain adaptation. As a model of neural network, marginalized stacked denoising autoencoders(mSDA) can extract and encode more robust feature space. mSDA tends to learn a common and robust feature representation to solve the problem of domain adaptation by marginalizing corruption with noise to the source and target domain data. However, mSDA uses the same marginalized and denoising method to corrupt all features. But in fact, features have different effects on the classification. This paper tries to corrupt the different features with a variant noise, and proposes the approach named multi-marginalized denoising autoencoders(M-MDA) for domain adaptation. Firstly, a polarity index WLLRU(weighted log-likelihood ratio update) which is improved from weight likelihood ratio, is proposed to distinguish the shared features from specific features. Then,the shared features and specific features are corrupted with different noises, and the noise is computed according to the distance of features between the source and target domain. And then marginalized denoising autoencoders(MDA) is used to learn a more robust feature space with the corrupted data. Lastly, the new feature space is corrupted again to enhance the proportion of shared features. The experimental results show that the proposed method outperforms state-of-the-art methods in cross-domain sentiment classification.
Neural network models are used to address domain adaptation. As a model of neural network, marginalized stacked denoising autoencoders(mSDA) can extract and encode more robust feature space. mSDA tends to learn a common and robust feature representation to solve the problem of domain adaptation by marginalizing corruption with noise to the source and target domain data. However, mSDA uses the same marginalized and denoising method to corrupt all features. But in fact, features have different effects on the classification. This paper tries to corrupt the different features with a variant noise, and proposes the approach named multi-marginalized denoising autoencoders(M-MDA) for domain adaptation. Firstly, a polarity index WLLRU(weighted log-likelihood ratio update) which is improved from weight likelihood ratio, is proposed to distinguish the shared features from specific features. Then,the shared features and specific features are corrupted with different noises, and the noise is computed according to the distance of features between the source and target domain. And then marginalized denoising autoencoders(MDA) is used to learn a more robust feature space with the corrupted data. Lastly, the new feature space is corrupted again to enhance the proportion of shared features. The experimental results show that the proposed method outperforms state-of-the-art methods in cross-domain sentiment classification.
引文
[1]Dai W Y,Xue G R,Yang Q,et al.Co-clustering based classification for out-of-domain documents[C]//Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,San Jose,Aug 12-15,2007.New York:ACM,2007:210-219.
[2]DauméIII H,Marcu D.Domain adaptation for statistical classifiers[J].Journal of Artificial Intelligence Research,2006,26:101-126.
[3]Zhuang F Z,Luo P,He Q,et al.Survey on transfer learning research[J].Journal of Software,2015,26(1):26-39.
[4]Xue G R,Dai W Y,Yang Q,et al.Topic-bridged PLSA for cross-domain text classification[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,Singapore,Jul20-24,2008.New York:ACM,2008:627-634.
[5]Ponomareva N,Thelwall M.Do neighbours help?An exploration of graph-based algorithms for cross-domain sentiment classification[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,Jeju Island,Jul 12-14,2012.Stroudsburg:ACL,2012:655-665.
[6]Blitzer J,McDonald R,Pereira F.Domain adaptation with structural correspondence learning[C]//Proceedings of the2006 Conference on Empirical Methods in Natural Language Processing,Sydney,Jul 22-23,2006.Stroudsburg:ACL,2006:120-128.
[7]Blitzer J,Dredze M,Pereira F.Biographies,bollywood,boom-boxes and blenders:domain adaptation for sentiment classification[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics,Prague,Jun 23-30,2007.Stroudsburg:ACL,2007:440-447.
[8]Pan S J,Ni X C,Sun J T,et al.Cross-domain sentiment classification via spectral feature alignment[C]//Proceedings of the 19th International Conference on World Wide Web,Raleigh,Apr 26-30,2010.New York:ACM,2010:751-760.
[9]Zhang Y H,Hu X G,Li P P,et al.Cross-domain sentiment classification-feature divergence,polarity divergence or both?[J].Pattern Recognition Letters,2015,65(C):44-50.
[10]Gao S,Li H Z.A cross-domain adaptation method for sentiment classification using probabilistic latent analysis[C]//Proceedings of the 20th ACM Conference on Information and Knowledge Management,Glasgow,Oct 24-28,2011.New York:ACM,2011:1047-1052.
[11]Socher R,Perelygin A,Wu J,et al.Recursive deep models for semantic compositionality over a sentiment treebank[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,Grand Hyatt Seattle,Oct 18-21,2013.Stroudsburg:ACL,2013:1631-1642.
[12]Dong L,Wei F R,Tan C Q,et al.Adaptive recursive neural network for target-dependent twitter sentiment classification[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,Baltimore,Jun 22-27,2014.Stroudsburg:ACL,2014:49-54.
[13]Kim Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,Doha,Oct 25-29,2014.Stroudsburg:ACL,2014:1746-1751.
[14]Tang D Y,Qin B,Liu T.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,Lisbon,Sep 17-21,2015.Stroudsburg:ACL,2015:1422-1432.
[15]Glorot X,Bordes A,Bengio Y.Domain adaptation for largescale sentiment classification:a deep learning approach[C]//Proceedings of the 28th International Conference on Machine Learning,Bellevue,Jun 28-Jul 2,2011.Madison:Omnipress,2011:513-520.
[16]Chen M M,Xu Z E,Weinberger K Q,et al.Marginalized denoising autoencoders for domain adaptation[C]//Proceedings of the 29th International Conference on Machine Learning,Edinburgh,Jun 26-Jul 1,2012.Madison:Omnipress,2012:1-8.
[17]Clinchant S,Csurka G,Chidlovskii B.A domain adaptation regularization for denoising autoencoders[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,Berlin,Aug 12,2016.Stroudsburg:ACL,2016:26-31.
[18]Ganin Y,Lempitsky V.Unsupervised domain adaptation by backpropagation[J].arXiv:1409.7495,2014.
[19]Hofmann T.Probabilistic latent semantic analysis[C]//Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence,Stockholm,Jul 30-Aug 1,1999.San Mateo:Morgan Kaufmann,1999:289-296.
[20]Li L H,Jin X M,Long M S.Topic correlation analysis for cross-domain text classification[C]//Proceedings of the 26th AAAI Conference on Artificial Intelligence,Toronto,Jul22-26,2012.Menlo Park:AAAI,2012:998-1004.
[21]Long M S,Wang J M,Ding G G,et al.Transfer learning with graph co-regularization[J].IEEE Transactions on Knowledge and Data Engineering,2014,26(7):1805-1818.
[22]Zhuang F Z,Luo P,Shen Z Y,et al.Collaborative dualPLSA:mining distinction and commonality across multiple domains for text classification[C]//Proceedings of the 19th ACM Conference on Information and Knowledge Management,Toronto,Oct 26-30,2010.New York:ACM,2010:359-368.
[23]Zhuang F Z,Luo P,Du C Y,et al.Triplex transfer learning:exploiting both shared and distinct concepts for text classification[J].IEEE Transactions on Cybernetics,2014,44(7):1191-1203.
[24]Pan J H,Hu X G,Zhang Y H,et al.Quadruple transfer learning:exploiting both shared and non-shared concepts for text classification[J].Knowledge-Based Systems,2015,90:199-210.
[25]Yang Y,Eisenstein J.Fast easy unsupervised domain adaptation with marginalized structured dropout[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,Baltimore,Jun 22-27,2014.Stroudsburg:ACL,2014:538-544.
[26]Ziser Y,Reichart R.Neural structural correspondence learning for domain adaptation[J].arXiv:1610.01588,2016.
[27]Jiang W H,Gao H C,Chung F,et al.The l2,1-norm stacked robust autoencoders for domain adaptation[C]//Proceedings of the 13th AAAI Conference on Artificial Intelligence,Phoenix,Feb 12-17,2016.Menlo Park:AAAI,2016:1723-1729.
[28]Louizos C,Swersky K,Li Y,et al.The variational fair autoencoder[J].arXiv:1511.00830,2015.
[3]庄福振,罗平,何清,等.迁移学习研究进展[J].软件学报,2015,26(1):26-39.
[2]DauméIII H,Marcu D.Domain adaptation for statistical classifiers[J].Journal of Artificial Intelligence Research,2006,26:101-126.
[3]Zhuang F Z,Luo P,He Q,et al.Survey on transfer learning research[J].Journal of Software,2015,26(1):26-39.
[4]Xue G R,Dai W Y,Yang Q,et al.Topic-bridged PLSA for cross-domain text classification[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,Singapore,Jul20-24,2008.New York:ACM,2008:627-634.
[5]Ponomareva N,Thelwall M.Do neighbours help?An exploration of graph-based algorithms for cross-domain sentiment classification[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,Jeju Island,Jul 12-14,2012.Stroudsburg:ACL,2012:655-665.
[6]Blitzer J,McDonald R,Pereira F.Domain adaptation with structural correspondence learning[C]//Proceedings of the2006 Conference on Empirical Methods in Natural Language Processing,Sydney,Jul 22-23,2006.Stroudsburg:ACL,2006:120-128.
[7]Blitzer J,Dredze M,Pereira F.Biographies,bollywood,boom-boxes and blenders:domain adaptation for sentiment classification[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics,Prague,Jun 23-30,2007.Stroudsburg:ACL,2007:440-447.
[8]Pan S J,Ni X C,Sun J T,et al.Cross-domain sentiment classification via spectral feature alignment[C]//Proceedings of the 19th International Conference on World Wide Web,Raleigh,Apr 26-30,2010.New York:ACM,2010:751-760.
[9]Zhang Y H,Hu X G,Li P P,et al.Cross-domain sentiment classification-feature divergence,polarity divergence or both?[J].Pattern Recognition Letters,2015,65(C):44-50.
[10]Gao S,Li H Z.A cross-domain adaptation method for sentiment classification using probabilistic latent analysis[C]//Proceedings of the 20th ACM Conference on Information and Knowledge Management,Glasgow,Oct 24-28,2011.New York:ACM,2011:1047-1052.
[11]Socher R,Perelygin A,Wu J,et al.Recursive deep models for semantic compositionality over a sentiment treebank[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,Grand Hyatt Seattle,Oct 18-21,2013.Stroudsburg:ACL,2013:1631-1642.
[12]Dong L,Wei F R,Tan C Q,et al.Adaptive recursive neural network for target-dependent twitter sentiment classification[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,Baltimore,Jun 22-27,2014.Stroudsburg:ACL,2014:49-54.
[13]Kim Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,Doha,Oct 25-29,2014.Stroudsburg:ACL,2014:1746-1751.
[14]Tang D Y,Qin B,Liu T.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,Lisbon,Sep 17-21,2015.Stroudsburg:ACL,2015:1422-1432.
[15]Glorot X,Bordes A,Bengio Y.Domain adaptation for largescale sentiment classification:a deep learning approach[C]//Proceedings of the 28th International Conference on Machine Learning,Bellevue,Jun 28-Jul 2,2011.Madison:Omnipress,2011:513-520.
[16]Chen M M,Xu Z E,Weinberger K Q,et al.Marginalized denoising autoencoders for domain adaptation[C]//Proceedings of the 29th International Conference on Machine Learning,Edinburgh,Jun 26-Jul 1,2012.Madison:Omnipress,2012:1-8.
[17]Clinchant S,Csurka G,Chidlovskii B.A domain adaptation regularization for denoising autoencoders[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,Berlin,Aug 12,2016.Stroudsburg:ACL,2016:26-31.
[18]Ganin Y,Lempitsky V.Unsupervised domain adaptation by backpropagation[J].arXiv:1409.7495,2014.
[19]Hofmann T.Probabilistic latent semantic analysis[C]//Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence,Stockholm,Jul 30-Aug 1,1999.San Mateo:Morgan Kaufmann,1999:289-296.
[20]Li L H,Jin X M,Long M S.Topic correlation analysis for cross-domain text classification[C]//Proceedings of the 26th AAAI Conference on Artificial Intelligence,Toronto,Jul22-26,2012.Menlo Park:AAAI,2012:998-1004.
[21]Long M S,Wang J M,Ding G G,et al.Transfer learning with graph co-regularization[J].IEEE Transactions on Knowledge and Data Engineering,2014,26(7):1805-1818.
[22]Zhuang F Z,Luo P,Shen Z Y,et al.Collaborative dualPLSA:mining distinction and commonality across multiple domains for text classification[C]//Proceedings of the 19th ACM Conference on Information and Knowledge Management,Toronto,Oct 26-30,2010.New York:ACM,2010:359-368.
[23]Zhuang F Z,Luo P,Du C Y,et al.Triplex transfer learning:exploiting both shared and distinct concepts for text classification[J].IEEE Transactions on Cybernetics,2014,44(7):1191-1203.
[24]Pan J H,Hu X G,Zhang Y H,et al.Quadruple transfer learning:exploiting both shared and non-shared concepts for text classification[J].Knowledge-Based Systems,2015,90:199-210.
[25]Yang Y,Eisenstein J.Fast easy unsupervised domain adaptation with marginalized structured dropout[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,Baltimore,Jun 22-27,2014.Stroudsburg:ACL,2014:538-544.
[26]Ziser Y,Reichart R.Neural structural correspondence learning for domain adaptation[J].arXiv:1610.01588,2016.
[27]Jiang W H,Gao H C,Chung F,et al.The l2,1-norm stacked robust autoencoders for domain adaptation[C]//Proceedings of the 13th AAAI Conference on Artificial Intelligence,Phoenix,Feb 12-17,2016.Menlo Park:AAAI,2016:1723-1729.
[28]Louizos C,Swersky K,Li Y,et al.The variational fair autoencoder[J].arXiv:1511.00830,2015.
[3]庄福振,罗平,何清,等.迁移学习研究进展[J].软件学报,2015,26(1):26-39.