聚类辅助特征对齐的域适应方法
|
摘要
有监督深度学习在有大量标记数据的领域可以取得不错的效果,但实际上很多领域只有大量未标记的数据。如何利用大量无标记数据,成为了深度学习发展的一个关键问题,领域自适应就是解决这一问题的一种有效方法。目前,基于对抗训练的域适应方法取得了较好的效果,这类方法利用领域分类损失对源域和目标域的特征分布进行对齐,降低了两个领域特征表示的分布差异,使采用源域数据训练的模型可以应用在目标域数据上。现有的域适应方法是在适配后的特征上进行模型训练的,没有充分利用目标域数据的原始信息,当两个领域差异较大时,会降低目标域特征的域内可鉴别性。针对现有方法的弱点,文中基于对抗判别域适应方法(ADDA),提出了一种基于对目标域数据聚类辅助特征对齐的域适应方法(CAFA-DA)。该方法通过聚类获得目标域数据伪标记,并在域适应阶段约束特征编码器训练,利用目标域数据的原始信息提高目标域特征的可鉴别性;将聚类和域适应两个过程训练的分类器进行集成学习,用高置信度样本进行训练,以提升模型的最终效果。CAFA-DA可用在任何基于对抗损失的领域自适应方法上。最后,在领域自适应的4个标准数据集上将CAFA-DA方法与目前几种先进的领域自适应方法进行了对比实验,结果表明:CAFA-DA方法的实验结果比其他几种方法都好;相对于ADDA方法,该方法在2个对比实验上的准确率分别提升了3.2%和17.2%。
Supervised deep learning can reach good results in the areas with large amounts of labeled data,but the rea-lity is that there are only a lot of unlabeled data in many areas.How to take advantages of large amounts of unlabeled data has become a key issue in the development of deep learning.Domain adaptation is an effective way to solve this problem.At present,domain adaptation methods based on adversarial training have achieved a good effect.This method uses domain classification loss to align the feature distribution of source domain,and target domain and reduce the difference of distribution between the feature representations of two domain,so the classifier trained with source domain data can be applied to target domain data.The existing domain adaptation method trains the model on the features after domain adaptation and does not make full use of the original information of the target domain data.When the differences between two domains are large,the intra-domain discriminability of target domain features will be reduced.In view of the disadvantages of the present methods,this paper proposed a method for clustering target domain data to assist feature alignment(CAFA-DA) based on the adversarial discriminative domain adaptation(ADDA).Pseudo-labels of target domain data are obtained by clustering and the feature encoder training is constrained in the domain adaptation stage,and the original information of the target domain data is used to improve the discriminability of target domain features.Classifiers trained in the two processes of clustering and domain adaptation are used for ensemble learning and high confidence samples are trained to improve the final effect of the model.The CAFA-DA can be applied to any domain adaption method based on adversarial loss.Finally,this paper compared CAFA-DA with several advanced domain adaption methods on four standard domain adaption data sets.The results show that the accuracy of the CAFA-DA method is better than other methods.Compared with the ADDA method,the results of two comparative experiments are improved by 3.2% and 17.2% respectively.
Supervised deep learning can reach good results in the areas with large amounts of labeled data,but the rea-lity is that there are only a lot of unlabeled data in many areas.How to take advantages of large amounts of unlabeled data has become a key issue in the development of deep learning.Domain adaptation is an effective way to solve this problem.At present,domain adaptation methods based on adversarial training have achieved a good effect.This method uses domain classification loss to align the feature distribution of source domain,and target domain and reduce the difference of distribution between the feature representations of two domain,so the classifier trained with source domain data can be applied to target domain data.The existing domain adaptation method trains the model on the features after domain adaptation and does not make full use of the original information of the target domain data.When the differences between two domains are large,the intra-domain discriminability of target domain features will be reduced.In view of the disadvantages of the present methods,this paper proposed a method for clustering target domain data to assist feature alignment(CAFA-DA) based on the adversarial discriminative domain adaptation(ADDA).Pseudo-labels of target domain data are obtained by clustering and the feature encoder training is constrained in the domain adaptation stage,and the original information of the target domain data is used to improve the discriminability of target domain features.Classifiers trained in the two processes of clustering and domain adaptation are used for ensemble learning and high confidence samples are trained to improve the final effect of the model.The CAFA-DA can be applied to any domain adaption method based on adversarial loss.Finally,this paper compared CAFA-DA with several advanced domain adaption methods on four standard domain adaption data sets.The results show that the accuracy of the CAFA-DA method is better than other methods.Compared with the ADDA method,the results of two comparative experiments are improved by 3.2% and 17.2% respectively.
引文
[1] BEN-DAVID S,BLITZER J,CRAMMER K,et al.A theory of learning from different domains[J].Machine learning,2010,79(1-2):151-175.
[2] TZENG E,HOFFMAN J,SAENKO K,et al.Adversarial dis- criminative domain adaptation[C]//Computer Vision and Pattern Recognition (CVPR).IEEE,2017:4.
[3] TZENG E,HOFFMAN J,ZHANG N,et al.Deep Domain Confusion:Maximizing for Domain Invariance[J].arXiv:1412.3474,2014.
[4] LONG M,CAO Y,WANG J,et al.Learning transferable fea- tures with deep adaptation networks[J].arXiv:1502.02791,2015.
[5] LONG M,ZHU H,WANG J,et al.Deep transfer learning with joint adaptation networks[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70.2017:2208-2217.
[6] GRETTON A,BORGWARDT K M,RASCH M J,et al.A kernel two-sample test[J].Journal of Machine Learning Research,2012,13(1):723-773.
[7] GANIN Y,USTINOVA E,AJAKAN H,et al.Domain-adver- sarial training of neural networks[J].The Journal of Machine Learning Research,2016,17(1):2096-2030.
[8] BOUSMALIS K,TRIGEORGIS G,SILBERMAN N,et al.Domain separation networks[C]//Advances in Neural Information Processing Systems.IEEE,2016:343-351.
[9] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gene- rative adversarial nets[C]//Advances in Neural Information Processing Systems.IEEE,2014:2672-2680.
[10] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved training of wasserstein gans[C]//Advances in Neural Information Processing Systems.IEEE,2017:5767-5777.
[11] SHU R,BUI H H,NARUI H,et al.A DIRT-T Approach to Unsupervised Domain Adaptation[J].arXiv:1802.08735,2018.
[12] TAKERU M,SHIN-ICHI M,SHIN I,et al.Virtual Adversarial Training:A Regularization Method for Supervised and Semi-Supervised Learning[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018:1.
[13] SAITO K,USHIKU Y,HARADA T.Asymmetric tri-training for unsupervised domain adaptation[J].arXiv:1702.08400,2017.
[14] CHEN C,XIE W,XU T,et al.Progressive Feature Alignment for Unsupervised Domain Adaptation[J].arXiv:1811.08585,2018.
[15] KUMAR A,SATTIGERI P,WADHAWAN K,et al.Co-regularized Alignment for Unsupervised Domain Adaptation[C]//Advances in Neural Information Processing Systems.IEEE,2018:9367-9378.
[16] GRANDVALET Y,BENGIO Y.Semi-supervised learning by entropy minimization[C]//Advances in Neural Information Processing Systems.IEEE,2005:529-536.
[17] SHIMODAIRA H.Improving predictive inference under cova- riate shift by weighting the log-likelihood function[J].Journal of statistical planning and inference,2000,90(2):227-244.
[18] MANSOUR Y,MOHRI M,ROSTAMIZADEH A.Domain ada- ptation:Learning bounds and algorithms[J].arXiv:0902.3430,2009.
[19] CHEN X,DUAN Y,HOUTHOOFT R,et al.Infogan:Interpretable representation learning by information maximizing ge-nerative adversarial nets[C]//Advances in Neural Information Processing Systems.IEEE,2016:2172-2180.
[20] TZENG E,HOFFMAN J,DARRELL T,et al.Simultaneous deep transfer across domains and tasks[C]//Proceedings of the IEEE International Conference on Computer Vision.IEEE,2015:4068-4076.
[21] GANIN Y,LEMPITSKY V.Unsupervised domain adaptation by backpropagation[J].arXiv:1409.7495,2014.
[22] GHIFARY M,KLEIJN W B,ZHANG M,et al.Deep reconstruction-classification networks for unsupervised domain adaptation[C]//European Conference on Computer Vision.Cham:Springer,2016:597-613.
[23] LIU M Y,TUZEL O.Coupled generative adversarial networks[C]//Advances in Neural Information Processing Systems.IEEE,2016:469-477.
[24] BOUSMALIS K,SILBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2017:7.
[25] TAIGMAN Y,POLYAK A,WOLF L.Unsupervised cross-do- main image generation[J].arXiv:1611.02200,2016.
[26] CHAPELLE O,ZIEN A.Semi-Supervised Classification by Low Density Separation[C]//AISTATS.2005:57-64.
[27] DAI Z,YANG Z,YANG F,et al.Good semi-supervised learning that requires a bad gan[C]//Advances in Neural Information Processing Systems.IEEE,2017:6510-6520.
[28] ZHOU Z H,LI M.Tri-training:Exploiting unlabeled data using three classifiers[J].IEEE Transactions on knowledge and Data Engineering,2005,17(11):1529-1541.
[29] CARON M,BOJANOWSKI P,JOULIN A,et al.Deep Clustering for Unsupervised Learning of Visual Features[M]//Computer Vision-ECCV 2018.Cham:Springer,2018:139-156.
[30] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[C]//Advances in Neural Information Processing Systems.IEEE,2012:1097-1105.
[31] SRIVASTAVA N,HINTON G,KRIZHEVSKY A,et al.Dropout:a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research,2014,15(1):1929-1958.
[32] JOHNSON J,DOUZE M,JéGOU H.Billion-scale similarity search with gpus[J].arXiv:1702.08734,2017.
[33] LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[34] ARBELAEZ P,MAIRE M,FOWLKES C,et al.Contour detection and hierarchical image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(5):898-916.
[35] NETZER Y,WANG T,COATES A,et al.Reading digits in na- tural images with unsupervised feature learning[C]//NIPS Workshop on Deep Learning and Unsupervised Feature Lear-ning.2011:5.
[36] WANG Y,WANG Q,LV Q F,et al.An Improved K-means Algorithm Based on Initial Clustering Center Optimization and Weighted Between Dimension[J].Journal of Chongqing University of Technology(Natural Science),2013,27(4):77-80.(in Chinese)王越,王泉,吕奇峰,等. 基于初始聚类中心优化和维间加权的改进K-means算法[J]. 重庆理工大学学报(自然科学),2013,27(4):77-80.
[2] TZENG E,HOFFMAN J,SAENKO K,et al.Adversarial dis- criminative domain adaptation[C]//Computer Vision and Pattern Recognition (CVPR).IEEE,2017:4.
[3] TZENG E,HOFFMAN J,ZHANG N,et al.Deep Domain Confusion:Maximizing for Domain Invariance[J].arXiv:1412.3474,2014.
[4] LONG M,CAO Y,WANG J,et al.Learning transferable fea- tures with deep adaptation networks[J].arXiv:1502.02791,2015.
[5] LONG M,ZHU H,WANG J,et al.Deep transfer learning with joint adaptation networks[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70.2017:2208-2217.
[6] GRETTON A,BORGWARDT K M,RASCH M J,et al.A kernel two-sample test[J].Journal of Machine Learning Research,2012,13(1):723-773.
[7] GANIN Y,USTINOVA E,AJAKAN H,et al.Domain-adver- sarial training of neural networks[J].The Journal of Machine Learning Research,2016,17(1):2096-2030.
[8] BOUSMALIS K,TRIGEORGIS G,SILBERMAN N,et al.Domain separation networks[C]//Advances in Neural Information Processing Systems.IEEE,2016:343-351.
[9] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gene- rative adversarial nets[C]//Advances in Neural Information Processing Systems.IEEE,2014:2672-2680.
[10] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved training of wasserstein gans[C]//Advances in Neural Information Processing Systems.IEEE,2017:5767-5777.
[11] SHU R,BUI H H,NARUI H,et al.A DIRT-T Approach to Unsupervised Domain Adaptation[J].arXiv:1802.08735,2018.
[12] TAKERU M,SHIN-ICHI M,SHIN I,et al.Virtual Adversarial Training:A Regularization Method for Supervised and Semi-Supervised Learning[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018:1.
[13] SAITO K,USHIKU Y,HARADA T.Asymmetric tri-training for unsupervised domain adaptation[J].arXiv:1702.08400,2017.
[14] CHEN C,XIE W,XU T,et al.Progressive Feature Alignment for Unsupervised Domain Adaptation[J].arXiv:1811.08585,2018.
[15] KUMAR A,SATTIGERI P,WADHAWAN K,et al.Co-regularized Alignment for Unsupervised Domain Adaptation[C]//Advances in Neural Information Processing Systems.IEEE,2018:9367-9378.
[16] GRANDVALET Y,BENGIO Y.Semi-supervised learning by entropy minimization[C]//Advances in Neural Information Processing Systems.IEEE,2005:529-536.
[17] SHIMODAIRA H.Improving predictive inference under cova- riate shift by weighting the log-likelihood function[J].Journal of statistical planning and inference,2000,90(2):227-244.
[18] MANSOUR Y,MOHRI M,ROSTAMIZADEH A.Domain ada- ptation:Learning bounds and algorithms[J].arXiv:0902.3430,2009.
[19] CHEN X,DUAN Y,HOUTHOOFT R,et al.Infogan:Interpretable representation learning by information maximizing ge-nerative adversarial nets[C]//Advances in Neural Information Processing Systems.IEEE,2016:2172-2180.
[20] TZENG E,HOFFMAN J,DARRELL T,et al.Simultaneous deep transfer across domains and tasks[C]//Proceedings of the IEEE International Conference on Computer Vision.IEEE,2015:4068-4076.
[21] GANIN Y,LEMPITSKY V.Unsupervised domain adaptation by backpropagation[J].arXiv:1409.7495,2014.
[22] GHIFARY M,KLEIJN W B,ZHANG M,et al.Deep reconstruction-classification networks for unsupervised domain adaptation[C]//European Conference on Computer Vision.Cham:Springer,2016:597-613.
[23] LIU M Y,TUZEL O.Coupled generative adversarial networks[C]//Advances in Neural Information Processing Systems.IEEE,2016:469-477.
[24] BOUSMALIS K,SILBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2017:7.
[25] TAIGMAN Y,POLYAK A,WOLF L.Unsupervised cross-do- main image generation[J].arXiv:1611.02200,2016.
[26] CHAPELLE O,ZIEN A.Semi-Supervised Classification by Low Density Separation[C]//AISTATS.2005:57-64.
[27] DAI Z,YANG Z,YANG F,et al.Good semi-supervised learning that requires a bad gan[C]//Advances in Neural Information Processing Systems.IEEE,2017:6510-6520.
[28] ZHOU Z H,LI M.Tri-training:Exploiting unlabeled data using three classifiers[J].IEEE Transactions on knowledge and Data Engineering,2005,17(11):1529-1541.
[29] CARON M,BOJANOWSKI P,JOULIN A,et al.Deep Clustering for Unsupervised Learning of Visual Features[M]//Computer Vision-ECCV 2018.Cham:Springer,2018:139-156.
[30] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[C]//Advances in Neural Information Processing Systems.IEEE,2012:1097-1105.
[31] SRIVASTAVA N,HINTON G,KRIZHEVSKY A,et al.Dropout:a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research,2014,15(1):1929-1958.
[32] JOHNSON J,DOUZE M,JéGOU H.Billion-scale similarity search with gpus[J].arXiv:1702.08734,2017.
[33] LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[34] ARBELAEZ P,MAIRE M,FOWLKES C,et al.Contour detection and hierarchical image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(5):898-916.
[35] NETZER Y,WANG T,COATES A,et al.Reading digits in na- tural images with unsupervised feature learning[C]//NIPS Workshop on Deep Learning and Unsupervised Feature Lear-ning.2011:5.
[36] WANG Y,WANG Q,LV Q F,et al.An Improved K-means Algorithm Based on Initial Clustering Center Optimization and Weighted Between Dimension[J].Journal of Chongqing University of Technology(Natural Science),2013,27(4):77-80.(in Chinese)王越,王泉,吕奇峰,等. 基于初始聚类中心优化和维间加权的改进K-means算法[J]. 重庆理工大学学报(自然科学),2013,27(4):77-80.