生成对抗网络图像处理综述
|
摘要
2014年提出的生成对抗网络(Generative Adversarial Networks,GAN)是近年来神经网络领域中为数不多的一项新锐技术。GAN在常见生成模型的基础上增加了一个判别模型,以形成巧妙的对抗学习机制,使它能够产生更高质量的图像。近年来各种改进型GAN在图像处理领域得到广泛应用,不但覆盖了几乎所有传统图像处理领域,还包括一些新应用,如图像编辑、图像翻译、风格转移等,普遍取得了胜过传统方法的良好结果。文中在简要分析GAN的系统结构、对抗生成和网络训练的基础上,重点介绍了为提高GAN性能、克服现存缺陷和满足不同应用而出现的多种改进型GAN,如DC-GAN、W-GAN、Big-GAN等。尽管如此,目前GAN尚处于初始发展阶段,将来的前途不可估量。
Generative adversarial networks(GAN)proposed in 2014 is one cutting-edge technique of several deep learning technologies in the neural networks in the past few years.A discriminative model is added to familiar generative model by GAN to form an adversarial competitive strategy in subtle way in order to produce higher quality images.Recently,many improved GANs are widely applied in image processing,not only covering nearly all applications of traditional image processing,but also including other new applications,such as image editing,image translation,style transfer,etc,and have gotten better performance than that the traditional methods got.Based on brief analysis on GAN system architecture,adversarial generating and networks training,this paper stresses on introducing several schemes of improved GANs,such as DC-GAN,W-GAN,Big-GAN,and so on,whose target is to promote GAN's performance,overcome GAN's defects,and adapt the environment requirements for deferent applications.Nowadays,GAN techniques are still in initial stage of vigorous development,and have a bright future for us.
Generative adversarial networks(GAN)proposed in 2014 is one cutting-edge technique of several deep learning technologies in the neural networks in the past few years.A discriminative model is added to familiar generative model by GAN to form an adversarial competitive strategy in subtle way in order to produce higher quality images.Recently,many improved GANs are widely applied in image processing,not only covering nearly all applications of traditional image processing,but also including other new applications,such as image editing,image translation,style transfer,etc,and have gotten better performance than that the traditional methods got.Based on brief analysis on GAN system architecture,adversarial generating and networks training,this paper stresses on introducing several schemes of improved GANs,such as DC-GAN,W-GAN,Big-GAN,and so on,whose target is to promote GAN's performance,overcome GAN's defects,and adapt the environment requirements for deferent applications.Nowadays,GAN techniques are still in initial stage of vigorous development,and have a bright future for us.
引文
[1] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Advances Neural Information Processing Systems Conference.2014:2672-2680.
[2] MIRZA M,OSINDERO S.Conditional generative adversarial nets[EB/OL].[2019-03-16].https://arxiv.org/pdf/1411.1784.pdf.
[3] CHEN X,DUAN Y,HOUTHOOFT R,et al.InfoGAN:interpretable representation learning by information maximizing generative adversarial nets[C]//Advances Neural Information Processing Systems Conference.2016:2172-2180.
[4] RADFORD A,METZ L,CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[C]//International Conference on Learning Representations Workshop Track.2016.
[5] ARJOVSKY M,BOTTOU L.Towards principled methods for training generative adversarial networks[C]//Neural Information Processing Systems Conference.2016.
[6] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein GAN[C]//34th International Conference on Machine Learning.2017:214-223.
[7] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved training of Wasserstein GANS[C]//Proceedings of Advances Neural Information Processing Systems Conference.2017.
[8] NOWOZIN S,CSEKE B,TOMIOKA R.F-GAN:training generative neural samplers using variational divergence minimization[C]//Advances Neural Information Processing Systems Conference.2016:271-279.
[9] BROCK A,DONAHUE J,SIMONYAN J.Large scale GAN training for high fidelity natural image synthesis[C]//International Conference on Learning Representations (ICLR).2019.
[10] MESCHEDER L,GEIGER A,NOWOZIN S.Which training methods for GANs do actually converge?[C]//International Conference on Machine Learning (ICML).2018.
[11] ZHAO J,MATHIEU M,LECUN Y.Energy-based generative adversarial network[C]//International Conference on Learning Representations (ICLR).2017.
[12] LEDIG C,THEIS L,HUSZáR F,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2017:4681-4690.
[13] WANG X,YU K,WU S,et al.ESRGAN:enhanced super-resolution generative adversarial networks[C]//European Conference on Computer Vision (ECCV Workshops).2018.
[14] JOLICOEUR-MARTINEAU A.The relativistic discriminator:a key element missing from standard GAN[EB/OL].[2019-03-21].https://arxiv.org/abs/1807.00734.
[15] ZHU Lin,CHEN Yushi ,GHAMISI P ,et al.Generative adversarial networks for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2018,56(9):5046-5063.
[16] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training GANS[C]//Advances Neural Information Processing Systems Conference.2016:2226-2234.
[17] SHRIVASTAVA A,PFISTER T,TUZEL O,et al.Learning from simulated and unsupervised images through adversarial training[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2016:2107-2116.
[18] ZHANG M,MA K T,LIM J H,et al.Deep future gaze:gaze anticipation on egocentric videos using adversarial networks[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2017:4372-4381.
[19] BOUSMALIS K,SIBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2016:3722-3731.
[20] DENTON E L,CHINTALA S,SZLAM A.Deep generative image models using a Laplacian pyramid of adversarial networks[C]//Advances Neural Information Processing Systems Conference.2015:1486-1494.
[21] REED S,AKATA Z,YAN X,et al.Generative adversarial text to image synthesis[C]//33rd International Conference on Machine Learning (ICML).2016.
[22] ZHU J Y,KR?HENBüHL P,SHECHTMAN E,et al.Generative visual manipulation on the natural image manifold[C]//European Conference on Computer Vision.2016:597-613.
[23] BROCK A,LIM T,RITCHIE J M,et al.Neural photo editing with introspective adversarial networks[C]//6th International Conference on Learning Representations (ICLR).2017:24-26.
[24] LI Chengcheng,WANG Zi,QI Hairong.Fast-converging conditional generative adversarial networks for image synthesis[C]//IEEE International Conference on Image Processing (ICIP).2018:2132-2136.
[25] ISOLA P,ZHU J Y,ZHOU T,et al.Image to image translation with conditional adversarial networks[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2017:5967-5976.
[26] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle consistent adversarial networks[C]//IEEE International Conference on Computer Vision (ICCV).2017:2242-2251.
[27] LI Yijun,LIU Sifei,YANG Jimei,et al.Generative face completion[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2017:5892-5900.
[28] LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//IEEE International Conference on Computer Vision (ICCV).2015:3730-3738.
[29] LIANG Xiaodan,LEE Lisa,DAI Wei,et al.Dual motion GAN for future-flow embedded video prediction[C]//IEEE International Conference on Computer Vision (ICCV).2017:1762-1770.
[30] YU Tingzhao,WANG Lingfeng,GUA Huxiang,et al.Deep generative video prediction[J].Pattern Recognition Letters,2018,110:58-65.
[31] CHEN Jingwen,CHEN Jiawei,CHAO Hongyang,et al.Image blind denoising with generative adversarial network based noise modeling[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2018:3155-3164.
[32] CHENG Zhengxue,SUN Heming,TAKEUCHI M,et al.Performance comparison of convolutional auto encoders,generative adversarial networks and super-resolution for image compression[C]//IEEE Conference on Computer Vision and Patter Recognition(CVPR).2018:2613-2616.
[33] SOULY N,SPAMPINATO C,SHAH M.Semi supervised semantic segmentation using generative adversarial network[C]//IEEE International Conference on Computer Vision (ICCV).2017:5689-5697.
[34] WU Xian,XU Kun,HALL P.A srvey of image synthesis and editing with Generative adversarial networks[J].Tsinghua Science and Technology,2017,22(6):660-674.
[35] PATRICIA L S,SAPPA A D,VINTIMILLA B X,et al.Deep learning based single image dehazing[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).2018:1250-12507.
[36] ZHU Junyan,ZHANG Zhoutong,ZHANG Chengkai,et al.Visual object networks:image generation with disentangled 3D representation[C]//32nd Conference on Neural Information Processing Systems (NeurIPS).2018.
[37] WU J,ZHANG C,XUE T,et al.Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling[C]//Advances Neural Information Processing Systems Conference.2016:82-90.
[38] LEE J D,SIMCHOWITZ M,JORDAN M I,et al.Gradient descent only converges to minimizers[C]//Proceedings of Conference on Learning Theory.2016:1246-1257.
[39] THEIS L,OORD A,BETHGE M.A note on the evaluation of generative models[C]//International Conference on Learning Representations (ICLR).2018.
[40] ZHOU Bolei,KHOSLA A,LAPEDRIZA A,et al.Learning deep features for discriminative localization[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:2921-2929.
[41] LEI N,LUO Z,YAU S T,et al.Geometric understanding of deep learning[EB/OL].[2019-03-22].https://arxiv.org/pdf/1805.10451.pdf
[42] LEI Na,SU Kehua,CUI Li,et al.A geometric view of optimal transportation and generative model[J].Computer Aided Geometric Design,2019,68:1-21.
[43] LUCIC M,KURACH K,MICHALSKI M,et al.Are GANs created equal a large-scale study[C]//32nd Conference on Neural Information Processing Systems (NIPS).2018.
[44] CRESWELL A,WHITE T,DUMOULIN V,et al.Generative adversarial networks:an overview[J].IEEE Signal Processing Magazine,2018,35(1):53-65.
[2] MIRZA M,OSINDERO S.Conditional generative adversarial nets[EB/OL].[2019-03-16].https://arxiv.org/pdf/1411.1784.pdf.
[3] CHEN X,DUAN Y,HOUTHOOFT R,et al.InfoGAN:interpretable representation learning by information maximizing generative adversarial nets[C]//Advances Neural Information Processing Systems Conference.2016:2172-2180.
[4] RADFORD A,METZ L,CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[C]//International Conference on Learning Representations Workshop Track.2016.
[5] ARJOVSKY M,BOTTOU L.Towards principled methods for training generative adversarial networks[C]//Neural Information Processing Systems Conference.2016.
[6] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein GAN[C]//34th International Conference on Machine Learning.2017:214-223.
[7] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved training of Wasserstein GANS[C]//Proceedings of Advances Neural Information Processing Systems Conference.2017.
[8] NOWOZIN S,CSEKE B,TOMIOKA R.F-GAN:training generative neural samplers using variational divergence minimization[C]//Advances Neural Information Processing Systems Conference.2016:271-279.
[9] BROCK A,DONAHUE J,SIMONYAN J.Large scale GAN training for high fidelity natural image synthesis[C]//International Conference on Learning Representations (ICLR).2019.
[10] MESCHEDER L,GEIGER A,NOWOZIN S.Which training methods for GANs do actually converge?[C]//International Conference on Machine Learning (ICML).2018.
[11] ZHAO J,MATHIEU M,LECUN Y.Energy-based generative adversarial network[C]//International Conference on Learning Representations (ICLR).2017.
[12] LEDIG C,THEIS L,HUSZáR F,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2017:4681-4690.
[13] WANG X,YU K,WU S,et al.ESRGAN:enhanced super-resolution generative adversarial networks[C]//European Conference on Computer Vision (ECCV Workshops).2018.
[14] JOLICOEUR-MARTINEAU A.The relativistic discriminator:a key element missing from standard GAN[EB/OL].[2019-03-21].https://arxiv.org/abs/1807.00734.
[15] ZHU Lin,CHEN Yushi ,GHAMISI P ,et al.Generative adversarial networks for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2018,56(9):5046-5063.
[16] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training GANS[C]//Advances Neural Information Processing Systems Conference.2016:2226-2234.
[17] SHRIVASTAVA A,PFISTER T,TUZEL O,et al.Learning from simulated and unsupervised images through adversarial training[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2016:2107-2116.
[18] ZHANG M,MA K T,LIM J H,et al.Deep future gaze:gaze anticipation on egocentric videos using adversarial networks[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2017:4372-4381.
[19] BOUSMALIS K,SIBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//IEEE Conference on Computer Vision Pattern Recognition (CVPR).2016:3722-3731.
[20] DENTON E L,CHINTALA S,SZLAM A.Deep generative image models using a Laplacian pyramid of adversarial networks[C]//Advances Neural Information Processing Systems Conference.2015:1486-1494.
[21] REED S,AKATA Z,YAN X,et al.Generative adversarial text to image synthesis[C]//33rd International Conference on Machine Learning (ICML).2016.
[22] ZHU J Y,KR?HENBüHL P,SHECHTMAN E,et al.Generative visual manipulation on the natural image manifold[C]//European Conference on Computer Vision.2016:597-613.
[23] BROCK A,LIM T,RITCHIE J M,et al.Neural photo editing with introspective adversarial networks[C]//6th International Conference on Learning Representations (ICLR).2017:24-26.
[24] LI Chengcheng,WANG Zi,QI Hairong.Fast-converging conditional generative adversarial networks for image synthesis[C]//IEEE International Conference on Image Processing (ICIP).2018:2132-2136.
[25] ISOLA P,ZHU J Y,ZHOU T,et al.Image to image translation with conditional adversarial networks[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2017:5967-5976.
[26] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle consistent adversarial networks[C]//IEEE International Conference on Computer Vision (ICCV).2017:2242-2251.
[27] LI Yijun,LIU Sifei,YANG Jimei,et al.Generative face completion[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2017:5892-5900.
[28] LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//IEEE International Conference on Computer Vision (ICCV).2015:3730-3738.
[29] LIANG Xiaodan,LEE Lisa,DAI Wei,et al.Dual motion GAN for future-flow embedded video prediction[C]//IEEE International Conference on Computer Vision (ICCV).2017:1762-1770.
[30] YU Tingzhao,WANG Lingfeng,GUA Huxiang,et al.Deep generative video prediction[J].Pattern Recognition Letters,2018,110:58-65.
[31] CHEN Jingwen,CHEN Jiawei,CHAO Hongyang,et al.Image blind denoising with generative adversarial network based noise modeling[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2018:3155-3164.
[32] CHENG Zhengxue,SUN Heming,TAKEUCHI M,et al.Performance comparison of convolutional auto encoders,generative adversarial networks and super-resolution for image compression[C]//IEEE Conference on Computer Vision and Patter Recognition(CVPR).2018:2613-2616.
[33] SOULY N,SPAMPINATO C,SHAH M.Semi supervised semantic segmentation using generative adversarial network[C]//IEEE International Conference on Computer Vision (ICCV).2017:5689-5697.
[34] WU Xian,XU Kun,HALL P.A srvey of image synthesis and editing with Generative adversarial networks[J].Tsinghua Science and Technology,2017,22(6):660-674.
[35] PATRICIA L S,SAPPA A D,VINTIMILLA B X,et al.Deep learning based single image dehazing[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).2018:1250-12507.
[36] ZHU Junyan,ZHANG Zhoutong,ZHANG Chengkai,et al.Visual object networks:image generation with disentangled 3D representation[C]//32nd Conference on Neural Information Processing Systems (NeurIPS).2018.
[37] WU J,ZHANG C,XUE T,et al.Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling[C]//Advances Neural Information Processing Systems Conference.2016:82-90.
[38] LEE J D,SIMCHOWITZ M,JORDAN M I,et al.Gradient descent only converges to minimizers[C]//Proceedings of Conference on Learning Theory.2016:1246-1257.
[39] THEIS L,OORD A,BETHGE M.A note on the evaluation of generative models[C]//International Conference on Learning Representations (ICLR).2018.
[40] ZHOU Bolei,KHOSLA A,LAPEDRIZA A,et al.Learning deep features for discriminative localization[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:2921-2929.
[41] LEI N,LUO Z,YAU S T,et al.Geometric understanding of deep learning[EB/OL].[2019-03-22].https://arxiv.org/pdf/1805.10451.pdf
[42] LEI Na,SU Kehua,CUI Li,et al.A geometric view of optimal transportation and generative model[J].Computer Aided Geometric Design,2019,68:1-21.
[43] LUCIC M,KURACH K,MICHALSKI M,et al.Are GANs created equal a large-scale study[C]//32nd Conference on Neural Information Processing Systems (NIPS).2018.
[44] CRESWELL A,WHITE T,DUMOULIN V,et al.Generative adversarial networks:an overview[J].IEEE Signal Processing Magazine,2018,35(1):53-65.