基于条件生成对抗网络的漫画手绘图上色方法
|
摘要
提出一种基于条件生成对抗网络(CGAN)的漫画手绘图自动上色方法。实验中,采用U型结构的生成器,对网络模型使用L1进行约束,在生成器和判别器的对抗式训练中,模型不断学习并优化手绘图到对应彩色图像间的映射关系,最后使用训练得到的条件GAN网络模型对手绘图上色。实验表明,使用这种方法可以有效并且快速地对漫画手绘图上色,同时保持可观的视觉效果。
This paper proposed a method to color manga sketch in unsupervised based conditional generative adversarial network( CGAN). In the experiments,it adopted a generator with an U-Net structure,constrained the model with L1 term,in the adversarial training between the generator and the discriminator,model continuously learned and optimized the mapping from manga sketch to its corresponding colorful image. At last,GAN that generated model from training could be used to color manga sketch. Experiment results show to demonstrate the effectiveness of rapid colorization for manga sketch as well as the plausibility of visual effects.
This paper proposed a method to color manga sketch in unsupervised based conditional generative adversarial network( CGAN). In the experiments,it adopted a generator with an U-Net structure,constrained the model with L1 term,in the adversarial training between the generator and the discriminator,model continuously learned and optimized the mapping from manga sketch to its corresponding colorful image. At last,GAN that generated model from training could be used to color manga sketch. Experiment results show to demonstrate the effectiveness of rapid colorization for manga sketch as well as the plausibility of visual effects.
引文
[1] Zhang R,Isola P,Efros A A. Colorful image colorization[C]//Proc of European Conference on Computer Vision. Berlin:Springer International Publishing,2016:649-666.
[2] Levin A,Lischinski D,Weiss Y. Colorization using optimization[J].ACM Trans on Graphics,2004,23(3):689-694.
[3] Cao Yun,Zhou Zhiming,Zhang Weinan,et al. Unsupervised diverse colorization via generative adversarial networks[EB/OL].(2017-07-04). https://arxiv. org/abs/1702. 06674.
[4] Gatys L A,Ecker A S,Bethge M. A neural algorithm of artistic style[EB/OL].(2015-09-03). https://arxiv. org/abs/1508. 06576.
[5] Gatys L A,Ecker A S,Bethge M. Image style transfer using convolutional neural networks[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition. Washington DC:IEEE Computer Society,2016:2414-2423.
[6] Johnson J,Alahi A,Li Feifei. Perceptual losses for real-time style transfer and super-resolution[C]//Proc of European Conference on Computer Vision. Berlin:Springer,2016:694-711.
[7] Goodfellow I,Pouget-Abadie J,Mirza M,et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. 2014:2672-2680.
[8] Mirza M,Osindero S. Conditional generative adversarial nets[EB/OL].(2014-11-10). https://arxiv. org/abs/1411. 1784.
[9] Ronneberger O,Fischer P,Brox T. U-net:convolutional networks for biomedical image segmentation[C]//Proc of International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin:Springer,2015:234-241.
[10]Hinton G E,Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J]. Science,2006,313(5786):504-507.
[11]Li Chuan,Wand M. Precomputed real-time texture synthesis with Markovian generative adversarial networks[C]//Proc of European Conference on Computer Vision. Berlin:Springer International Publishing,2016:702-716.
[12]Pathak D,Krahenbuhl P,Donahue J,et al. Context encoders:feature learning by inpainting[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway,NJ:IEEE Press,2016:2536-2544.
[13] Ioffe S,Szegedy C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proc of International Conference on Machine Learning. 2015:448-456.
[14] Kingma D,Ba J. Adam:a method for stochastic optimization[EB/OL].(2017-01-31). https://arxiv. org/abs/1412. 6980.
[15]Ulyanov D,Vedaldi A,Lempitsky V. Instance normalization:the missing ingredient for fast stylization[EB/OL].(2017-11-07). https://arxiv. org/pdf/1607. 08022. pdf.
[2] Levin A,Lischinski D,Weiss Y. Colorization using optimization[J].ACM Trans on Graphics,2004,23(3):689-694.
[3] Cao Yun,Zhou Zhiming,Zhang Weinan,et al. Unsupervised diverse colorization via generative adversarial networks[EB/OL].(2017-07-04). https://arxiv. org/abs/1702. 06674.
[4] Gatys L A,Ecker A S,Bethge M. A neural algorithm of artistic style[EB/OL].(2015-09-03). https://arxiv. org/abs/1508. 06576.
[5] Gatys L A,Ecker A S,Bethge M. Image style transfer using convolutional neural networks[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition. Washington DC:IEEE Computer Society,2016:2414-2423.
[6] Johnson J,Alahi A,Li Feifei. Perceptual losses for real-time style transfer and super-resolution[C]//Proc of European Conference on Computer Vision. Berlin:Springer,2016:694-711.
[7] Goodfellow I,Pouget-Abadie J,Mirza M,et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. 2014:2672-2680.
[8] Mirza M,Osindero S. Conditional generative adversarial nets[EB/OL].(2014-11-10). https://arxiv. org/abs/1411. 1784.
[9] Ronneberger O,Fischer P,Brox T. U-net:convolutional networks for biomedical image segmentation[C]//Proc of International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin:Springer,2015:234-241.
[10]Hinton G E,Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J]. Science,2006,313(5786):504-507.
[11]Li Chuan,Wand M. Precomputed real-time texture synthesis with Markovian generative adversarial networks[C]//Proc of European Conference on Computer Vision. Berlin:Springer International Publishing,2016:702-716.
[12]Pathak D,Krahenbuhl P,Donahue J,et al. Context encoders:feature learning by inpainting[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway,NJ:IEEE Press,2016:2536-2544.
[13] Ioffe S,Szegedy C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proc of International Conference on Machine Learning. 2015:448-456.
[14] Kingma D,Ba J. Adam:a method for stochastic optimization[EB/OL].(2017-01-31). https://arxiv. org/abs/1412. 6980.
[15]Ulyanov D,Vedaldi A,Lempitsky V. Instance normalization:the missing ingredient for fast stylization[EB/OL].(2017-11-07). https://arxiv. org/pdf/1607. 08022. pdf.