基于卷积神经网络的刺绣风格数字合成
|
摘要
针对刺绣风格数字化模拟方法立体感不强、缺少线条方向等问题,提出了一种基于深度学习和卷积神经网络的算法,将刺绣艺术风格传输到目标图像。利用图像语义分割网络及风格迁移网络,分别对目标内容图像与刺绣艺术风格图像进行目标提取和风格迁移。首先,输入目标内容图像与刺绣艺术风格图像,采用基于条件随机场的图像语义分割,将目标内容图与刺绣艺术风格图的前景与背景分离,并进行二值化处理,形成掩模图像;其次,将目标内容图与刺绣艺术风格图的RGB颜色空间转换为YIQ;最后,参照掩模图像使用VGG19网络模型提取目标内容图的内容特征及刺绣艺术风格图的风格纹理特征进行目标区域内的风格迁移,从而对刺绣艺术进行数字化模拟。该算法能模拟出具有刺绣艺术效果的结果图像,能更好地模拟真实刺绣艺术的线条方向,突出了线条的立体感。通过使用语义分割与风格迁移相结合的方法,有效模拟了色彩艳丽、立体感强的刺绣艺术风格图像,是对非真实感绘制的有效补充,为刺绣数字化保护与非物质文化传承奠定了基础。
To remedy the deficiency of the current digital embroidery algorithm in arousing the original linear and stereo perception, this paper proposes an artificial system based on deep-learning and convolutional neural network to synthetize the style of embroidery. First, we input the content image and the embroidery style image, perform image semantic segmentation which is based on conditional random field to separate the foreground and the background of both images and construct masks by image binarization. Then, we convert the color spaces of both input images from RGB into YIQ, extract the features of embroidery by VGG19 and transfer the content image into embroidery style in the foreground by using mask, meanwhile emphasizing the gorgeous colors and the stereoscopic textures of the embroidery.Experimental results show that the proposed method can enhance images with embroidery style effectively. It lays a foundation for digital inheritance of the traditional embroidery.
To remedy the deficiency of the current digital embroidery algorithm in arousing the original linear and stereo perception, this paper proposes an artificial system based on deep-learning and convolutional neural network to synthetize the style of embroidery. First, we input the content image and the embroidery style image, perform image semantic segmentation which is based on conditional random field to separate the foreground and the background of both images and construct masks by image binarization. Then, we convert the color spaces of both input images from RGB into YIQ, extract the features of embroidery by VGG19 and transfer the content image into embroidery style in the foreground by using mask, meanwhile emphasizing the gorgeous colors and the stereoscopic textures of the embroidery.Experimental results show that the proposed method can enhance images with embroidery style effectively. It lays a foundation for digital inheritance of the traditional embroidery.
引文
[1]WINKENBACH G,SALESIN D H.Computergenerated pen-and-ink illustration[C]//Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM,1994:91-100.
[2]LI Y J,FANG C,YANG J M,et al.Universal style transfer via feature transforms[C]//Advances in Neural Information Processing Systems.California:NIPS,2017:386-396.
[3]GUAY M,RONFARD R,GLEICHER M L.Space-time sketching of character animation[J].Acm Transactions on Graphics,2015,34(4):1-10.
[4]SHENG K,DONG W,KONG Y,et al.Evaluating the quality of face alignment without ground truth[J].Computer Graphics Forum,2015,34(7):213-223.
[5]钱文华,徐丹,官铮,等.粉笔画艺术风格模拟[J].中国图象图形学报,2017,22(5):622-630.QIAN W H,XU D,GUAN Z,et al.Simulating chalk art style painting[J].Journal of Image and Graphics,2017,22(5):620-630.
[6]QIAN W H,XU D,YUE K,et al.Gourd pyrography art simulating based on non-photorealistic rendering[J].Multimedia Tools&Applications,2017,76(13):14559-14579.
[7]杨丽洁,徐添辰,吴恩华.动态模拟中国水墨画中的笔画绘制[J].计算机辅助设计与图形学学报,2015,28(5):742-749.YANG L J,XU T C,WU E H.Animating strokes in drawing process of Chinese ink painting[J].Journal of Computer-Aided Design and Computer Graphics,2016,28(5):742-749.
[8]陈圣国,孙正兴,项建华,等.计算机辅助乱针绣制作技术研究[J].计算机学报,2011,34(3):526-538.CHEN S G,SUN Z X,XIANG J H,et al.Research on the technology of computer aided irregular needling embroidery[J].Chinese Journal of Computers,2011,34(3):526-538.
[9]周杰,孙正兴,杨克微,等.乱针绣模拟的参数化生成方法[J].计算机辅助设计与图形学学报,2014,26(3):436-444.ZHOU J,SUN Z X,YANG K W,et al.Parametric generation method for irregular needling embroidery rendering[J].Journal of Computer-Aided Design and Computer Graphics,2014,26(3):436-444.
[10]项建华,杨克微,周杰,等.一种基于图像分解的乱针绣模拟生成方法[J].图学学报,2013,34(4):16-23.XIANG J H,YANG K W,ZHOU J,et al.A novel image disintegration-based computerized embroidery method for random stitch[J].Journal of Graphics,2013,34(4):16-23.
[11]杨克微,孙正兴.物像空间依赖的乱针绣针法定义与风格生成[J].计算机辅助设计与图形学学报,2018,30(5):778-790.YANG K W,SUN Z X.Vector and pixel space depended stitch definition and style transfer for randomneedle embroidery[J].Journal of Computer-Aided Design&Computer Graphics,2018,30(5):778-790.
[12]张盼,梁昭华.传统刺绣与电脑刺绣的艺术特点比较[J].艺术与设计(理论版),2010(7):229-231.ZHANG P,LIANG Z H.Tradition embroidery comparing with computer embroidery art characteristic[J].Art and Design,2010(7):229-231.
[13]LECUN Y L,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[14]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems.Nevada,NIPS,2012:1097-1105.
[15]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[16]ZEILER M D,FERGUS R.Visualizing and understanding convolutional networks[C]//European Conference on Computer Vision.Zurich:Springer Verlag,2014:818-833.
[17]GATYS L A,ECKER A S,BETHGE M.Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks[C]//International Conference on Neural Information Processing Systems.Heidelberg:Bernstein Conference,2015.
[18]GATYS L A,ECKER A S,BETHGE M.A neural algorithm of artistic style[J].Nature Communications,arXiv:1508.0657,2015.
[19]ULYANOV D,LEBEDEV V,VEDALDI A,et al.Texture networks:Feed-forward synthesis of textures and stylized images[C]//Proceedings of the 33rd International Conference on Machine Learning.New York:ICML,2016:1349-1357.
[20]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:694-711.
[21]CHEN T Q,SCHMIDT M.Fast patch-based style transfer of arbitrary style[J].Computer Vision and Pattern Recognition,arXiv:1612.04337,2016.
[22]LUAN F J,PARIS S,SHECHTMAN E,et al.Deep photo style transfer[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu:IEEE Computer Society,2017:6997-7005.
[23]OTSU N.A threshold selection method from graylevel histograms[J].IEEE Transactions on Systems,Man,and Cybernetics,2007,9(1):62-66.
[24]JIAO S,LI X,LU X.An improved Ostu method for image segmentation[C]//International Conference on Signal Processing Proceeding.Beijing:IEEE,2007.
[25]SHI J,MALIK J.Normalized cuts and image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(8):888-905.
[26]ROTHER C,KOLMOGOROV V,BLAKE A.Grabcut:Interactive foreground extraction using iterated graph cuts[J].ACM Transactions on Graphics(TOG),2004,23(3):309-314.
[27]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEEComputer Society,2015:3431-3440.
[28]ZHENG S,JAYASUMANA S,ROMERA-PAREDES B,et al.Conditional random fields as recurrent neural networks[C]//Proceedings of the2015 IEEE International Conference on Computer Vision.Washington DC:IEEE Computer Society,2015:1529-1537.
[29]ARNAB A,JAYASUMANA S,ZHENG S,et al.Higher order conditional random fields in deep neural networks[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:524-540.
[30]EVERINGHAM M,VAN GOOL L,WILLIAMSC K I,et al.The pascal visual object classes(voc)challenge[J].International Journal of Computer Vision,2010,88(2):303-338.
[31]SCHWARZ M W,COWAN W B,BEATTY J C.An experimental comparison of RGB,YIQ,LAB,HSV,and opponent color models[J].ACMTransactions on Graphics,1987,6(2):123-158.
[32]GATYS L A,ECKER A S,BETHGE M,et al.Controlling perceptual factors in neural style transfer[C]//IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu:IEEE Computer Society,2017:3730-3738.
[33]CHAMPANDARD A J.Semantic style transfer and turning two-bit doodles into fine artworks[J].Semantic Style Transfer and Turning Two-Bit Poodles into Fine Artworks,arXiv:1603.01768,2016.
[2]LI Y J,FANG C,YANG J M,et al.Universal style transfer via feature transforms[C]//Advances in Neural Information Processing Systems.California:NIPS,2017:386-396.
[3]GUAY M,RONFARD R,GLEICHER M L.Space-time sketching of character animation[J].Acm Transactions on Graphics,2015,34(4):1-10.
[4]SHENG K,DONG W,KONG Y,et al.Evaluating the quality of face alignment without ground truth[J].Computer Graphics Forum,2015,34(7):213-223.
[5]钱文华,徐丹,官铮,等.粉笔画艺术风格模拟[J].中国图象图形学报,2017,22(5):622-630.QIAN W H,XU D,GUAN Z,et al.Simulating chalk art style painting[J].Journal of Image and Graphics,2017,22(5):620-630.
[6]QIAN W H,XU D,YUE K,et al.Gourd pyrography art simulating based on non-photorealistic rendering[J].Multimedia Tools&Applications,2017,76(13):14559-14579.
[7]杨丽洁,徐添辰,吴恩华.动态模拟中国水墨画中的笔画绘制[J].计算机辅助设计与图形学学报,2015,28(5):742-749.YANG L J,XU T C,WU E H.Animating strokes in drawing process of Chinese ink painting[J].Journal of Computer-Aided Design and Computer Graphics,2016,28(5):742-749.
[8]陈圣国,孙正兴,项建华,等.计算机辅助乱针绣制作技术研究[J].计算机学报,2011,34(3):526-538.CHEN S G,SUN Z X,XIANG J H,et al.Research on the technology of computer aided irregular needling embroidery[J].Chinese Journal of Computers,2011,34(3):526-538.
[9]周杰,孙正兴,杨克微,等.乱针绣模拟的参数化生成方法[J].计算机辅助设计与图形学学报,2014,26(3):436-444.ZHOU J,SUN Z X,YANG K W,et al.Parametric generation method for irregular needling embroidery rendering[J].Journal of Computer-Aided Design and Computer Graphics,2014,26(3):436-444.
[10]项建华,杨克微,周杰,等.一种基于图像分解的乱针绣模拟生成方法[J].图学学报,2013,34(4):16-23.XIANG J H,YANG K W,ZHOU J,et al.A novel image disintegration-based computerized embroidery method for random stitch[J].Journal of Graphics,2013,34(4):16-23.
[11]杨克微,孙正兴.物像空间依赖的乱针绣针法定义与风格生成[J].计算机辅助设计与图形学学报,2018,30(5):778-790.YANG K W,SUN Z X.Vector and pixel space depended stitch definition and style transfer for randomneedle embroidery[J].Journal of Computer-Aided Design&Computer Graphics,2018,30(5):778-790.
[12]张盼,梁昭华.传统刺绣与电脑刺绣的艺术特点比较[J].艺术与设计(理论版),2010(7):229-231.ZHANG P,LIANG Z H.Tradition embroidery comparing with computer embroidery art characteristic[J].Art and Design,2010(7):229-231.
[13]LECUN Y L,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[14]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems.Nevada,NIPS,2012:1097-1105.
[15]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[16]ZEILER M D,FERGUS R.Visualizing and understanding convolutional networks[C]//European Conference on Computer Vision.Zurich:Springer Verlag,2014:818-833.
[17]GATYS L A,ECKER A S,BETHGE M.Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks[C]//International Conference on Neural Information Processing Systems.Heidelberg:Bernstein Conference,2015.
[18]GATYS L A,ECKER A S,BETHGE M.A neural algorithm of artistic style[J].Nature Communications,arXiv:1508.0657,2015.
[19]ULYANOV D,LEBEDEV V,VEDALDI A,et al.Texture networks:Feed-forward synthesis of textures and stylized images[C]//Proceedings of the 33rd International Conference on Machine Learning.New York:ICML,2016:1349-1357.
[20]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:694-711.
[21]CHEN T Q,SCHMIDT M.Fast patch-based style transfer of arbitrary style[J].Computer Vision and Pattern Recognition,arXiv:1612.04337,2016.
[22]LUAN F J,PARIS S,SHECHTMAN E,et al.Deep photo style transfer[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu:IEEE Computer Society,2017:6997-7005.
[23]OTSU N.A threshold selection method from graylevel histograms[J].IEEE Transactions on Systems,Man,and Cybernetics,2007,9(1):62-66.
[24]JIAO S,LI X,LU X.An improved Ostu method for image segmentation[C]//International Conference on Signal Processing Proceeding.Beijing:IEEE,2007.
[25]SHI J,MALIK J.Normalized cuts and image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(8):888-905.
[26]ROTHER C,KOLMOGOROV V,BLAKE A.Grabcut:Interactive foreground extraction using iterated graph cuts[J].ACM Transactions on Graphics(TOG),2004,23(3):309-314.
[27]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEEComputer Society,2015:3431-3440.
[28]ZHENG S,JAYASUMANA S,ROMERA-PAREDES B,et al.Conditional random fields as recurrent neural networks[C]//Proceedings of the2015 IEEE International Conference on Computer Vision.Washington DC:IEEE Computer Society,2015:1529-1537.
[29]ARNAB A,JAYASUMANA S,ZHENG S,et al.Higher order conditional random fields in deep neural networks[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:524-540.
[30]EVERINGHAM M,VAN GOOL L,WILLIAMSC K I,et al.The pascal visual object classes(voc)challenge[J].International Journal of Computer Vision,2010,88(2):303-338.
[31]SCHWARZ M W,COWAN W B,BEATTY J C.An experimental comparison of RGB,YIQ,LAB,HSV,and opponent color models[J].ACMTransactions on Graphics,1987,6(2):123-158.
[32]GATYS L A,ECKER A S,BETHGE M,et al.Controlling perceptual factors in neural style transfer[C]//IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu:IEEE Computer Society,2017:3730-3738.
[33]CHAMPANDARD A J.Semantic style transfer and turning two-bit doodles into fine artworks[J].Semantic Style Transfer and Turning Two-Bit Poodles into Fine Artworks,arXiv:1603.01768,2016.