双向时域特征流盲去运动模糊方法
|
摘要
便携式成像设备在日常的生活中已经无处不在,但是因为拍摄时的抖动或者场景中的快速运动物体,所拍摄的图像或视频经常存在模糊现象,造成图像重要细节丢失。为了将模糊的视频图像恢复到清晰的状态,结合近年来的研究热点——生成对抗网络,本文提出了一种新颖的端到端的双向时域特征流盲去运动模糊方法。该方法充分利用时空连续特征信息,在三连帧图像间建立双向的时域特征传输通道。多阶段自编码去模糊网络结构和并行编码-混合解码融合方案能够融合三连帧图像多通道内容信息,并恢复出更加清晰的视频图像。实验结果表明,在不牺牲较大时间代价前提下,本文提出的方法在传统的质量评价指标(峰值信噪比和结构相似性)和视觉质量上均优于现有的去模糊算法。
Portable imaging devices are ubiquitous in everyday life. However,as the hand jitter or the fast moving objects in the scene during shooting process,the captured image or video is often blurred,causing important details loss. In order to restore the blurred video and image to a clear state,we combine the recent research hotspots——Generative adversarial network,and propose a novel end-to-end bidirectional time-domain feature flow blind motion deblurring algorithm. The algorithm makes full use of the feature information of spatio-temporal continuity constraint to establish a bidirectional transmission channel of timedomain features between the adjacent frames. The multi-stage autoencoder deblurring network structure and the parallel coding and hybrid decoding fusion solution can fuse the multi-channel content information of a frame triplet and restore a clearer frame for a video. Experimental results show that the proposed algorithm is superior to the existing advanced algorithms on the traditional image quality evaluation indexes,i. e.,peak signal to noise ratio(PSNR) and structural similarity(SSIM),and visual quality within acceptable time cost.
Portable imaging devices are ubiquitous in everyday life. However,as the hand jitter or the fast moving objects in the scene during shooting process,the captured image or video is often blurred,causing important details loss. In order to restore the blurred video and image to a clear state,we combine the recent research hotspots——Generative adversarial network,and propose a novel end-to-end bidirectional time-domain feature flow blind motion deblurring algorithm. The algorithm makes full use of the feature information of spatio-temporal continuity constraint to establish a bidirectional transmission channel of timedomain features between the adjacent frames. The multi-stage autoencoder deblurring network structure and the parallel coding and hybrid decoding fusion solution can fuse the multi-channel content information of a frame triplet and restore a clearer frame for a video. Experimental results show that the proposed algorithm is superior to the existing advanced algorithms on the traditional image quality evaluation indexes,i. e.,peak signal to noise ratio(PSNR) and structural similarity(SSIM),and visual quality within acceptable time cost.
引文
[1]Kupyn O,Budzan V,Mykhailych M,et al.DeblurGAN:Blind motion deblurring using conditional adversarial networks[EB/OL].https://arxiv.org/abs/1711.07064,2017.
[2]Proakis J G,Manolakis D G.数字信号处理——原理、算法与应用[M].4版.北京:电子工业出版社,2013:863-871.Proakis J G,Manolakis D G.Digital signal processing-Principles,algorithms and applications[M].4th ed.Beijing:Electronic Industry Press,2013:863-871.
[3]Lam E Y,Goodman J W.Iterative statistical approach to blind image deconvolution[J].Journal of the Optical Society of America:A Optics Image Science&Vision,2000,17(7):1177.
[4]Ma L,Zhang R,Qu Z,et al.Blind image deconvolution using sparse and redundant representation[J].Optik-International Journal for Light and Electron Optics,2014,125(23):6942-6945.
[5]Xu L,Ren J S J,Liu C,et al.Deep convolutional neural network for image deconvolution[C]//NIPS'14 Proceedings of the27th International Conference on Neural Information Processing Systems.Cambridge,MA,USA:MIT Press,2014:1790-1798.
[6]Sun J,Cao W,Xu Z,et al.Learning a convolutional neural network for non-uniform motion blur removal[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2015:769-777.
[7]Nah S,Kim T H,Lee K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//2017 IEEEConference on Computer Vision and Pattern Recognition(CVPR).Honolulu,Hawaii,USA:IEEE,2017,2:3.
[8]Gulrajani I,Ahmed F,Arjovsky M,et al.Improved training of Wasserstein GANs[EB/OL].https://arxiv.org/abs/1704.00028v3,2017.
[9]Kim T H,Lee K M.Generalized video deblurring for dynamic scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2015:5426-5434.
[10]Hyun Kim T,Ahn B,Mu Lee K.Dynamic scene deblurring[C]//Proceedings of the IEEE International Conference on Computer Vision.[S.l.]:IEEE,2013:3160-3167.
[11]Su S,Delbracio M,Wang J,et al.Deep video deblurring for hand-held cameras[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu,Hawaii,USA:IEEE,2017,2:6.
[12]Goodfellow I,Pouget-Abadie J,Mirza M,et al.Generative adversarial nets[EB/OL].https://arxiv.org/abs/1406.2661,2014.
[13]Johnson J,Alahi A,Fei-Fei L.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Cham,Switzerland:Springer,2016:694-711.
[14]Mao X,Li Q,Xie H,et al.Least squares generative adversarial networks[C]//Computer Vision(ICCV),2017 IEEEInternational Conference on.[S.l]:IEEE,2017:2813-2821.
[15]Mao X,Shen C,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[EB/OL].https://arxiv.org/abs/1603.09056,2016.
[16]Tao X,Gao H,Wang Y,et al.Scale-recurrent network for deep image deblurring[EB/OL].https://arxiv.org/abs/1802.01770,2018.
[17]He K,Zhang X,Ren S,et al.Identity mappings in deep residual networks[C]//European Conference on Computer Vision.Cham,Switzerland:Springer,2016:630-645.
[18]He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2016:770-778.
[19]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[EB/OL].https://arxiv.org/abs/1409.1556,2014.
[20]Ulyanov D,Vedaldi A,Lempitsky V S.Instance normalization:The missing ingredient for fast stylization[EB/OL].https://arxiv.org/abs/1607.08022,2016.
[21]Ba J L,Kiros J R,Hinton G E.Layer normalization[EB/OL].https://arxiv.org/abs/1607.06450,2016.
[22]Kingma D P,Ba J.Adam:A method for stochastic optimization[EB/OL].https://arxiv.org/abs/1412.6980,2014.
[23]Wieschollek P,Hirsch M,Sch?lkopf B,et al.Learning blind motion deblurring[C]//2017 IEEE International Conference on Computer Vision.Venice,Italy:IEEE,2017:231-240
[2]Proakis J G,Manolakis D G.数字信号处理——原理、算法与应用[M].4版.北京:电子工业出版社,2013:863-871.Proakis J G,Manolakis D G.Digital signal processing-Principles,algorithms and applications[M].4th ed.Beijing:Electronic Industry Press,2013:863-871.
[3]Lam E Y,Goodman J W.Iterative statistical approach to blind image deconvolution[J].Journal of the Optical Society of America:A Optics Image Science&Vision,2000,17(7):1177.
[4]Ma L,Zhang R,Qu Z,et al.Blind image deconvolution using sparse and redundant representation[J].Optik-International Journal for Light and Electron Optics,2014,125(23):6942-6945.
[5]Xu L,Ren J S J,Liu C,et al.Deep convolutional neural network for image deconvolution[C]//NIPS'14 Proceedings of the27th International Conference on Neural Information Processing Systems.Cambridge,MA,USA:MIT Press,2014:1790-1798.
[6]Sun J,Cao W,Xu Z,et al.Learning a convolutional neural network for non-uniform motion blur removal[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2015:769-777.
[7]Nah S,Kim T H,Lee K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//2017 IEEEConference on Computer Vision and Pattern Recognition(CVPR).Honolulu,Hawaii,USA:IEEE,2017,2:3.
[8]Gulrajani I,Ahmed F,Arjovsky M,et al.Improved training of Wasserstein GANs[EB/OL].https://arxiv.org/abs/1704.00028v3,2017.
[9]Kim T H,Lee K M.Generalized video deblurring for dynamic scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2015:5426-5434.
[10]Hyun Kim T,Ahn B,Mu Lee K.Dynamic scene deblurring[C]//Proceedings of the IEEE International Conference on Computer Vision.[S.l.]:IEEE,2013:3160-3167.
[11]Su S,Delbracio M,Wang J,et al.Deep video deblurring for hand-held cameras[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Honolulu,Hawaii,USA:IEEE,2017,2:6.
[12]Goodfellow I,Pouget-Abadie J,Mirza M,et al.Generative adversarial nets[EB/OL].https://arxiv.org/abs/1406.2661,2014.
[13]Johnson J,Alahi A,Fei-Fei L.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Cham,Switzerland:Springer,2016:694-711.
[14]Mao X,Li Q,Xie H,et al.Least squares generative adversarial networks[C]//Computer Vision(ICCV),2017 IEEEInternational Conference on.[S.l]:IEEE,2017:2813-2821.
[15]Mao X,Shen C,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[EB/OL].https://arxiv.org/abs/1603.09056,2016.
[16]Tao X,Gao H,Wang Y,et al.Scale-recurrent network for deep image deblurring[EB/OL].https://arxiv.org/abs/1802.01770,2018.
[17]He K,Zhang X,Ren S,et al.Identity mappings in deep residual networks[C]//European Conference on Computer Vision.Cham,Switzerland:Springer,2016:630-645.
[18]He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2016:770-778.
[19]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[EB/OL].https://arxiv.org/abs/1409.1556,2014.
[20]Ulyanov D,Vedaldi A,Lempitsky V S.Instance normalization:The missing ingredient for fast stylization[EB/OL].https://arxiv.org/abs/1607.08022,2016.
[21]Ba J L,Kiros J R,Hinton G E.Layer normalization[EB/OL].https://arxiv.org/abs/1607.06450,2016.
[22]Kingma D P,Ba J.Adam:A method for stochastic optimization[EB/OL].https://arxiv.org/abs/1412.6980,2014.
[23]Wieschollek P,Hirsch M,Sch?lkopf B,et al.Learning blind motion deblurring[C]//2017 IEEE International Conference on Computer Vision.Venice,Italy:IEEE,2017:231-240