基于卷积神经网络的视频图像超分辨率重建方法
|
摘要
为了进一步增强视频图像超分辨率重建的效果,研究利用卷积神经网络的特性进行视频图像的空间分辨率重建,提出了一种基于卷积神经网络的视频图像重建模型。采取预训练的策略用于重建模型参数的初始化,同时在多帧视频图像的空间和时间维度上进行训练,提取描述主要运动信息的特征进行学习,充分利用视频帧间图像的信息互补进行中间帧的重建。针对帧间图像的运动模糊,采用自适应运动补偿加以处理,对通道进行优化输出得到高分辨率的重建图像。实验表明,重建视频图像在平均客观评价指标上均有较大提升(PSNR+0. 4 d B/SSIM+0. 02),并且有效减少了图像在主观视觉效果上的边缘模糊现象。与其他传统算法相比,在图像评价的客观指标和主观视觉效果上均有明显的提升,为视频图像的超分辨率重建提供了一种基于卷积神经网络的新颖架构,也为进一步探索基于深度学习的视频图像超分辨率重建方法提供了思路。
In order to further improve the performance of video image super-resolution reconstruction and study the reconstruction of spatial resolution of video images by using the characteristics of convolution neural network,this paper proposed a video image reconstruction model based on convolution neural network. The model adopted the pre-training strategy to initialize the parameters. And it carried out the training processing both on the spatial and temporal dimensions of the multi-frame video images at the same time. It extracted the characteristics of the main motion information,learnt and made full use of the information inter the frames for improved performance. And it used the adaptive motion compensation algorithm to optimize the output of the channel to obtain the reconstructed center frame image with high resolution. The experimental results show that the average of objective evaluation indexes for video image reconstruction improves with a rather clear margin( PSNR + 0. 4 dB/SSIM +0. 02),and the edge of the fuzzy phenomenon in video reconstruction image for the subjective visual effect is effectively reduced. Compared with other traditional algorithms,it both obviously improved the evaluation of the objective indexes and subjective visual effect of the reconstructed image. Providing a novel architecture based on convolution neural network for video image super-resolution,which provides an exploration for the further study of video image super-resolution reconstruction based on the deep learning method.
In order to further improve the performance of video image super-resolution reconstruction and study the reconstruction of spatial resolution of video images by using the characteristics of convolution neural network,this paper proposed a video image reconstruction model based on convolution neural network. The model adopted the pre-training strategy to initialize the parameters. And it carried out the training processing both on the spatial and temporal dimensions of the multi-frame video images at the same time. It extracted the characteristics of the main motion information,learnt and made full use of the information inter the frames for improved performance. And it used the adaptive motion compensation algorithm to optimize the output of the channel to obtain the reconstructed center frame image with high resolution. The experimental results show that the average of objective evaluation indexes for video image reconstruction improves with a rather clear margin( PSNR + 0. 4 dB/SSIM +0. 02),and the edge of the fuzzy phenomenon in video reconstruction image for the subjective visual effect is effectively reduced. Compared with other traditional algorithms,it both obviously improved the evaluation of the objective indexes and subjective visual effect of the reconstructed image. Providing a novel architecture based on convolution neural network for video image super-resolution,which provides an exploration for the further study of video image super-resolution reconstruction based on the deep learning method.
引文
[1]何小海,吴媛媛,陈为龙,等.视频超分辨率重建技术综述[J].信息与电子工程,2011,9(1):1-6.(He Xiaohai,Wu Yuanyuan,Chen Weilong,et al.Review of video super-resolution reconstruction technology[J].Information and Electronic Engineering,2011,9(1):1-6.)
[2]Belekos S P,Galatsanos N P,Katsaggelos A K.Maximum a posteriori video super-resolution using a new multichannel image prior[J].IEEE Trans on Image Processing,2010,19(6):1451-1464.
[3]Babacan S D,Molina R,Katsaggelos A K.Variational Bayesian super resolution[J].IEEE Trans on Image Processing,2011,20(4):984-999.
[4]Liu Ce,Sun Deqing.On Bayesian adaptive video super resolution[J].IEEE Trans on Pattern Analysis and Machine Intelligence,2014,36(2):346-360.
[5]Yang Jianchao,Wright J,Huang T,et al.Image super-resolution via sparse representation[J].IEEE Trans on Image Processing,2010,19(11):2861-2873.
[6]Aharon M,Elad M,Bruckstein A.K-SVD:an algorithm for designing overcomplete dictionaries for sparse representation[J].IEEE Trans on Signal Processing,2006,54(11):4311-4322.
[7]Song B C,Jeong S C,Choi Y.Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training[J].IEEE Trans on Circuits and Systems for Video Technology,2011,21(3):274-285.
[8]张岩,李建增,李德良,等.无人机侦察视频超分辨率重建方法[J].中国图象图形学报,2016,21(7):967-976.(Zhang Yan,Li Jianzheng,Li Deliang,et al.Super-resolution reconstruction method for UAV reconnaissance video[J].Journal of Image and Graphics,2016,21(7):967-976.)
[9]Timofte R,De Smet V,Van Gool L.A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proc of the 12th IEEEAsian Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:1920-1927.
[10]Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:3791-3799.
[11]Glasner D,Bagon S,Irani M.Super-resolution from a single image[C]//Proc of IEEE International Conference on Computer Vision.Piscataway,NJ:IEEE Press,2009:349-356.
[12]覃凤清,何小海,陈为龙,等.一种基于子像素配准视频超分辨率重建方法[J].光电子·激光,2009,20(7):972-976.(Qin Fengqing,He Xiaohai,Chen Weilong,et al.A video super-resolution reconstruction method based on sub-pixel registration[J].Journal of Optoelectronics·Laser,2009,20(7):972-976.)
[13]Ma Ziyang,Liao Renjie,Xu Li.Handling motion blur in multi-frame super-resolution[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:5224-5232.
[14]Takeda H,Milanfar P,Protter M,et al.Super-resolution without explicit subpixel motion estimation[J].IEEE Trans on Image Processing,2009,18(9):1958-1975.
[15]Shi Wenzhe,Caballero J,Huszar F,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2016:1874-1883.
[16]Kappeler A,Yoo S,Dai Qiqin,et al.Video super-resolution with convolutional neural networks[J].IEEE Trans on Computational Imaging,2016,2(2):109-122.
[17]Dong Chao,Loy C C,He Kaiming,et al.Learning a deep convolutional network for image super-resolution[C]//Proc of IEEE European Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:184-199.
[18]Wang Zhangyang,Yang Yingzhen,Wang Zhaowen,et al.Self-tuned deep super resolution[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:1-8.
[19]Cui Zhen,Chang Hong,Shan Shiguang,et al.Deep network cascade for image super-resolution[C]//Proc of IEEE European Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:1-16.
[20]Cheng Minghui,Lin Naiwei,Hwang K,et al.Fast video super-resolution using artificial neural networks[C]//Proc of the 8th International Symposium on Communication Systems,Networks&Digital Signal Processing.Piscataway,NJ:IEEE Press,2012:1-4.
[21]Liao Renjie,Tao Xin,Li Ruiyu,et al.Video super-resolution via deep draft-ensemble learning[C]//Proc of IEEE International Conference on Computer Vision.Piscataway,NJ:IEEE Press,2015:531-539.
[22]黄璇,杨晓梅.基于低秩及全变分的视频超分辨率重建[J].计算机应用研究,2015,32(3):938-941.(Huang Xuan,Yang Xiaomei.Video super-resolution reconstruction based on low rank and total variation[J].Application Research of Computers,2015,32(3):938-941.)
[23]Maas A,Hannun A,Ng A.Rectifier nonlinearities improve neural network acoustic models[C]//Proc of IEEE International Conference on Machine Learning.Piscataway,NJ:IEEE Press,2013:1-8.
[2]Belekos S P,Galatsanos N P,Katsaggelos A K.Maximum a posteriori video super-resolution using a new multichannel image prior[J].IEEE Trans on Image Processing,2010,19(6):1451-1464.
[3]Babacan S D,Molina R,Katsaggelos A K.Variational Bayesian super resolution[J].IEEE Trans on Image Processing,2011,20(4):984-999.
[4]Liu Ce,Sun Deqing.On Bayesian adaptive video super resolution[J].IEEE Trans on Pattern Analysis and Machine Intelligence,2014,36(2):346-360.
[5]Yang Jianchao,Wright J,Huang T,et al.Image super-resolution via sparse representation[J].IEEE Trans on Image Processing,2010,19(11):2861-2873.
[6]Aharon M,Elad M,Bruckstein A.K-SVD:an algorithm for designing overcomplete dictionaries for sparse representation[J].IEEE Trans on Signal Processing,2006,54(11):4311-4322.
[7]Song B C,Jeong S C,Choi Y.Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training[J].IEEE Trans on Circuits and Systems for Video Technology,2011,21(3):274-285.
[8]张岩,李建增,李德良,等.无人机侦察视频超分辨率重建方法[J].中国图象图形学报,2016,21(7):967-976.(Zhang Yan,Li Jianzheng,Li Deliang,et al.Super-resolution reconstruction method for UAV reconnaissance video[J].Journal of Image and Graphics,2016,21(7):967-976.)
[9]Timofte R,De Smet V,Van Gool L.A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proc of the 12th IEEEAsian Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:1920-1927.
[10]Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:3791-3799.
[11]Glasner D,Bagon S,Irani M.Super-resolution from a single image[C]//Proc of IEEE International Conference on Computer Vision.Piscataway,NJ:IEEE Press,2009:349-356.
[12]覃凤清,何小海,陈为龙,等.一种基于子像素配准视频超分辨率重建方法[J].光电子·激光,2009,20(7):972-976.(Qin Fengqing,He Xiaohai,Chen Weilong,et al.A video super-resolution reconstruction method based on sub-pixel registration[J].Journal of Optoelectronics·Laser,2009,20(7):972-976.)
[13]Ma Ziyang,Liao Renjie,Xu Li.Handling motion blur in multi-frame super-resolution[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:5224-5232.
[14]Takeda H,Milanfar P,Protter M,et al.Super-resolution without explicit subpixel motion estimation[J].IEEE Trans on Image Processing,2009,18(9):1958-1975.
[15]Shi Wenzhe,Caballero J,Huszar F,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2016:1874-1883.
[16]Kappeler A,Yoo S,Dai Qiqin,et al.Video super-resolution with convolutional neural networks[J].IEEE Trans on Computational Imaging,2016,2(2):109-122.
[17]Dong Chao,Loy C C,He Kaiming,et al.Learning a deep convolutional network for image super-resolution[C]//Proc of IEEE European Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:184-199.
[18]Wang Zhangyang,Yang Yingzhen,Wang Zhaowen,et al.Self-tuned deep super resolution[C]//Proc of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2015:1-8.
[19]Cui Zhen,Chang Hong,Shan Shiguang,et al.Deep network cascade for image super-resolution[C]//Proc of IEEE European Conference on Computer Vision.Piscataway,NJ:IEEE Press,2014:1-16.
[20]Cheng Minghui,Lin Naiwei,Hwang K,et al.Fast video super-resolution using artificial neural networks[C]//Proc of the 8th International Symposium on Communication Systems,Networks&Digital Signal Processing.Piscataway,NJ:IEEE Press,2012:1-4.
[21]Liao Renjie,Tao Xin,Li Ruiyu,et al.Video super-resolution via deep draft-ensemble learning[C]//Proc of IEEE International Conference on Computer Vision.Piscataway,NJ:IEEE Press,2015:531-539.
[22]黄璇,杨晓梅.基于低秩及全变分的视频超分辨率重建[J].计算机应用研究,2015,32(3):938-941.(Huang Xuan,Yang Xiaomei.Video super-resolution reconstruction based on low rank and total variation[J].Application Research of Computers,2015,32(3):938-941.)
[23]Maas A,Hannun A,Ng A.Rectifier nonlinearities improve neural network acoustic models[C]//Proc of IEEE International Conference on Machine Learning.Piscataway,NJ:IEEE Press,2013:1-8.