利用多尺度卷积神经网络的图像超分辨率算法
|
摘要
单幅图像超分辨率问题是典型的图像反问题。近年来深度学习广泛应用于图像超分辨率重建。为提高超分辨率算法的性能,本文利用多尺度和残差训练的思想,提出一种利用多尺度卷积神经网络的图像超分辨率算法。该算法采用多尺度的卷积核及收缩--扩展的网络结构来提取图像多尺度的信息,并在网络结构中使用跳跃连接,以便更好的传递信息并弥补由于使用下采样和上采样而造成的图像细节信息的损失,来提高图像的重建质量。通过与其他算法的对比实验表明了本文算法不仅可以取得更好的性能,并且训练的收敛速度较快。
The single image super-resolution problem is a typical image inverse problem. In recent years,deep learning is widely used in image super-resolution. In order to improve the performance of super-resolution algorithms,in this paper,the ideas of multi-scale and residual training are utilized. An image super-resolution algorithm that exploits the multi-scale convolution neural network is proposed. This algorithm exploits the multi-scale convolution kernels and the shrinkage-extension structure to extract image multi-scale information. Skip connection is used in the network structure to improve the quality of image reconstruction,which can transmit information effectively. Moreover,it can compensate for the loss of image details resulting from the use of down-sampling and up-sampling. Compared with other algorithms,the experiments shows that our algorithm can not only achieve better performance,but also has the faster convergence speed.
The single image super-resolution problem is a typical image inverse problem. In recent years,deep learning is widely used in image super-resolution. In order to improve the performance of super-resolution algorithms,in this paper,the ideas of multi-scale and residual training are utilized. An image super-resolution algorithm that exploits the multi-scale convolution neural network is proposed. This algorithm exploits the multi-scale convolution kernels and the shrinkage-extension structure to extract image multi-scale information. Skip connection is used in the network structure to improve the quality of image reconstruction,which can transmit information effectively. Moreover,it can compensate for the loss of image details resulting from the use of down-sampling and up-sampling. Compared with other algorithms,the experiments shows that our algorithm can not only achieve better performance,but also has the faster convergence speed.
引文
[1]Shi W Z,Caballero J,Ledig C,et al.Cardiac image superresolution with global correspondence using multi-atlas patch match[C]∥Medical Image Computing and ComputerAssisted Intervention,Nagoya,Japan,2013:9-16.
[2]Thornton M W,Atkinson P M,Holland D A.Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J].International Journal of Remote Sensing,2006,27(3):473-491.
[3]王民,刘可心,刘利,等.基于优化卷积神经网络的图像超分辨率重建[J].激光与光电子学进展,2017,54(11):111005.Wang M,Liu K X,Liu L,et al.Super-resolution reconstruction of image based on optimized convolution neural network[J].Laser&Optoelectronics Progress,2017,54(11):111005.(in Chinese)
[4]肖进胜,刘恩雨,朱力,等.改进的基于卷积神经网络的图像超分辨率算法[J].光学学报,2017,37(3):0318011.Xiao J S,Liu E Y,Zhu L,et al.Improved image super-resolution algorithm based on convolutional neural network[J].Acta Optica Sinica,2017,37(3):0318011.(in Chinese)
[5]Harris J L.Diffraction and resolving power[J].Journal of the Optical Society of America,1964,54(7):931-936.
[6]Zhu Y,Zhang Y,Yuille A L.Single image super-resolution using deformable patches[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Columbus,USA,2014:2917-2924.
[7]Qu Y,Shi C,Liu J,et al.Single Image super-resolution via convolutional neural network and total variation regularization[C]∥International Conference on Multimedia Modeling,MIAMI,USA,2016:28-38.
[8]Hou H,Andrews H.Cubic splines for image interpolation and digital filtering[J].IEEE Transactions on Acoustics,Speech,and Signal Processing,1978,26(6):508-517.
[9]Keys R.Cubic convolution interpolation for digital image processing[J].IEEE Transactions on Acoustics,Speech,and Signal Processing,1981,29(6):1153-1160.
[10]Yang J,Wright J,Huang T S,et al.Image super-resolution via sparse representation[J].IEEE Transactions on Image Processing,2010,19(11):2861-2873.
[11]Yang J,Wright J,Huang T,et al.Image super-resolution as sparse representation of raw image patches[C]∥Computer Vision and Pattern Recognition,Anchorage,AK,USA,2008:1-8.
[12]曹明明,干宗良,陈杰,等.自适应邻域选取的邻域嵌入超分辨率重建算法[J].信号处理,2015,31(1):8-16.Cao M M,Gan Z L,Chen J,et al.Neighborhood embedding super-resolution reconstruction algorithm based on adaptive neighborhood selection[J].Journal of Signal Processing,2015,31(1):8-16.(in Chinese)
[13]Yang J,Lin Z,Cohen S.Fast image super-resolution based on in-place example regression[C]∥IEEE Conference on Computer Vision and Pattern Recognition,Portland,USA,2013:1059-1066.
[14]Chang H,Yeung D Y,Xiong Y M.Super-resolution through neighbor embedding[C]∥Proceedings of the2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,Washington,DC,USA,2004,1:275-282.
[15]Dong C,Loy C C,He K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-304.
[16]Liang Y,Timofte R,Wang J,et al.Single image super resolution-when model adaptation matters[OL].http:∥arxiv.org/abs/1703.10889,2017.
[17]Dong C,Loy C C,Tang X.Accelerating the super-resolution convolutional neural network[C]∥European Conference on Computer Vision,Amsterdam,Netherlands,2016:391-407.
[18]Kim J,Kwon Lee J,Mu Lee K.Accurate image superresolution using very deep convolutional networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,USA,2016:1646-1654.
[19]Kim J,Kwon Lee J,Mu Lee K.Deeply-recursive convolutional network for image super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,USA,2016:1637-1645.
[20]Lai W S,Huang J B,Ahuja N,et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]∥The IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA,2017:624-632.
[21]Tai Y,Yang J,Liu X.Image super-resolution via deep recursive residual network[C]∥The IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA,2017:3147-3155.
[22]Ronneberger O,Fischer P,Brox T.U-net:Convolutional networks for biomedical image segmentation[C]∥International Conference on Medical Image Computing and Computer-Assisted Intervention.Munich,Germany,2015:234-241.
[23]Krizhevsky A,Sutskever I,Hinton G E.Imagenet classification with deep convolutional neural networks[C]∥Advances in Neural Information Processing Systems,USA,2012:1097-1105.
[24]Mao X J,Shen C,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[C]∥Advances in Neural Information Processing Systems,Barcelona,Spain,2016:2802-2810.
[25]Jia Y,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[C]∥Proceedings of the 22nd ACM International Conference on Multimedia,Orlando,Florida,USA,2014:675-678.
[26]Martin D,Fowlkes C,Tal D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]∥IEEE International Conference on Computer Vision,Vancouver,BC,Canada,2001,2:416-423.
[27]Kingma D,Ba J.Adam:A method for stochastic optimization[C]∥The International Conference on Learning Representations,San Diego,USA,2015.
[28]Zhao H,Gallo O,Frosio I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2017,3(1):47-57.
[29]Lim B,Son S,Kim H,et al.Enhanced deep residual networks for single image super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu,HI,USA,2017:136-144.
[2]Thornton M W,Atkinson P M,Holland D A.Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J].International Journal of Remote Sensing,2006,27(3):473-491.
[3]王民,刘可心,刘利,等.基于优化卷积神经网络的图像超分辨率重建[J].激光与光电子学进展,2017,54(11):111005.Wang M,Liu K X,Liu L,et al.Super-resolution reconstruction of image based on optimized convolution neural network[J].Laser&Optoelectronics Progress,2017,54(11):111005.(in Chinese)
[4]肖进胜,刘恩雨,朱力,等.改进的基于卷积神经网络的图像超分辨率算法[J].光学学报,2017,37(3):0318011.Xiao J S,Liu E Y,Zhu L,et al.Improved image super-resolution algorithm based on convolutional neural network[J].Acta Optica Sinica,2017,37(3):0318011.(in Chinese)
[5]Harris J L.Diffraction and resolving power[J].Journal of the Optical Society of America,1964,54(7):931-936.
[6]Zhu Y,Zhang Y,Yuille A L.Single image super-resolution using deformable patches[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Columbus,USA,2014:2917-2924.
[7]Qu Y,Shi C,Liu J,et al.Single Image super-resolution via convolutional neural network and total variation regularization[C]∥International Conference on Multimedia Modeling,MIAMI,USA,2016:28-38.
[8]Hou H,Andrews H.Cubic splines for image interpolation and digital filtering[J].IEEE Transactions on Acoustics,Speech,and Signal Processing,1978,26(6):508-517.
[9]Keys R.Cubic convolution interpolation for digital image processing[J].IEEE Transactions on Acoustics,Speech,and Signal Processing,1981,29(6):1153-1160.
[10]Yang J,Wright J,Huang T S,et al.Image super-resolution via sparse representation[J].IEEE Transactions on Image Processing,2010,19(11):2861-2873.
[11]Yang J,Wright J,Huang T,et al.Image super-resolution as sparse representation of raw image patches[C]∥Computer Vision and Pattern Recognition,Anchorage,AK,USA,2008:1-8.
[12]曹明明,干宗良,陈杰,等.自适应邻域选取的邻域嵌入超分辨率重建算法[J].信号处理,2015,31(1):8-16.Cao M M,Gan Z L,Chen J,et al.Neighborhood embedding super-resolution reconstruction algorithm based on adaptive neighborhood selection[J].Journal of Signal Processing,2015,31(1):8-16.(in Chinese)
[13]Yang J,Lin Z,Cohen S.Fast image super-resolution based on in-place example regression[C]∥IEEE Conference on Computer Vision and Pattern Recognition,Portland,USA,2013:1059-1066.
[14]Chang H,Yeung D Y,Xiong Y M.Super-resolution through neighbor embedding[C]∥Proceedings of the2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,Washington,DC,USA,2004,1:275-282.
[15]Dong C,Loy C C,He K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-304.
[16]Liang Y,Timofte R,Wang J,et al.Single image super resolution-when model adaptation matters[OL].http:∥arxiv.org/abs/1703.10889,2017.
[17]Dong C,Loy C C,Tang X.Accelerating the super-resolution convolutional neural network[C]∥European Conference on Computer Vision,Amsterdam,Netherlands,2016:391-407.
[18]Kim J,Kwon Lee J,Mu Lee K.Accurate image superresolution using very deep convolutional networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,USA,2016:1646-1654.
[19]Kim J,Kwon Lee J,Mu Lee K.Deeply-recursive convolutional network for image super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,USA,2016:1637-1645.
[20]Lai W S,Huang J B,Ahuja N,et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]∥The IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA,2017:624-632.
[21]Tai Y,Yang J,Liu X.Image super-resolution via deep recursive residual network[C]∥The IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA,2017:3147-3155.
[22]Ronneberger O,Fischer P,Brox T.U-net:Convolutional networks for biomedical image segmentation[C]∥International Conference on Medical Image Computing and Computer-Assisted Intervention.Munich,Germany,2015:234-241.
[23]Krizhevsky A,Sutskever I,Hinton G E.Imagenet classification with deep convolutional neural networks[C]∥Advances in Neural Information Processing Systems,USA,2012:1097-1105.
[24]Mao X J,Shen C,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[C]∥Advances in Neural Information Processing Systems,Barcelona,Spain,2016:2802-2810.
[25]Jia Y,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[C]∥Proceedings of the 22nd ACM International Conference on Multimedia,Orlando,Florida,USA,2014:675-678.
[26]Martin D,Fowlkes C,Tal D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]∥IEEE International Conference on Computer Vision,Vancouver,BC,Canada,2001,2:416-423.
[27]Kingma D,Ba J.Adam:A method for stochastic optimization[C]∥The International Conference on Learning Representations,San Diego,USA,2015.
[28]Zhao H,Gallo O,Frosio I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2017,3(1):47-57.
[29]Lim B,Son S,Kim H,et al.Enhanced deep residual networks for single image super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu,HI,USA,2017:136-144.