基于递归残差网络的图像超分辨率重建
|
摘要
深度卷积神经网络在单图像超分辨率重建方面取得了卓越成就,但其良好表现通常以巨大的参数数量为代价.本文提出一种简洁紧凑型递归残差网络结构,该网络通过局部残差学习减轻训练深层网络的困难,引入递归结构保证增加深度的同时控制模型参数数量,采用可调梯度裁剪方法防止产生梯度消失/梯度爆炸,使用反卷积层在网络末端直接上采样图像到超分辨率输出图像.基准测试表明,本文在重建出同等质量超分辨率图像的前提下,参数数量及计算复杂度分别仅为VDSR方法的1/10和1/(2n~2).
Despite the great success in single image super-resolution reconstruction achieved by deep convolutional neural network, the number of the computational parameters is often very large. This paper proposes a concise and compact recursive residual network. The local residual learning method is adopted to mitigate the difficulty of training very deep network, the recursive structure is configured to control the number of model parameters while increasing the model depth, the adjustable gradient clipping strategy is applied to prevent the gradient disappearance/gradient explosion, and a deconvolutional layer is set to directly up sample the image to a super-resolution image at the end of the residual network. According to benchmark tests, in the premise that the same quality super-resolution image is reconstructed, the number of parameters and the computational complexity of the proposed method are reduced to about 1/10 and 1/(2n~2)of VDSR, respectively.
Despite the great success in single image super-resolution reconstruction achieved by deep convolutional neural network, the number of the computational parameters is often very large. This paper proposes a concise and compact recursive residual network. The local residual learning method is adopted to mitigate the difficulty of training very deep network, the recursive structure is configured to control the number of model parameters while increasing the model depth, the adjustable gradient clipping strategy is applied to prevent the gradient disappearance/gradient explosion, and a deconvolutional layer is set to directly up sample the image to a super-resolution image at the end of the residual network. According to benchmark tests, in the premise that the same quality super-resolution image is reconstructed, the number of parameters and the computational complexity of the proposed method are reduced to about 1/10 and 1/(2n~2)of VDSR, respectively.
引文
1 Oktay O,Bai W,Lee M,Guerrero R,Kamnitsas K,Caballero J,et al.Multi-input cardiac image super-resolution using convolutional neural networks.In:Proceedings of the2016 International Conference on Medical Image Computing and Computer-assisted Intervention.Athens,Greece:Springer,Cham,2016.246-254
2 Luo Y,Zhou L,Wang S,Wang Z Y.Video satellite imagery super resolution via convolutional neural networks.IEEE Geoscience and Remote Sensing Letters,2017,14(12):2398-2402
3 Rasti P,Uiboupin T,Escalera S,Anbarjafari G.Convolutional neural network super resolution for face recognition in surveillance monitoring.In:Proceedings of the 2016 International Conference on Articulated Motion and Deformable Objects.Palma de Mallorca,Spain:Springer,Cham,2016.175-184
4 Lu Zhi-Fang,Zhong Bao-Jiang.Image interpolation with predicted gradients.Acta Automatica Sinica,2018,44(6):1072-1085(陆志芳,钟宝江.基于预测梯度的图像插值算法.自动化学报,2018,44(6):1072-1085)
5 Xiong Jiao-Jiao,Lu Hong-Yang,Zhang Ming-Hui,Liu QieGen.Convolutional sparse coding in gradient domain for MRI reconstruction.Acta Automatica Sinica,2017,43(10):1841-1849(熊娇娇,卢红阳,张明辉,刘且根.基于梯度域的卷积稀疏编码磁共振成像重建.自动化学报,2017,43(10):1841-1849)
6 Dong C,Chen C L,He K M,Tang X O.Learning a deep convolutional network for image super-resolution.In:Proceedings of the 13th European Conference on Computer Vision,Zurich Switzerland:Springer,Cham,2014.184-199
7 Dong C,Chen C L,Tang X O.Accelerating the supersesolution convolutional neural network.In:Proceedings of the 14th European Conference on Computer Vision.Amsterdam,The Netherlands:Springer,Cham,2016.391-407
8 Kim J,Lee J K,Lee K M.Accurate image super-resolution using very deep convolutional networks.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1646-1654
9 Lim B,Son S,Kim H,Nah S,Lee K M.Enhanced deep residual networks for single image super-resolution.In:Proceedings of the 2017 IEEE Computer Vision and Pattern Recognition Workshops.Honolulu,USA:IEEE,2017.1132-1140
10 Lai W S,Huang J B,Ahuja N,Yang M H.Deep laplacian pyramid networks for fast and accurate super-resolution.In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.5835-5843
11 Ledig C,Theis L,Huszar F,Caballero J,Cunningham A,Acosta A,et al.Photo-realistic single image super-resolution using a generative adversarial network.In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.105-114
12 Dong C,Chen C L,He K M,Tang X O.Image superresolution using deep convolutional networks.IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(2):295-307
13 Timofte R,De Smet V,Van Gool L.A+:Adjusted anchored neighborhood regression for fast super-resolution.In:Proceedings of the 2015 Asian Conference on Computer Vision.Singapore,Singapore:Springer,Cham.2015.111-126
14 Hu Chang-Sheng,Zhan Shu,Wu Cong-Zhong.Image superresolution based on deep learning features.Acta Automatica Sinica,2017,43(5):814-821(胡长胜,詹曙,吴从中.基于深度特征学习的图像超分辨率重建.自动化学报,2017,43(5):814-821)
15 Kim J,Lee J K,Lee K M.Deeply-recursive convolutional network for image super-resolution.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,Nevada,USA:IEEE,2016.1637-1645
16 Yang W H,Feng J S,Xie G S,Liu J Y,Guo Z M,Yan S C.Video super-resolution based on spatial-temporal recurrent residual networks.Computer Vision and Image Understanding,2018,168:79-92
17 LeCun Y,Bottou L,Bengio Y,Haffner P.Gradient-based learning applied to document recognition.Proceedings of the IEEE,1998,86(11):2278-2324
18 He K M,Zhang X Y,Ren S Q,Sun J.Deep residual learning for image recognition.In:Proceedings of the 2016 IEEEConference on Computer Vision and Pattern Recognition.Las Vegas,Nevada,USA:IEEE,2016.770-778
19 Mao X J,Shen C H,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.In:Proceedings of the 30th International Conference on Neural Information Processing Systems.Barcelona,Spain:NIPS,2016.2810-2818
20 Nair V,Hinton G E.Rectified linear units improve restricted boltzmann machines.In:Proceedings of the 27th International Conference on Machine Learning.Haifa,Israel:ICML,2010.807-814
21 Pascanu R,Mikolov T,Bengio Y.On the difficulty of training recurrent neural networks.In:Proceedings of the 30th International Conference on Machine Learning.Atlanta,USA:ICML,2013,28(3):1310-1318
22 Perez-Pellitero E,Salvador J,Ruiz-Hidalgo J,Rosenhahn B.PSyCo:Manifold span reduction for super resolution.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1837-1845
23 Huang Y,Wang W,Wang L.Video super-resolution via bidirectional recurrent convolutional networks.IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(4):1015-1028
24 Yang J C,Wright J,Huang T S,Ma Y.Image superresolution via sparse representation.IEEE Transactions on Image Processing,2010,19(11):2861-2873
25 Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests.In:Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA:IEEE,2015.3791-3799
26 Martin D,Fowlkes C,Tal D,Malik J.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.In:Proceedings of the 2002 International Conference on Computer Vision.Vancouver,BC,Canada:IEEE,2002.416-423
27 Bevilacqua M,Roumy A,Guillemot C,Alberi Morel ML.Low-complexity single-image super-resolution based on nonnegative neighbor embedding.In:Proceedings of the23rd British Machine Vision Conference.Surrey UK:BMVAPress,2012.1-10
28 Zeyde R,Elad M,Protter M.On single image scaleup using sparse-representations.In:Proceedings of the 2012 International Conference on Curves and Surfaces.Avignon,France:Springer,Berlin,Heidelberg,2012.711-730
29 Jia Y Q,Shelhamer E,Donahue J,Karayev S,Long J,Girshick R,et al.Caffe:convolutional architecture for fast feature embedding.In:Proceedings of the 22nd ACM International Conference on Multimedia.New York,USA:ACM,2014.675-678
30 He K M,Zhang X Y,Ren S Q,Sun J.Delving deep into rectifiers:surpassing human-level performance on imagenet classification.In:Proceedings of the 2015 International Conference on Computer Vision.Santiago,Chile:IEEE,2015.1026-1034
31 Huang J B,Singh A,Ahuja N.Single image super-resolution from transformed self-exemplars.In:Proceedings of the2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA:IEEE,2015.5197-5206
32 Shi W Z,Caballero J,Huszar F,Totz J,P Aitken A,Bishop R,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1874-1883
33 Wang Z W,Liu D,Yang J C,Han W,Huang T.Deep networks for image super-resolution with sparse prior.In:Proceedings of the 2015 IEEE International Conference on Computer Vision.Santiago,Chile:OALib Journal,2015.370-378
34 Vedaldi A,Lenc K.MatConvNet:convolutional neural networks for MATLAB.In:Proceedings of the 23rd ACM International Conference on Multimedia.New York,USA:ACM,2015.689-692
2 Luo Y,Zhou L,Wang S,Wang Z Y.Video satellite imagery super resolution via convolutional neural networks.IEEE Geoscience and Remote Sensing Letters,2017,14(12):2398-2402
3 Rasti P,Uiboupin T,Escalera S,Anbarjafari G.Convolutional neural network super resolution for face recognition in surveillance monitoring.In:Proceedings of the 2016 International Conference on Articulated Motion and Deformable Objects.Palma de Mallorca,Spain:Springer,Cham,2016.175-184
4 Lu Zhi-Fang,Zhong Bao-Jiang.Image interpolation with predicted gradients.Acta Automatica Sinica,2018,44(6):1072-1085(陆志芳,钟宝江.基于预测梯度的图像插值算法.自动化学报,2018,44(6):1072-1085)
5 Xiong Jiao-Jiao,Lu Hong-Yang,Zhang Ming-Hui,Liu QieGen.Convolutional sparse coding in gradient domain for MRI reconstruction.Acta Automatica Sinica,2017,43(10):1841-1849(熊娇娇,卢红阳,张明辉,刘且根.基于梯度域的卷积稀疏编码磁共振成像重建.自动化学报,2017,43(10):1841-1849)
6 Dong C,Chen C L,He K M,Tang X O.Learning a deep convolutional network for image super-resolution.In:Proceedings of the 13th European Conference on Computer Vision,Zurich Switzerland:Springer,Cham,2014.184-199
7 Dong C,Chen C L,Tang X O.Accelerating the supersesolution convolutional neural network.In:Proceedings of the 14th European Conference on Computer Vision.Amsterdam,The Netherlands:Springer,Cham,2016.391-407
8 Kim J,Lee J K,Lee K M.Accurate image super-resolution using very deep convolutional networks.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1646-1654
9 Lim B,Son S,Kim H,Nah S,Lee K M.Enhanced deep residual networks for single image super-resolution.In:Proceedings of the 2017 IEEE Computer Vision and Pattern Recognition Workshops.Honolulu,USA:IEEE,2017.1132-1140
10 Lai W S,Huang J B,Ahuja N,Yang M H.Deep laplacian pyramid networks for fast and accurate super-resolution.In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.5835-5843
11 Ledig C,Theis L,Huszar F,Caballero J,Cunningham A,Acosta A,et al.Photo-realistic single image super-resolution using a generative adversarial network.In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.105-114
12 Dong C,Chen C L,He K M,Tang X O.Image superresolution using deep convolutional networks.IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(2):295-307
13 Timofte R,De Smet V,Van Gool L.A+:Adjusted anchored neighborhood regression for fast super-resolution.In:Proceedings of the 2015 Asian Conference on Computer Vision.Singapore,Singapore:Springer,Cham.2015.111-126
14 Hu Chang-Sheng,Zhan Shu,Wu Cong-Zhong.Image superresolution based on deep learning features.Acta Automatica Sinica,2017,43(5):814-821(胡长胜,詹曙,吴从中.基于深度特征学习的图像超分辨率重建.自动化学报,2017,43(5):814-821)
15 Kim J,Lee J K,Lee K M.Deeply-recursive convolutional network for image super-resolution.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,Nevada,USA:IEEE,2016.1637-1645
16 Yang W H,Feng J S,Xie G S,Liu J Y,Guo Z M,Yan S C.Video super-resolution based on spatial-temporal recurrent residual networks.Computer Vision and Image Understanding,2018,168:79-92
17 LeCun Y,Bottou L,Bengio Y,Haffner P.Gradient-based learning applied to document recognition.Proceedings of the IEEE,1998,86(11):2278-2324
18 He K M,Zhang X Y,Ren S Q,Sun J.Deep residual learning for image recognition.In:Proceedings of the 2016 IEEEConference on Computer Vision and Pattern Recognition.Las Vegas,Nevada,USA:IEEE,2016.770-778
19 Mao X J,Shen C H,Yang Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.In:Proceedings of the 30th International Conference on Neural Information Processing Systems.Barcelona,Spain:NIPS,2016.2810-2818
20 Nair V,Hinton G E.Rectified linear units improve restricted boltzmann machines.In:Proceedings of the 27th International Conference on Machine Learning.Haifa,Israel:ICML,2010.807-814
21 Pascanu R,Mikolov T,Bengio Y.On the difficulty of training recurrent neural networks.In:Proceedings of the 30th International Conference on Machine Learning.Atlanta,USA:ICML,2013,28(3):1310-1318
22 Perez-Pellitero E,Salvador J,Ruiz-Hidalgo J,Rosenhahn B.PSyCo:Manifold span reduction for super resolution.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1837-1845
23 Huang Y,Wang W,Wang L.Video super-resolution via bidirectional recurrent convolutional networks.IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(4):1015-1028
24 Yang J C,Wright J,Huang T S,Ma Y.Image superresolution via sparse representation.IEEE Transactions on Image Processing,2010,19(11):2861-2873
25 Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests.In:Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA:IEEE,2015.3791-3799
26 Martin D,Fowlkes C,Tal D,Malik J.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.In:Proceedings of the 2002 International Conference on Computer Vision.Vancouver,BC,Canada:IEEE,2002.416-423
27 Bevilacqua M,Roumy A,Guillemot C,Alberi Morel ML.Low-complexity single-image super-resolution based on nonnegative neighbor embedding.In:Proceedings of the23rd British Machine Vision Conference.Surrey UK:BMVAPress,2012.1-10
28 Zeyde R,Elad M,Protter M.On single image scaleup using sparse-representations.In:Proceedings of the 2012 International Conference on Curves and Surfaces.Avignon,France:Springer,Berlin,Heidelberg,2012.711-730
29 Jia Y Q,Shelhamer E,Donahue J,Karayev S,Long J,Girshick R,et al.Caffe:convolutional architecture for fast feature embedding.In:Proceedings of the 22nd ACM International Conference on Multimedia.New York,USA:ACM,2014.675-678
30 He K M,Zhang X Y,Ren S Q,Sun J.Delving deep into rectifiers:surpassing human-level performance on imagenet classification.In:Proceedings of the 2015 International Conference on Computer Vision.Santiago,Chile:IEEE,2015.1026-1034
31 Huang J B,Singh A,Ahuja N.Single image super-resolution from transformed self-exemplars.In:Proceedings of the2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA:IEEE,2015.5197-5206
32 Shi W Z,Caballero J,Huszar F,Totz J,P Aitken A,Bishop R,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.In:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA:IEEE,2016.1874-1883
33 Wang Z W,Liu D,Yang J C,Han W,Huang T.Deep networks for image super-resolution with sparse prior.In:Proceedings of the 2015 IEEE International Conference on Computer Vision.Santiago,Chile:OALib Journal,2015.370-378
34 Vedaldi A,Lenc K.MatConvNet:convolutional neural networks for MATLAB.In:Proceedings of the 23rd ACM International Conference on Multimedia.New York,USA:ACM,2015.689-692