基于多尺度特征映射网络的图像超分辨率重建
|
摘要
针对基于卷积神经网络的图像超分辨率重建(SRCNN)方法存在的重建网络浅、特征利用率低以及重建图像模糊等问题,提出基于多尺度特征映射网络的图像超分辨率重建方法.多尺度特征映射网络通过学习低分辨率(LR)特征与高分辨率(HR)特征之间的映射关系,将多个尺度的LR特征映射到HR特征空间,通过特征融合来提高重建过程中对特征的利用率;该方法定义了结合逐像素损失、感知损失和对抗损失的联合损失函数,从低频内容、图像边缘和局部纹理等方面均衡提升重建图像质量.对数据集Set5、Set14和BSD100的图片4倍下采样后进行测试,与当前主流方法进行比较和分析.实验证明,基于生成对抗的多尺度特征映射网络在提高图像感知质量方面表现优秀,重建的图像具有更加清晰的边缘和纹理,在客观评价上具有较好的评分.
An image super-resolution reconstruction method based on multi-scale feature mapping network was proposed for the problems of shallow network, low utilization rate of features and fuzzy reconstructed images, which existed in the super-resolution convolutional neural network(SRCNN). Multi-scale low-resolution(LR) features were mapped into high-resolution(HR) feature space by learning the mapping relation between LR features and HR features, and the utilization rate of features in the reconstruction process was improved by using feature concatenation. A joint loss function consisting of the pixel-wise loss, the perceptual loss and the adversarial loss was defined, which performed well in restoring the low-frequency content, the sharp edges and the high-frequency textures of the reconstructed images. The experimental results of datasets Set5, Set14 and BSD100 for the upscaling factor 4 were compared with those of state-of-the-art methods. The proposed method performs well in improving the perceptual quality of the reconstructed images in order to achieve clearer edges and textures, and has better scores in the objective evaluation.
An image super-resolution reconstruction method based on multi-scale feature mapping network was proposed for the problems of shallow network, low utilization rate of features and fuzzy reconstructed images, which existed in the super-resolution convolutional neural network(SRCNN). Multi-scale low-resolution(LR) features were mapped into high-resolution(HR) feature space by learning the mapping relation between LR features and HR features, and the utilization rate of features in the reconstruction process was improved by using feature concatenation. A joint loss function consisting of the pixel-wise loss, the perceptual loss and the adversarial loss was defined, which performed well in restoring the low-frequency content, the sharp edges and the high-frequency textures of the reconstructed images. The experimental results of datasets Set5, Set14 and BSD100 for the upscaling factor 4 were compared with those of state-of-the-art methods. The proposed method performs well in improving the perceptual quality of the reconstructed images in order to achieve clearer edges and textures, and has better scores in the objective evaluation.
引文
[1]YUE L W,SHEN H F,LI J,et al.Image superresolution:the techniques,applications,and future[J].Signal Processing,2016,128(11):389-408.
[2]陆志芳,钟宝江.基于预测梯度的图像插值算法[J].自动化学报,2018,44(06):1072-1085.LU Zhi-fang,ZHONG Bao-jiang.Image interpolation with predicted gradients[J].Acta Automatica Sinica,2018,44(06):1072-1085.
[3]孙京,袁强强,李冀玮,等.亮度--梯度联合约束的车牌图像超分辨率重建[J].中国图象图形学报,2018,23(06):802-813.SUN Jing,YUAN Qiang-qiang,LI Ji-wei,et al.License plate image super-resolution based on intensity-gradient prior combination[J].Journal of Image and Graphics,2018,23(06):802-813.
[4]YANG J,WRIGHT J,HUANG T,et al.Image superresolution as sparse representation of raw image patches[C]//IEEE Conference on Computer Vision and Pattern Recognition.Anchorage:IEEE,2008:1-8.
[5]孙旭,李晓光,李嘉锋,等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报,2017,43(5):697-709.SUN Xu,LI Xiao-guang,LI Jia-feng,et al.Review on deep learning based image super-resolution restoration algorithms[J].Acta Automatica Sinica,2017,43(5):697-709.
[6]DONG C,CHEN C L,HE K,et al.Image superresolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[7]YANG S,SUN Y,CHEN Y,et al.Structural similarity regularized and sparse coding based superresolution for medical images[J].Biomedical Signal Processing and Control,2012,7(6):579-590.
[8]DONG C,CHEN C L,TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:391-407.
[9]KIM J,LEE J K,LEE K M.Accurate image superresolution using very deep convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:1646-1654.
[10]谢珍珠,吴从中,詹曙.边缘增强深层网络的图像超分辨率重建[J].中国图象图形学报,2018,23(1):114-122.XIE Zhen-zhu,WU Cong-zhong,ZHAN Shu.Image super-resolution reconstruction via deep network based on edge-enhancement[J].Journal of Image and Graphics,2018,23(1):114-122.
[11]李云飞,符冉迪,金炜,等.多通道卷积的图像超分辨率方法[J].中国图象图形学报,2017,22(12):1690-1700.LI Yun-fei,FU Ran-di,JIN Wei,et al.Image superresolution using multi-channel convolution[J].Journal of Image and Graphics,2017,22(12):1690-1700.
[12]陈书贞,解小会,杨郁池,等.利用多尺度卷积神经网络的图像超分辨率算法[J].信号处理,2018,34(09):1033-1044.CHEN Shu-zhen,XIE Xiao-hui,YANG Yu-chi,et al.Image super-resolution algorithm based on multiscale convolution neural network[J].Journal of Signal Processing,2018,34(09):1033-1044.
[13]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:2414-2423.
[14]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:694-711.
[15]LEDIG C,WANG Z,SHI W,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Hawaii:IEEE,2017:105-114.
[16]GOODFELLOW I J,POUGET-ABADIE J,MIRZAM,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.Montreal:ACM,2014:2672-2680.
[17]DENG X.Enhancing image quality via style transfer for single image super-resolution[J].IEEE Signal Processing Letters,2018,25(4):571-575.
[18]SHI W,CABALLERO J,HUSZáR F,et al.Realtime single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:1874-1883.
[19]REED S,AKATA Z,YAN X,et al.Generative adversarial text to image synthesis[C]//International Conference on International Conference on Machine Learning.New York City:ACM,2016:1060-1069.
[20]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA:IEEE,2017:5967-5976.
[21]SAJJADI M S,SCHOLKOPF B,HIRSCH M,et al.EnhanceNet:single image super-resolution through automated texture synthesis[C]//IEEEInternational Conference on Computer Vision.Zurich:IEEE,2017:4501-4510.
[22]LI C,WAND M.Precomputed real-time texture synthesis with Markovian generative adversarial networks[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:702-716.
[23]YANG J,WRIGHT J,HUANG T S,et al.Image super-resolution via sparse representation[J].IEEETransactions on Image Processing,2010,19(11):2861-2873.
[24]TIMOFTE R,LEE K M,WANG X,et al.NTIRE2017 challenge on single image super-resolution:methods and results[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops.Hawaii,USA:IEEE,2017:1110-1121.
[25]HE K,ZHANG X,REN S,et al.Delving deep into rectifiers:surpassing human-level performance on imageNet classification[C]//IEEE International Conference on Computer Vision.Santiago:IEEE,2015:1026-1034.
[26]TIMOFTE R,SMET V D,GOOL L V.A+:adjusted anchored neighborhood regression for fast superresolution[C]//Asian Conference on Computer Vision.Singapore:Springer,2014:111-126.
[2]陆志芳,钟宝江.基于预测梯度的图像插值算法[J].自动化学报,2018,44(06):1072-1085.LU Zhi-fang,ZHONG Bao-jiang.Image interpolation with predicted gradients[J].Acta Automatica Sinica,2018,44(06):1072-1085.
[3]孙京,袁强强,李冀玮,等.亮度--梯度联合约束的车牌图像超分辨率重建[J].中国图象图形学报,2018,23(06):802-813.SUN Jing,YUAN Qiang-qiang,LI Ji-wei,et al.License plate image super-resolution based on intensity-gradient prior combination[J].Journal of Image and Graphics,2018,23(06):802-813.
[4]YANG J,WRIGHT J,HUANG T,et al.Image superresolution as sparse representation of raw image patches[C]//IEEE Conference on Computer Vision and Pattern Recognition.Anchorage:IEEE,2008:1-8.
[5]孙旭,李晓光,李嘉锋,等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报,2017,43(5):697-709.SUN Xu,LI Xiao-guang,LI Jia-feng,et al.Review on deep learning based image super-resolution restoration algorithms[J].Acta Automatica Sinica,2017,43(5):697-709.
[6]DONG C,CHEN C L,HE K,et al.Image superresolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[7]YANG S,SUN Y,CHEN Y,et al.Structural similarity regularized and sparse coding based superresolution for medical images[J].Biomedical Signal Processing and Control,2012,7(6):579-590.
[8]DONG C,CHEN C L,TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:391-407.
[9]KIM J,LEE J K,LEE K M.Accurate image superresolution using very deep convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:1646-1654.
[10]谢珍珠,吴从中,詹曙.边缘增强深层网络的图像超分辨率重建[J].中国图象图形学报,2018,23(1):114-122.XIE Zhen-zhu,WU Cong-zhong,ZHAN Shu.Image super-resolution reconstruction via deep network based on edge-enhancement[J].Journal of Image and Graphics,2018,23(1):114-122.
[11]李云飞,符冉迪,金炜,等.多通道卷积的图像超分辨率方法[J].中国图象图形学报,2017,22(12):1690-1700.LI Yun-fei,FU Ran-di,JIN Wei,et al.Image superresolution using multi-channel convolution[J].Journal of Image and Graphics,2017,22(12):1690-1700.
[12]陈书贞,解小会,杨郁池,等.利用多尺度卷积神经网络的图像超分辨率算法[J].信号处理,2018,34(09):1033-1044.CHEN Shu-zhen,XIE Xiao-hui,YANG Yu-chi,et al.Image super-resolution algorithm based on multiscale convolution neural network[J].Journal of Signal Processing,2018,34(09):1033-1044.
[13]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:2414-2423.
[14]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:694-711.
[15]LEDIG C,WANG Z,SHI W,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Hawaii:IEEE,2017:105-114.
[16]GOODFELLOW I J,POUGET-ABADIE J,MIRZAM,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.Montreal:ACM,2014:2672-2680.
[17]DENG X.Enhancing image quality via style transfer for single image super-resolution[J].IEEE Signal Processing Letters,2018,25(4):571-575.
[18]SHI W,CABALLERO J,HUSZáR F,et al.Realtime single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:1874-1883.
[19]REED S,AKATA Z,YAN X,et al.Generative adversarial text to image synthesis[C]//International Conference on International Conference on Machine Learning.New York City:ACM,2016:1060-1069.
[20]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Hawaii,USA:IEEE,2017:5967-5976.
[21]SAJJADI M S,SCHOLKOPF B,HIRSCH M,et al.EnhanceNet:single image super-resolution through automated texture synthesis[C]//IEEEInternational Conference on Computer Vision.Zurich:IEEE,2017:4501-4510.
[22]LI C,WAND M.Precomputed real-time texture synthesis with Markovian generative adversarial networks[C]//European Conference on Computer Vision.Amsterdam:Springer,2016:702-716.
[23]YANG J,WRIGHT J,HUANG T S,et al.Image super-resolution via sparse representation[J].IEEETransactions on Image Processing,2010,19(11):2861-2873.
[24]TIMOFTE R,LEE K M,WANG X,et al.NTIRE2017 challenge on single image super-resolution:methods and results[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops.Hawaii,USA:IEEE,2017:1110-1121.
[25]HE K,ZHANG X,REN S,et al.Delving deep into rectifiers:surpassing human-level performance on imageNet classification[C]//IEEE International Conference on Computer Vision.Santiago:IEEE,2015:1026-1034.
[26]TIMOFTE R,SMET V D,GOOL L V.A+:adjusted anchored neighborhood regression for fast superresolution[C]//Asian Conference on Computer Vision.Singapore:Springer,2014:111-126.