视频序列超分辨率重构技术综述
|
摘要
视频序列超分辨率重构技术可以利用来自一个或多个低分辨率视频序列信息重构一个具有高空间分辨率或高时空分辨率的视频序列。本文从视频序列的空间超分辨率、时间超分辨率以及时空超分辨率重构三个方面对视频序列超分辨率重构技术进行了详细的阐述,对视频序列超分辨率重构技术中的关键问题和应用前景进行了分析与展望。
The video sequence super-resolution reconstruction technique can reconstruct a video sequence with high spatial resolution or high temporal resolution from one or more low resolution video sequence information.In this paper, the concept of super-resolution reconstruction is introduced. The super-resolution reconstruction technique of video sequence is described in detail from three aspects: spatial super-resolution, temporal super-resolution and spatial-temporal super-resolution. Sequence super-resolution reconstruction technology in the key issues and application prospects were analyzed and prospects.
The video sequence super-resolution reconstruction technique can reconstruct a video sequence with high spatial resolution or high temporal resolution from one or more low resolution video sequence information.In this paper, the concept of super-resolution reconstruction is introduced. The super-resolution reconstruction technique of video sequence is described in detail from three aspects: spatial super-resolution, temporal super-resolution and spatial-temporal super-resolution. Sequence super-resolution reconstruction technology in the key issues and application prospects were analyzed and prospects.
引文
[1]Tsai R Y,Huang T S.Multi-frame image restoration and registration[J].Advances in Computer Vision and Image Processing,1984(1):317-339.
[2]Sanchez-Beato A,Pajares G.Noniterative InterpolationBased Super-Resolution Minimizing Aliasing in the Reconstructed Image[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2008,17(10):1817-1826.
[3]Nasonov A V,Krylov A S.Fast Super-Resolution Using Weighted Median Filtering[C]//International Conference on Pattern Recognition.IEEE Computer Society,2010:2230-2233.
[4]Lin S C,Chen C T.Reconstructing Vehicle License Plate Image from Low Resolution Images using Nonuniform Interpolation Method[J].International Journal of Image Processing,2007,1(2):21-28.
[5]Tsai R Y,Huang T S.Multi-frame image restoration and registration[J].Advances in Computer Vision and Image Processing,1984(1):317-339.
[6]Kim S P,Su W Y.Recursive high-resolution reconstruction of blurred multiframe images[J].IEEE Transactions on Image Processing,1993,2(4):534-9.
[7]Ur H,Gross D.Improved resolution form subpixel shifted pictures[J].CVGIP:Graphical Models and Image Processing,1992,54(2):181-186.
[8]Nguyen N,Milanfar P.An efficient wavelet-based algorithm for image superresolution[J].2000,2:351-354 vol.2.
[9]Ji H,Ller C.Robust Wavelet-Based Super-Resolution Reconstruction:Theory and Algorithm[J].Pattern Analysis&Machine Intelligence IEEE Transactions on,2008,31(4):649-660.
[10]Ozkan M K,Tekalp A M,Sezan M I.POCS-based restoration of space-varying blurred images[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,1994,3(4):450-454.
[11]Stark H,Oskoui P.High-resolution image recovery from image-plane arrays,using convex projections[J].Journal of the Optical Society of America A,1989,6(11):1715.
[12]Banham M R,Katsaggelos A K.Digital image restoration.IEEE Signal Processing Magazine,1997,14(2):24-41.
[13]Tekalp A M,Ozkan M K,Sezan M I.High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration[J].1992,3:169-172.
[14]Eren P E,Sezan M I,Tekalp A M.Robust,object-based high-resolution image reconstruction from low-resolution video[J].IEEE Transactions on Image Processing,1997,6(10):1446-51.
[15]Patti A J,Altunbasak Y.Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants[J].IEEE Transactions on Image Processing,2001,10(1):179-86.
[16]A.N.Tikhonov,Regularization of incorrectly posed problems,Soviet Mathematical Doklady 4(1963)1624–1627.
[17]Marquina,S.J.Osher,Image super-resolution by TV-regularization and Bregman iteration,Journal of Scientific Computing 37(2008)367–382.
[18]Farsiu S,Robinson M D,Elad M,et al.Fast and Robust Multiframe Super Resolution[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2004,13(10):1327-1344.
[19]Chantas G,Galatsanos N P,Molina R,et al.Variational Bayesian image restoration based on a product of t-distributions image prior[J].IEEE Transactions on Image Processing,2008,17(10):1795.
[20]Schultz R R,Stevenson R L.Video resolution enhancement[J].Proc Spie,1995:23-34.
[21]Schultz R R,Stevenson R L.Motion-compensated scan conversion of interlaced video sequences[J].Proceedings of SPIE-The International Society for Optical Engineering,1996.
[22]Elad M,Feuer A.Restoration of single super-resolution image from several blurred[J].IEEE Transactions on Image Processing,1997.
[23]Elad M,Feuer A.Super-resolution reconstruction of an image[C]//Electrical and Electronics Engineers in Israel,1996.Nineteenth Convention of.IEEE,1996:391-394.
[24]Irani M,Peleg S.Improving resolution by image registration[J].Cvgip Graphical Models&Image Processing,1991,53(3):231-239.
[25]Irani M,Peleg S.Motion Analysis for Image Enhancement:Resolution,Occlusion,and Transparency[J].Journal of Visual Communication&Image Representation,1993,4(4):324-335.
[26]Mann S,Picard R W.Virtual bellows:constructing high quality stills from video[C]//Image Processing,1994.Proceedings.ICIP-94.IEEE International Conference.IEEE Xplore,1994:363-367 vol.1.
[27]覃凤清,陈为龙,余永松,等.视频超分辨率重建技术研究[C]//2007中国西部青年通信学术会议.2007.QIN,CHEN,YU.Research on Video Super-Resolution Reconstruction Technology[C]//2007 China Youth Academic Conference in Western China.2007.
[28]Elad M.On the origin of the bilateral filter and ways to improve it[J].IEEE Transactions on Image Processing,2002,11(10):1141.
[29]Pentland A,Horowitz B.A practical approach to fractalbased image compression[C]//Data Compression Conference.IEEE,1991:176-185.
[30]Baker S,Kanade T.Limits on Super-Resolution and How to Break Them[J].Pattern Analysis&Machine Intelligence IEEE Transactions on,2002,24(9):1167-1183.
[31]Freeman W T,Pasztor E C,Carmichael O T.Learning Low-Level Vision[J].International Journal of Computer Vision,2000,40(1):25-47.
[32]Freeman W T,Jones T R,Pasztor E C.Example-Based Super-Resolution[J].Computer Graphics&Applications IEEE,2002,22(2):56-65.
[33]Wang Q,Tang X,Shum H.Patch Based Blind Image Super Resolution[C]//Tenth IEEE International Conference on Computer Vision.IEEE,2005:709-716 Vol.1.
[34]Qiu G.Interresolution Look-up Table for Improved Spatial Magnification of Image[J].Journal of Visual Communication&Image Representation,2000,11(4):360-373.
[35]Yang J,Wright J,Huang T,et al.Image super-resolution as sparse representation of raw image patches[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.DBLP,2008:1-8.
[36]Adler A,Helor Y,Elad M.A Shrinkage Learning Approach for Single Image Super-Resolution with Overcomplete Representations[M]//Computer Vision–ECCV 2010.Springer Berlin Heidelberg,2010:622-635.
[37]Zhang Y,Zhao D,Ma S,et al.A motion-aligned autoregressive model for frame rate up conversion[J].IEEE Transactions on Image Processing,2010,19(5):1248-1258.
[38]Huang A M,Nguyen T.Correlation-based motion vector processing for motion compensated frame interpolation[J].2008:1244-1247.
[39]Vinh T Q,Kim Y C,Hong S H.Frame rate up-conversion using forward-backward jointing motion estimation and spatio-temporal motion vector smoothing[C]//International Conference on Computer Engineering&Systems.IEEE Xplore,2010:605-609.
[40]Yoo D G,Kang S J,Lee S K,et al.Motion Vector Smoothing for Motion-Compensated Frame Rate Up-Conversion[J].???????isocc,2010:358-359.
[41]Shimano M,Okabe T,Sato I,et al.Video Temporal Super-Resolution Based on Self-similarity[J].Ieice Transactions on Information&Systems,2010,94(8):1376-1386.
[42]Shechtman E,Caspi Y,Irani M.Space-time super-resolution[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2005,27(4):531.
[43]陈为龙,郭黎,雷刚,等.多视频超分辨率的时间重建倍数[J].光学精密工程,2014,22(9):2518-2527.CHEN,GUO,LEI.Time Reconstruction of Multi-Video Super-resolution[J].Editorial Office of Optics and Precision Engineering,2014,22(9):2518-2527.
[44]Mudenagudi U,Banerjee S,Kalra P.On Improving SpaceTime Super Resolution Using a Small Set of Video Inputs[C]//Computer Vision,Graphics&Image Processing,2008.ICVGIP'08.Sixth Indian Conference on.IEEE,2008:320-327.
[45]宋海英,何小海,陈为龙,等.多视频的时空超分辨率重建算法[J].北京邮电大学学报,2011,34(4):85-88.SONG,HE,CHEN.A Space-Time Super-Resolution Reconstruction Algorithm of Video Sequences[J].Journal of Beijing University of Posts and Telecommunications,2011,34(4):85-88.
[46]Shahar O,Faktor A,Irani M.Space-time super-resolution from a single video[C]//Computer Vision and Pattern Recognition.IEEE,2011:3353-3360.
[47]Glasner D,Bagon S,Irani M.Super-resolution from a single image[C]//IEEE,International Conference on Computer Vision.IEEE Xplore,2009:349-356.
[48]Takeda H,Milanfar P,Protter M,et al.Super-resolution without explicit subpixel motion estimation[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2009,18(9):1958.
[49]Hazen D,Puri R,Ramchandran K.Multi-camera Video Resolution Enhancement by Fusion of Spatial Disparity and Temporal Motion fields[C]//IEEE International Conference on Computer Vision Systems.IEEE Xplore,2006:38-38.
[50]Zhao W Y,Sawhney H S.Is Super-Resolution with Optical Flow Feasible?[M]//Computer Vision—ECCV 2002.Springer Berlin Heidelberg,2002:599-613.
[51]任忠良.一种基于SIFT特征的快速图像匹配算法[J].软件,2015,36(6):53-57.REN.A Fast Image Matching Algorithm Based on SIFT Features[J].COMPUTER ENGINEERING&SOFTWARE,2015,36(6):53-57.
[52]黄全亮,孙坤,刘水清.超分辨率技术对基于特征图像配准算法的影响[J].新型工业化,2011,1(4):1-7.HUANG,SUN,LIU.Infulence of using super-resolution in feature-based image registration[J].New Industrialization Straregy,2011,1(4):1-7.
[53]王新年,张涛,张毓良.改进的基于SIFT和RANSAC的图像配准方法[J].新型工业化,2011,1(6):72-77.WANG,ZHANGTao,ZHANG.An improved image registration method based on SIFT and RANSAC[J].New Industrialization Straregy,2011,1(6):72-77.
[2]Sanchez-Beato A,Pajares G.Noniterative InterpolationBased Super-Resolution Minimizing Aliasing in the Reconstructed Image[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2008,17(10):1817-1826.
[3]Nasonov A V,Krylov A S.Fast Super-Resolution Using Weighted Median Filtering[C]//International Conference on Pattern Recognition.IEEE Computer Society,2010:2230-2233.
[4]Lin S C,Chen C T.Reconstructing Vehicle License Plate Image from Low Resolution Images using Nonuniform Interpolation Method[J].International Journal of Image Processing,2007,1(2):21-28.
[5]Tsai R Y,Huang T S.Multi-frame image restoration and registration[J].Advances in Computer Vision and Image Processing,1984(1):317-339.
[6]Kim S P,Su W Y.Recursive high-resolution reconstruction of blurred multiframe images[J].IEEE Transactions on Image Processing,1993,2(4):534-9.
[7]Ur H,Gross D.Improved resolution form subpixel shifted pictures[J].CVGIP:Graphical Models and Image Processing,1992,54(2):181-186.
[8]Nguyen N,Milanfar P.An efficient wavelet-based algorithm for image superresolution[J].2000,2:351-354 vol.2.
[9]Ji H,Ller C.Robust Wavelet-Based Super-Resolution Reconstruction:Theory and Algorithm[J].Pattern Analysis&Machine Intelligence IEEE Transactions on,2008,31(4):649-660.
[10]Ozkan M K,Tekalp A M,Sezan M I.POCS-based restoration of space-varying blurred images[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,1994,3(4):450-454.
[11]Stark H,Oskoui P.High-resolution image recovery from image-plane arrays,using convex projections[J].Journal of the Optical Society of America A,1989,6(11):1715.
[12]Banham M R,Katsaggelos A K.Digital image restoration.IEEE Signal Processing Magazine,1997,14(2):24-41.
[13]Tekalp A M,Ozkan M K,Sezan M I.High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration[J].1992,3:169-172.
[14]Eren P E,Sezan M I,Tekalp A M.Robust,object-based high-resolution image reconstruction from low-resolution video[J].IEEE Transactions on Image Processing,1997,6(10):1446-51.
[15]Patti A J,Altunbasak Y.Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants[J].IEEE Transactions on Image Processing,2001,10(1):179-86.
[16]A.N.Tikhonov,Regularization of incorrectly posed problems,Soviet Mathematical Doklady 4(1963)1624–1627.
[17]Marquina,S.J.Osher,Image super-resolution by TV-regularization and Bregman iteration,Journal of Scientific Computing 37(2008)367–382.
[18]Farsiu S,Robinson M D,Elad M,et al.Fast and Robust Multiframe Super Resolution[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2004,13(10):1327-1344.
[19]Chantas G,Galatsanos N P,Molina R,et al.Variational Bayesian image restoration based on a product of t-distributions image prior[J].IEEE Transactions on Image Processing,2008,17(10):1795.
[20]Schultz R R,Stevenson R L.Video resolution enhancement[J].Proc Spie,1995:23-34.
[21]Schultz R R,Stevenson R L.Motion-compensated scan conversion of interlaced video sequences[J].Proceedings of SPIE-The International Society for Optical Engineering,1996.
[22]Elad M,Feuer A.Restoration of single super-resolution image from several blurred[J].IEEE Transactions on Image Processing,1997.
[23]Elad M,Feuer A.Super-resolution reconstruction of an image[C]//Electrical and Electronics Engineers in Israel,1996.Nineteenth Convention of.IEEE,1996:391-394.
[24]Irani M,Peleg S.Improving resolution by image registration[J].Cvgip Graphical Models&Image Processing,1991,53(3):231-239.
[25]Irani M,Peleg S.Motion Analysis for Image Enhancement:Resolution,Occlusion,and Transparency[J].Journal of Visual Communication&Image Representation,1993,4(4):324-335.
[26]Mann S,Picard R W.Virtual bellows:constructing high quality stills from video[C]//Image Processing,1994.Proceedings.ICIP-94.IEEE International Conference.IEEE Xplore,1994:363-367 vol.1.
[27]覃凤清,陈为龙,余永松,等.视频超分辨率重建技术研究[C]//2007中国西部青年通信学术会议.2007.QIN,CHEN,YU.Research on Video Super-Resolution Reconstruction Technology[C]//2007 China Youth Academic Conference in Western China.2007.
[28]Elad M.On the origin of the bilateral filter and ways to improve it[J].IEEE Transactions on Image Processing,2002,11(10):1141.
[29]Pentland A,Horowitz B.A practical approach to fractalbased image compression[C]//Data Compression Conference.IEEE,1991:176-185.
[30]Baker S,Kanade T.Limits on Super-Resolution and How to Break Them[J].Pattern Analysis&Machine Intelligence IEEE Transactions on,2002,24(9):1167-1183.
[31]Freeman W T,Pasztor E C,Carmichael O T.Learning Low-Level Vision[J].International Journal of Computer Vision,2000,40(1):25-47.
[32]Freeman W T,Jones T R,Pasztor E C.Example-Based Super-Resolution[J].Computer Graphics&Applications IEEE,2002,22(2):56-65.
[33]Wang Q,Tang X,Shum H.Patch Based Blind Image Super Resolution[C]//Tenth IEEE International Conference on Computer Vision.IEEE,2005:709-716 Vol.1.
[34]Qiu G.Interresolution Look-up Table for Improved Spatial Magnification of Image[J].Journal of Visual Communication&Image Representation,2000,11(4):360-373.
[35]Yang J,Wright J,Huang T,et al.Image super-resolution as sparse representation of raw image patches[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.DBLP,2008:1-8.
[36]Adler A,Helor Y,Elad M.A Shrinkage Learning Approach for Single Image Super-Resolution with Overcomplete Representations[M]//Computer Vision–ECCV 2010.Springer Berlin Heidelberg,2010:622-635.
[37]Zhang Y,Zhao D,Ma S,et al.A motion-aligned autoregressive model for frame rate up conversion[J].IEEE Transactions on Image Processing,2010,19(5):1248-1258.
[38]Huang A M,Nguyen T.Correlation-based motion vector processing for motion compensated frame interpolation[J].2008:1244-1247.
[39]Vinh T Q,Kim Y C,Hong S H.Frame rate up-conversion using forward-backward jointing motion estimation and spatio-temporal motion vector smoothing[C]//International Conference on Computer Engineering&Systems.IEEE Xplore,2010:605-609.
[40]Yoo D G,Kang S J,Lee S K,et al.Motion Vector Smoothing for Motion-Compensated Frame Rate Up-Conversion[J].???????isocc,2010:358-359.
[41]Shimano M,Okabe T,Sato I,et al.Video Temporal Super-Resolution Based on Self-similarity[J].Ieice Transactions on Information&Systems,2010,94(8):1376-1386.
[42]Shechtman E,Caspi Y,Irani M.Space-time super-resolution[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2005,27(4):531.
[43]陈为龙,郭黎,雷刚,等.多视频超分辨率的时间重建倍数[J].光学精密工程,2014,22(9):2518-2527.CHEN,GUO,LEI.Time Reconstruction of Multi-Video Super-resolution[J].Editorial Office of Optics and Precision Engineering,2014,22(9):2518-2527.
[44]Mudenagudi U,Banerjee S,Kalra P.On Improving SpaceTime Super Resolution Using a Small Set of Video Inputs[C]//Computer Vision,Graphics&Image Processing,2008.ICVGIP'08.Sixth Indian Conference on.IEEE,2008:320-327.
[45]宋海英,何小海,陈为龙,等.多视频的时空超分辨率重建算法[J].北京邮电大学学报,2011,34(4):85-88.SONG,HE,CHEN.A Space-Time Super-Resolution Reconstruction Algorithm of Video Sequences[J].Journal of Beijing University of Posts and Telecommunications,2011,34(4):85-88.
[46]Shahar O,Faktor A,Irani M.Space-time super-resolution from a single video[C]//Computer Vision and Pattern Recognition.IEEE,2011:3353-3360.
[47]Glasner D,Bagon S,Irani M.Super-resolution from a single image[C]//IEEE,International Conference on Computer Vision.IEEE Xplore,2009:349-356.
[48]Takeda H,Milanfar P,Protter M,et al.Super-resolution without explicit subpixel motion estimation[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2009,18(9):1958.
[49]Hazen D,Puri R,Ramchandran K.Multi-camera Video Resolution Enhancement by Fusion of Spatial Disparity and Temporal Motion fields[C]//IEEE International Conference on Computer Vision Systems.IEEE Xplore,2006:38-38.
[50]Zhao W Y,Sawhney H S.Is Super-Resolution with Optical Flow Feasible?[M]//Computer Vision—ECCV 2002.Springer Berlin Heidelberg,2002:599-613.
[51]任忠良.一种基于SIFT特征的快速图像匹配算法[J].软件,2015,36(6):53-57.REN.A Fast Image Matching Algorithm Based on SIFT Features[J].COMPUTER ENGINEERING&SOFTWARE,2015,36(6):53-57.
[52]黄全亮,孙坤,刘水清.超分辨率技术对基于特征图像配准算法的影响[J].新型工业化,2011,1(4):1-7.HUANG,SUN,LIU.Infulence of using super-resolution in feature-based image registration[J].New Industrialization Straregy,2011,1(4):1-7.
[53]王新年,张涛,张毓良.改进的基于SIFT和RANSAC的图像配准方法[J].新型工业化,2011,1(6):72-77.WANG,ZHANGTao,ZHANG.An improved image registration method based on SIFT and RANSAC[J].New Industrialization Straregy,2011,1(6):72-77.