结构约束的全局目标函数视频修复方法
|
摘要
针对一致性修复方法仅利用视频的颜色和运动特征来优化目标函数,导致无法准确预测未知区域像素值的问题,将视频的结构信息引入到目标函数中,提出结构约束的视频修复方法.首先计算破损视频的时空金字塔,在从上到下的第1层金字塔中先修复破损部分边界上的像素;然后采用加权平均方式完成破损像素的重建,对权值重新定义,再更新边界,修复新边界上的像素,完成第1层金字塔的修复;最后用相同方法依次修复各层金字塔,直到所有破损像素都被修复.采用不同方法对不同复杂场景视频进行修复并用峰值信噪比和结构相似度进行评价,实验结果表明该方法取得了更好的视觉效果,峰值信噪比平均提高1～3 dB.
To optimize the objective function by using only the color and motion features of video, the consistent completion method was unable to predict the pixel value of unknown region accurately. In this paper,the structure information of video is introduced into the objective function. Firstly, calculate the space-time pyramid of damaged video. In the first layer of the pyramid from top to bottom, repair the pixels on the boundary of the broken part. The broken pixel can be inpainted by a weighted average method, update the boundary and inpaint the pixels on the new boundary by redefining weight, complete the first pyramid. Finally, the same method is used to inpaint each layer pyramid in turn, until all damaged pixels are completed.The various methods are used to repair the videos of different complex scenes and evaluate them with peak signal to noise ratio and structural similarity. Experimental results show that the proposed method achieves better visual effect, and the peak signal to noise ratio(PSNR) increases by 1 to 3 dB on average.
To optimize the objective function by using only the color and motion features of video, the consistent completion method was unable to predict the pixel value of unknown region accurately. In this paper,the structure information of video is introduced into the objective function. Firstly, calculate the space-time pyramid of damaged video. In the first layer of the pyramid from top to bottom, repair the pixels on the boundary of the broken part. The broken pixel can be inpainted by a weighted average method, update the boundary and inpaint the pixels on the new boundary by redefining weight, complete the first pyramid. Finally, the same method is used to inpaint each layer pyramid in turn, until all damaged pixels are completed.The various methods are used to repair the videos of different complex scenes and evaluate them with peak signal to noise ratio and structural similarity. Experimental results show that the proposed method achieves better visual effect, and the peak signal to noise ratio(PSNR) increases by 1 to 3 dB on average.
引文
[1]Pawar A J, Phatale A P. A comparative study of effective way to modifymovingobjectinvideo:usingdifferentinpainting methods[C]//Proceedings of the 10th International Conference on Intelligent Systems and Control. Los Alamitos:IEEE Computer Society Press, 2016:1-5
[2]Patwardhan K A, Sapiro G, Bertalmio M. Video inpainting of occluding and occluded objects[C]//Proceedings of the IEEE International Conference on Image Processing. Los Alamitos:IEEE Computer Society Press, 2005, 2:69-72
[3]Jia Y T, Hu S M, Martin R R. Video completion using tracking and fragment merging[J]. The Visual Computer, 2005, 21(8-10):601-610
[4]Patwardhan K A, Sapiro G, Bertalmio M. Video inpainting under constrained camera motion[J]. IEEE Transactions on Image Processing, 2007, 16(2):545-553
[5]AhireBA,DeshpandeNA.Videoinpaintingofobjectsusing modified patch based image inpainting algorithm[C]//Proceedings of Conference on IT in Business, Industry and Government. Los Alamitos:IEEE Computer Society Press, 2014:1-5
[6]EbdelliM,GuillemotC,LeMeurO.Examplar-basedvideo inpaintingwithmotion-compensatedneighborembedding[C]//Proceedingsofthe19thIEEEInternationalConferenceon Image Processing. Los Alamitos:IEEE Computer Society Press,2012:1737-1740
[7]Kokaram A C, Morris R D, Fitzgerald W J, et al. Interpolation of missing data in image sequences[J]. IEEE Transactions on Image Processing, 1995, 4(11):1509-1519
[8]WexlerY,ShechtmanE,IraniM.Space-timecompletionof video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3):463-476
[9]XiaoChunxia,LiuShu,LinChengchun,etal.Aglobal space-timeoptimizationframeworkforvideocompletion[J].JournalofComputer-AidedDesign&ComputerGraphics,2008, 20(9):1204-1211(in Chinese)(肖春霞,刘舒,林成春,等.基于时空全局优化的视频修复[J].计算机辅助设计与图形学学报, 2008, 20(9):1204-1211)
[10]NewsonA,AlmansaA,FradetM,etal.Videoinpaintingof complex scenes[J]. SIAM Journal on Imaging Sciences, 2014,7(4):1993-2019
[11]ChoTS,ZitnickCL,JoshiN,etal.Imagerestorationby matchinggradientdistributions[J].IEEETransactionsonPattern Analysis and Machine Intelligence, 2012, 34(4):683-694
[12]Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//ProceedingsoftheACMSIGGRAPHConferenceonComputer Graphics. New York:ACM Press, 2000:417-424
[13]Arya S, Mount D M. Approximate nearest neighbor queries in fixeddimensions[C]//Proceedingsofthe4thAnnual ACM-SIAMSymposiumonDiscreteAlgorithms.NewYork:ACM Press, 1993:271-280
[14]Xie Yaohua, Tang Xiaoan, Sun Maoyin. Image pyramid structures without truncation and data expansion[J]. Journal of Image and Graphics, 2009, 14(6):1070-1074(in Chinese)(谢耀华,汤晓安,孙茂印.不产生精度截断及数据膨胀的图像金字塔[J].中国图象图形学报, 2009, 14(6):1070-1074)
[15]Somkantha K, Theera-Umpon N, Auephanwiriyakul S. Boundary detection in medical images using edge following algorithm based on intensity gradient and texture gradient features[J]. IEEE Transactions on Biomedical Engineering, 2011, 58(3):567-573
[16]Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment:fromerrorvisibilitytostructuralsimilarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612
[2]Patwardhan K A, Sapiro G, Bertalmio M. Video inpainting of occluding and occluded objects[C]//Proceedings of the IEEE International Conference on Image Processing. Los Alamitos:IEEE Computer Society Press, 2005, 2:69-72
[3]Jia Y T, Hu S M, Martin R R. Video completion using tracking and fragment merging[J]. The Visual Computer, 2005, 21(8-10):601-610
[4]Patwardhan K A, Sapiro G, Bertalmio M. Video inpainting under constrained camera motion[J]. IEEE Transactions on Image Processing, 2007, 16(2):545-553
[5]AhireBA,DeshpandeNA.Videoinpaintingofobjectsusing modified patch based image inpainting algorithm[C]//Proceedings of Conference on IT in Business, Industry and Government. Los Alamitos:IEEE Computer Society Press, 2014:1-5
[6]EbdelliM,GuillemotC,LeMeurO.Examplar-basedvideo inpaintingwithmotion-compensatedneighborembedding[C]//Proceedingsofthe19thIEEEInternationalConferenceon Image Processing. Los Alamitos:IEEE Computer Society Press,2012:1737-1740
[7]Kokaram A C, Morris R D, Fitzgerald W J, et al. Interpolation of missing data in image sequences[J]. IEEE Transactions on Image Processing, 1995, 4(11):1509-1519
[8]WexlerY,ShechtmanE,IraniM.Space-timecompletionof video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3):463-476
[9]XiaoChunxia,LiuShu,LinChengchun,etal.Aglobal space-timeoptimizationframeworkforvideocompletion[J].JournalofComputer-AidedDesign&ComputerGraphics,2008, 20(9):1204-1211(in Chinese)(肖春霞,刘舒,林成春,等.基于时空全局优化的视频修复[J].计算机辅助设计与图形学学报, 2008, 20(9):1204-1211)
[10]NewsonA,AlmansaA,FradetM,etal.Videoinpaintingof complex scenes[J]. SIAM Journal on Imaging Sciences, 2014,7(4):1993-2019
[11]ChoTS,ZitnickCL,JoshiN,etal.Imagerestorationby matchinggradientdistributions[J].IEEETransactionsonPattern Analysis and Machine Intelligence, 2012, 34(4):683-694
[12]Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//ProceedingsoftheACMSIGGRAPHConferenceonComputer Graphics. New York:ACM Press, 2000:417-424
[13]Arya S, Mount D M. Approximate nearest neighbor queries in fixeddimensions[C]//Proceedingsofthe4thAnnual ACM-SIAMSymposiumonDiscreteAlgorithms.NewYork:ACM Press, 1993:271-280
[14]Xie Yaohua, Tang Xiaoan, Sun Maoyin. Image pyramid structures without truncation and data expansion[J]. Journal of Image and Graphics, 2009, 14(6):1070-1074(in Chinese)(谢耀华,汤晓安,孙茂印.不产生精度截断及数据膨胀的图像金字塔[J].中国图象图形学报, 2009, 14(6):1070-1074)
[15]Somkantha K, Theera-Umpon N, Auephanwiriyakul S. Boundary detection in medical images using edge following algorithm based on intensity gradient and texture gradient features[J]. IEEE Transactions on Biomedical Engineering, 2011, 58(3):567-573
[16]Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment:fromerrorvisibilitytostructuralsimilarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612