No-reference synthetic image quality assessment with convolutional neural network and local image saliency
|
摘要
Depth-image-based rendering(DIBR) is widely used in 3 DTV, free-viewpoint video, and interactive 3 D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2 D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2 D image quality metrics and state-of-the-art DIBR-related metrics.
Depth-image-based rendering(DIBR) is widely used in 3 DTV, free-viewpoint video, and interactive 3 D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2 D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2 D image quality metrics and state-of-the-art DIBR-related metrics.
Depth-image-based rendering(DIBR) is widely used in 3 DTV, free-viewpoint video, and interactive 3 D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2 D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2 D image quality metrics and state-of-the-art DIBR-related metrics.
引文
[1]Fehn,C.Depth-image-based rendering(DIBR),compression,and transmission for a new approach on 3D-TV.In:Proceedings of the SPIE 5291,Stereoscopic Displays and Virtual Reality Systems XI,93-104,2004.
[2]Smolic,A.3D video and free viewpoint video:From capture to display.Pattern Recognition Vol.44,No.9,1958-1968,2011.
[3]Smolic,A.;Mueller,K.;Merkle,P.;Kauff,P.;Wiegand,T.An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution.In:Proceedings of the Picture Coding Symposium,1-4,2009.
[4]Wang,X.;Liang,X.;Yang,B.;Li,F.W.Scalable remote rendering using synthesized image quality assessment.IEEE Access Vol.6,36595-36610,2018.
[5]Mark,W.Post-rendering 3D image warping:Visibility,reconstruction,and performance for depth-image warping.Technical Report.Chapel Hill,NC,USA,1999.
[6]Zhou,Y.;Li,L.;Gu,K.;Fang,Y.;Lin,W.Quality assessment of 3D synthesized images via disoccluded region discovery.In:Proceedings of the IEEEInternational Conference on Image Processing,1012-1016,2016.
[7]Battisti,F.;Bosc,E.;Carli,M.;Le Callet,P.;Perugia,S.Objective image quality assessment of 3D synthesized views.Signal Processing:Image Communication Vol.30,78-88,2015.
[8]Gu,K.;Jakhetiya,V.;Qiao,J.F.;Li,X.;Lin,W.;Thalmann,D.Model-based referenceless quality metric of 3D synthesized images using local image description.IEEE Transactions on Image Processing Vol.27,No.1,394-405,2018.
[9]Tian,S.;Zhang,L.;Morin,L.;D′eforges,O.NIQSV+:A no-reference synthesized view quality assessment metric.IEEE Transactions on Image Processing Vol.27,No.4,1652-1664,2018.
[10]Bosc,E.;Pepion,R.;Le Callet,P.;Koppel,M.;NdjikiNya,P.;Pressigout,M.;Morin,L.Towards a new quality metric for 3-D synthesized view assessment.IEEE Journal of Selected Topics in Signal Processing Vol.5,No.7,1332-1343,2011.
[11]Wang,Z.;Bovik,A.C.;Sheikh,H.R.;Simoncelli,E.P.Image quality assessment:From error visibility to structural similarity.IEEE Transactions on Image Processing Vol.13,No.4,600-612,2004.
[12]Sharifi,K.;Leon-Garcia,A.Estimation of shape parameter for generalized Gaussian distributions in subband decompositions of video.IEEE Transactions on Circuits and Systems for Video Technology Vol.5,No.1,52-56,1995.
[13]Mittal,A.;Moorthy,A.K.;Bovik,A.C.No-reference image quality assessment in the spatial domain.IEEETransactions on Image Processing Vol.21,No.12,4695-4708,2012.
[14]Mittal,A.;Soundararajan,R.;Bovik,A.C.Making a“completely blind”image quality analyzer.IEEE Signal Processing Letters Vol.20,No.3,209-212,2013.
[15]Kang,L.;Ye,P.;Li,Y.;Doermann,D.Convolutional neural networks for no-reference image quality assessment.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,1733-1740,2014.
[16]Bosse,S.;Maniry,D.;Wiegand,T.;Samek,W.Adeep neural network for image quality assessment.In:Proceedings of the IEEE International Conference on Image Processing,3773-3777,2016.
[17]Bare,B.;Li,K.;Yan,B.An accurate deep convolutional neural networks model for no-reference image quality assessment.In:Proceedings of the IEEEInternational Conference on Multimedia and Expo,1356-1361,2017.
[18]Kim,J.;Nguyen,A.;Ahn,S.;Luo,C.;Lee,S.Multiple level feature-based universal blind image quality assessment model.In:Proceedings of the 25th IEEE International Conference on Image Processing,291-295,2018.
[19]Lin,K.-Y.;Wang,G.Hallucinated-IQA:No-reference image quality assessment via adversarial learning.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,732-741,2018.
[20]Zhang,F.-L.;Wu,X.;Li,R.-L.;Wang,J.;Zheng,Z.-H.;Hu,S.-M.Detecting and removing visual distractors for video aesthetic enhancement.IEEE Transactions on Multimedia Vol.20,No.8,1987-1999,2018.
[21]Sheikh,H.R.;Wang,Z.;Cormack,L.;Bovik,A.C.Live image quality assessment database release 2(2005).2016.Available at http://live.ece.utexas.edu/research/quality.
[22]Ponomarenko,N.;Lukin,V.;Zelensky,A.;Egiazarian,K.;Carli,M.;Battisti,F.TID2008-a database for evaluation of full-reference visual quality assessment metrics.Advances of Modern Radioelectronics Vol.10,No.4,30-45,2009.
[23]Ponomarenko,N.;Jin,L.;Ieremeiev,O.;Lukin,V.;Egiazarian,K.;Astola,J.;Vozel,B.;Chehdi,K.;Carli,M.;Battisti,F.;Jay Kuo,C.-C.Image database TID2013:Peculiarities,results and perspectives.Signal Processing:Image Communication Vol.30,57-77,2015.
[24]Conze,P.-H.;Robert,P.;Morin,L.Objective view synthesis quality assessment.In:Proceedings of the SPIE 8288,Stereoscopic Displays and Applications XXIII,82881M,2012.
[25]Sandi′c Stankovi′c,D.;Kukolj,D.;Le Callet,P.DIBR synthesized image quality assessment based on morphological wavelets.In:Proceedings of the7th International Workshop on Quality of Multimedia Experience,1-6,2015.
[26]Sandi′c Stankovi′c,D.;Kukolj,D.;Le Callet,P.DIBR-synthesized image quality assessment based on morphological multi-scale approach.EURASIP Journal on Image and Video Processing Vol.2017,4,2017.
[27]Heng,W.;Jiang.T.From image quality to patch quality:An image-patch model for no-reference image quality assessment.In:Proceedings of the IEEEInternational Conference on Acoustics,Speech and Signal Processing,1238-1242,2017.
[28]Zhu,W.;Liang,S.;Wei,Y.;Sun,J.Saliency optimization from robust background detection.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2814-2821,2014.
[29]Yang,X.;Ling,W.;Lu,Z.;Ong,E.P.;Yao,S.Just noticeable distortion model and its applications in video coding.Signal Processing:Image Communication Vol.20,No.7,662-680,2005.
[30]Kimata,H.;Kitahara,M.;Kamikura,K.;Yashima,Y.Free-viewpoint video communication using multiview video coding.NTT Technical Review Vol.2,No.8,21-26,2004.
[31]Zitnick,C.L.;Kang,S.B.;Uyttendaele,M.;Winder,S.;Szeliski,R.High-quality video view interpolation using a layered representation.ACM Transactions on Graphics Vol.23,No.3,600-608,2004.
[32]Doma?nski,M.;Grajek,T.;Klimaszewski,K.;Kurc,M.;Stankiewicz,O.;Stankowski,J.;Wegner,K.Poznan multiview video test sequences and camera parameters.ISO/IEC JTC1/SC29/WG11 MPEG,M17050,2009.
[33]Mayer,N.;Ilg,E.;Hausser,P.;Fischer,P.;Cremers,D.;Dosovitskiy,A.;Brox,T.A large dataset to train convolutional networks for disparity,optical flow,and scene flow estimation.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,4040-4048,2016.
[34]Hirschmuller,H.;Scharstein,D.Evaluation of cost functions for stereo matching.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,1-8,2007.
[35]Bosc,E.;P′epion,R.;Le Callet,P.;K¨oppel,M.;NdjikiNya,P.;Morin,L.;Pressigout,M.Perceived quality of DIBR-based synthesized views.In:Proceedings of SPIE 8135,Applications of Digital Image Processing XXXIV,81350I,2011.
[36]Song,R.;Ko,H.;Jay Kuo,C.-C.MCL-3D:A database for stereoscopic image quality assessment using2D-image-plus-depth source.Journal of Information Science and Engineering Vol.31,1593-1611,2015.
[37]Winkler,S.Analysis of public image and video databases for quality assessment.IEEE Journal of Selected Topics in Signal Processing Vol.6,No.6,616-625,2012.
[38]I.T.Union.ITU-R BT.910.In:Subjective video quality assessment methods for multimedia applications.1999.
[39]I.T.Union.ITU-R BT.500-12.In:Recommendation:Methodology for the subjective assessment of the quality of television pictures.1993.
[40]Chandler,D.M.;Hemami,S.S.VSNR:A waveletbased visual signal-to-noise ratio for natural images.IEEE Transactions on Image Processing Vol.16,No.9,2284-2298,2007.
[41]Zhang,L.;Zhang,L.;Mou,X.;Zhang,D.FSIM:Afeature similarity index for image quality assessment.IEEE Transactions on Image Processing Vol.20,No.8,2378-2386,2011.
[42]Liu,L.;Liu,B.;Huang,H.;Bovik,A.C.No-reference image quality assessment based on spatial and spectral entropies.Signal Processing:Image Communication Vol.29,No.8,856-863,2014.
[43]Bao,P.;Gourlay,D.Low bandwidth remote rendering using 3D image warping.In:Proceedings of the International Conference on Visual Information Engineering.Ideas,Applications,Experience,61-64,2003.
[44]Bao,P.;Gourlay,D.A framework for remote rendering of 3-D scenes on limited mobile devices.IEEETransactions on Multimedia Vol.8,No.2,382-389,2006.
[45]Shi,S.;Nahrstedt,K.;Campbell,R.A real-time remote rendering system for interactive mobile graphics.ACM Transactions on Multimedia Computing,Communications,and Applications Vol.8,No.3s,Article No.46,2012.
[2]Smolic,A.3D video and free viewpoint video:From capture to display.Pattern Recognition Vol.44,No.9,1958-1968,2011.
[3]Smolic,A.;Mueller,K.;Merkle,P.;Kauff,P.;Wiegand,T.An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution.In:Proceedings of the Picture Coding Symposium,1-4,2009.
[4]Wang,X.;Liang,X.;Yang,B.;Li,F.W.Scalable remote rendering using synthesized image quality assessment.IEEE Access Vol.6,36595-36610,2018.
[5]Mark,W.Post-rendering 3D image warping:Visibility,reconstruction,and performance for depth-image warping.Technical Report.Chapel Hill,NC,USA,1999.
[6]Zhou,Y.;Li,L.;Gu,K.;Fang,Y.;Lin,W.Quality assessment of 3D synthesized images via disoccluded region discovery.In:Proceedings of the IEEEInternational Conference on Image Processing,1012-1016,2016.
[7]Battisti,F.;Bosc,E.;Carli,M.;Le Callet,P.;Perugia,S.Objective image quality assessment of 3D synthesized views.Signal Processing:Image Communication Vol.30,78-88,2015.
[8]Gu,K.;Jakhetiya,V.;Qiao,J.F.;Li,X.;Lin,W.;Thalmann,D.Model-based referenceless quality metric of 3D synthesized images using local image description.IEEE Transactions on Image Processing Vol.27,No.1,394-405,2018.
[9]Tian,S.;Zhang,L.;Morin,L.;D′eforges,O.NIQSV+:A no-reference synthesized view quality assessment metric.IEEE Transactions on Image Processing Vol.27,No.4,1652-1664,2018.
[10]Bosc,E.;Pepion,R.;Le Callet,P.;Koppel,M.;NdjikiNya,P.;Pressigout,M.;Morin,L.Towards a new quality metric for 3-D synthesized view assessment.IEEE Journal of Selected Topics in Signal Processing Vol.5,No.7,1332-1343,2011.
[11]Wang,Z.;Bovik,A.C.;Sheikh,H.R.;Simoncelli,E.P.Image quality assessment:From error visibility to structural similarity.IEEE Transactions on Image Processing Vol.13,No.4,600-612,2004.
[12]Sharifi,K.;Leon-Garcia,A.Estimation of shape parameter for generalized Gaussian distributions in subband decompositions of video.IEEE Transactions on Circuits and Systems for Video Technology Vol.5,No.1,52-56,1995.
[13]Mittal,A.;Moorthy,A.K.;Bovik,A.C.No-reference image quality assessment in the spatial domain.IEEETransactions on Image Processing Vol.21,No.12,4695-4708,2012.
[14]Mittal,A.;Soundararajan,R.;Bovik,A.C.Making a“completely blind”image quality analyzer.IEEE Signal Processing Letters Vol.20,No.3,209-212,2013.
[15]Kang,L.;Ye,P.;Li,Y.;Doermann,D.Convolutional neural networks for no-reference image quality assessment.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,1733-1740,2014.
[16]Bosse,S.;Maniry,D.;Wiegand,T.;Samek,W.Adeep neural network for image quality assessment.In:Proceedings of the IEEE International Conference on Image Processing,3773-3777,2016.
[17]Bare,B.;Li,K.;Yan,B.An accurate deep convolutional neural networks model for no-reference image quality assessment.In:Proceedings of the IEEEInternational Conference on Multimedia and Expo,1356-1361,2017.
[18]Kim,J.;Nguyen,A.;Ahn,S.;Luo,C.;Lee,S.Multiple level feature-based universal blind image quality assessment model.In:Proceedings of the 25th IEEE International Conference on Image Processing,291-295,2018.
[19]Lin,K.-Y.;Wang,G.Hallucinated-IQA:No-reference image quality assessment via adversarial learning.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,732-741,2018.
[20]Zhang,F.-L.;Wu,X.;Li,R.-L.;Wang,J.;Zheng,Z.-H.;Hu,S.-M.Detecting and removing visual distractors for video aesthetic enhancement.IEEE Transactions on Multimedia Vol.20,No.8,1987-1999,2018.
[21]Sheikh,H.R.;Wang,Z.;Cormack,L.;Bovik,A.C.Live image quality assessment database release 2(2005).2016.Available at http://live.ece.utexas.edu/research/quality.
[22]Ponomarenko,N.;Lukin,V.;Zelensky,A.;Egiazarian,K.;Carli,M.;Battisti,F.TID2008-a database for evaluation of full-reference visual quality assessment metrics.Advances of Modern Radioelectronics Vol.10,No.4,30-45,2009.
[23]Ponomarenko,N.;Jin,L.;Ieremeiev,O.;Lukin,V.;Egiazarian,K.;Astola,J.;Vozel,B.;Chehdi,K.;Carli,M.;Battisti,F.;Jay Kuo,C.-C.Image database TID2013:Peculiarities,results and perspectives.Signal Processing:Image Communication Vol.30,57-77,2015.
[24]Conze,P.-H.;Robert,P.;Morin,L.Objective view synthesis quality assessment.In:Proceedings of the SPIE 8288,Stereoscopic Displays and Applications XXIII,82881M,2012.
[25]Sandi′c Stankovi′c,D.;Kukolj,D.;Le Callet,P.DIBR synthesized image quality assessment based on morphological wavelets.In:Proceedings of the7th International Workshop on Quality of Multimedia Experience,1-6,2015.
[26]Sandi′c Stankovi′c,D.;Kukolj,D.;Le Callet,P.DIBR-synthesized image quality assessment based on morphological multi-scale approach.EURASIP Journal on Image and Video Processing Vol.2017,4,2017.
[27]Heng,W.;Jiang.T.From image quality to patch quality:An image-patch model for no-reference image quality assessment.In:Proceedings of the IEEEInternational Conference on Acoustics,Speech and Signal Processing,1238-1242,2017.
[28]Zhu,W.;Liang,S.;Wei,Y.;Sun,J.Saliency optimization from robust background detection.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2814-2821,2014.
[29]Yang,X.;Ling,W.;Lu,Z.;Ong,E.P.;Yao,S.Just noticeable distortion model and its applications in video coding.Signal Processing:Image Communication Vol.20,No.7,662-680,2005.
[30]Kimata,H.;Kitahara,M.;Kamikura,K.;Yashima,Y.Free-viewpoint video communication using multiview video coding.NTT Technical Review Vol.2,No.8,21-26,2004.
[31]Zitnick,C.L.;Kang,S.B.;Uyttendaele,M.;Winder,S.;Szeliski,R.High-quality video view interpolation using a layered representation.ACM Transactions on Graphics Vol.23,No.3,600-608,2004.
[32]Doma?nski,M.;Grajek,T.;Klimaszewski,K.;Kurc,M.;Stankiewicz,O.;Stankowski,J.;Wegner,K.Poznan multiview video test sequences and camera parameters.ISO/IEC JTC1/SC29/WG11 MPEG,M17050,2009.
[33]Mayer,N.;Ilg,E.;Hausser,P.;Fischer,P.;Cremers,D.;Dosovitskiy,A.;Brox,T.A large dataset to train convolutional networks for disparity,optical flow,and scene flow estimation.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,4040-4048,2016.
[34]Hirschmuller,H.;Scharstein,D.Evaluation of cost functions for stereo matching.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,1-8,2007.
[35]Bosc,E.;P′epion,R.;Le Callet,P.;K¨oppel,M.;NdjikiNya,P.;Morin,L.;Pressigout,M.Perceived quality of DIBR-based synthesized views.In:Proceedings of SPIE 8135,Applications of Digital Image Processing XXXIV,81350I,2011.
[36]Song,R.;Ko,H.;Jay Kuo,C.-C.MCL-3D:A database for stereoscopic image quality assessment using2D-image-plus-depth source.Journal of Information Science and Engineering Vol.31,1593-1611,2015.
[37]Winkler,S.Analysis of public image and video databases for quality assessment.IEEE Journal of Selected Topics in Signal Processing Vol.6,No.6,616-625,2012.
[38]I.T.Union.ITU-R BT.910.In:Subjective video quality assessment methods for multimedia applications.1999.
[39]I.T.Union.ITU-R BT.500-12.In:Recommendation:Methodology for the subjective assessment of the quality of television pictures.1993.
[40]Chandler,D.M.;Hemami,S.S.VSNR:A waveletbased visual signal-to-noise ratio for natural images.IEEE Transactions on Image Processing Vol.16,No.9,2284-2298,2007.
[41]Zhang,L.;Zhang,L.;Mou,X.;Zhang,D.FSIM:Afeature similarity index for image quality assessment.IEEE Transactions on Image Processing Vol.20,No.8,2378-2386,2011.
[42]Liu,L.;Liu,B.;Huang,H.;Bovik,A.C.No-reference image quality assessment based on spatial and spectral entropies.Signal Processing:Image Communication Vol.29,No.8,856-863,2014.
[43]Bao,P.;Gourlay,D.Low bandwidth remote rendering using 3D image warping.In:Proceedings of the International Conference on Visual Information Engineering.Ideas,Applications,Experience,61-64,2003.
[44]Bao,P.;Gourlay,D.A framework for remote rendering of 3-D scenes on limited mobile devices.IEEETransactions on Multimedia Vol.8,No.2,382-389,2006.
[45]Shi,S.;Nahrstedt,K.;Campbell,R.A real-time remote rendering system for interactive mobile graphics.ACM Transactions on Multimedia Computing,Communications,and Applications Vol.8,No.3s,Article No.46,2012.