用户多维感知的3D图像体验质量评价
|
摘要
目的符合用户视觉特性的3维图像体验质量评价方法有助于准确、客观地体现用户观看3D图像或视频时的视觉感知体验,从而给优化3维内容提供一定的思路。现有的评价方法仅从图像失真、深度感知和视觉舒适度中的一个维度或两个维度出发对立体图像进行评价,评价结果的准确性有待进一步提升。为了更加全面和准确地评价3D图像的视觉感知体验,提出了一种用户多维感知的3D图像体验质量评价算法。方法首先对左右图像的差异图像和融合图像提取自然场景统计参数表示失真特征;然后对深度图像提取敏感区域,对敏感区域绘制失真前后深度变换直方图,统计深度变化情况以及利用尺度不变特征变换(SIFT)关键点匹配算法计算匹配点数目,两者共同表示深度感知特征;接下来对视觉显著区域提取视差均值、幅值表示舒适度特征;最后综合考虑图像失真、深度感知和视觉舒适度3个维度特征,将3个维度特征归一化后联合成体验质量特征向量,采用支持向量回归(SVR)训练评价模型,并得到最终的体验质量得分。结果在LIVE和Waterloo IVC数据库上的实验结果表明,所提出的方法与人们的主观感知的相关性达到了0. 942和0. 858。结论该方法充分利用了立体图像的特性,评价结果优于比较的几种经典算法,所构建模型的评价结果与用户的主观体验有更好的一致性。
Objective Although 3 D technology is increasingly used in film and television,the development of 3 D display technology has been stagnant in recent years. The main reason is that image degradation occurs in the process of 3 D information transmission,along with the decline of the depth perception of stereo images to a certain extent. These conditions affect users' immersion experience. Furthermore,the discomfort caused by 3 D display limits the development of 3 D technology. On the one hand,an effective stereo image quality evaluation technology can provide new ideas for image compression standards. On the other hand,the technology provides a reference for the rational improvement of the quality of 3 D videos,thereby accelerating the development of 3 D multimedia application technology. Providing content that conforms to users' viewing experience is of paramount importance for the further promotion of 3 D multimedia technology. The stereoscopic image quality evaluation method that conforms to users' visual characteristics helps to accurately and objectively reflect the visual perception experience when users watch 3 D images or videos. On the basis of different dimensions thataffect the quality of stereoscopic image experience,four categories of image quality assessment algorithms are used: image quality evaluation based on distortion,quality of experience based on depth perception,quality of experience based on comfort,and comprehensive dimensions. The quality of experience( Qo E) represents the quality of the stereoscopic visual experience of users. Qo E is an objective result that takes users as the core and considers the multi-dimensional perception factors that comprehensively affect it. The stereo image quality is the result of the three perceptions of distortion,depth,and comfort of 3 D images. Distortion quality indicates the degree of image degradation caused by image distortion. Depth quality indicates the depth and immersion feeling experienced when viewing 3 D content. Visual comfort indicates the degree of visual fatigue experienced when viewing stereoscopic images. The existing research on the objective evaluation of 3 D Qo E only evaluates results beginning from one or two dimensions of image distortion,stereoscopic perception,and visual comfort. However,in an actual subjective experiment,we found that any change in dimensions leads to changes in the quality of the stereo image experience and that existing methods do not comprehensively consider the three factors of image distortion,depth perception,and comfort. To evaluate the visual perception experience of 3 D images comprehensively and accurately,this study proposes a stereoscopic image experience quality evaluation method that is based on users multi-dimensional perception. Method A distortion-free natural scene image has a certain regularity in distribution,and image distortion causes its distribution law to change; thus,image quality can be estimated from the extracted feature parameters. The left and right eye images are subtracted and added to obtain the difference image and fused image. Then,the difference image and fused image are fitted by the generalized Gaussian distribution function,and the fitting parameters are obtained as the distortion quality features. Distortion reduces the depth perception quality of stereo images. It exerts two main effects on depth perception. First,the relative depth information between objects is lost,and the position of the object consequently becomes blurred,thereby affecting depth perception. Second,distortion reduces the feature points at which the left and right viewpoint images are matched; thus,the binocular depth perception information is reduced,thereby diminishing the sense of depth. Then,the distortion-sensitive pixel map is obtained. For each distortion-sensitive pixel,the neighborhood brightness distribution is calculated. SIFT( scale invariant feature transform) key point matching is performed on the left and right views. The statistical result of the neighborhood brightness distribution and the key point matching quantity are used as the depth quality feature. When the parallax of the stereoscopic image exceeds a certain range,the human eye may generate a convergence conflict,thereby resulting in visual fatigue. The human eye is only sensitive to the comfort/discomfort characteristics of the significant area. Thus,we adopt the comfort evaluation model based on the visual important regions and extract the mean parallax value of the significant area. Finally,the three-dimensional features are combined as the experience quality feature vector,and the objective prediction model is constructed by support vector regression. Result Experimental results on the LIVE database and Waterloo IVC database show that the proposed method correlates with people's subjective perception at values of 0. 942 and 0. 858,which are better than those of other methods. Conclusion The method fully uses the characteristics of the stereo image,and the evaluation result is better than that of several classical algorithms. Therefore,the evaluation result of the constructed model shows improved consistency with the subjective experience of the user. In the future,we will combine the evaluation process with the stereo image quality optimization process and guide quality optimization to stereo images from various dimensions.
Objective Although 3 D technology is increasingly used in film and television,the development of 3 D display technology has been stagnant in recent years. The main reason is that image degradation occurs in the process of 3 D information transmission,along with the decline of the depth perception of stereo images to a certain extent. These conditions affect users' immersion experience. Furthermore,the discomfort caused by 3 D display limits the development of 3 D technology. On the one hand,an effective stereo image quality evaluation technology can provide new ideas for image compression standards. On the other hand,the technology provides a reference for the rational improvement of the quality of 3 D videos,thereby accelerating the development of 3 D multimedia application technology. Providing content that conforms to users' viewing experience is of paramount importance for the further promotion of 3 D multimedia technology. The stereoscopic image quality evaluation method that conforms to users' visual characteristics helps to accurately and objectively reflect the visual perception experience when users watch 3 D images or videos. On the basis of different dimensions thataffect the quality of stereoscopic image experience,four categories of image quality assessment algorithms are used: image quality evaluation based on distortion,quality of experience based on depth perception,quality of experience based on comfort,and comprehensive dimensions. The quality of experience( Qo E) represents the quality of the stereoscopic visual experience of users. Qo E is an objective result that takes users as the core and considers the multi-dimensional perception factors that comprehensively affect it. The stereo image quality is the result of the three perceptions of distortion,depth,and comfort of 3 D images. Distortion quality indicates the degree of image degradation caused by image distortion. Depth quality indicates the depth and immersion feeling experienced when viewing 3 D content. Visual comfort indicates the degree of visual fatigue experienced when viewing stereoscopic images. The existing research on the objective evaluation of 3 D Qo E only evaluates results beginning from one or two dimensions of image distortion,stereoscopic perception,and visual comfort. However,in an actual subjective experiment,we found that any change in dimensions leads to changes in the quality of the stereo image experience and that existing methods do not comprehensively consider the three factors of image distortion,depth perception,and comfort. To evaluate the visual perception experience of 3 D images comprehensively and accurately,this study proposes a stereoscopic image experience quality evaluation method that is based on users multi-dimensional perception. Method A distortion-free natural scene image has a certain regularity in distribution,and image distortion causes its distribution law to change; thus,image quality can be estimated from the extracted feature parameters. The left and right eye images are subtracted and added to obtain the difference image and fused image. Then,the difference image and fused image are fitted by the generalized Gaussian distribution function,and the fitting parameters are obtained as the distortion quality features. Distortion reduces the depth perception quality of stereo images. It exerts two main effects on depth perception. First,the relative depth information between objects is lost,and the position of the object consequently becomes blurred,thereby affecting depth perception. Second,distortion reduces the feature points at which the left and right viewpoint images are matched; thus,the binocular depth perception information is reduced,thereby diminishing the sense of depth. Then,the distortion-sensitive pixel map is obtained. For each distortion-sensitive pixel,the neighborhood brightness distribution is calculated. SIFT( scale invariant feature transform) key point matching is performed on the left and right views. The statistical result of the neighborhood brightness distribution and the key point matching quantity are used as the depth quality feature. When the parallax of the stereoscopic image exceeds a certain range,the human eye may generate a convergence conflict,thereby resulting in visual fatigue. The human eye is only sensitive to the comfort/discomfort characteristics of the significant area. Thus,we adopt the comfort evaluation model based on the visual important regions and extract the mean parallax value of the significant area. Finally,the three-dimensional features are combined as the experience quality feature vector,and the objective prediction model is constructed by support vector regression. Result Experimental results on the LIVE database and Waterloo IVC database show that the proposed method correlates with people's subjective perception at values of 0. 942 and 0. 858,which are better than those of other methods. Conclusion The method fully uses the characteristics of the stereo image,and the evaluation result is better than that of several classical algorithms. Therefore,the evaluation result of the constructed model shows improved consistency with the subjective experience of the user. In the future,we will combine the evaluation process with the stereo image quality optimization process and guide quality optimization to stereo images from various dimensions.
引文
[1]Bovik A C.Automatic prediction of perceptual image and video quality[J].Proceedings of the IEEE,2013,101(9):2008-2024.[DOI:10.1109/JPROC.2013.2257632]
[2]Liu L X,Liu B,Su C C,et al.Binocular spatial activity and reverse saliency driven no-reference stereopair quality assessment[J].Signal Processing:Image Communication,2017,58:287-299.[DOI:10.1016/j.image.2017.08.011]
[3]Liu X K.Research of subjective/objective quality assessment and perceptual optimized coding for 3D video[D].Chengdu:Southwest Jiaotong University,2016.[刘祥凯.三维视频主客观质量评价方法与感知优化编码研究[D].成都:西南交通大学,2016.]
[4]Kim D,Sohn K.Visual fatigue prediction for stereoscopic image[J].IEEE Transactions on Circuits and Systems for Video Technology,2011,21(2):231-236.[DOI:10.1109/TCSVT.2011.2106275]
[5]Zeri F,Livi S,et al.Visual discomfort while watching stereoscopic three-dimensional movies at the cinema[J].Ophthalmic Physiol Opt,2015,35(3):271-282.[DOI:10.1111/opo.12194]
[6]Danli W,Xinpan Y,Haichen H,et al.Visual fatigue during continuous viewing the 3D movie[J].Electronic Imaging,2016,2016(5):1-6.[DOI:10.2352/ISSN.2470-1173.2016.5.SDA-442]
[7]Chen M J,Cormack L K,Bovik A C.No-reference quality assessment of natural stereopairs[J].IEEE Transactions on Image Processing,2013,22(9):3379-3391.[DOI:10.1109/TIP.2013.2267393]
[8]Lin Y H,Wu J L.Quality assessment of stereoscopic 3D image compression by binocular integration behaviors[J].IEEE Transactions on Image Processing,2014,23(4):1527-1542.[DOI:10.1109/TIP.2014.2302686]
[9]Farid M S,Lucenteforte M,Grangetto M.Blind depth quality assessment using histogram shape analysis[C]//Proceedings of2015 3DTV-Conference:the True Vision-Capture,Transmission and Display of 3D Video.Lisbon,Portugal:IEEE,2015:1-5.[DOI:10.1109/3DTV.2015.7169352]
[10]Sohn H,Jung Y J,Lee S I,et al.Predicting visual discomfort using object size and disparity information in stereoscopic images[J].IEEE Transactions on Broadcasting,2013,59(1):28-37.[DOI:10.1109/TBC.2013.2238413]
[11]Jung Y J,Sohn H,Lee S I,et al.Predicting visual discomfort of stereoscopic images using human attention model[J].IEEETransactions on Circuits and Systems for Video Technology,2013,23(12):2077-2082.[DOI:10.1109/TCSVT.2013.2270394]
[12]Wang J H,Wang S Q,Ma K D,et al.Perceptual depth quality in distorted stereoscopic images[J].IEEE Transactions on Image Processing,2017,26(3):1202-1215.[DOI:10.1109/TIP.2016.2642791]
[13]Shao F,Lin W S,Li Z T,et al.Toward simultaneous visual comfort and depth sensation optimization for stereoscopic 3D experience[J].IEEE Transactions on Cybernetics,2017,47(12):4521-4533.[DOI:10.1109/TCYB.2016.2615856]
[14]爦enol E,zbek N.Quality of experience measurement of compressed multi-view video[J].Signal Processing:Image Communication,2017,57:147-156.[DOI:10.1016/j.image.2017.05.003]
[15]Mittal A,Moorthy A K,Bovik A C.No-reference image quality assessment in the spatial domain[J].IEEE Transactions on Image Processing,2012,21(12):4695-4708.[DOI:10.1109/TIP.2012.2214050]
[16]Chen Z B,Zhou W,Li W P.Blind stereoscopic video quality assessment:from depth perception to overall experience[J].IEEETransactions on Image Processing,2018,27(2):721-734.[DOI:10.1109/TIP.2017.2766780]
[17]Sharifi K.Estimation of shape parameter for generalized Gaussian distributions in subband decomposition of video[J].IEEETrans.Circuits,Syst.Video Technol.1995,5.[DOI:10.1109/76.350779]
[18]Chen M J,Su C C,Kwon D K,et al.Full-reference quality assessment of stereopairs accounting for rivalry[J].Signal Processing:Image Communication,2013,28(9):1143-1155.[DOI:10.1016/j.image.2013.05.006]
[19]Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEETransactions on Image Process,2004,13(4):600-612.[DOI:10.1109/TIP.2003.819861]
[20]Wang Z,Simoncelli E P,Bovik A C.Multiscale structural similarity for image quality assessment[C]//Proceedings of the 37th Asilomar Conference on Signals,Systems&Computers.Pacific Grove,CA,USA:IEEE,2003:1398-1402.[DOI:10.1109/ACSSC.2003.1292216]
[21]Sheikh H R,Bovik A C.Image information and visual quality[J].IEEE Transactions on Image Processing,2006,15(2):430-444.[DOI:10.1109/TIP.2005.859378]
[2]Liu L X,Liu B,Su C C,et al.Binocular spatial activity and reverse saliency driven no-reference stereopair quality assessment[J].Signal Processing:Image Communication,2017,58:287-299.[DOI:10.1016/j.image.2017.08.011]
[3]Liu X K.Research of subjective/objective quality assessment and perceptual optimized coding for 3D video[D].Chengdu:Southwest Jiaotong University,2016.[刘祥凯.三维视频主客观质量评价方法与感知优化编码研究[D].成都:西南交通大学,2016.]
[4]Kim D,Sohn K.Visual fatigue prediction for stereoscopic image[J].IEEE Transactions on Circuits and Systems for Video Technology,2011,21(2):231-236.[DOI:10.1109/TCSVT.2011.2106275]
[5]Zeri F,Livi S,et al.Visual discomfort while watching stereoscopic three-dimensional movies at the cinema[J].Ophthalmic Physiol Opt,2015,35(3):271-282.[DOI:10.1111/opo.12194]
[6]Danli W,Xinpan Y,Haichen H,et al.Visual fatigue during continuous viewing the 3D movie[J].Electronic Imaging,2016,2016(5):1-6.[DOI:10.2352/ISSN.2470-1173.2016.5.SDA-442]
[7]Chen M J,Cormack L K,Bovik A C.No-reference quality assessment of natural stereopairs[J].IEEE Transactions on Image Processing,2013,22(9):3379-3391.[DOI:10.1109/TIP.2013.2267393]
[8]Lin Y H,Wu J L.Quality assessment of stereoscopic 3D image compression by binocular integration behaviors[J].IEEE Transactions on Image Processing,2014,23(4):1527-1542.[DOI:10.1109/TIP.2014.2302686]
[9]Farid M S,Lucenteforte M,Grangetto M.Blind depth quality assessment using histogram shape analysis[C]//Proceedings of2015 3DTV-Conference:the True Vision-Capture,Transmission and Display of 3D Video.Lisbon,Portugal:IEEE,2015:1-5.[DOI:10.1109/3DTV.2015.7169352]
[10]Sohn H,Jung Y J,Lee S I,et al.Predicting visual discomfort using object size and disparity information in stereoscopic images[J].IEEE Transactions on Broadcasting,2013,59(1):28-37.[DOI:10.1109/TBC.2013.2238413]
[11]Jung Y J,Sohn H,Lee S I,et al.Predicting visual discomfort of stereoscopic images using human attention model[J].IEEETransactions on Circuits and Systems for Video Technology,2013,23(12):2077-2082.[DOI:10.1109/TCSVT.2013.2270394]
[12]Wang J H,Wang S Q,Ma K D,et al.Perceptual depth quality in distorted stereoscopic images[J].IEEE Transactions on Image Processing,2017,26(3):1202-1215.[DOI:10.1109/TIP.2016.2642791]
[13]Shao F,Lin W S,Li Z T,et al.Toward simultaneous visual comfort and depth sensation optimization for stereoscopic 3D experience[J].IEEE Transactions on Cybernetics,2017,47(12):4521-4533.[DOI:10.1109/TCYB.2016.2615856]
[14]爦enol E,zbek N.Quality of experience measurement of compressed multi-view video[J].Signal Processing:Image Communication,2017,57:147-156.[DOI:10.1016/j.image.2017.05.003]
[15]Mittal A,Moorthy A K,Bovik A C.No-reference image quality assessment in the spatial domain[J].IEEE Transactions on Image Processing,2012,21(12):4695-4708.[DOI:10.1109/TIP.2012.2214050]
[16]Chen Z B,Zhou W,Li W P.Blind stereoscopic video quality assessment:from depth perception to overall experience[J].IEEETransactions on Image Processing,2018,27(2):721-734.[DOI:10.1109/TIP.2017.2766780]
[17]Sharifi K.Estimation of shape parameter for generalized Gaussian distributions in subband decomposition of video[J].IEEETrans.Circuits,Syst.Video Technol.1995,5.[DOI:10.1109/76.350779]
[18]Chen M J,Su C C,Kwon D K,et al.Full-reference quality assessment of stereopairs accounting for rivalry[J].Signal Processing:Image Communication,2013,28(9):1143-1155.[DOI:10.1016/j.image.2013.05.006]
[19]Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEETransactions on Image Process,2004,13(4):600-612.[DOI:10.1109/TIP.2003.819861]
[20]Wang Z,Simoncelli E P,Bovik A C.Multiscale structural similarity for image quality assessment[C]//Proceedings of the 37th Asilomar Conference on Signals,Systems&Computers.Pacific Grove,CA,USA:IEEE,2003:1398-1402.[DOI:10.1109/ACSSC.2003.1292216]
[21]Sheikh H R,Bovik A C.Image information and visual quality[J].IEEE Transactions on Image Processing,2006,15(2):430-444.[DOI:10.1109/TIP.2005.859378]