基于结构相似性的视频质量评价方法及其在视频通信中的应用
摘要
随着多媒体通信技术的发展,数字图像以及视频的应用深入到了人类生活的各个方面。数字图像视频在采集、处理、压缩、传输以及重建的过程中可能会引入各种各样的失真,因此经常需要测量数字图像视频的质量。在实际应用中,主观质量评价方法通常不方便,并且费时费力;而传统的客观质量评价方法,如MSE和PSNR,在很多情况下不符合人的主观感受。因此,有必要研究更符合人的主观感受的客观质量评价方法。
结构相似性理论认为人眼视觉系统(HVS)的主要功能是从视觉场景中提取结构信息,因此对结构信息变化的度量可以作为图像感知失真的很好的近似。本文深入研究了结构相似理论的实现结构相似度(SSIM)指数在图像和视频评价方面的应用,改进了基于结构相似性的视频质量评价方法并将结构相似性应用于视频通信。
一般来说,运动表征的质量将决定视频的感知质量,然而现有的大多数视频质量评价方法很少直接用到运动信息,因而限制了其有效性。另外,HVS对观察到的视频的质量较差的区域比较敏感。为了弥补这些缺陷,改进一种基于结构相似性的空时视频质量评价方法(stVSSIM),其分别在时域和空域上对视频的质量进行估计,并根据HVS的感知特性对质量指数进行相应的处理。实验结果表明,stVSSIM方法比MOVIE方法以及PSNR等方法的有效性和准确性更高。
在视频通信方面,本文改进一种基于结构相似性的H.264快速运动估计算法( FMESS)。该算法应用SSIM值较大的两个图像块之间的差信号通常是一个低频信号,而低频信号通常比较容易压缩的特点以及SSIM值是图像块的结构信息的反映,非常适合于提前结束算法的特性。通过在运动估计过程中设置固定的SSIM阈值,减少不必要的运动搜索,从而达到减小运动估计复杂度、改善压缩性能的目的。实验结果表明,本文所改进的FMESS算法在保证视频的压缩质量的前提下,不仅降低了视频编码的码率,而且节省了编码时间。
Recent advances in multimedia communication technology have resulted in the proliferation of digital images, both still and video. However, digital image and video are subject to a wide variety of distortions during acquisition, processing, compression, storage, transmission and reproduction, so it’s always need to measurement its quality. In practice, however, subjective evaluation is usually too inconvenient, time-consuming and expensive. The traditional objective methods such as MSE and PSNR have low correlation with the perceptual visual quality. So it’s necessary to develop new objective methods which can correspond better to subjective feelings.
The philosophy of the structural similarity based on the assumption that the human visual system is highly adapted to extract structural information from the viewing field. It follows that a measure of structural information change can provide a good approximation to perceived image distortion. This paper studies the applications of the Structural SIMilarity(SSIM)index on image and video quality assessment, and develop its applications on the video communication.
Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. On the other hand, the human visual system is very sensitive to the low quality of the video detected. We seek to ameliorate this by developing a new VQA algorithm based on the structural similarity. Video quality is evaluated not only in space, but also in time. It is found that the stVSSIM index delivers VQA scores that correlate quite closely with human subjective judgment.
On the other hand, we propose a novel H.264 fast motion estimation algorithm based on the structural similarity, the characteristics of SSIM are considered, a SSIM threshold is set for the motion estimation process and the unnecessary searching positions are eliminated, thus reducing the complexity of motion estimation and improving the coding performance. It is found that the FMESS effectively saves the coding time and improves the compression ratio without reducing the video quality.
结构相似性理论认为人眼视觉系统(HVS)的主要功能是从视觉场景中提取结构信息,因此对结构信息变化的度量可以作为图像感知失真的很好的近似。本文深入研究了结构相似理论的实现结构相似度(SSIM)指数在图像和视频评价方面的应用,改进了基于结构相似性的视频质量评价方法并将结构相似性应用于视频通信。
一般来说,运动表征的质量将决定视频的感知质量,然而现有的大多数视频质量评价方法很少直接用到运动信息,因而限制了其有效性。另外,HVS对观察到的视频的质量较差的区域比较敏感。为了弥补这些缺陷,改进一种基于结构相似性的空时视频质量评价方法(stVSSIM),其分别在时域和空域上对视频的质量进行估计,并根据HVS的感知特性对质量指数进行相应的处理。实验结果表明,stVSSIM方法比MOVIE方法以及PSNR等方法的有效性和准确性更高。
在视频通信方面,本文改进一种基于结构相似性的H.264快速运动估计算法( FMESS)。该算法应用SSIM值较大的两个图像块之间的差信号通常是一个低频信号,而低频信号通常比较容易压缩的特点以及SSIM值是图像块的结构信息的反映,非常适合于提前结束算法的特性。通过在运动估计过程中设置固定的SSIM阈值,减少不必要的运动搜索,从而达到减小运动估计复杂度、改善压缩性能的目的。实验结果表明,本文所改进的FMESS算法在保证视频的压缩质量的前提下,不仅降低了视频编码的码率,而且节省了编码时间。
Recent advances in multimedia communication technology have resulted in the proliferation of digital images, both still and video. However, digital image and video are subject to a wide variety of distortions during acquisition, processing, compression, storage, transmission and reproduction, so it’s always need to measurement its quality. In practice, however, subjective evaluation is usually too inconvenient, time-consuming and expensive. The traditional objective methods such as MSE and PSNR have low correlation with the perceptual visual quality. So it’s necessary to develop new objective methods which can correspond better to subjective feelings.
The philosophy of the structural similarity based on the assumption that the human visual system is highly adapted to extract structural information from the viewing field. It follows that a measure of structural information change can provide a good approximation to perceived image distortion. This paper studies the applications of the Structural SIMilarity(SSIM)index on image and video quality assessment, and develop its applications on the video communication.
Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. On the other hand, the human visual system is very sensitive to the low quality of the video detected. We seek to ameliorate this by developing a new VQA algorithm based on the structural similarity. Video quality is evaluated not only in space, but also in time. It is found that the stVSSIM index delivers VQA scores that correlate quite closely with human subjective judgment.
On the other hand, we propose a novel H.264 fast motion estimation algorithm based on the structural similarity, the characteristics of SSIM are considered, a SSIM threshold is set for the motion estimation process and the unnecessary searching positions are eliminated, thus reducing the complexity of motion estimation and improving the coding performance. It is found that the FMESS effectively saves the coding time and improves the compression ratio without reducing the video quality.
引文
[1] B. Girod. What’s Wrong With Mean-Squared Error. In A.B.Watson(ed.), Digital Images and Human Vision, The MIT Press,Cambridge,Massachusetts,1993:207~220
[2] Z. Wang, A. C. Bovik and L. Lu. Why is image quality assessment so difficult. IEEE International Conference on Acoustics, Speech, & Signal Processing, May 2002.
[3] J.L.Mannos,D.J.Sakrison. The Effects of a Visual Fidelity Criterion on the Encoding of Images. IEEE Transactions on Information Theory,1974,20(4):525~536
[4] VQEG. Evaluation of New Methods for Objective Testing of Video Quality: Objective Test Plan.1998
[5] P.Corriveau, et al. Video Quality Experts Group: Current Results and Future Directions. Proceedings of SPIE Visual Comm.and Image Processing.2000,4067(7)
[6] Z.Wang,A.C.Bovik,H.R.Sheikh and E.P.Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing.vol.13, no.4, pp.600-612. Apr. 2004.
[7] Z.Wang, E.P.Simoncelli and A.C.Bovik. Multi-scale structural similarity for image quality assessment. IEEE Asilomar Conference Signals, Systems and Computers, Nov.2003.
[8] Z.Wang, H. R. Sheikh, and A. C. Bovik. Objective video quality assessment in The Handbook of Video Databases: Design and Applications, B. Furht and O. Marques, Eds. Boca Raton, FL: CRC Press, 2003.
[9] D.V.Rao,et.al. An Image Quality Assessment Technique Based on Visual Regions of Interest Weighted Structural Similarity. GVIP Journal.2006,6(2):69~75
[10]胡良梅,高隽,何柯峰.图像融合质量评价方法的研究.电子学报.2004,32(12A):218~221
[11]王超,叶中付.基于相似性的图像融合质量的客观评估方法.软件学报.2006,17(7):1580~158
[12] X.B.Gao, T.Wang, J.Li. A content-based image quality metric. Lecture Notes in Artificial Intelligence. LNAI 3642,pp.231-240,2005.
[13] Yang chun-ling, Kuang kai-zhi, Chen guan-hao, Xie sheng-li. Gradient based structural similarity for image quality assessment.Journal of South China University of Technology,2006,34(9):22-25
[14] Z. Wang, L. Lu, and A. C. Bovik. Video quality assessment based on structural distortion measurement. Signal Processing: Image Communication, special issue on Objective video quality metrics, vol. 19, no. 2, pp. 121-132, Feb. 2004.
[15] Z. Wang and Q. Li. Video quality assessment using a statistical model of human visual speed perception. Journal of the Optical Society of America A, vol. 24, no. 12, pp. B61-B69, Dec. 2007
[16] Kalpana Seshadrinathan and Alan C. Bovik. Statistical video models and their application to quality assessment. In Second International Wor ocessing and Quality Metrics for Consumer Electronics, Scottsdale, Arizona, January 2006. .2007. cessing,.Feb. 2010. omputers, v32, n8, p 1603-1610, August 2009. of roup on the Validation of Objective Models of ality cessing, vol.14, no.12, December 2005. rmation. Int. J. Electronic Imaging XIV. Proceedings of the SPIE 7240 .January.2009 pril.2009. y Experts Group. http://www.its.bldrdoc.gov/vqeg/ Annual Review of kshop on Video Pr
[17] Kalpana Seshadrinathan and Alan C. Bovik. A structural similarity metric for video based on motion models. In IEEE Intl. Conf. on Acoustics, Speech, and Signal Proc
[18] Kalpana Seshadrinathan and A.C. Bovik. Motion Tuned Spatio-temporal Quality Assessment of Natural Videos. vol. 19, no. 2, pp. 335-350, IEEE Transactions on Image Pro
[19] Yang Chun-Ling, Wang Hua-Xing, Liang Rong-Kun. Improved inter prediction based on structural similarity for H.264.Jisuanji Xuebao/Chinese Journal of C
[20] VQEG. Final Report from the Video Quality Expert Group on the Validation of Objective ModelsVideo Quality Assessment.2000,3
[21] VQEG. Final Report from the Video Quality Expert GVideo Quality Assessment, Phase II.2003.8
[22] H.R. Sheikh, A.C. Bovik and G. de Veciana. An information fidelity criterion for image quassessment using natural scene statistics. IEEE Transactions on Image Processing , vol.14, no.12pp. 2117- 2128, Dec. 2005.
[23] H.R. Sheikh.and A.C. Bovik. Image information and visual quality. IEEE Transactions on Image Processing, vol.15, no.2,pp. 430- 444, Feb. 2006.
[24] H. R. Sheikh, A. C. Bovik, and L. K. Cormack. No-Reference Quality Assessment Using Natural Scene Statistics: JPEG2000. IEEE Transactions on Image Pro
[25] D. Fleet and A. Jepson. Computation of component image velocity from local phase infoComput. Vis., vol. 5, no. 1, pp.77–104, 1990.
[26] Moorthy, A. K. and Bovik, A. C. Perceptually significant spatial pooling techniques for image quality assessment. Human Vision and
[27] Moorthy, A. K. and Bovik, A. C. Visual importance pooling for image quality assessment. IEEE Journalof Selected Topics in Signal Processing, Issue on Visual Media Quality Assessment 3, 193–201.A
[28] Nie, Y. and Ma, K. Adaptive rood pattern search for fast block-matching motion estimation. IEEE Transactions on Image Processing 11(12), 1442–1449.2002.
[29] VQEG. The Video Qualit
[30] Image & Video Quality Assessment at LIVE. http://live.ece.utexas.edu/research/quality/
[31] Simoncelli E, Olshausen B. Natural Image Statistics and Neural Representation.Improved inter prediction based on structural similarity in Neuroscience 24(1), 1193–1216.2001.
[32]毕厚杰.新一代视频压缩编码标准—H.264/AVC[M].北京:人民邮电出版社.2005
[33] Chun-Ling Yang,Hua-Xing Wang ,Lai-Man Po. H.264. IEEE International Conference on Signal Processing and Communications (ICSPC 2007), 24-27 November 2007, Dubai
[34] ITU-T Rec. H.264 / ISO/IEC 11496-10,Advanced Video Coding, Final Committee Draft, Document JVTG050,March.2003
[35] http://iphome.hhi.de/suehring/tml/download/
[2] Z. Wang, A. C. Bovik and L. Lu. Why is image quality assessment so difficult. IEEE International Conference on Acoustics, Speech, & Signal Processing, May 2002.
[3] J.L.Mannos,D.J.Sakrison. The Effects of a Visual Fidelity Criterion on the Encoding of Images. IEEE Transactions on Information Theory,1974,20(4):525~536
[4] VQEG. Evaluation of New Methods for Objective Testing of Video Quality: Objective Test Plan.1998
[5] P.Corriveau, et al. Video Quality Experts Group: Current Results and Future Directions. Proceedings of SPIE Visual Comm.and Image Processing.2000,4067(7)
[6] Z.Wang,A.C.Bovik,H.R.Sheikh and E.P.Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing.vol.13, no.4, pp.600-612. Apr. 2004.
[7] Z.Wang, E.P.Simoncelli and A.C.Bovik. Multi-scale structural similarity for image quality assessment. IEEE Asilomar Conference Signals, Systems and Computers, Nov.2003.
[8] Z.Wang, H. R. Sheikh, and A. C. Bovik. Objective video quality assessment in The Handbook of Video Databases: Design and Applications, B. Furht and O. Marques, Eds. Boca Raton, FL: CRC Press, 2003.
[9] D.V.Rao,et.al. An Image Quality Assessment Technique Based on Visual Regions of Interest Weighted Structural Similarity. GVIP Journal.2006,6(2):69~75
[10]胡良梅,高隽,何柯峰.图像融合质量评价方法的研究.电子学报.2004,32(12A):218~221
[11]王超,叶中付.基于相似性的图像融合质量的客观评估方法.软件学报.2006,17(7):1580~158
[12] X.B.Gao, T.Wang, J.Li. A content-based image quality metric. Lecture Notes in Artificial Intelligence. LNAI 3642,pp.231-240,2005.
[13] Yang chun-ling, Kuang kai-zhi, Chen guan-hao, Xie sheng-li. Gradient based structural similarity for image quality assessment.Journal of South China University of Technology,2006,34(9):22-25
[14] Z. Wang, L. Lu, and A. C. Bovik. Video quality assessment based on structural distortion measurement. Signal Processing: Image Communication, special issue on Objective video quality metrics, vol. 19, no. 2, pp. 121-132, Feb. 2004.
[15] Z. Wang and Q. Li. Video quality assessment using a statistical model of human visual speed perception. Journal of the Optical Society of America A, vol. 24, no. 12, pp. B61-B69, Dec. 2007
[16] Kalpana Seshadrinathan and Alan C. Bovik. Statistical video models and their application to quality assessment. In Second International Wor ocessing and Quality Metrics for Consumer Electronics, Scottsdale, Arizona, January 2006. .2007. cessing,.Feb. 2010. omputers, v32, n8, p 1603-1610, August 2009. of roup on the Validation of Objective Models of ality cessing, vol.14, no.12, December 2005. rmation. Int. J. Electronic Imaging XIV. Proceedings of the SPIE 7240 .January.2009 pril.2009. y Experts Group. http://www.its.bldrdoc.gov/vqeg/ Annual Review of kshop on Video Pr
[17] Kalpana Seshadrinathan and Alan C. Bovik. A structural similarity metric for video based on motion models. In IEEE Intl. Conf. on Acoustics, Speech, and Signal Proc
[18] Kalpana Seshadrinathan and A.C. Bovik. Motion Tuned Spatio-temporal Quality Assessment of Natural Videos. vol. 19, no. 2, pp. 335-350, IEEE Transactions on Image Pro
[19] Yang Chun-Ling, Wang Hua-Xing, Liang Rong-Kun. Improved inter prediction based on structural similarity for H.264.Jisuanji Xuebao/Chinese Journal of C
[20] VQEG. Final Report from the Video Quality Expert Group on the Validation of Objective ModelsVideo Quality Assessment.2000,3
[21] VQEG. Final Report from the Video Quality Expert GVideo Quality Assessment, Phase II.2003.8
[22] H.R. Sheikh, A.C. Bovik and G. de Veciana. An information fidelity criterion for image quassessment using natural scene statistics. IEEE Transactions on Image Processing , vol.14, no.12pp. 2117- 2128, Dec. 2005.
[23] H.R. Sheikh.and A.C. Bovik. Image information and visual quality. IEEE Transactions on Image Processing, vol.15, no.2,pp. 430- 444, Feb. 2006.
[24] H. R. Sheikh, A. C. Bovik, and L. K. Cormack. No-Reference Quality Assessment Using Natural Scene Statistics: JPEG2000. IEEE Transactions on Image Pro
[25] D. Fleet and A. Jepson. Computation of component image velocity from local phase infoComput. Vis., vol. 5, no. 1, pp.77–104, 1990.
[26] Moorthy, A. K. and Bovik, A. C. Perceptually significant spatial pooling techniques for image quality assessment. Human Vision and
[27] Moorthy, A. K. and Bovik, A. C. Visual importance pooling for image quality assessment. IEEE Journalof Selected Topics in Signal Processing, Issue on Visual Media Quality Assessment 3, 193–201.A
[28] Nie, Y. and Ma, K. Adaptive rood pattern search for fast block-matching motion estimation. IEEE Transactions on Image Processing 11(12), 1442–1449.2002.
[29] VQEG. The Video Qualit
[30] Image & Video Quality Assessment at LIVE. http://live.ece.utexas.edu/research/quality/
[31] Simoncelli E, Olshausen B. Natural Image Statistics and Neural Representation.Improved inter prediction based on structural similarity in Neuroscience 24(1), 1193–1216.2001.
[32]毕厚杰.新一代视频压缩编码标准—H.264/AVC[M].北京:人民邮电出版社.2005
[33] Chun-Ling Yang,Hua-Xing Wang ,Lai-Man Po. H.264. IEEE International Conference on Signal Processing and Communications (ICSPC 2007), 24-27 November 2007, Dubai
[34] ITU-T Rec. H.264 / ISO/IEC 11496-10,Advanced Video Coding, Final Committee Draft, Document JVTG050,March.2003
[35] http://iphome.hhi.de/suehring/tml/download/