基于H.264标准的网络视频编码器关键技术的研究与实现
|
摘要
随着多媒体和信息技术的发展与完善,各种新的视频应用不断出现,人们对图像质量的要求越来越高。其中实时视频应用倍受大家青睐,应用前景相当广阔。在视频应用领域,图像质量和编码效率一直是两难问题。H.264视频编码标准成功的解决了这一问题,但是编码过程的计算复杂性很高。因此降低编码过程的计算复杂性,提高编码速度是H.264视频应用的重点和难点。
本课题从三个方面研究了网络视频编码器:H.264视频编码算法优化;DSP系统中视频编码应用的硬件加速;H.264视频报文的网络实时传输。
首先,分析算法优化难点,提出了启发式预测模式确定算法,对帧间预测、帧内预测和最佳预测模式的确定过程进行综合优化。该算法利用视频序列在空间和时间上的相关性,以及相邻的图像单位在预测失真上的相似性,推测当前宏块中接近最佳性能的预测块划分方法和预测模式。算法提供了推测错误处理机制,能够保证预测块与原始图像之间实现最佳匹配。实验证明经过优化之后,以微小的编码图像失真为代价,较大幅度地提高了编码速度,编码器的性能得到显著改善。
其次,研究了DSP上实现H.264的关键技术。将保证DSP系统定点运算精度和利用DSP算法实现加速的关键技术作为研究重点。提出了多层次优化设计方案,在程序、算法和指令三个不同层次上,通过优化程序结构、改善存储管理、合理分配资源、优化指令调度、减少cache失效,以及利用多种DMA传输机制实现视频处理与数据传输的重叠作业等优化关键技术,提高DSP系统中的功能部件、存储资源和外部设备的工作效率,实现了视频图像的实时编码。实验结果表明,通过关键技术优化的DSP视频编码器性能有大幅度的提高。
最后,利用德州仪器提供的网络开发工具包实现了RTP/RTCP实时网络传输协议,主要包括TCP/IP网络协议栈的配置与使用,以及实现基于RTP/RTCP协议的应用服务。在网络应用的基础上,对视频编码器进行性能进行测试,检验视频编码器的实时效果;完成了进行数据完整性测试,验证了视频编码器的正确性。测试结果表明本课题研究和实现的视频编码器具有良好的实时性和稳定性。
With the development of multimedia and communication technology, a variety of new applications with high image resolution have been taken into practice. With great perspective, realtime video applications are very popular, in which image quality and bitrates are in the dilemma. H.264 video coding standard can solve this intractability problem successfully. However, there is much complexity in the encoding processes of H.264 encoder. Therefore, how to reduce the complexities of encoding process, as well as to speed up the encoder, becomes the keystone of video application, which is hard to achieve. Therefore, this dissertation researches on three aspects of network video encoder, which include the optimizing algorithm in video encoder, DSP hardware speedup and the real-time transfer of H.264 video packets.
Firstly, by researching on the optimizing algorithms, we propose an algorithm based on method of elicitation to optimize intra prediction, inter prediction and mode decision. By exploiting spacial or temporal correlation of neighbor macroblocks, as well as distortion of neighbor macroblock, the algorithm speculates that the partition method and prediction mode of current macroblock is similar to its neighbors. The mechanism provided by the algorithm ensures the best match between prediction block and original block. The results of test prove that the algorithm improves the encoding rate with little image distortion.
Secondly, our researches focus on high accuracy fix-point arithmetic and DSP hardware speedup of H.264 application. By optimizing the structure of program, improving memory management, and assigning resources properly, we realize optimizations on the program, algorithm and instruction level respectively. High speed data transfers between DSP and peripherals are implemented by EDMA operations. An overlap between data transfers and data processing are introduced by EDMA mechanisms and the flexible configuration of the on chip SRAM of DSP. After the optimization, ALUs, memory and peripherals are sufficiently utilized. The testing results indicate that the high performance of video encoder optimized by DSP optimized algorithms.
Finally, we implement RTP protocol based on the network develop kit provided by TI corporation, which includes the configuration and utilization of TCP/IP protocol. Experimental results are given to evaluate the performance of the video encoder. According to the results, our video encoder is considered to be an application with good performance and high reliability.
本课题从三个方面研究了网络视频编码器:H.264视频编码算法优化;DSP系统中视频编码应用的硬件加速;H.264视频报文的网络实时传输。
首先,分析算法优化难点,提出了启发式预测模式确定算法,对帧间预测、帧内预测和最佳预测模式的确定过程进行综合优化。该算法利用视频序列在空间和时间上的相关性,以及相邻的图像单位在预测失真上的相似性,推测当前宏块中接近最佳性能的预测块划分方法和预测模式。算法提供了推测错误处理机制,能够保证预测块与原始图像之间实现最佳匹配。实验证明经过优化之后,以微小的编码图像失真为代价,较大幅度地提高了编码速度,编码器的性能得到显著改善。
其次,研究了DSP上实现H.264的关键技术。将保证DSP系统定点运算精度和利用DSP算法实现加速的关键技术作为研究重点。提出了多层次优化设计方案,在程序、算法和指令三个不同层次上,通过优化程序结构、改善存储管理、合理分配资源、优化指令调度、减少cache失效,以及利用多种DMA传输机制实现视频处理与数据传输的重叠作业等优化关键技术,提高DSP系统中的功能部件、存储资源和外部设备的工作效率,实现了视频图像的实时编码。实验结果表明,通过关键技术优化的DSP视频编码器性能有大幅度的提高。
最后,利用德州仪器提供的网络开发工具包实现了RTP/RTCP实时网络传输协议,主要包括TCP/IP网络协议栈的配置与使用,以及实现基于RTP/RTCP协议的应用服务。在网络应用的基础上,对视频编码器进行性能进行测试,检验视频编码器的实时效果;完成了进行数据完整性测试,验证了视频编码器的正确性。测试结果表明本课题研究和实现的视频编码器具有良好的实时性和稳定性。
With the development of multimedia and communication technology, a variety of new applications with high image resolution have been taken into practice. With great perspective, realtime video applications are very popular, in which image quality and bitrates are in the dilemma. H.264 video coding standard can solve this intractability problem successfully. However, there is much complexity in the encoding processes of H.264 encoder. Therefore, how to reduce the complexities of encoding process, as well as to speed up the encoder, becomes the keystone of video application, which is hard to achieve. Therefore, this dissertation researches on three aspects of network video encoder, which include the optimizing algorithm in video encoder, DSP hardware speedup and the real-time transfer of H.264 video packets.
Firstly, by researching on the optimizing algorithms, we propose an algorithm based on method of elicitation to optimize intra prediction, inter prediction and mode decision. By exploiting spacial or temporal correlation of neighbor macroblocks, as well as distortion of neighbor macroblock, the algorithm speculates that the partition method and prediction mode of current macroblock is similar to its neighbors. The mechanism provided by the algorithm ensures the best match between prediction block and original block. The results of test prove that the algorithm improves the encoding rate with little image distortion.
Secondly, our researches focus on high accuracy fix-point arithmetic and DSP hardware speedup of H.264 application. By optimizing the structure of program, improving memory management, and assigning resources properly, we realize optimizations on the program, algorithm and instruction level respectively. High speed data transfers between DSP and peripherals are implemented by EDMA operations. An overlap between data transfers and data processing are introduced by EDMA mechanisms and the flexible configuration of the on chip SRAM of DSP. After the optimization, ALUs, memory and peripherals are sufficiently utilized. The testing results indicate that the high performance of video encoder optimized by DSP optimized algorithms.
Finally, we implement RTP protocol based on the network develop kit provided by TI corporation, which includes the configuration and utilization of TCP/IP protocol. Experimental results are given to evaluate the performance of the video encoder. According to the results, our video encoder is considered to be an application with good performance and high reliability.
引文
[1]Iain E.G.Richardson,Video Codec Design,John Wiley & Sons,Ltd,2002
[2]胡广书,数字信号处理--理论、算法与实现,清华大学出版社,1997年8月
[3]Wiegand,T.;Sullivan,G.J.;Bjntegaard,G.;Luthra,A.,"Overview of the H.264/AVC video coding standard",IEEE Transactions on Circuits and Systems for Video Technology,Vol.13,No.7,pp.560-576,July 2003
[4]José Roberto Alvarez,"Discrepancies in Documentation and Implementation of Sub-pel Interpolation in TML-8(Draft 0)",VCEG document VCEG-N71,Santa Barbara,CA,USA,24-27 Sep.,2001
[5]程云鹏,矩阵论,西北工业大学出版社,2000
[6]A.Hallapuro,M.Karczewicz and H.Malvar,Low Complexity Transform and Quantization - PartⅠ:Basic Implementation,JVT document JVT-B038,Geneva,February 2002
[7]Alexis Michael,Tourapis Docomo Labs USA,Revised H.264/MPEG-4 AVC Reference Software Manual,JVT document JVT-O017,April,2005
[8]Peter List,Anthony Joch,Jani Lainema,Gisle Bjontegaard,and Marta Karczewicz,Adaptive Deblocking Filter,IEEE Transactions on Circuits and Systems for Video Technology,Vol.13,NO.7,July 2003
[9]B.Jeon,"Entropy Coding for H.26L",ITU-T Doc.Q15-J-57,May 2000
[10]D.Marpe,H.Schwarz and T.Wiegand,Context-Based Adaptive Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard,IEEE Transactions on Circuits and Systems for Video Technology,to be published in 2003
[11]Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification(ITU-T Rec.H.264 | ISO/IEC 14496-10 AVC),May,2003
[12]G.Bjφntegaard and K.Lillevold,Context-adaptive VLC coding of coefficients,JVT document JVT-C028,Fairfax,May 2002
[13]S.W.Golomb,Run-length encoding,IEEE Trans.on Inf.Theory,IT-12,pp.399-401,1966
[14]T.Koga,K.Iinuma,A.Hirano,Y.Iijima,and T.Ishiguro,Motion Compensated Interframe Coding for Video Conferencing,Proc.Nat.Telecommun.Conf.,New Orleans,LA,pp.G5.3.1-G5.3.5,Dec'81
[15]J.R.Jain and A.K.Jain,Displacement measurement and its application in interframe image coding,IEEE Trans.on Communications,vol.COM-29,pp.1799-808,Dec'81
[16]R.Li,B.Zeng,and M.L.Liou,A New Three-Step Search Algorithm for Block Motion Estimation, IEEE Trans. Circuits Syst. Video Technol., vol. 4, pp. 438-442,Aug. 1994
[17] Shan Zhu and Kai-Kuang Ma, A New Diamond Search Algorithm for Fast Block-Matching Motion Estimation, IEEE Transactions on Image Processing, Vol.9, No. 2, February 2000
[18] A. M. Tourapis, O. C. Au, and M. L. Liou, New Results on Zonal Based Motion Estimation Algorithms - Advanced Predictive Diamond Zonal Search, in proceedings of 2001 IEEE International Symposium on Circuits and Systems (ISCAS-2001), vol.5, pp.183-186, Sydney, Australia, May 6-9,2001
[19] A. M. Tourapis, O. C. Au, and M. L. Liou, Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) - Enhancing Block Based Motion Estimation, in proceedings of Visual Communications and Image Processing 2001 (VCIP-2001), pp.883-892, San Jose, CA, January 2001
[20] H. Y. Cheong, A. M. Tourapis, and P. Topiwala, Fast Motion Estimation within the JVT codec, ISO/EEC JTC1/SC29/WG11 and ITU-T Q6/SG16, document JVT-E023,Oct'02
[21]Zhibo Chen, Peng Zhou, Yun He, "Fast Integer Pel and Fractional Pel Motion estimation in for JVT", JVT-F017rl.doc, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, 6th meeting, Awaji, Island, JP, 5-13 December, 2002
[22] Zhibo Chen, Peng Zhou, Yun He , Guozhong Wang, Fast Motion Estimation for JVT, JVT-G016, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG(ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), Pattaya, March, 2003
[23] T. Halbach, Performance Comparison: H.26L Intra Coding vs. JPEG2000, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, JVT 4th Meeting Klagenfurt, Austria,July 2002
[24] C. Kim, H.-H. Shih, and C.-C. J. Kuo, Feature-based intra-prediction mode decision for h.264, in Proc. IEEE International Conference on Image Processing,2004
[25] A. K. Jain, Fundamentals of Digital Image Processing Englewood Cliffs, NJ:Prentice-Hall, 1989
[26] Rafael C. Gonzalez, Richard E. Woods, Digital Image Processing, Prentice Hall,2002
[27] K. P. Lim, S. Wu, D. J. Wu, S. Rahardja, X. Lin, F. Pan, Z. G. Li, "Fast INTER Mode Selection," Doc. 1020, Sep. 2003
[28] TI Corp., TVP5150A/TVP5150AM1 Data Manual, May. 2004
[29] 彭启琮,关庆等编著, DSP集成开发环境——CCS 及DSP/BIOS 的原理与应用
[30] TI Corp., TMS320C6000 CPU and Instruction Set Reference Guide, Oct. 2004
[31] TI Corp., Manual Update Sheet of TMS320C6000 CPU and Instruction Set Reference Guide,June. 2004
[32] IETF RFC3550, RTP: A Transport Protocol for Real-Time Applications, July 2003
[33] IETF RFC3984, RTP Payload Format for H.264 Video, Feb. 2005
[34] Microsoft MSDN 2003 Platform SDK
[2]胡广书,数字信号处理--理论、算法与实现,清华大学出版社,1997年8月
[3]Wiegand,T.;Sullivan,G.J.;Bjntegaard,G.;Luthra,A.,"Overview of the H.264/AVC video coding standard",IEEE Transactions on Circuits and Systems for Video Technology,Vol.13,No.7,pp.560-576,July 2003
[4]José Roberto Alvarez,"Discrepancies in Documentation and Implementation of Sub-pel Interpolation in TML-8(Draft 0)",VCEG document VCEG-N71,Santa Barbara,CA,USA,24-27 Sep.,2001
[5]程云鹏,矩阵论,西北工业大学出版社,2000
[6]A.Hallapuro,M.Karczewicz and H.Malvar,Low Complexity Transform and Quantization - PartⅠ:Basic Implementation,JVT document JVT-B038,Geneva,February 2002
[7]Alexis Michael,Tourapis Docomo Labs USA,Revised H.264/MPEG-4 AVC Reference Software Manual,JVT document JVT-O017,April,2005
[8]Peter List,Anthony Joch,Jani Lainema,Gisle Bjontegaard,and Marta Karczewicz,Adaptive Deblocking Filter,IEEE Transactions on Circuits and Systems for Video Technology,Vol.13,NO.7,July 2003
[9]B.Jeon,"Entropy Coding for H.26L",ITU-T Doc.Q15-J-57,May 2000
[10]D.Marpe,H.Schwarz and T.Wiegand,Context-Based Adaptive Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard,IEEE Transactions on Circuits and Systems for Video Technology,to be published in 2003
[11]Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification(ITU-T Rec.H.264 | ISO/IEC 14496-10 AVC),May,2003
[12]G.Bjφntegaard and K.Lillevold,Context-adaptive VLC coding of coefficients,JVT document JVT-C028,Fairfax,May 2002
[13]S.W.Golomb,Run-length encoding,IEEE Trans.on Inf.Theory,IT-12,pp.399-401,1966
[14]T.Koga,K.Iinuma,A.Hirano,Y.Iijima,and T.Ishiguro,Motion Compensated Interframe Coding for Video Conferencing,Proc.Nat.Telecommun.Conf.,New Orleans,LA,pp.G5.3.1-G5.3.5,Dec'81
[15]J.R.Jain and A.K.Jain,Displacement measurement and its application in interframe image coding,IEEE Trans.on Communications,vol.COM-29,pp.1799-808,Dec'81
[16]R.Li,B.Zeng,and M.L.Liou,A New Three-Step Search Algorithm for Block Motion Estimation, IEEE Trans. Circuits Syst. Video Technol., vol. 4, pp. 438-442,Aug. 1994
[17] Shan Zhu and Kai-Kuang Ma, A New Diamond Search Algorithm for Fast Block-Matching Motion Estimation, IEEE Transactions on Image Processing, Vol.9, No. 2, February 2000
[18] A. M. Tourapis, O. C. Au, and M. L. Liou, New Results on Zonal Based Motion Estimation Algorithms - Advanced Predictive Diamond Zonal Search, in proceedings of 2001 IEEE International Symposium on Circuits and Systems (ISCAS-2001), vol.5, pp.183-186, Sydney, Australia, May 6-9,2001
[19] A. M. Tourapis, O. C. Au, and M. L. Liou, Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) - Enhancing Block Based Motion Estimation, in proceedings of Visual Communications and Image Processing 2001 (VCIP-2001), pp.883-892, San Jose, CA, January 2001
[20] H. Y. Cheong, A. M. Tourapis, and P. Topiwala, Fast Motion Estimation within the JVT codec, ISO/EEC JTC1/SC29/WG11 and ITU-T Q6/SG16, document JVT-E023,Oct'02
[21]Zhibo Chen, Peng Zhou, Yun He, "Fast Integer Pel and Fractional Pel Motion estimation in for JVT", JVT-F017rl.doc, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, 6th meeting, Awaji, Island, JP, 5-13 December, 2002
[22] Zhibo Chen, Peng Zhou, Yun He , Guozhong Wang, Fast Motion Estimation for JVT, JVT-G016, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG(ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), Pattaya, March, 2003
[23] T. Halbach, Performance Comparison: H.26L Intra Coding vs. JPEG2000, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, JVT 4th Meeting Klagenfurt, Austria,July 2002
[24] C. Kim, H.-H. Shih, and C.-C. J. Kuo, Feature-based intra-prediction mode decision for h.264, in Proc. IEEE International Conference on Image Processing,2004
[25] A. K. Jain, Fundamentals of Digital Image Processing Englewood Cliffs, NJ:Prentice-Hall, 1989
[26] Rafael C. Gonzalez, Richard E. Woods, Digital Image Processing, Prentice Hall,2002
[27] K. P. Lim, S. Wu, D. J. Wu, S. Rahardja, X. Lin, F. Pan, Z. G. Li, "Fast INTER Mode Selection," Doc. 1020, Sep. 2003
[28] TI Corp., TVP5150A/TVP5150AM1 Data Manual, May. 2004
[29] 彭启琮,关庆等编著, DSP集成开发环境——CCS 及DSP/BIOS 的原理与应用
[30] TI Corp., TMS320C6000 CPU and Instruction Set Reference Guide, Oct. 2004
[31] TI Corp., Manual Update Sheet of TMS320C6000 CPU and Instruction Set Reference Guide,June. 2004
[32] IETF RFC3550, RTP: A Transport Protocol for Real-Time Applications, July 2003
[33] IETF RFC3984, RTP Payload Format for H.264 Video, Feb. 2005
[34] Microsoft MSDN 2003 Platform SDK