双向预测帧率变换算法的研究及硬件电路的设计和功能实现
摘要
随着科技的进步与社会的发展,我国将于2015年全面进入数字电视时代。液晶面板电视将取代传统的显像管电视成为数字电视时代中的佼佼者,但是液晶面板在器件物理特性上的原因会造成图像的残影(shadow)、运动模糊(motion blur)等现象,尤其是当画面表现快速运动场景时运动模糊现象会非常突出。
为解决液晶电视图像在快速运动中的残影,以及图像模糊闪烁等技术问题,本文提出采用双向预测算法来提升帧率。将基于经验阈值的双向运动估计运动补偿算法应用于帧率倍频处理中,解决提升帧率时的内插帧误差过大,减少块效应和图像空洞的问题。并且提出该算法的硬件架构,用图像序列进行软件仿真,得出主观评价良好的内插帧图像效果。
本文正是在上述背景下提出来,主要研究双向运动补偿算法的实现,运动估计器及DDR控制模块的硬件实现架构研究。本文主要工作如下:
(1)实现了用双向预测运动估计的算法提升帧率;
(2)提出了双向预测的运动估计器的硬件架构;验证奇偶阵列运动估计器的功能;
(3)实现了DDR内存的读写功能,测试通过了DDR内存的读写的功能;
(4)提出了系统的流水化处理方案,降低硬件开销,并且给出了每个子模块的状态机的逻辑控制转移图;
(5)提出了实现系统的硬件架构,设计输入输出接口,以及硬件环境的调试。
本文首先介绍运动估计和运动补偿的理论算法,接着阐述实现双向预测运动估计的算法,并且得到软件仿真的实验结果,然后提出基于双向预测运动估计的帧率提升算法的硬件架构,包括运动估计器,DDR存储设备的读写,各层次流水线的架构设计,以及每个子模块的状态机。最后给出帧率提升算法的仿真结果,以及系统基本调试平台的搭建结果。
最后本文进行了帧率提升算法的总结和未来的发展趋势。
With technology inproving and society developing, our country will totally use digital TV in 2015. LCD Panel will replace the tradional electric image tabe TV, and it may become a super one in the digital TV area. But LCD panel will cause some image shadow, motion blur problems with its physical structure, espsially the image moves so rapidly.
To solve the image shadow in rapid moving, and image flick problems. This thesise report an bidirection predict algorithm to up frame which based on experience value. It can reduce the torlance about insert the additional frame, and it can also reduce the block effect and white hole issues. And this thesis also report the hardware architecture for the algorithm,then using image seirial to do software simulation. Concluding a nice comment for the inserting additional frame.
This thesis is published in this background, it focus on how to realistic bidirectional motion compensate algorithm. And study the hardware architecture for motion estimate and double frequency. This article focus on some expects below:
1:realistic bidirectional predict frame up algorithm
2: propose the motion estimate of the hardware architecture for bidirectional predict, verified the function and timing of even-odd array motion estimator;
3: realistic the funciton for DDR Ram read and wright,and verified the funtion
4: propose the system architecture by using pipeline, it can reduce the hardware cost, and give each module status machine logical control transfer chart.
5: To propose the system hardware architecture, design the input and output interace, and the debug hardware environment.
This thesis reviewed motion estimate and motion conpensate algorithm, then state the bidirectional predict motion estimate algorithm meanwhile get the test conclution. In Charpter 3 this paper propose the hardware architecture for frame up algorithm which based on bidirectional predict motin estimate, include motion estimate module, DDR read and wright function. In Charpter 4, it presents each module’s pipeline architecture and their status machine. In the last this thesis shows the result of software simulation about frame up algorithm by the test bench.
Last but not least this paper predict the future of the frame up algorithm and give its concluetion.
为解决液晶电视图像在快速运动中的残影,以及图像模糊闪烁等技术问题,本文提出采用双向预测算法来提升帧率。将基于经验阈值的双向运动估计运动补偿算法应用于帧率倍频处理中,解决提升帧率时的内插帧误差过大,减少块效应和图像空洞的问题。并且提出该算法的硬件架构,用图像序列进行软件仿真,得出主观评价良好的内插帧图像效果。
本文正是在上述背景下提出来,主要研究双向运动补偿算法的实现,运动估计器及DDR控制模块的硬件实现架构研究。本文主要工作如下:
(1)实现了用双向预测运动估计的算法提升帧率;
(2)提出了双向预测的运动估计器的硬件架构;验证奇偶阵列运动估计器的功能;
(3)实现了DDR内存的读写功能,测试通过了DDR内存的读写的功能;
(4)提出了系统的流水化处理方案,降低硬件开销,并且给出了每个子模块的状态机的逻辑控制转移图;
(5)提出了实现系统的硬件架构,设计输入输出接口,以及硬件环境的调试。
本文首先介绍运动估计和运动补偿的理论算法,接着阐述实现双向预测运动估计的算法,并且得到软件仿真的实验结果,然后提出基于双向预测运动估计的帧率提升算法的硬件架构,包括运动估计器,DDR存储设备的读写,各层次流水线的架构设计,以及每个子模块的状态机。最后给出帧率提升算法的仿真结果,以及系统基本调试平台的搭建结果。
最后本文进行了帧率提升算法的总结和未来的发展趋势。
With technology inproving and society developing, our country will totally use digital TV in 2015. LCD Panel will replace the tradional electric image tabe TV, and it may become a super one in the digital TV area. But LCD panel will cause some image shadow, motion blur problems with its physical structure, espsially the image moves so rapidly.
To solve the image shadow in rapid moving, and image flick problems. This thesise report an bidirection predict algorithm to up frame which based on experience value. It can reduce the torlance about insert the additional frame, and it can also reduce the block effect and white hole issues. And this thesis also report the hardware architecture for the algorithm,then using image seirial to do software simulation. Concluding a nice comment for the inserting additional frame.
This thesis is published in this background, it focus on how to realistic bidirectional motion compensate algorithm. And study the hardware architecture for motion estimate and double frequency. This article focus on some expects below:
1:realistic bidirectional predict frame up algorithm
2: propose the motion estimate of the hardware architecture for bidirectional predict, verified the function and timing of even-odd array motion estimator;
3: realistic the funciton for DDR Ram read and wright,and verified the funtion
4: propose the system architecture by using pipeline, it can reduce the hardware cost, and give each module status machine logical control transfer chart.
5: To propose the system hardware architecture, design the input and output interace, and the debug hardware environment.
This thesis reviewed motion estimate and motion conpensate algorithm, then state the bidirectional predict motion estimate algorithm meanwhile get the test conclution. In Charpter 3 this paper propose the hardware architecture for frame up algorithm which based on bidirectional predict motin estimate, include motion estimate module, DDR read and wright function. In Charpter 4, it presents each module’s pipeline architecture and their status machine. In the last this thesis shows the result of software simulation about frame up algorithm by the test bench.
Last but not least this paper predict the future of the frame up algorithm and give its concluetion.
引文
[1] Dane and Truong Q. Nguyen. Motion vector processing for frame rate up conversion. IEEE ICASSP 2004
[2] Taehyeun Ha. Motion Compensated Frame Interpolation by new Block-based Motion Estimation Algorithm. IEEE Transactions on Consumer 754 Electronics, Vol. 50, No. 2, MAY 2004
[3] Lei Zhang, Ci Wang, Wenjun Zhang et al,“Frame rate up-conversion using edge-weighted motion estimation and trilateral Interpolation”, in IEEE International Symposium on Circuits and Systems, 2009, 5: 1637–1640
[4] L. C. Liu, J. C. Chien, Y. H. Chuang and C. C. Li,“A frame-level FSBM motion estimation architecture with large search range,”IEEE Conference on Advanced Video and Signal Based Surveillance, Miami, USA, 2003, pp. 327-333.
[5] H. Yeo, Y. H. Hu,“A novel modular systolic array architecture for full-search block-matching motion estimation,”IEEE Transactions on Circuits and Systems for Video Technology, 5(5): 407-416, 1995.
[6] J. C. Tuan, T. S. Chang, and C. W. Jen,“On the data reuse and memory bandwidth analysis for full-search block-matching VLSI architecture,”IEEE Transactions on Circuits and Systems for Video Technology, 12(1): 61-72, 2002.
[7]株式会社东芝,运动矢量检测设备、运动矢量检测方法及内插帧创建设备,中国,CN101316365A,2008
[8] Konstantinos et.al. A real-time motion estimation FPGA architecture. J Real-Time Image Proc (2008) 3:3–20.
[9] Scott Fischaber, Roger Woods, John McAllister. SoC Memory Hierarchy Derivation from Dataflow Graphs. The Journal of Signal Processing Systems. SiPS '07
[10] Y.K. Chen, A. Vetro, H. Sun et al, Frame-rate up-conversion using transmitted true motion vectors, in Proc. IEEE Workshop Multimedia Signal Process., 1998, 12(2): 622–627
[11] T. Y. Kuo and C.C. J. Kuo. Motion-compensated interpolation for low-bit-rate video quality enhancement, Proc. SPIE Vis. Commun Image Process., 1998, 6: 277–288 [12 A. Kaup and T. Aach, Efficient prediction of uncovered background in interframe coding spatial extraprediction, in Proc. ICASSP, 1994, 4(5): 501–504
[13] Y.B. Zhang, D.B. Zhao, X.Y. Ji et al, A spatio-temporal autoregressive frame rate up conversion scheme, in Proc. ICIP, 2007, 7(1):441–444
[14]何小海.图像通信.西安:西安电子科技大学出版社. 2005:149-151.
[15]何小海,王正勇.数字图像通信及其应用.四川:四川大学出版社. 2006:193-196.
[16]毕厚杰.新一代视频压缩编码标准--H.264/AVC.北京人民邮电出版社. 2005:37-38
[17] Steve Kilts. Advanced FPGA Design, Architecture, Implementation, and Optimization. John Wiley & Sons, Inc., Hoboken, New Jersey.
[18] B.T. Choi, S.H. Lee and S.J. Ko, New frame rate up-conversion using bi-directional motion estimation, IEEE Transaction on Consumer Electronics, 2000, 46(3): 603-609
[19] B.D. Choi, J.W. Han, C.S. Kim et al, Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation, IEEE Trans. Circuits Syst. Video Technol., 2007, 17(4):407–416
[20]李莉,侯正信,一种自适应帧频提升算法研究,计算机应用研究,2009,26(4):1575-1578
[21] G. Dane and T.Q. Nguyen, Smooth motion vector resampling for standard compatible video post-procssing, Proc. Asilomar Conf. Signals, Systems and Computers, 2004, 2: 1731-1734 [22 M.E.A. Mualla, C.N. Canagarajah and D.R. Bull, Motion field interpolation for temporal error concealment, IEEE proc. Vis. Image Signal Process, 2000, 147(5): 445-453
[23] G. Dane, and T.Q. Nguyen, Optimal temporal interpolation filter for motion-compensated frame rate up comversion, IEEE Trans. Image Process., 2006, 15(4): 978–991, 2006
[24] S.J. Kang and Y.H. Kim, Motion compensated frame up-conversion using extended bilateral motion estimation, IEEE Transactions on Consumer Electronics, 2007, 53(4): 1759-1767
[25] K. M. Yang, M. T. Sun, and L. Wu, A family of VLSI designs for the motion compensation block-matching algorithm, IEEE Trans. Circuits Syst. Video Technol., vol. 36, pp. 1317–1325, Oct. 1989.
[26] S. H. Nam and M. K. Lee, Flexible VLSI architecture of motion estimator for video image application, IEEE Trans. Circuits Syst. II, vol. 43, pp. 467–470, June 1996.
[27] L. De Vos and M. Stegherr, Parameterizable VLSI architectures for the full-search block-matching algorithm, IEEE Trans. Circuits Syst., vol. 36, pp. 1309–1316, Oct. 1989.
[28] C. H. Hsieh and T. P. Lin, VLSI architecture for block-matching motion estimation algorithm, IEEE Trans. Circuits Syst. Video Technol., vol. 2, pp. 169–175, June 1992.
[29] Yeong-Kang Lai and Liang-Gee Chen,A Data-Interlacing Architecture with Two-Dimensional Data-Reuse for Full-Search Block-Matching Algorithm, IEEE Trans. Circuits Syst. Video Technol, vol. 8, NO. 2, April 1998
[30] Wolf, W., High-Performance Embedded Computing: Architectures, Applications, and Methodologies, Morgan Kaufman, San Francisco, CA, USA, 2006.
[31] Fischaber, S., Memory-Centric System Level Design of Heterogeneous Embedded DSP Systems, PhD Thesis, Queen’s University Belfast, 2007.
[32] S. Stuijk, et al., Exploring Trade-offs in Buffer Require-ments and Throughput Constraints for Synchronous Dataflow Graphs, in Conf. on Design Automation, pp. 899-904, 2006.
[33] Fischaber, S., Memory-Centric System Level Design of Heterogeneous Embedded DSP Systems, PhD Thesis, Queen’s University Belfast, 2007.
[34] E. Brockmeyer et al., Low Power Memory Storage and Transfer Organization for the MPEG-4 Full Pel Motion Estimation on a Multi Media Processor, IEEE Trans. on Multimedia, Vol. 1, No. 2, pp. 202-216, June 1999.
[35] Virtex 4 Family Overview, Version 1.6, Xilinx, Inc., 2006, available at www.xilinx.com, April, 2007.
[36] R. M. Ali, DDR2 SDRAM Interfaces for Next-gen Systems, in Electronic Engineering Times-Asia, Oct. 16, 2006.
[37] J.-C. Tuan, et al., On the Data Reuse and Memory Bandwidth Analysis for Full-Search Block–Matching BLSI Architecture, in IEEE Trans. On Circuits and Systems for Video Technology, Vol. 12, No. 1, Jan. 2002.
[38] Carl Hamacher等著.张红光,张健民译.北京:机械工业出版社. 2004:319-321.
[39]吴继华,王诚.设计与验证:Verilog HDL.北京:人民邮电出版社. 2006:199-201.
[40] John F. Wakerly著.林生,葛红,金京林译.数字设计原理与实践.北京:机械工业出版社. 2007:387-388.
[41] Michael D. Ciletti著.张雅绮,李锵等译.高级数字设计.北京:电子工业出版社. 2005:56-57.
[42] Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest, Introduction to Algorithms (MIT Press, 2000).
[43]孙世鹏,李斌桥.用于帧频提升的三步搜索及其硬件实现.北京:电子应用技术. 2006.2.
[2] Taehyeun Ha. Motion Compensated Frame Interpolation by new Block-based Motion Estimation Algorithm. IEEE Transactions on Consumer 754 Electronics, Vol. 50, No. 2, MAY 2004
[3] Lei Zhang, Ci Wang, Wenjun Zhang et al,“Frame rate up-conversion using edge-weighted motion estimation and trilateral Interpolation”, in IEEE International Symposium on Circuits and Systems, 2009, 5: 1637–1640
[4] L. C. Liu, J. C. Chien, Y. H. Chuang and C. C. Li,“A frame-level FSBM motion estimation architecture with large search range,”IEEE Conference on Advanced Video and Signal Based Surveillance, Miami, USA, 2003, pp. 327-333.
[5] H. Yeo, Y. H. Hu,“A novel modular systolic array architecture for full-search block-matching motion estimation,”IEEE Transactions on Circuits and Systems for Video Technology, 5(5): 407-416, 1995.
[6] J. C. Tuan, T. S. Chang, and C. W. Jen,“On the data reuse and memory bandwidth analysis for full-search block-matching VLSI architecture,”IEEE Transactions on Circuits and Systems for Video Technology, 12(1): 61-72, 2002.
[7]株式会社东芝,运动矢量检测设备、运动矢量检测方法及内插帧创建设备,中国,CN101316365A,2008
[8] Konstantinos et.al. A real-time motion estimation FPGA architecture. J Real-Time Image Proc (2008) 3:3–20.
[9] Scott Fischaber, Roger Woods, John McAllister. SoC Memory Hierarchy Derivation from Dataflow Graphs. The Journal of Signal Processing Systems. SiPS '07
[10] Y.K. Chen, A. Vetro, H. Sun et al, Frame-rate up-conversion using transmitted true motion vectors, in Proc. IEEE Workshop Multimedia Signal Process., 1998, 12(2): 622–627
[11] T. Y. Kuo and C.C. J. Kuo. Motion-compensated interpolation for low-bit-rate video quality enhancement, Proc. SPIE Vis. Commun Image Process., 1998, 6: 277–288 [12 A. Kaup and T. Aach, Efficient prediction of uncovered background in interframe coding spatial extraprediction, in Proc. ICASSP, 1994, 4(5): 501–504
[13] Y.B. Zhang, D.B. Zhao, X.Y. Ji et al, A spatio-temporal autoregressive frame rate up conversion scheme, in Proc. ICIP, 2007, 7(1):441–444
[14]何小海.图像通信.西安:西安电子科技大学出版社. 2005:149-151.
[15]何小海,王正勇.数字图像通信及其应用.四川:四川大学出版社. 2006:193-196.
[16]毕厚杰.新一代视频压缩编码标准--H.264/AVC.北京人民邮电出版社. 2005:37-38
[17] Steve Kilts. Advanced FPGA Design, Architecture, Implementation, and Optimization. John Wiley & Sons, Inc., Hoboken, New Jersey.
[18] B.T. Choi, S.H. Lee and S.J. Ko, New frame rate up-conversion using bi-directional motion estimation, IEEE Transaction on Consumer Electronics, 2000, 46(3): 603-609
[19] B.D. Choi, J.W. Han, C.S. Kim et al, Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation, IEEE Trans. Circuits Syst. Video Technol., 2007, 17(4):407–416
[20]李莉,侯正信,一种自适应帧频提升算法研究,计算机应用研究,2009,26(4):1575-1578
[21] G. Dane and T.Q. Nguyen, Smooth motion vector resampling for standard compatible video post-procssing, Proc. Asilomar Conf. Signals, Systems and Computers, 2004, 2: 1731-1734 [22 M.E.A. Mualla, C.N. Canagarajah and D.R. Bull, Motion field interpolation for temporal error concealment, IEEE proc. Vis. Image Signal Process, 2000, 147(5): 445-453
[23] G. Dane, and T.Q. Nguyen, Optimal temporal interpolation filter for motion-compensated frame rate up comversion, IEEE Trans. Image Process., 2006, 15(4): 978–991, 2006
[24] S.J. Kang and Y.H. Kim, Motion compensated frame up-conversion using extended bilateral motion estimation, IEEE Transactions on Consumer Electronics, 2007, 53(4): 1759-1767
[25] K. M. Yang, M. T. Sun, and L. Wu, A family of VLSI designs for the motion compensation block-matching algorithm, IEEE Trans. Circuits Syst. Video Technol., vol. 36, pp. 1317–1325, Oct. 1989.
[26] S. H. Nam and M. K. Lee, Flexible VLSI architecture of motion estimator for video image application, IEEE Trans. Circuits Syst. II, vol. 43, pp. 467–470, June 1996.
[27] L. De Vos and M. Stegherr, Parameterizable VLSI architectures for the full-search block-matching algorithm, IEEE Trans. Circuits Syst., vol. 36, pp. 1309–1316, Oct. 1989.
[28] C. H. Hsieh and T. P. Lin, VLSI architecture for block-matching motion estimation algorithm, IEEE Trans. Circuits Syst. Video Technol., vol. 2, pp. 169–175, June 1992.
[29] Yeong-Kang Lai and Liang-Gee Chen,A Data-Interlacing Architecture with Two-Dimensional Data-Reuse for Full-Search Block-Matching Algorithm, IEEE Trans. Circuits Syst. Video Technol, vol. 8, NO. 2, April 1998
[30] Wolf, W., High-Performance Embedded Computing: Architectures, Applications, and Methodologies, Morgan Kaufman, San Francisco, CA, USA, 2006.
[31] Fischaber, S., Memory-Centric System Level Design of Heterogeneous Embedded DSP Systems, PhD Thesis, Queen’s University Belfast, 2007.
[32] S. Stuijk, et al., Exploring Trade-offs in Buffer Require-ments and Throughput Constraints for Synchronous Dataflow Graphs, in Conf. on Design Automation, pp. 899-904, 2006.
[33] Fischaber, S., Memory-Centric System Level Design of Heterogeneous Embedded DSP Systems, PhD Thesis, Queen’s University Belfast, 2007.
[34] E. Brockmeyer et al., Low Power Memory Storage and Transfer Organization for the MPEG-4 Full Pel Motion Estimation on a Multi Media Processor, IEEE Trans. on Multimedia, Vol. 1, No. 2, pp. 202-216, June 1999.
[35] Virtex 4 Family Overview, Version 1.6, Xilinx, Inc., 2006, available at www.xilinx.com, April, 2007.
[36] R. M. Ali, DDR2 SDRAM Interfaces for Next-gen Systems, in Electronic Engineering Times-Asia, Oct. 16, 2006.
[37] J.-C. Tuan, et al., On the Data Reuse and Memory Bandwidth Analysis for Full-Search Block–Matching BLSI Architecture, in IEEE Trans. On Circuits and Systems for Video Technology, Vol. 12, No. 1, Jan. 2002.
[38] Carl Hamacher等著.张红光,张健民译.北京:机械工业出版社. 2004:319-321.
[39]吴继华,王诚.设计与验证:Verilog HDL.北京:人民邮电出版社. 2006:199-201.
[40] John F. Wakerly著.林生,葛红,金京林译.数字设计原理与实践.北京:机械工业出版社. 2007:387-388.
[41] Michael D. Ciletti著.张雅绮,李锵等译.高级数字设计.北京:电子工业出版社. 2005:56-57.
[42] Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest, Introduction to Algorithms (MIT Press, 2000).
[43]孙世鹏,李斌桥.用于帧频提升的三步搜索及其硬件实现.北京:电子应用技术. 2006.2.