基于FPGA的大场景图像融合可视化系统的研究与设计
摘要
随着图像处理技术和投影技术的不断发展,人们对高沉浸感的虚拟现实场景提出了更高的要求,这种虚拟显示的场景往往由多通道的投影仪器同时在屏幕上投影出多幅高清晰的图像,再把这些单独的图像拼接在一起组成一幅大场景的图像。而为了给人以逼真的效果,投影的屏幕往往被设计为柱面屏幕,甚至是球面屏幕。当图像投影在柱面屏幕的时候就会发生几何形状的变化,而避免这种几何变形的就是图像拼接过程中的几何校正和边缘融合技术。
一个大场景可视化系统由投影机、投影屏幕、图像融合机等主要模块组成。在虚拟现实应用系统中,要实现高临感的多屏幕无缝拼接以及曲面组合显示,显示系统还需要运用几何数字变形及边缘融合等图像处理技术,实现诸如在平面、柱面、球面等投影显示面上显示图像。而关键设备在于图像融合机,它实时采集图形服务器,或者PC的图像信号,通过图像处理模块对图像信息进行几何校正和边缘融合,在处理完成后再送到显示设备。
本课题提出了一种基于FPGA技术的图像处理系统。该系统实现图像数据的A/D采集、图像数据在SRAM以及SDRAM中的存取、图像在FPGA内部的DSP运算以及图像数据的D/A输出。系统设计的核心部分在于系统的控制以及数字信号的处理。本课题采用Xilinx Virtex4系列FPGA作为主处理芯片,并利用Verilog HDL硬件描述语言在FPGA内部设计了A/D模块、D/A模块、SRAM、SDRAM以及ARM处理器的控制器逻辑。
本课题在FPGA图像处理系统中设计了一个ARM处理器模块,用于上电时对系统在图像变化处理时所需参数进行传递,并能实时从上位机更新参数。该设计在提高了系统性能的同时也便于系统扩展。
本文首先介绍了图像处理过程中的几何变化和图像融合的算法,接着提出了系统的设计方案及模块划分,然后围绕FPGA的设计介绍了SDRAM控制器的设计方法,最后介绍了ARM处理器的接口及外围电路的设计。
Along with the development of picture processing technology and projection technology, people request to high immersed feeling of virtual reality scene. This kind of virtual reality scene with high sharp image often needs to be projected on the screen by multichannel projection instrument, then compose these independent pictures to be a large scene picture. In order to present a lifelike effect, the projection screens are designed as cylinder screens, or even spherical surface screens, this may bring the result of geometrical change when picture was projected in the cylinder screen. The key to affect imagery effect is the geometrical correction and edge blending technology used in projection splicing process of the picture.
A large scene visible system is composed by the projecting camera, the projection screen, the picture process machine and so on. In the virtual reality application system, in order to realize vivid multi-screens seamless splicing as well as the curved combination demonstration, the display system also needs to utilize picture processing technology, such as geometrical distortion and edge fusion to demonstrate the picture on the plane, cylinder, spherical surface and other projection screens. The essential technology lies in the picture fusion machine which will complete real-time data gathering from the graph server or the PC, and process the image data with geometrical correction and edge blending, then transfer the data to display device.
The thesis presents one kind of picture process system bases on the FPGA technology including following functions: image data’s A/D gathering, deposit and withdrawal in SRAM or SDRAM, DSP in the FPGA and D/A output. The core part of system design lies in the system control and digital signal processing. This project uses Xilinx Virtex4 series FPGA as primarily processor. The A/D module, D/A module, SRAM controller, SDRAM controller and the ARM processor controller logic are designed using Verilog HDL language.
This project designed an ARM processor in the FPGA picture processing system, used to tranfer parameter for image process when system power on, and updata new parameter from PC immediately. Such design enhanced system performance at the same time to be also advantageous for the expansion.
This thesis firstly introduces the geometrical distortion algorithm and the edge fusion algorithm which used in the image processing, secondly proposed the system design proposal and the module division, thirdly introduced the design method of SDRAM controller, finally introduced the ARM processor connection and the periphery electric circuit design.
一个大场景可视化系统由投影机、投影屏幕、图像融合机等主要模块组成。在虚拟现实应用系统中,要实现高临感的多屏幕无缝拼接以及曲面组合显示,显示系统还需要运用几何数字变形及边缘融合等图像处理技术,实现诸如在平面、柱面、球面等投影显示面上显示图像。而关键设备在于图像融合机,它实时采集图形服务器,或者PC的图像信号,通过图像处理模块对图像信息进行几何校正和边缘融合,在处理完成后再送到显示设备。
本课题提出了一种基于FPGA技术的图像处理系统。该系统实现图像数据的A/D采集、图像数据在SRAM以及SDRAM中的存取、图像在FPGA内部的DSP运算以及图像数据的D/A输出。系统设计的核心部分在于系统的控制以及数字信号的处理。本课题采用Xilinx Virtex4系列FPGA作为主处理芯片,并利用Verilog HDL硬件描述语言在FPGA内部设计了A/D模块、D/A模块、SRAM、SDRAM以及ARM处理器的控制器逻辑。
本课题在FPGA图像处理系统中设计了一个ARM处理器模块,用于上电时对系统在图像变化处理时所需参数进行传递,并能实时从上位机更新参数。该设计在提高了系统性能的同时也便于系统扩展。
本文首先介绍了图像处理过程中的几何变化和图像融合的算法,接着提出了系统的设计方案及模块划分,然后围绕FPGA的设计介绍了SDRAM控制器的设计方法,最后介绍了ARM处理器的接口及外围电路的设计。
Along with the development of picture processing technology and projection technology, people request to high immersed feeling of virtual reality scene. This kind of virtual reality scene with high sharp image often needs to be projected on the screen by multichannel projection instrument, then compose these independent pictures to be a large scene picture. In order to present a lifelike effect, the projection screens are designed as cylinder screens, or even spherical surface screens, this may bring the result of geometrical change when picture was projected in the cylinder screen. The key to affect imagery effect is the geometrical correction and edge blending technology used in projection splicing process of the picture.
A large scene visible system is composed by the projecting camera, the projection screen, the picture process machine and so on. In the virtual reality application system, in order to realize vivid multi-screens seamless splicing as well as the curved combination demonstration, the display system also needs to utilize picture processing technology, such as geometrical distortion and edge fusion to demonstrate the picture on the plane, cylinder, spherical surface and other projection screens. The essential technology lies in the picture fusion machine which will complete real-time data gathering from the graph server or the PC, and process the image data with geometrical correction and edge blending, then transfer the data to display device.
The thesis presents one kind of picture process system bases on the FPGA technology including following functions: image data’s A/D gathering, deposit and withdrawal in SRAM or SDRAM, DSP in the FPGA and D/A output. The core part of system design lies in the system control and digital signal processing. This project uses Xilinx Virtex4 series FPGA as primarily processor. The A/D module, D/A module, SRAM controller, SDRAM controller and the ARM processor controller logic are designed using Verilog HDL language.
This project designed an ARM processor in the FPGA picture processing system, used to tranfer parameter for image process when system power on, and updata new parameter from PC immediately. Such design enhanced system performance at the same time to be also advantageous for the expansion.
This thesis firstly introduces the geometrical distortion algorithm and the edge fusion algorithm which used in the image processing, secondly proposed the system design proposal and the module division, thirdly introduced the design method of SDRAM controller, finally introduced the ARM processor connection and the periphery electric circuit design.
引文
[1] Kai Li, Han Chen. Building and Using A Scalable Display Wall System. IEEE Computer Graphics and Applications, July/August 2000, P29-37
[2] Daniel R. Schikore, Richard A. Fischer. High-Resolution Multiprojector Display Walls. IEEE Computer Graphics and Applications, July/August 2000, P38-44
[3] Shigeru Muraki, Masato Ogata. Next-Generation Visual Supercomputing using PC Clusters with Volume Graphics Hardware Device. Proceedings of Supercomputing 2001 Conference, November 10-16, 2001
[4] Princeton Display Wall. http://www.cs.princeton.edu/omnimedia/
[5] Raskar, R. 2000. Immersive Planar Display using Roughly Aligned Projectors. In IEEE VR 2000: P27-34
[6]林柏纬.PC构架的多投影面沉浸式虚拟环境及其应用:[研究学位论文].杭州:浙江大学,2003
[7]王纪会.图像校正及图像镶嵌:[研究学位论文].西安:西北工业大学,2002
[8] Kenneth R. Castleman著.朱志刚,林学訚,石定机译.数字图像处理.北京:电子工业出版社.2002.2
[9]刘勇奎.计算机图形学的基础算法.北京:科学出版社,2001.8
[10]巨安丽,郝重阳,徐建荣.边缘融合组合屏图像显示技术与系统研究.计算机仿真.2006,23(2): P267-269
[11]贾庆轩,宋荆洲,孙汉旭等.高临场感多投影面虚拟环境系统的设计与实现.中国工程科学.2006,8(8): P33-38
[12] M.C.Stone. Color balancing experimental projection displays. In 9th IS&t/SID Color Imaging Conference, April 2001.
[13] G. Wallace, H. Chen and K. Li . Color Gamut Matching for Tiled Display Walls. Eurographics Workshop on Virtual Environments, 2003.
[14]孙颖力.图像融合算法的研究及可重构FPGA实现:[硕士学位论文].西安:西北工业大学,2007
[15]贾庆轩,阮瑞卿,孙汉旭等.多投影面显示系统亮度均衡的实现.系统仿真学报.2006, 18: P478-482
[16]姜雪松,张海风.可编程逻辑器件和EDA设计技术.北京:机械工业出版社,2006,15-44,231-237
[17]吴继华,王诚著.设计与验证Verilog HDL.北京.人民邮电出版社.2006.8
[18] J.Bhasker著.孙海平等译.Verilog HDL综合实用教程.北京:清华大学出版社,2004.1
[19] Samir Palnitkar著.夏宇闻,胡燕祥,刁岚松等译.Verilog HDL数字设计与综合.北京.电子工业出版社.2004.11
[20]黄思敏.视频图像几何校正卡的设计与实现:[研究学位论文].成都:电子科技大学,2005
[21]王海涛.高分辨率实时图像处理系统的FPGA设计及实现:[硕士学位论文].成都:电子科技大学,2006
[22]万鸣华.基于FPGA的图像处理加速研究:[硕士学位论文].南京:南京理工大学,2007
[23]徐欣,于红旗,易凡等.基于FPGA的嵌入式系统设计.北京:机械工业出版社, 2004.9: 1-18
[24]龚涛.基于FPGA的图像采集处理系统:[研究学位论文] .武汉:华中科技大学,2005
[25]刘虹,黄涛.基于FPGA的高速图像采集系统的研究与实现.工业控制计算机.2004,17(8): P18-19
[26] Xilinx Inc. DVI, VGA and Component Video Demonstration (ug208) .Oct. 2006
[27] Analog Devices. Dual Interface for Flat Panel Displays: AD9887A, 2003
[28] Analog Devices. COMS, 240 MHZ Triple 10-Bit High Speed Video DAC: ADV7123 , 1998
[29] Silicon Image. SiI164 PanelLink Transmitter Data Sheet, 2002
[30] Xilinx inc. Virtex-4 user guide(ug070).Apr.2007
[31] Xilinx Inc. XtremeDSP for Virtex-4 FPGAs User Guide UG073 (v2.5), 2007
[32]刘明章.基于FPGA的嵌入式系统设计.北京:国防工业出版社,2007
[33]曹华.一种高分辨率视频图像处理系统的数据流控制设计实现:[硕士学位论文].成都:电子科技大学,2005
[34]刘斌兵,刘云海,汪燮彬.用FPGA实现高速大图像采集系统.电子技术,2006,60-63
[35]尹乐.基于FPGA和DSP的图像多功能卡的设计与实现:[硕士学位论文].成都:电子科技大学,2004
[36] Uwe Meyer-Baese著.刘凌,胡永生译.数字信号处理的FPGA实现.北京:清华大学出版社.2003.1
[37] CYPRESS.72Mb (2M×36/4M×18/1M×72) Pipelined Sync SRAM: CY7C1480V33, 2004
[38] MICRON. 128Mb (1M×32×4) Synchronous DRAM: MT48LC4M32B2, 2002
[39]段小晨,何小刚,程永强.实时视频SDRAM控制器的FPGA设计与实现.太原理工大学学报.2006, 37: P5-8
[40]李卫,王杉,魏急波.SDRAM控制器的FPGA设计与实现.电子工程师.2004,30(10): P29-31
[41] Xilinx Inc. Memory Interface Generator (MIG) 1.7 User Guide, 2007
[42]薛小刚,葛毅敏.Xilinx ISE 9.X FPGA/CPLD设计指南.北京:人民邮电出版社,2007,303-315
[43] Xilinx Inc.synthesizable high performance SDRAM controller. Dec.2005
[44]于明,范书瑞,曾祥烨. ARM9嵌入式系统设计与开发教程.北京:电子工业出版社.2006: P108-116
[45]吴做勇,韩新洁,孟守尧.基于FPGA和ARM的彩色图像处理系统.计算机应用,2007,26(4):67-69
[46] Samsung Electronics. USER’S MANUAL S3C2410A Revision 1.0. 2005: P190-221
[47] ARM Limited. ARM920T Technical Reference Manual Rev 1. 2001: P24-26
[48]周立功. ARM嵌入式系统软件开发实例.北京:北京航空航天大学出版社.2005: P242-265
[49]马洪连,丁男,李屹璐等著.嵌入式系统设计教程.北京.电子工业出版社.2006.6
[50]王应莉.基于ARM+DSP构架的视频处理系统:[硕士学位论文].成都:电子科技大学,2007
[51] HYNIX.HY57V561620 4 Bank×4M×16bit Synchronous DRAM. 2004: P2-5
[52] Samsung Electronics. K9F1208U0M Flash Memory. 2001-4: P5-9
[53] Cirrus logic inc. Crystal LAN ISA Ethernet Controller: CS8900. Mar.1999
[2] Daniel R. Schikore, Richard A. Fischer. High-Resolution Multiprojector Display Walls. IEEE Computer Graphics and Applications, July/August 2000, P38-44
[3] Shigeru Muraki, Masato Ogata. Next-Generation Visual Supercomputing using PC Clusters with Volume Graphics Hardware Device. Proceedings of Supercomputing 2001 Conference, November 10-16, 2001
[4] Princeton Display Wall. http://www.cs.princeton.edu/omnimedia/
[5] Raskar, R. 2000. Immersive Planar Display using Roughly Aligned Projectors. In IEEE VR 2000: P27-34
[6]林柏纬.PC构架的多投影面沉浸式虚拟环境及其应用:[研究学位论文].杭州:浙江大学,2003
[7]王纪会.图像校正及图像镶嵌:[研究学位论文].西安:西北工业大学,2002
[8] Kenneth R. Castleman著.朱志刚,林学訚,石定机译.数字图像处理.北京:电子工业出版社.2002.2
[9]刘勇奎.计算机图形学的基础算法.北京:科学出版社,2001.8
[10]巨安丽,郝重阳,徐建荣.边缘融合组合屏图像显示技术与系统研究.计算机仿真.2006,23(2): P267-269
[11]贾庆轩,宋荆洲,孙汉旭等.高临场感多投影面虚拟环境系统的设计与实现.中国工程科学.2006,8(8): P33-38
[12] M.C.Stone. Color balancing experimental projection displays. In 9th IS&t/SID Color Imaging Conference, April 2001.
[13] G. Wallace, H. Chen and K. Li . Color Gamut Matching for Tiled Display Walls. Eurographics Workshop on Virtual Environments, 2003.
[14]孙颖力.图像融合算法的研究及可重构FPGA实现:[硕士学位论文].西安:西北工业大学,2007
[15]贾庆轩,阮瑞卿,孙汉旭等.多投影面显示系统亮度均衡的实现.系统仿真学报.2006, 18: P478-482
[16]姜雪松,张海风.可编程逻辑器件和EDA设计技术.北京:机械工业出版社,2006,15-44,231-237
[17]吴继华,王诚著.设计与验证Verilog HDL.北京.人民邮电出版社.2006.8
[18] J.Bhasker著.孙海平等译.Verilog HDL综合实用教程.北京:清华大学出版社,2004.1
[19] Samir Palnitkar著.夏宇闻,胡燕祥,刁岚松等译.Verilog HDL数字设计与综合.北京.电子工业出版社.2004.11
[20]黄思敏.视频图像几何校正卡的设计与实现:[研究学位论文].成都:电子科技大学,2005
[21]王海涛.高分辨率实时图像处理系统的FPGA设计及实现:[硕士学位论文].成都:电子科技大学,2006
[22]万鸣华.基于FPGA的图像处理加速研究:[硕士学位论文].南京:南京理工大学,2007
[23]徐欣,于红旗,易凡等.基于FPGA的嵌入式系统设计.北京:机械工业出版社, 2004.9: 1-18
[24]龚涛.基于FPGA的图像采集处理系统:[研究学位论文] .武汉:华中科技大学,2005
[25]刘虹,黄涛.基于FPGA的高速图像采集系统的研究与实现.工业控制计算机.2004,17(8): P18-19
[26] Xilinx Inc. DVI, VGA and Component Video Demonstration (ug208) .Oct. 2006
[27] Analog Devices. Dual Interface for Flat Panel Displays: AD9887A, 2003
[28] Analog Devices. COMS, 240 MHZ Triple 10-Bit High Speed Video DAC: ADV7123 , 1998
[29] Silicon Image. SiI164 PanelLink Transmitter Data Sheet, 2002
[30] Xilinx inc. Virtex-4 user guide(ug070).Apr.2007
[31] Xilinx Inc. XtremeDSP for Virtex-4 FPGAs User Guide UG073 (v2.5), 2007
[32]刘明章.基于FPGA的嵌入式系统设计.北京:国防工业出版社,2007
[33]曹华.一种高分辨率视频图像处理系统的数据流控制设计实现:[硕士学位论文].成都:电子科技大学,2005
[34]刘斌兵,刘云海,汪燮彬.用FPGA实现高速大图像采集系统.电子技术,2006,60-63
[35]尹乐.基于FPGA和DSP的图像多功能卡的设计与实现:[硕士学位论文].成都:电子科技大学,2004
[36] Uwe Meyer-Baese著.刘凌,胡永生译.数字信号处理的FPGA实现.北京:清华大学出版社.2003.1
[37] CYPRESS.72Mb (2M×36/4M×18/1M×72) Pipelined Sync SRAM: CY7C1480V33, 2004
[38] MICRON. 128Mb (1M×32×4) Synchronous DRAM: MT48LC4M32B2, 2002
[39]段小晨,何小刚,程永强.实时视频SDRAM控制器的FPGA设计与实现.太原理工大学学报.2006, 37: P5-8
[40]李卫,王杉,魏急波.SDRAM控制器的FPGA设计与实现.电子工程师.2004,30(10): P29-31
[41] Xilinx Inc. Memory Interface Generator (MIG) 1.7 User Guide, 2007
[42]薛小刚,葛毅敏.Xilinx ISE 9.X FPGA/CPLD设计指南.北京:人民邮电出版社,2007,303-315
[43] Xilinx Inc.synthesizable high performance SDRAM controller. Dec.2005
[44]于明,范书瑞,曾祥烨. ARM9嵌入式系统设计与开发教程.北京:电子工业出版社.2006: P108-116
[45]吴做勇,韩新洁,孟守尧.基于FPGA和ARM的彩色图像处理系统.计算机应用,2007,26(4):67-69
[46] Samsung Electronics. USER’S MANUAL S3C2410A Revision 1.0. 2005: P190-221
[47] ARM Limited. ARM920T Technical Reference Manual Rev 1. 2001: P24-26
[48]周立功. ARM嵌入式系统软件开发实例.北京:北京航空航天大学出版社.2005: P242-265
[49]马洪连,丁男,李屹璐等著.嵌入式系统设计教程.北京.电子工业出版社.2006.6
[50]王应莉.基于ARM+DSP构架的视频处理系统:[硕士学位论文].成都:电子科技大学,2007
[51] HYNIX.HY57V561620 4 Bank×4M×16bit Synchronous DRAM. 2004: P2-5
[52] Samsung Electronics. K9F1208U0M Flash Memory. 2001-4: P5-9
[53] Cirrus logic inc. Crystal LAN ISA Ethernet Controller: CS8900. Mar.1999