基于ARToolKit平台的增强现实人机交互系统的若干问题研究
摘要
增强现实(AR)是一种将计算机产生的虚拟物体或信息与真实环境进行合成并对景象加以增强或扩充的技术。文中介绍了AR的基本概念、主要应用及研究现状，分析了AR的关键技术及开发平台。
本文分析了ARToolKit平台的基本建模与渲染方法，指出了其存在的问题并进行了改进，将基于3DS的建模方法与基于可编程图形硬件的Cg图形渲染方法应用于ARToolKit平台，设计并实现了一个基于单目视觉的桌面AR人机交互系统。本文的主要工作包括：
1．研究了AR中基于视觉的跟踪注册方法，主要分析了标识物的识别及多个坐标系的矩阵转换等问题。基于视觉的跟踪注册方法识别效率高，能够满足AR系统对实时性的要求。
2．分析了ARToolKit平台的基本建模方法并对其进行了改进，将基于3DS的建模方法引入ARToolKit平台。分析了3DS文件的基本结构、3D编辑器块结构及关键帧块结构，给出了系统实现以及实验结果与分析。将3DS模型信息应用于AR系统的虚拟物体建模，提高了模型的复杂度和建模的效率。
3．分析了ARToolKit平台的基本渲染方法并对其进行了改进，将基于可编程图形硬件与Cg语言的渲染方法引入ARToolKit平台。分析并实现了Cg光照渲染、雾化渲染和变形动画效果，给出了各部分的实验结果及分析。将Cg应用于AR系统的图形渲染，提高了渲染效率，增强了AR场景的真实感。
4．将3DS模型信息的导入、基于Cg的图形渲染、交互信息的分析与处理等模块与ARToolKit平台进行整合，设计并实现了一个基于单目视觉的桌面AR人机交互系统，给出了系统实验结果及分析。该系统兼容性好，对硬件要求低，性能稳定，具有可扩展性以及良好的可移植性。
Augmented Reality (AR) is a kind of skill which can realize the fusion of virtual objects into real environment and expand it. The present thesis introduces the basic concept of AR and its main applications. It also analyzes the key technologies of AR and development platform.
This thesis applies the modeling of 3DS and the graphics rendering which is based on programmable graphics hardware into the AR system, and further develops the platform of ARToolKit. Besides that, this paper designs a human-computer interaction system based on monocular vision, and discusses its realizations. The present thesis includes:
1. Study the augmented reality of tracking registration, mainly on the realization of identification marker and the conversion of the matrix of coordinate system. Augmented reality of tracking registration is of high efficiency and thus can be used to meet the need of real-time performance.
2. Analyze the basic modeling of ARToolKit and ameliorate it, introduce the modeling of 3DS into the platform of ARToolKit. It also analyzes basic structures of 3DS files type, 3D editor chunk and keyframer chunk, presents the realization of this kind of method and the results of the experiment. The application of 3DS file type into virtual models increases the complexity and efficiency of modeling.
3. Research the basic rendering of the platform of ARToolKit and ameliorate it. Put forward the application of programmable graphics hardware and platform-cross of Cg language into the platform of ARToolKit. Analyze and then realize the Cg lightening rendering, fog effect and the effects of distortion animation. Provide the experiment result of each section and its analysis. The application of Cg into the AR system can further improve the realities of virtual objects, and its combination of real objects.
4. Import 3DS modeling, graphics rendering based on Cg, the disposal of mutual information into the platform of ARToolKit. Introduce and design the realization of a human-computer interaction system based on monocular vision. The system has a good compatibility, low requirement on the hardware, stable capability, better expansibility and transferability.
本文分析了ARToolKit平台的基本建模与渲染方法，指出了其存在的问题并进行了改进，将基于3DS的建模方法与基于可编程图形硬件的Cg图形渲染方法应用于ARToolKit平台，设计并实现了一个基于单目视觉的桌面AR人机交互系统。本文的主要工作包括：
1．研究了AR中基于视觉的跟踪注册方法，主要分析了标识物的识别及多个坐标系的矩阵转换等问题。基于视觉的跟踪注册方法识别效率高，能够满足AR系统对实时性的要求。
2．分析了ARToolKit平台的基本建模方法并对其进行了改进，将基于3DS的建模方法引入ARToolKit平台。分析了3DS文件的基本结构、3D编辑器块结构及关键帧块结构，给出了系统实现以及实验结果与分析。将3DS模型信息应用于AR系统的虚拟物体建模，提高了模型的复杂度和建模的效率。
3．分析了ARToolKit平台的基本渲染方法并对其进行了改进，将基于可编程图形硬件与Cg语言的渲染方法引入ARToolKit平台。分析并实现了Cg光照渲染、雾化渲染和变形动画效果，给出了各部分的实验结果及分析。将Cg应用于AR系统的图形渲染，提高了渲染效率，增强了AR场景的真实感。
4．将3DS模型信息的导入、基于Cg的图形渲染、交互信息的分析与处理等模块与ARToolKit平台进行整合，设计并实现了一个基于单目视觉的桌面AR人机交互系统，给出了系统实验结果及分析。该系统兼容性好，对硬件要求低，性能稳定，具有可扩展性以及良好的可移植性。
Augmented Reality (AR) is a kind of skill which can realize the fusion of virtual objects into real environment and expand it. The present thesis introduces the basic concept of AR and its main applications. It also analyzes the key technologies of AR and development platform.
This thesis applies the modeling of 3DS and the graphics rendering which is based on programmable graphics hardware into the AR system, and further develops the platform of ARToolKit. Besides that, this paper designs a human-computer interaction system based on monocular vision, and discusses its realizations. The present thesis includes:
1. Study the augmented reality of tracking registration, mainly on the realization of identification marker and the conversion of the matrix of coordinate system. Augmented reality of tracking registration is of high efficiency and thus can be used to meet the need of real-time performance.
2. Analyze the basic modeling of ARToolKit and ameliorate it, introduce the modeling of 3DS into the platform of ARToolKit. It also analyzes basic structures of 3DS files type, 3D editor chunk and keyframer chunk, presents the realization of this kind of method and the results of the experiment. The application of 3DS file type into virtual models increases the complexity and efficiency of modeling.
3. Research the basic rendering of the platform of ARToolKit and ameliorate it. Put forward the application of programmable graphics hardware and platform-cross of Cg language into the platform of ARToolKit. Analyze and then realize the Cg lightening rendering, fog effect and the effects of distortion animation. Provide the experiment result of each section and its analysis. The application of Cg into the AR system can further improve the realities of virtual objects, and its combination of real objects.
4. Import 3DS modeling, graphics rendering based on Cg, the disposal of mutual information into the platform of ARToolKit. Introduce and design the realization of a human-computer interaction system based on monocular vision. The system has a good compatibility, low requirement on the hardware, stable capability, better expansibility and transferability.
引文
[1] Azuma R T. A survey of augmented reality[J]. Teleoperators and Virtual Environments. MIT Press, 1997, 6(4): 355-385.
[2] 朱淼良，姚远，蒋云良．增强现实综述[J]．中国图象图形学报，2004，9(7)：767—774．
[3] Milgram P, Kishino F. A taxonomy of mixed reality visual displays[J]. IEICE Transaction on Information Systems, 1994.
[4] Bajura Michael, Henry Fuchs, Ryutarou Ohbuchi. Merging Virtual Reality with the Real World: Seeing Ultrasound Imagery Within the Patient[A]. Proceedings of SIGGRAPH 92[C]. 1992, 153-161.
[5] Grimson W T et al. An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery and Enhanced Reality Visualization[A]. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition[C]. 1994, 232-243.
[6] Feiner S, MacIntyre B, Seligmann D. Knowledge-based augmented reality[J]. Communications of the ACM, 1993, 36(7), 52-62.
[7] Dirk Reimers, Didier Stricker, Gudrun Klinker. Augmented Reality for Construction Tasks: Doorlock Assembly[A]. Proceedings of UIST'93[C]. Atlanta, GA, 1993, 83-94.
[8] Rastogi A, Milgram P, Drascic D. Telerobotic Control with Stereoscopic Augmented Reality[J]. SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems Ⅲ, San Jose, Feb 1996, 135-146.
[9] Rastogi A, Milgram P, Grodski J, Augmented Telerobotic Control: a visual display for unstructured environments[A]. 1995 KBS/Robotics Conference[C]. Oct, 1995, 16-18.
[10] R Azuma, Y Baillot et al. Recent advances in augmented reality[J]. IEEE Computer Graphics and Applications, Dec 2001, 34-47.
[11] J P Rolland ,R L Holloway, H Fuchs. A comparison of optical and video see-through head-mounted display[J]. SPIE, Telemanipulator, Telepresence and Technologies, 1994,97-114.
[12] Mukaigawa Y, Mihashi Setal. Photometric image-based rendering for virtual lighting image synthesis[A]. Proc. 2nd Int'l Workshop Augmented Reality[C]. 1999, 115-124.
[13] Debevec P. Rendering synthetic objects into real scenes: bridging traditional and image-based graph ics with global illumination and high dynamic range photography[A]. Proc. SIGGRAPH 98 [C]. 1998,189-198.
[14] O Bimber and B Frohlich, Occlusion shadows: Using projected light to generate realistic occlusion effects for view-dependent optical see-through displays[A]. In Proceedings of the International Symposion on Mixed and Augmented Reality (ISMAR) [C]. 2002.
[15] Raskar R, et al. Shader lamps: animating real objects with image based illumination [A]. Proceedings of Eurographics Workshop on Rendering[C]. London, 2001, 89-102.
[16] Naemura T, et al Virtual shadows-enhanced interaction in mixed reality environment [A]. Proceedings of IEEE Virtual Reality(VR'02) [C]. Orlando, 2002, 293-294.
[17] Haller M, Drab S, Hartmann W. A realtime shadow approach for an augmented reality application using shadow volumes[A]. Proceedings of the ACM Symposium on Virtual Reality Software and Technology[C]. Osaka, 2003, 56-65.
[18] Loscos C, et al. Interactive virtual relighting and remodeling of real scenes[A]. Proceedings of the 10th Eurographics Workshop on Rendering[C]. Granada, 1999, 3-18.
[19] Loscos C, et al. Interactive virtual relighting of real scenes [J]. IEEE Transactions on Visualization and Computer Graphic, 2000, 6(3): 289-305.
[20] Jacobs Katrien, et al. Automatic consistent shadow generatio for augmented reality [A]. Proceedings of Graphics Interface[C]. Vancouver, 2005, 113-120.
[21] Jaakko Konttinen, Sumanta Pattanaik, Hughes Charles E. Realtime illumination and shadowing by virtual lights in a mixed reality setting [OL]. http://graphics.cs.ucf.edu/MAR-Sumant, 2007.
[22] Kanbara Masayuki, Yokoya Naokazu. Real-time estimation of light source environment for photorealistic augmented reality[A]. Proceedings of the 17th International Conference on Pattern Recognition[C]. Cambridge, 2004:911-914.
[23] Human Interface Technology Laboratory. ARToolKit [OL]. http://www.hitl.washington.edu/projects/artoolkit/, 2007.
[24] 周雅，晏磊，赵虎．增强现实系统光照模型建立研究[J]．中国图象图形学报，2004，9(8)：968-972．
[25] H Kato, et al. Virtual Object Manipulation of a Table-Top AR Environment[A]. Proceeding of International Symposium on Augmented Reality 2000 (ISAR 00) [C]. Los Alamitos, Calif, 2000, 119.
[26] K Dorfmueller-Ulhaas, D Schrnalstieg. Finger tracking for interaction in augmented environments[A], the 2nd ACM/IEEE International Symposium on Augmented Reality (ISAR'01) [C]. 2001.
[27] Yuan Yao, Miaoliang Zhu, Yunliang Jiang, and Guang Lu. A bare hand controlled AR map navigation system[A]. Systems, Man and Cybernetics, 2004 IEEE International Conference[C]. 2004.
[28] W Daniel, S Dieter. First Steps Towards Handheld Augmented Reality[A]. Proceedings of the 7th International Conference on Wearable Computers[C]. White Plains, NY, USA.
[29] D Schmalstieg, A Fuhrmann, G. Hesina, et al. The Studierstube Augmented Reality Project [J]. Teleoperators and Virtual Environments, MIT Press, 2002, 11(1):33-54.
[30] 贾志刚．精通OpenGL[M]．北京：电子工业出版社，1998．
[31] Michitaka Hirose,image-Base virtual world generation[J]. IEEE Multimedia January. March, 1997,27-33.
[32] Mason Woo,Jackie Neider,Tom Davis.OpenGL编程权威指南[M]．北京：中国电力出版社，2001．
[33] 乔林，费广正．OpenGL程序设计[M]．北京：清华大学出版社，2000．
[34] OpenGL ARB. OpenGL Programming Guide [M].Prentice Hall, 1999.
[35] 李颖，薛海斌，朱伯立等编著．OpenGL函数与范例解析手册[M]．北京：国防工业出版社，2002．
[36] Fellner D W. Modeling of and navigation in complex 3D documents[J]. Computer Graphics, Dec 1998, 22(6), 647-653.
[37] Randima Fernando,Mark J.Kilgard著，洪伟，刘亚妮等译，Cg教程—可编程实时图形权威指南[M]．人民邮电出版社，2004．9：21．
[38] Reevs W T. Particle System-a Technique for Modeling a Class of Fuzzy Objects[J]. Computer Graphics, 1983, 17(3):359-376.
[39] Didier J Y, Roussel D, Mallem M. A texture based time delay compensation method for augmented reality[A]. The Int'l Symposium on Mixed and Augmented Reality(ISMAR 2004)[C]. Arlington, 2004, 262-263.
[40] Coelho E M, MacIntyre B, Julier S J. OSGAR-A scene graph with uncertain transformations[A]. The 3rd IEEE and ACM Int'l Symposium on Mixed and Augmented Reality[C]. Arlington, 2004, 6-15.
[41] Piekarski W, Thomas B H. Augmented reality working planes: a foundation for action and construction at a distance[A]. The Third IEEE and ACM Int'l Symposium on Mixed and Augmented Reality[C]. Arlington, 2004, 162-171.
[42] Malik S, Roth G, McDonald C. Robust Comer Tracking for Real-Time Augmented Reality [Z]. In Vision Interface 2002. Calgary, Alberta, Canada. May 2002.
[43] T. Reicher et al. Results of a Study on Software Architectures for Augmented Reality Systems[A]. IEEE ISMAR 2003 [C]. Tokyo, Japan, 2003.
[2] 朱淼良，姚远，蒋云良．增强现实综述[J]．中国图象图形学报，2004，9(7)：767—774．
[3] Milgram P, Kishino F. A taxonomy of mixed reality visual displays[J]. IEICE Transaction on Information Systems, 1994.
[4] Bajura Michael, Henry Fuchs, Ryutarou Ohbuchi. Merging Virtual Reality with the Real World: Seeing Ultrasound Imagery Within the Patient[A]. Proceedings of SIGGRAPH 92[C]. 1992, 153-161.
[5] Grimson W T et al. An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery and Enhanced Reality Visualization[A]. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition[C]. 1994, 232-243.
[6] Feiner S, MacIntyre B, Seligmann D. Knowledge-based augmented reality[J]. Communications of the ACM, 1993, 36(7), 52-62.
[7] Dirk Reimers, Didier Stricker, Gudrun Klinker. Augmented Reality for Construction Tasks: Doorlock Assembly[A]. Proceedings of UIST'93[C]. Atlanta, GA, 1993, 83-94.
[8] Rastogi A, Milgram P, Drascic D. Telerobotic Control with Stereoscopic Augmented Reality[J]. SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems Ⅲ, San Jose, Feb 1996, 135-146.
[9] Rastogi A, Milgram P, Grodski J, Augmented Telerobotic Control: a visual display for unstructured environments[A]. 1995 KBS/Robotics Conference[C]. Oct, 1995, 16-18.
[10] R Azuma, Y Baillot et al. Recent advances in augmented reality[J]. IEEE Computer Graphics and Applications, Dec 2001, 34-47.
[11] J P Rolland ,R L Holloway, H Fuchs. A comparison of optical and video see-through head-mounted display[J]. SPIE, Telemanipulator, Telepresence and Technologies, 1994,97-114.
[12] Mukaigawa Y, Mihashi Setal. Photometric image-based rendering for virtual lighting image synthesis[A]. Proc. 2nd Int'l Workshop Augmented Reality[C]. 1999, 115-124.
[13] Debevec P. Rendering synthetic objects into real scenes: bridging traditional and image-based graph ics with global illumination and high dynamic range photography[A]. Proc. SIGGRAPH 98 [C]. 1998,189-198.
[14] O Bimber and B Frohlich, Occlusion shadows: Using projected light to generate realistic occlusion effects for view-dependent optical see-through displays[A]. In Proceedings of the International Symposion on Mixed and Augmented Reality (ISMAR) [C]. 2002.
[15] Raskar R, et al. Shader lamps: animating real objects with image based illumination [A]. Proceedings of Eurographics Workshop on Rendering[C]. London, 2001, 89-102.
[16] Naemura T, et al Virtual shadows-enhanced interaction in mixed reality environment [A]. Proceedings of IEEE Virtual Reality(VR'02) [C]. Orlando, 2002, 293-294.
[17] Haller M, Drab S, Hartmann W. A realtime shadow approach for an augmented reality application using shadow volumes[A]. Proceedings of the ACM Symposium on Virtual Reality Software and Technology[C]. Osaka, 2003, 56-65.
[18] Loscos C, et al. Interactive virtual relighting and remodeling of real scenes[A]. Proceedings of the 10th Eurographics Workshop on Rendering[C]. Granada, 1999, 3-18.
[19] Loscos C, et al. Interactive virtual relighting of real scenes [J]. IEEE Transactions on Visualization and Computer Graphic, 2000, 6(3): 289-305.
[20] Jacobs Katrien, et al. Automatic consistent shadow generatio for augmented reality [A]. Proceedings of Graphics Interface[C]. Vancouver, 2005, 113-120.
[21] Jaakko Konttinen, Sumanta Pattanaik, Hughes Charles E. Realtime illumination and shadowing by virtual lights in a mixed reality setting [OL]. http://graphics.cs.ucf.edu/MAR-Sumant, 2007.
[22] Kanbara Masayuki, Yokoya Naokazu. Real-time estimation of light source environment for photorealistic augmented reality[A]. Proceedings of the 17th International Conference on Pattern Recognition[C]. Cambridge, 2004:911-914.
[23] Human Interface Technology Laboratory. ARToolKit [OL]. http://www.hitl.washington.edu/projects/artoolkit/, 2007.
[24] 周雅，晏磊，赵虎．增强现实系统光照模型建立研究[J]．中国图象图形学报，2004，9(8)：968-972．
[25] H Kato, et al. Virtual Object Manipulation of a Table-Top AR Environment[A]. Proceeding of International Symposium on Augmented Reality 2000 (ISAR 00) [C]. Los Alamitos, Calif, 2000, 119.
[26] K Dorfmueller-Ulhaas, D Schrnalstieg. Finger tracking for interaction in augmented environments[A], the 2nd ACM/IEEE International Symposium on Augmented Reality (ISAR'01) [C]. 2001.
[27] Yuan Yao, Miaoliang Zhu, Yunliang Jiang, and Guang Lu. A bare hand controlled AR map navigation system[A]. Systems, Man and Cybernetics, 2004 IEEE International Conference[C]. 2004.
[28] W Daniel, S Dieter. First Steps Towards Handheld Augmented Reality[A]. Proceedings of the 7th International Conference on Wearable Computers[C]. White Plains, NY, USA.
[29] D Schmalstieg, A Fuhrmann, G. Hesina, et al. The Studierstube Augmented Reality Project [J]. Teleoperators and Virtual Environments, MIT Press, 2002, 11(1):33-54.
[30] 贾志刚．精通OpenGL[M]．北京：电子工业出版社，1998．
[31] Michitaka Hirose,image-Base virtual world generation[J]. IEEE Multimedia January. March, 1997,27-33.
[32] Mason Woo,Jackie Neider,Tom Davis.OpenGL编程权威指南[M]．北京：中国电力出版社，2001．
[33] 乔林，费广正．OpenGL程序设计[M]．北京：清华大学出版社，2000．
[34] OpenGL ARB. OpenGL Programming Guide [M].Prentice Hall, 1999.
[35] 李颖，薛海斌，朱伯立等编著．OpenGL函数与范例解析手册[M]．北京：国防工业出版社，2002．
[36] Fellner D W. Modeling of and navigation in complex 3D documents[J]. Computer Graphics, Dec 1998, 22(6), 647-653.
[37] Randima Fernando,Mark J.Kilgard著，洪伟，刘亚妮等译，Cg教程—可编程实时图形权威指南[M]．人民邮电出版社，2004．9：21．
[38] Reevs W T. Particle System-a Technique for Modeling a Class of Fuzzy Objects[J]. Computer Graphics, 1983, 17(3):359-376.
[39] Didier J Y, Roussel D, Mallem M. A texture based time delay compensation method for augmented reality[A]. The Int'l Symposium on Mixed and Augmented Reality(ISMAR 2004)[C]. Arlington, 2004, 262-263.
[40] Coelho E M, MacIntyre B, Julier S J. OSGAR-A scene graph with uncertain transformations[A]. The 3rd IEEE and ACM Int'l Symposium on Mixed and Augmented Reality[C]. Arlington, 2004, 6-15.
[41] Piekarski W, Thomas B H. Augmented reality working planes: a foundation for action and construction at a distance[A]. The Third IEEE and ACM Int'l Symposium on Mixed and Augmented Reality[C]. Arlington, 2004, 162-171.
[42] Malik S, Roth G, McDonald C. Robust Comer Tracking for Real-Time Augmented Reality [Z]. In Vision Interface 2002. Calgary, Alberta, Canada. May 2002.
[43] T. Reicher et al. Results of a Study on Software Architectures for Augmented Reality Systems[A]. IEEE ISMAR 2003 [C]. Tokyo, Japan, 2003.