摘要
增强现实(AR),也被称之为混合现实。它通过电脑技术,将虚拟的信息应用到真实世界,真实的环境和虚拟的物体实时地叠加到了同一个画面或空间同时存在,呈现给使用者一个感官效果真实的新环境。在显示方法方面,本文着重研究基于真实场景的空间增强现实方法,该方法将用户从头戴式显示器等光学器件中解放出来,直接改变用户周围的自然环境,达到所见即所得的效果。
空间增强现实技术在近年来取得许多成果,其中基于模型的投影增强现实是研究的热点。通过对真实的物体投影虚拟的图像,使之呈现不同的效果,可以增强观看者的感受。另一方面,光照渲染一直是计算机图形学中研究的热点,研究人员已经可以在计算机中渲染出各种逼真绚丽的场景。将光照技术应用到空间增强现实中,可以极大地提升增强后物体的视觉效果,另外,将渲染场景从计算机中搬到了真实世界,可以说是对图形学渲染技术的延伸。
本文首先介绍基于投影仪-摄像机的空间增强现实技术,以及空间增强现实中图形渲染的研究成果,着重说明真实物体模型的获取,全局光照渲染,以及将计算机中的场景与真实场景进行对准,即如何将渲染图像贴合到物体上,最后给出实验环境设置,实验结果及总结。
本文主要完成了如下工作:
(1)调研投影光照在增强现实中的应用。如结合实验室的环境,研究空间增强现实的桌面等技术,分析实验可行性,从而确定实验以及工作方案。
(2)研究基于模型的投影纹理贴合技术。通过棋盘格角点检测得到摄像机内参,通过结构光技术进行投影仪的内参,摄像机和投影仪外参的标定以及对物体进行建模,得到物体的三角面片信息以及纹理坐标。借助第三方工具对物体的纹理,材质进行编辑替换,使之呈现不同的属性及外观表现。
(3)研究投影场景的光照渲染方法。将计算机图形学中的全局光照渲染技术移植到投影场景中,将投影仪设定为可控数字光源,采用常用的全局光照渲染技术,对场景进行预处理,如面片的划分和简化,采用辐射度算法,配合由标定出的外参确定的摄像机位置作为视点,对整个场景进行渲染。
(4)光照补偿的数学建模与计算。对于给定场景的每一个面片,根据需要呈现出的颜色,考虑面片之间互反射的影响,计算实际投影到每个面片的颜色。根据辐射度算法中计算出的形状因子矩阵,决定面片之间的互相影响程度。最终划归为二次规划问题,在投影仪的实际参数限制范围内求最优解。
(5)给出了基于真实场景的投影光照渲染结果,包括建模效果,光照渲染结果以及光照补偿结果,展示并讨论了补偿前后实际增强场景在亮度与色度上的差异,以及利用光照补偿技术使得实际增强结果与期望结果之间的差异最小化。
本文主要贡献在于改进了增强现实中的显示效果,并非简单地利用投影仪将画面投影到物体表面上,而是将场景建模结果,在计算机中进行处理,经过模型的识别、划分等手段后,利用图形学中的光照渲染手段对建模场景进行渲染,采用三维对准技术,将渲染结果投影到真实场景中,增强了场景的视觉效果,并且采用了光照补偿技术,减弱了真实场景中面片之间互反射的影响,使得实际获得的增强效果与在计算机中渲染出的期望效果较为一致。
Augmented Reality (AR), also mentioned as Mixed Reality, brings virtual information into real world through computer technology, real environment and virtual object are mixed together in the same space at the same time, thus a new and real-visualized environment is displayed to visitors. On the aspect of displaying, this thesis mainly focus on Spatial Augmented Reality (SAR) based on real scenes, this method helps users get rid of optic equipments such as head-mounted monitors, and directly changes user’s surrounding environment, achieving the effect of what-you-see-is-what-you-get.
Many research achievements have been made in Spatial Augmented Reality (SAR), among which model-based projective Augmented Reality (AR) is the main research focus. The appearance of real object can be changed by projecting virtual image onto them in order to enhance user’s visual experience. On the other hand, global illumination (GI) has always been the hot spot in computer graphics, realistic and dazzling scene can be rendered by researchers. By introducing GI into SAR, the visual effect of augmented object can be greatly enhanced; In addition, by moving rendered scene from computer screen to real world, it is an extension of CG rendering technique.
This thesis accomplishes research works as follows:
(1)Survey the applications of projective illumination in SAR. Study the demo and methods taken to implement projective illumination rendering in real scenes, based on our lab conditions, study SAR desktop applications, analyze the feasibility of experiment, confirm the experiment and operation scheme.
(2)Study the method of model based projective texture mapping. Obtain camera intrinsic parameters through chessboard corners detection, calibration of projector’s intrinsic parameters, camera-projector’s extrinsic parameters and modeling is done with structured light. After doing this, object’s triangular patches information and uv coordinates is obtained, we modify model’s texture and material through third-party tools, so as to change its appearance and visual properties.
(3)Apply Global Illumination (GI) in projective area. Treat projectors as controllable digital light source, after preprocessing of the scene, such as patch-dividing and simplifying, use radiosity algorithm, according to the position of camera and projector decided by the extrinsic parameters that is calibrated, render the whole scene.
(4)Mathematical modeling and calculation of illumination compensation. For each patch in a certain scene, according to the desired color, taking account of the affect by inter-reflection, calculate the real color that is projected to objects. The form factor matrix calculated in radiosity algorithm decided the degree of interaction with each other. Finally the problem is converted to a quadratic programming problem, the optimal solution is solved under the restriction of the real parameters of projectors.
(5)Demonstrate the projective illumination augmenting result based on real scene, including modeling result, illumination rendering and illumination compensation result. Demonstrate and discuss the difference of real augmented scene before and after compensation in luminance of chrominance, and the use of compensation technique to minimum the difference between real augmented scene and desired scene.
The contribution of this thesis is to improve the displaying effect in spatial augmented reality, rather than simply projecting images onto objects, but process scene model result in computer, after recognizing and dividing of the model, render the scene with illumination techniques, and project the rendering results back into real scene using 3D align technique, improving the visual effect, and by using illumination compensation technique, the effect of interreflection of real scene is reduced, making the real augmented effect obtained and desired effect in computer rendering consistent.
引文
[1] I. E. Sutherland. A Head Mounted Three Dimensional Display. Proceedings of the Fall Joint Computer Conference (AFIPS) 33:1 .757–764. 1968.
[2] O. Bimber and R. Raskar. Spatial Augmented Reality: Merging Real and Virtual Worlds. A. K. Peters, Ltd., Natick, MA, USA, 3. 2005.
[3] Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. Shader lamps : Animating real objects with image-based illumination. In Eurographics Rendering Workshop 2001, August 2001.
[4] D. Bandyopadhyay, R. Raskar, and H. Fuchs. Dynamic shaderlamps: Painting on real objects. In The Second IEEE and ACM International Symposium on Augmented Reality (ISAR01).Citeseer, 2001.
[5] Low, K., Welch, G., Lastra, A., and Fuchs, H. Life-Sized Projector-Based Dioramas. Symposium on Virtual Reality Software and Technology, 2001.
[6] Cossairt O., Nayar S., Ramamoorthi R.: Lightfield transfer: Global illumination between real and synthetic objects. ACM Transactions on Graphics 27, 3 (July 2008).
[7] Wetzstein, G. and Bimber, O. Radiometric Compensation of Global Illumination Effects with Projector-Camera Systems. ACM SIGGRAPH`06 (Poster), 2006.
[8] B. C. Yu Sheng, Theodore C. Yapo. Global Illumination Compensation for Spatially Augmented Reality. In Computer Graphics Forum, Eurographics.2010.
[9] Rabiner L R. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceeding of IEEE,77(2):257-286.1989
[10]Zhang Z Y. A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334. 2000.
[11]百度百科. Available.相机标定. http://baike.baidu.com/view/380491.htm.2009
[12]陈胜勇,刘盛等,基于OpenCV的计算机视觉技术实现,北京,科学出版社,364-383,2008.
[13] Mendonca, P.R.D.S. and Cipolla, R. A simple technique for self-calibration. In Proc. IEEE Computer Vision and Pattern Recognition, pp. I:500–505. 1999.
[14] Weng Juyang, Cohen P, Herniou M. Camera Calibration with Distortion Models and Accuracy Evaluation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(10): 965 - 980. 1992.
[15] Legarda-Saenz R, Bothe T, Juptner W P. Accurate Procedure for the Calibration of a Structured Light System [J]. Optical Engineering , 43 (2) : 464 - 471. 2004.
[16] Zhang S, Huang P S. Novel method for structured light system calibration [J].Optical Engineering, 45 (8) :08360121 - 08360128. 2006.
[17] Joaquim Salvi, Jordi Pag'es, Joan Batlle. Pattern codification strategies in structured light systems Pattern Recognition 37 (2004) 827 -849.2004
[18]于晓洋,张加海,董秋玲.结构光三维测量系统的建模与标定.传感器世界,2007年13卷11期:15-18
[19]王宇超,增强现实系统中的嵌入结构光技术研究,上海交通大学硕士学位论文,28,2009.
[20] M.-A. Drouin, G. Godin, and S. Roy. An Energy Formulation for Establishing the Correspondence Used in Projector Calibration. In Proceedings of the Fourth International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), June 18–20, 2008.
[21]J. T. Kajiya. The Rendering Equation. Computer Graphics (Proc. SIGGRAPH’86) 20:4, 143–151. 1986.
[22] Tomas Nikodym Ray Tracing Algorithm For Interactive Applications. Czech Technical University, FEE. June 2010.
[23]Turner Whitted. An Improved Illumination Model for Shaded Display Communications of the ACM, v.23 n.6, p.343-349, June 1980.
[24] Yu Y., Debevec P., Malik J., Hawkins T.: Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs. In Proceedings of SIGGRAPH 99, Computer Graphics Proceedings, Annual Conference Series, pp. 215–224. Aug. 1999.
[25] Contensin M.: Inverse Lighting Problem in Radiosity. Inverse Problems in Engineering 10, 2, 131–152. January 2002.
[26] Bimber O., Emmerling A., Klemmer T.: Embedded Entertainment with Smart Projectors. Computer 38, 1,48–55. 2005.
[27] Ashdown M., Okabe T., Sato I., Sato Y. Robust Content-Dependent Photometric Projector Compensation. In CVPRW’06: Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop (Washington, DC, USA), IEEE Computer Society, p. 6. 2006.
[28]Grundhofer A.,Bimber O. Real-time Adaptive Radiometric Compensation. IEEE Transactions on Visualization and Computer Graphics 14, 1, 97–108. Jan./Feb. 2008.
[29] Bimber O., Grundhofer A., Zeidler T., Danch D., Kapakos P. Compensating Indirect Scattering for Immersive and Semi-immersive Projection Displays. In VR’06: Proceedings of the IEEE conference on Virtual Reality (Washington, DC, USA), IEEE Computer Society, pp. 151–158. 2006.
[30] Seitz S. M., Matsushita Y., Kutulakos K. N. A Theory of Inverse Light Transport. InICCV’05: Proceedings of the Tenth IEEE International Conference on Computer Vision (Washington, DC, USA), IEEE Computer Society, pp. 1440–1447. 2005.
[31]EQUASYS.ColorConversion.http://www.equasys.de/colorconversion.html#YPbPr-RGBColorFormatConversion. 2010
[32]Murty, K. G.Linear Complementarily, Linear and Nonlinear Programming. Sigma Series in Applied Mathematics. 3. Berlin: Heldermann Verlag. pp. xlviii+629 pp.. ISBN 3-88538-403-5. 1988.
[33]Nocedal, Jorge; Wright, Stephen J. Numerical Optimization (2nd ed.). Berlin, New York: Springer-Verlag. ISBN 978-0-387-30303-1. 2006.
[34]朱克强,贺力群.大规模简单界约束的凸二次规划新算法.北方交通大学学报.BEIFANG JIAOTONG DAXUE XUEBAO第3期1998年6月第22卷(总第79期).1998
[35]Coleman, T.F. and Y. Li, "A Reflective Newton Method for Minimizing a Quadratic Function Subject to Bounds on some of the Variables," SIAM Journal on Optimization, Vol. 6, Number 4, pp. 1040-1058, 1996.
[36]Fletcher R, Reeves C M. Function Minimization by Conjugate Gradients. Comput. J., 7:149. 1964.
[37]Klessig R, Polak E. Efficient Implementation of the Polak-Ribiere Conjugate Gradient algorithm, SIAM J. Control, 10:524～549, 1972.
[38]Murty B S N, Asghar Husain. Orthogonality Correction in the Conjugate-gradient Method. J. Comput & Appl. Math., 9:299～30, 1983.
[39]Murty B S N, Reddy P J. Numerical experiente with Conjugate direction methods in Constrained Minimization. European Journal of Operational Research, 42:203～2 1989.