手持设备上基于增强现实的虚实交互技术的研究与应用
|
摘要
增强现实技术(Augmented Reality,简称AR)是在虚拟现实技术(Virtual Reality,简称VR)的基础上发展起来的一个新的研究领域,成为人机界面技术发展的一个重要方向。增强现实技术将计算机生成的虚拟对象及其相关信息准确实时的叠加到真实环境中,可以通过显示设备看到一个真实场景中无缝融入虚拟对象的虚实结合空间,为人们带来一个视觉效果更真实、场景信息更丰富的新环境。随着移动设备和技术的发展,在这类通用性高、普及率大、便携性好、能耗低的手持设备设计开发增强现实系统对增强现实技术的推进和发展具有重要的理论意义和现实意义。
人机交互技术的发展将技术核心由计算机转向用户,而人手动作和手势是人类最自然、直观、易于掌握的人际交流方式,并含有丰富的交互信息,用户可以直接将手作为计算机的输入设备,通过手势发布命令,不需要交互媒介实现人机通讯,因此基于手势识别的交互技术已逐渐成为人机交互领域的研究热点。
因此,研究便携式手持设备上实现增强现实手势交互技术具有很大实用价值,论文的主要内容有：
1、研究了自适应三帧差分运动目标检测算法。运动目标检测是实现虚实交互的前提条件。论文研究了目前主流的视频检测算法,分析了这些算法的优势和不足,提出了一种通过运动目标速度自适应调整帧间隔的三帧差分检测方法,实验表明,与目前主流的视频检测算法相比,本文方法在运动物体方向不改变的条件下,能够快速准确的定位运动目标区域,有效降低手持设备处理延迟性,适用于计算性能较低的手持设备。
2、手持设备上增强现实的虚实交互系统需要利用手持设备内置摄像头实时采集视频图像,由于使用者生理上无意识的手部抖动或移动手持设备,都可能导致视频帧背景发生变化。本文研究了Surendra背景更新算法,实现能够自适应更新背景,并与自适应三帧差分和背景差分法相结合,解决了因手持设备抖动而引起的检测误差问题,提高检测精度和效果。
3、提出了基于区域相割的指尖检测算法,将交互点的范围缩小到一个特定的区域,能够确定人手指尖与虚拟物体交互的有效区域,使人手对虚拟物体的动作更加精确和协调。
Augmented Reality (Augmented Reality, referred to as AR) is a new field of research developed on the basis of virtual reality technology (Virtual Reality, referred to as VR). It also is an important development direction of human-machine interface technology. Augmented reality technology accurately adds the computer-generated virtual objects and their related information to the real environment, you can see a reality-virtual space by display device which virtual objects seamlessly integrated into a real scene. It can brings a more real and new environment. As the development of mobile devices and technology, design augmented reality system on such handheld device of high universality, high diffusion rate, portable, low energy consumption will promote the development of augmented reality technology and bring important theoretical significance and practical significance.
Human-computer interaction technology's technical core shifted from computer to users. The gestures of human is interpersonal communication methods which is the most natural, intuitive and easy to master and contains much more interactive information. Users can directly use hand as a computer input device, issued an order through the gestures, does not require the media to achieve human-computer interaction. So, the interaction based on gesture recognition technology has gradually become a research highlight in human-computer interaction research field.
Therefore, the study of interactive technology of AR based on handheld devices will bring great practical value. The main work in this paper is as followed:
Firstly, a algorithm of adaptive three frame difference moving target detection is proposed. Moving target detection is a prerequisite for interaction. By analyzing the strengths and weaknesses of mainstream video detection methods, a new algorithm based on moving target rate which could adaptive adjust frame interval is proposed. The experiments show that this algorithm is faster than other existed methods used aimed at moving target detection. Therefore it is feasible for end-users'devices with low computational performance effectively and reduce the processing delays of handheld devices.
The reality interactive system based on handheld devices needs integrated cameras capture real-time video image. As the user unconscious physiological hand shake or move mobile handheld devices are likely to result the background changes. Surendra's algorithm which could achieve adaptive update the background, combined with adaptive three frame difference and background difference solved detection error problems be caused by jitter on hand-held device. It can improve the detection accuracy and effectiveness.
人机交互技术的发展将技术核心由计算机转向用户,而人手动作和手势是人类最自然、直观、易于掌握的人际交流方式,并含有丰富的交互信息,用户可以直接将手作为计算机的输入设备,通过手势发布命令,不需要交互媒介实现人机通讯,因此基于手势识别的交互技术已逐渐成为人机交互领域的研究热点。
因此,研究便携式手持设备上实现增强现实手势交互技术具有很大实用价值,论文的主要内容有：
1、研究了自适应三帧差分运动目标检测算法。运动目标检测是实现虚实交互的前提条件。论文研究了目前主流的视频检测算法,分析了这些算法的优势和不足,提出了一种通过运动目标速度自适应调整帧间隔的三帧差分检测方法,实验表明,与目前主流的视频检测算法相比,本文方法在运动物体方向不改变的条件下,能够快速准确的定位运动目标区域,有效降低手持设备处理延迟性,适用于计算性能较低的手持设备。
2、手持设备上增强现实的虚实交互系统需要利用手持设备内置摄像头实时采集视频图像,由于使用者生理上无意识的手部抖动或移动手持设备,都可能导致视频帧背景发生变化。本文研究了Surendra背景更新算法,实现能够自适应更新背景,并与自适应三帧差分和背景差分法相结合,解决了因手持设备抖动而引起的检测误差问题,提高检测精度和效果。
3、提出了基于区域相割的指尖检测算法,将交互点的范围缩小到一个特定的区域,能够确定人手指尖与虚拟物体交互的有效区域,使人手对虚拟物体的动作更加精确和协调。
Augmented Reality (Augmented Reality, referred to as AR) is a new field of research developed on the basis of virtual reality technology (Virtual Reality, referred to as VR). It also is an important development direction of human-machine interface technology. Augmented reality technology accurately adds the computer-generated virtual objects and their related information to the real environment, you can see a reality-virtual space by display device which virtual objects seamlessly integrated into a real scene. It can brings a more real and new environment. As the development of mobile devices and technology, design augmented reality system on such handheld device of high universality, high diffusion rate, portable, low energy consumption will promote the development of augmented reality technology and bring important theoretical significance and practical significance.
Human-computer interaction technology's technical core shifted from computer to users. The gestures of human is interpersonal communication methods which is the most natural, intuitive and easy to master and contains much more interactive information. Users can directly use hand as a computer input device, issued an order through the gestures, does not require the media to achieve human-computer interaction. So, the interaction based on gesture recognition technology has gradually become a research highlight in human-computer interaction research field.
Therefore, the study of interactive technology of AR based on handheld devices will bring great practical value. The main work in this paper is as followed:
Firstly, a algorithm of adaptive three frame difference moving target detection is proposed. Moving target detection is a prerequisite for interaction. By analyzing the strengths and weaknesses of mainstream video detection methods, a new algorithm based on moving target rate which could adaptive adjust frame interval is proposed. The experiments show that this algorithm is faster than other existed methods used aimed at moving target detection. Therefore it is feasible for end-users'devices with low computational performance effectively and reduce the processing delays of handheld devices.
The reality interactive system based on handheld devices needs integrated cameras capture real-time video image. As the user unconscious physiological hand shake or move mobile handheld devices are likely to result the background changes. Surendra's algorithm which could achieve adaptive update the background, combined with adaptive three frame difference and background difference solved detection error problems be caused by jitter on hand-held device. It can improve the detection accuracy and effectiveness.
引文
[1]Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.. Recent Advances in Augmented Reality. Computer Graphics and Application, IEEE Volume 21, Issue 6, Nov/Dec 2001 Page(s):34-47
[2]Bimber. O. and Raskar. R., Spatial Augmented Reality Merging Real and Virtual Worlds.2005:A. K. Peters Ltd.
[3]Billinghurst, M.. Research Directions in Augmented Reality. In Proc. of 16th International Conf. on Artificial Reality and Telexistence (ICAT'06).2006
[4]Ronald T. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments,1997.6(4):355-385
[5]Feiner, S., MacIntyre, B., Seligmann D.. Knowledge-based augmented reality[J]. Communications of the ACM,1993,36(7),52-62
[6]Thalmann, N. M., and Papagiannakis, P.. Virtual worlds and augmented reality in cultural Heritage Applications. In Proc. of International Workshop on Recording, Modeling and Visualization of Cultural Heritage.2005
[7]Pentenrieder. K. and Meier. P.. The need for accuracy statements in industrial Augmented Reality applications. In Proc. of Sixth International Symposium on Mixed and Augmented Reality.2007
[8]朱淼良,姚远,蒋云良.增强现实综述[J].中国图像图形学报,2004,7：767-773
[9]Loscos, C., et al. Interactive virtual relighting of real scenes[J]. IEEE Transactions on Visualization and Computer Graphic,2000,6(3):289-305
[10]James R Vallino. Interactive Augmented Reality[D]. PhD thesis. New York: University of Rochester,1998, Kiriakos
[11]Azuma, Ronald T.. The Challenge of Making Augmented Reality Work outdoors[M]. In Mixed Reality: Merging Real and Virtual Worlds, Yuichi Ohta and Hideyuki tamura, editors. Springer-Verlag, 1999, ISBN 3-540-65623-5. Chaper 21,pp.379-390
[12]Jun Rekimoto and Katashi Nagao. The world through the computer: computer augmented interaction with real world envirments. In Proceedings of the 8th annual ACM symposium on User interface and software technology(UIST'95),1995:29-36
[13]Mark Billinghurst, Hirkazu Kato, Ivan Pougyrev. The magicbook-moving seamlessly between reality and virtuality. IEEE Computer Graphics and Applications,21(3),2001:6-8
[14]Kanvara Masayuki, Yoloya Naokazu. Real-time estimation of light source environment for photorealistic augmented reality[A]. Proceedings of the 17th International Conference on Pattern Recognition[C]. Cambridge,2004:911-914
[15]Daniel Wagner, Istvan Barakonyi. Augmented reality kanji learning. In Proceedings of the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality(ISMAR'03),2003: 335-336
[16]Shahzad Malik, Avhishek Ranjan, Ravin balakrishnan. Interacting with Large Displays form a Distance with Vision-Tracked Multi-Finger gestural Input. In Proceedings of ACM Symposium on User Interface software technology (UIST) 2005[C], Seattle, WA, USA. P43-52
[17]Chris McDonald, Shahzad Malik, Gerhard Roth. Hand-based Interaction in Augmented Reality[A]. In Proceedings of IEEE International Workshop on Haptic Audio Visual Environments and their Application(HAVE)[C], Ottawa, Ontario, Canada.2002, pp.55-59
[18]Kato H., Billinghurst M., Poupyrev I., et al. Virtual object manipulation of a tavle-top AR environment[A]. In Proceedings of International Symposium on Augmented Reality 2000 (ISAR 00) [C], USA,2000:111-119
[19]柳祖国,李世其,李作清.增强现实技术的研究进展及应用[J].系统仿真学报,2003,15(2)：222-225
[20]任波,管涛,李利军等.基于ARToolKit的增强现实系统开发与应用[J].计算机系统应用,2006(1)：81-84
[21]任波,李利军,曹伟民等.基于增强现实技术的交互式城市设计研究[J].华中科技大学学报,2006,23(2)：32-34
[22]Shahzad Malik. Robust Registration of Virtual Ovjects for Real-Time Augmented Reality. Master thesis, Carleton University Ottawa, Ontario, Canada,2002
[23]William A. Hoff, Khoi Nguyen. Computer vision-based registration techniques for augmented reality, Proc. of Intelligent Robots and Computer Vision ⅩⅤ, Vol.2904, in Intellignet Systems & Advanced Manufacturing, SPIE, Boston, Massachusetts, Nov.19-21,1996, pp.538-548
[24]姚远.增强现实应用技术研究：[研究学位论文]杭州：浙江大学,2006
[25]WEI Zhiqiang, Ji Xiaopeng, FENG Yewei. A moving object detection method based on self adaptive updating of background[J]. Acta Electronica Sinica,2005,33 (12):2261-2264
[26]Barron J., Fleet D., Neauchemin S. Performance of optical flow techniques[J]. International Computer Vision,1994,12(1):42-77
[27]Richard P. Wildes. A measure of motion salience for surveillance application. In Proc of Image Processing,1998
[28]Elgammal A, Hawood D, Davis L S. Non-Parametric model for background subtraction Proc. ECCV,2000,7:751-767
[29]A. Aydin Alatan, Levent Onural., Michael Wollborn, et al. Image Sequence Analysis for Emerging Interactive Multimedia Services. The European COST 211 Framework[J]. IEEE Transactions on Circuits and Systems for Video Technology.1998-11, (8,7):802-813
[30]Jwu-Sheng Hu, Tzung-Min Su and Shr-Chi Jeng, "Robust Backgroud Subtraction with Shadow and Highlight Removal for Indoor Surveillance," Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems October 9-15,2006,Beijing, China,4545-4551
[31]Daniel Diggins. ARLib: A C++ Augmented Reality Software Development Kit:[硕士学位论文]英国：Bournemouth University,2005
[32]Chang C J, HSIEH J W, CHEN Y S, et al. Tracking multiple moving objects using a level-set method[J]. International Journal of Pattern Recognition and Artificial Intelligence,2004,18(2): 101-125
[33]Jong Bae Kim, Hang Joon kim. Efficient region-based motion segmentation for video monitoring system. Pattern Recognition Letters 2003,24(1-3):113-128
[34]Qu You-shan, Tian Wei-jian, Li Ying-cai. The moving target detecting on the parallelable discontinuous frame difference optical field integrated with gray intensity analysis. Acta Photonica Sinica,2003,32(2):182-186
[35]Remagnino P, Tan T, Baker K. Mufti-agent visual surveillance of dynamic scenes. Image and Vision Computing,1998,16(8):529-532
[36]Lipton A., Fujiyoshi H., Patil R.. Moving target classification and tracking form real-time video[C]. IEEE workshop on Application of Computer Vision, Princeton: IEEE Press,1998:8-14
[37]危水根等.一种基于时间差分运动检测的改进方法.南昌航空工业学院学报.2005,3：15-19
[38]代科学等.监控视频运动目标检测减背景技术的研究现状和展望.中国图像图形
[39]郝志成,朱明.智能目标检测与跟踪系统的设计与实现[J].光电工程,2007,34(1)：27-3
[40]付萍,方帅,徐心和等,视频监控系统中运动目标检测的阴影去除方法.计算机工程,2007,33(10)：22-24
[41]魏志强,纪筱鹏,冯业伟.基于自适应背景图像更新的运动目标检测方法[J].电子学报,2005,33(12)：2261-2264
[42]秦涛,周泽魁.一种检测序列图像中运动目标的新方法[J].计算机应用与软件.2004,21(9)：105-107
[43]文灏,红涛.基于减背景与对称查分的运动目标检测[J].微计算机信息,2007,9-1：99-101
[44]王陈阳,周明全,耿国华.基于自适应背景模型运动目标检测[J].计算机技术与发展,2007,17(4)：21-23
[45]吕国亮,赵曙光,赵俊.基于三帧差分和连通性检验的图像运动目标检测新方法[J].液晶与显示,2007,22(1)：87-93
[46]贺贵明,李凌娟,贾振堂.一种快速的基于对称差分的视频分割算法[J].小型微型计算机系统,2003,24(6)：96-98
[47]任海兵,祝远新,徐光,张晓平,林学訚.复杂背景下的手势分割与识别.自动化学报,2002.3(28)：256-261
[48]Lew, Y. P, Ramli, A. R, Koay, S. Y, Ali, R. and Prakash, V. A Hand Segmentation Scheme Using Clustering Technique in Homogeneous Background[A]. Student Conference Research and Development Proceedings.2002,305-308
[49]李清水,方志刚等.手势识别技术及其在人机交互中的应用[J].人类工效学,2002,80
[50]隋毅.基于手持设备的增强现实技术研究与应用：[研究学位论文]山东：青岛大学,2009
[2]Bimber. O. and Raskar. R., Spatial Augmented Reality Merging Real and Virtual Worlds.2005:A. K. Peters Ltd.
[3]Billinghurst, M.. Research Directions in Augmented Reality. In Proc. of 16th International Conf. on Artificial Reality and Telexistence (ICAT'06).2006
[4]Ronald T. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments,1997.6(4):355-385
[5]Feiner, S., MacIntyre, B., Seligmann D.. Knowledge-based augmented reality[J]. Communications of the ACM,1993,36(7),52-62
[6]Thalmann, N. M., and Papagiannakis, P.. Virtual worlds and augmented reality in cultural Heritage Applications. In Proc. of International Workshop on Recording, Modeling and Visualization of Cultural Heritage.2005
[7]Pentenrieder. K. and Meier. P.. The need for accuracy statements in industrial Augmented Reality applications. In Proc. of Sixth International Symposium on Mixed and Augmented Reality.2007
[8]朱淼良,姚远,蒋云良.增强现实综述[J].中国图像图形学报,2004,7：767-773
[9]Loscos, C., et al. Interactive virtual relighting of real scenes[J]. IEEE Transactions on Visualization and Computer Graphic,2000,6(3):289-305
[10]James R Vallino. Interactive Augmented Reality[D]. PhD thesis. New York: University of Rochester,1998, Kiriakos
[11]Azuma, Ronald T.. The Challenge of Making Augmented Reality Work outdoors[M]. In Mixed Reality: Merging Real and Virtual Worlds, Yuichi Ohta and Hideyuki tamura, editors. Springer-Verlag, 1999, ISBN 3-540-65623-5. Chaper 21,pp.379-390
[12]Jun Rekimoto and Katashi Nagao. The world through the computer: computer augmented interaction with real world envirments. In Proceedings of the 8th annual ACM symposium on User interface and software technology(UIST'95),1995:29-36
[13]Mark Billinghurst, Hirkazu Kato, Ivan Pougyrev. The magicbook-moving seamlessly between reality and virtuality. IEEE Computer Graphics and Applications,21(3),2001:6-8
[14]Kanvara Masayuki, Yoloya Naokazu. Real-time estimation of light source environment for photorealistic augmented reality[A]. Proceedings of the 17th International Conference on Pattern Recognition[C]. Cambridge,2004:911-914
[15]Daniel Wagner, Istvan Barakonyi. Augmented reality kanji learning. In Proceedings of the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality(ISMAR'03),2003: 335-336
[16]Shahzad Malik, Avhishek Ranjan, Ravin balakrishnan. Interacting with Large Displays form a Distance with Vision-Tracked Multi-Finger gestural Input. In Proceedings of ACM Symposium on User Interface software technology (UIST) 2005[C], Seattle, WA, USA. P43-52
[17]Chris McDonald, Shahzad Malik, Gerhard Roth. Hand-based Interaction in Augmented Reality[A]. In Proceedings of IEEE International Workshop on Haptic Audio Visual Environments and their Application(HAVE)[C], Ottawa, Ontario, Canada.2002, pp.55-59
[18]Kato H., Billinghurst M., Poupyrev I., et al. Virtual object manipulation of a tavle-top AR environment[A]. In Proceedings of International Symposium on Augmented Reality 2000 (ISAR 00) [C], USA,2000:111-119
[19]柳祖国,李世其,李作清.增强现实技术的研究进展及应用[J].系统仿真学报,2003,15(2)：222-225
[20]任波,管涛,李利军等.基于ARToolKit的增强现实系统开发与应用[J].计算机系统应用,2006(1)：81-84
[21]任波,李利军,曹伟民等.基于增强现实技术的交互式城市设计研究[J].华中科技大学学报,2006,23(2)：32-34
[22]Shahzad Malik. Robust Registration of Virtual Ovjects for Real-Time Augmented Reality. Master thesis, Carleton University Ottawa, Ontario, Canada,2002
[23]William A. Hoff, Khoi Nguyen. Computer vision-based registration techniques for augmented reality, Proc. of Intelligent Robots and Computer Vision ⅩⅤ, Vol.2904, in Intellignet Systems & Advanced Manufacturing, SPIE, Boston, Massachusetts, Nov.19-21,1996, pp.538-548
[24]姚远.增强现实应用技术研究：[研究学位论文]杭州：浙江大学,2006
[25]WEI Zhiqiang, Ji Xiaopeng, FENG Yewei. A moving object detection method based on self adaptive updating of background[J]. Acta Electronica Sinica,2005,33 (12):2261-2264
[26]Barron J., Fleet D., Neauchemin S. Performance of optical flow techniques[J]. International Computer Vision,1994,12(1):42-77
[27]Richard P. Wildes. A measure of motion salience for surveillance application. In Proc of Image Processing,1998
[28]Elgammal A, Hawood D, Davis L S. Non-Parametric model for background subtraction Proc. ECCV,2000,7:751-767
[29]A. Aydin Alatan, Levent Onural., Michael Wollborn, et al. Image Sequence Analysis for Emerging Interactive Multimedia Services. The European COST 211 Framework[J]. IEEE Transactions on Circuits and Systems for Video Technology.1998-11, (8,7):802-813
[30]Jwu-Sheng Hu, Tzung-Min Su and Shr-Chi Jeng, "Robust Backgroud Subtraction with Shadow and Highlight Removal for Indoor Surveillance," Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems October 9-15,2006,Beijing, China,4545-4551
[31]Daniel Diggins. ARLib: A C++ Augmented Reality Software Development Kit:[硕士学位论文]英国：Bournemouth University,2005
[32]Chang C J, HSIEH J W, CHEN Y S, et al. Tracking multiple moving objects using a level-set method[J]. International Journal of Pattern Recognition and Artificial Intelligence,2004,18(2): 101-125
[33]Jong Bae Kim, Hang Joon kim. Efficient region-based motion segmentation for video monitoring system. Pattern Recognition Letters 2003,24(1-3):113-128
[34]Qu You-shan, Tian Wei-jian, Li Ying-cai. The moving target detecting on the parallelable discontinuous frame difference optical field integrated with gray intensity analysis. Acta Photonica Sinica,2003,32(2):182-186
[35]Remagnino P, Tan T, Baker K. Mufti-agent visual surveillance of dynamic scenes. Image and Vision Computing,1998,16(8):529-532
[36]Lipton A., Fujiyoshi H., Patil R.. Moving target classification and tracking form real-time video[C]. IEEE workshop on Application of Computer Vision, Princeton: IEEE Press,1998:8-14
[37]危水根等.一种基于时间差分运动检测的改进方法.南昌航空工业学院学报.2005,3：15-19
[38]代科学等.监控视频运动目标检测减背景技术的研究现状和展望.中国图像图形
[39]郝志成,朱明.智能目标检测与跟踪系统的设计与实现[J].光电工程,2007,34(1)：27-3
[40]付萍,方帅,徐心和等,视频监控系统中运动目标检测的阴影去除方法.计算机工程,2007,33(10)：22-24
[41]魏志强,纪筱鹏,冯业伟.基于自适应背景图像更新的运动目标检测方法[J].电子学报,2005,33(12)：2261-2264
[42]秦涛,周泽魁.一种检测序列图像中运动目标的新方法[J].计算机应用与软件.2004,21(9)：105-107
[43]文灏,红涛.基于减背景与对称查分的运动目标检测[J].微计算机信息,2007,9-1：99-101
[44]王陈阳,周明全,耿国华.基于自适应背景模型运动目标检测[J].计算机技术与发展,2007,17(4)：21-23
[45]吕国亮,赵曙光,赵俊.基于三帧差分和连通性检验的图像运动目标检测新方法[J].液晶与显示,2007,22(1)：87-93
[46]贺贵明,李凌娟,贾振堂.一种快速的基于对称差分的视频分割算法[J].小型微型计算机系统,2003,24(6)：96-98
[47]任海兵,祝远新,徐光,张晓平,林学訚.复杂背景下的手势分割与识别.自动化学报,2002.3(28)：256-261
[48]Lew, Y. P, Ramli, A. R, Koay, S. Y, Ali, R. and Prakash, V. A Hand Segmentation Scheme Using Clustering Technique in Homogeneous Background[A]. Student Conference Research and Development Proceedings.2002,305-308
[49]李清水,方志刚等.手势识别技术及其在人机交互中的应用[J].人类工效学,2002,80
[50]隋毅.基于手持设备的增强现实技术研究与应用：[研究学位论文]山东：青岛大学,2009