基于计算机视觉的多点触摸技术研究与实现
|
摘要
多点触摸技术是自2007年起兴起的一项多路输入技术,是计算机人机交互领域的一项重要创新。多点触摸技术提供了良好的用户体验,在社会服务领域有着很大的应用潜力,且其相关技术的推广应用甚至正作为一个新兴产业在蓬勃发展,具有明显的商业价值和十分诱人的发展前景。
多触摸技术的实现方案之一是基于计算机视觉技术。这种实现方案具有成本低廉、易于扩展的特点,尤其适合于大型的互动场合应用。
主要研究工作和创新点如下:
介绍了几种常用的多点触摸技术的实现原理,研究与讨论了多点触摸技术实现中的一般理论和技术性问题,并基于计算机视觉技术和红外光学技术实现了一个完整的多点触摸系统。该系统的体系架构可以划分为六个层次。若将系统建立成C/S模型,则系统包含一个通用的输出触摸事件的服务端驱动,和多个支持多点触摸输入的客户端应用程序。
介绍了多路触摸信号检测模块各个流程的实现方法,其中提出了一种基于Blob的触摸信息分析方法,使得系统能感应多触摸点的二维坐标和压力信息。
讨论了多触摸点运动分析技术。采用了一种将视频帧间的多目标运动跟踪问题归约到帧间触摸点匹配问题的新方法,在该方法的基础上首次提出一种基于二分图最优匹配的方法来解决帧间触摸点的匹配问题。实验证明,该算法能正确地判断多触摸点的运动状态,且能满足实时性的要求。
讨论了多触摸技术中的多用户判别技术。采用了一种面向应用对象的多用户判别策略,研究并实现了支持单用户的鼠标模拟技术,及支持多用户的多指移动、双指缩放、双指旋转及压力感应等多触摸手势的识别方法。
经过实验验证,提出的相关算法和策略能实现多用户多触摸手势识别功能,且实时性强,交互方式人性化,具有市场推广和应用价值。
Multi-touch technology is a multi-channel input technology which has been rising since 2007. It is an important innovation in Human and Computer Interaction (HCI) field. Multi-touch technology provides good user experience, and has great potential to apply on many social service fields. Its related technology has significant commercial potential and attractive development prospect.
One of the Multi-touch solutions is based on Computer Vision technology. This solution is low cost, and very easy to expand, and is especially fit for large situation of interaction. Follow is the main research work and innovations:
Introduce some technology which are used on multi-touch product now, and discuss the general problems during the implement of a multi-touch system which based on both Computer Vision and infrared optics technology. The system architecture can be divided into six levels, and if build as C/S model, the architecture can be composed of a common driver which outputs the touch events in real time, and some client applications which support the Multi-touch interaction.
Introduce the algorithm of each step in the multi-touch signal detection module. Especially, propose a Blob based method to analyze the touch information, which enables the system be sensitive to both position and pressure of touch.
Discuss the motion analysis problem of touch points. Apply an idea that the motion analysis problem can be transformed into a match problem between consecutive frames. According to this idea, propose a new method based on Optimal Matching of Bipartite Graph Algorithm to solve the match problem. The experiment proves that the new methods could correctly judge the motion status of touch points, and can run in real time.
Discuss the multi-user support technology in Multi-touch system, and propose a strategy based on different application object to distinguish multi-user. Show some Multi-touch applications, such as cursor simulation, and multi fingers to move, zoom and rotate something, also show an application which is pressure sensitive.
The experiments show that the algorithms and strategies discussed in this paper can support Multi-touch technology well, and useful for many relative applications.
多触摸技术的实现方案之一是基于计算机视觉技术。这种实现方案具有成本低廉、易于扩展的特点,尤其适合于大型的互动场合应用。
主要研究工作和创新点如下:
介绍了几种常用的多点触摸技术的实现原理,研究与讨论了多点触摸技术实现中的一般理论和技术性问题,并基于计算机视觉技术和红外光学技术实现了一个完整的多点触摸系统。该系统的体系架构可以划分为六个层次。若将系统建立成C/S模型,则系统包含一个通用的输出触摸事件的服务端驱动,和多个支持多点触摸输入的客户端应用程序。
介绍了多路触摸信号检测模块各个流程的实现方法,其中提出了一种基于Blob的触摸信息分析方法,使得系统能感应多触摸点的二维坐标和压力信息。
讨论了多触摸点运动分析技术。采用了一种将视频帧间的多目标运动跟踪问题归约到帧间触摸点匹配问题的新方法,在该方法的基础上首次提出一种基于二分图最优匹配的方法来解决帧间触摸点的匹配问题。实验证明,该算法能正确地判断多触摸点的运动状态,且能满足实时性的要求。
讨论了多触摸技术中的多用户判别技术。采用了一种面向应用对象的多用户判别策略,研究并实现了支持单用户的鼠标模拟技术,及支持多用户的多指移动、双指缩放、双指旋转及压力感应等多触摸手势的识别方法。
经过实验验证,提出的相关算法和策略能实现多用户多触摸手势识别功能,且实时性强,交互方式人性化,具有市场推广和应用价值。
Multi-touch technology is a multi-channel input technology which has been rising since 2007. It is an important innovation in Human and Computer Interaction (HCI) field. Multi-touch technology provides good user experience, and has great potential to apply on many social service fields. Its related technology has significant commercial potential and attractive development prospect.
One of the Multi-touch solutions is based on Computer Vision technology. This solution is low cost, and very easy to expand, and is especially fit for large situation of interaction. Follow is the main research work and innovations:
Introduce some technology which are used on multi-touch product now, and discuss the general problems during the implement of a multi-touch system which based on both Computer Vision and infrared optics technology. The system architecture can be divided into six levels, and if build as C/S model, the architecture can be composed of a common driver which outputs the touch events in real time, and some client applications which support the Multi-touch interaction.
Introduce the algorithm of each step in the multi-touch signal detection module. Especially, propose a Blob based method to analyze the touch information, which enables the system be sensitive to both position and pressure of touch.
Discuss the motion analysis problem of touch points. Apply an idea that the motion analysis problem can be transformed into a match problem between consecutive frames. According to this idea, propose a new method based on Optimal Matching of Bipartite Graph Algorithm to solve the match problem. The experiment proves that the new methods could correctly judge the motion status of touch points, and can run in real time.
Discuss the multi-user support technology in Multi-touch system, and propose a strategy based on different application object to distinguish multi-user. Show some Multi-touch applications, such as cursor simulation, and multi fingers to move, zoom and rotate something, also show an application which is pressure sensitive.
The experiments show that the algorithms and strategies discussed in this paper can support Multi-touch technology well, and useful for many relative applications.
引文
[1]崔鸿雁.多通道用户界面.微处理机, 2006, (5): 71~76
[2]张毅,覃京燕,李威.基于手语语义学的多点触摸交互系统的用户行为研究.包装工程, 2008, 29(5): 144~145, 156
[3] Wolfeld, Jeffrey A. Real Time Control of a Robot Tactile Sensor: [MSc Thesis]. Philadelphia: Moore School of Electrical Engineering, 1981
[4] Mehta, Nimish. A Flexible Machine Interface: [M.A.Sc. Thesis]. Department of Electrical Engineering, University of Toronto, 1982
[5] Nakatani, L. H., Rohrlich, John A. Soft Machines: A Philosophy of User-Computer Interface Design. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’83), 1983. 12~15
[6] Krueger, Myron, W., Gionfriddo et al. VIDEOPLACE - An Artificial Reality, In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’85), 1985. 35 ~ 40
[7] Leonard Kasday, Bell Labs. Touch position sensitive surface. US. Patent 4484179, 1984.
[8] SK Lee , William Buxton, K. C. Smith. A Multi-Touch Three Dimensional Touch-Sensitive Tablet, ACM Press, 1985. 21~25
[9] William Buxton, Ralph Hill, Peter Rowley. Issues and techniques in touch-sensitive tablet input. ACM Press, 1985. 215~224
[10] McAvinney, P. The Sensor Frame - A Gesture-Based Device for the Manipulation of Graphic Objects. Carnegie-Mellon University, 1986.
[11] Buxton, W., Myers, B. A study in two-handed input. Proceedings of CHI '86, 1986. 321~326
[12] Kabbash, P., Buxton, W., Sellen, A. Two-Handed Input in a Compound Task. Proceedings of CHI'94, 1994:417~423
[13] Buxton, W. Combined keyboard / touch tablet input device. XEROX Disclosure Journal, 1994, 19(2): 109~111
[14] Leganchuk, A., Zhai, S., Buxton, W. Manual and Cognitive Benefits of Two-Handed Input: An Experimental Study. Transactions on Human-Computer Interaction, 1998, 5(4): 326~359
[15] Westerman, Wayne. Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface: [PhD Thesis]. University of Delaware, 1999.
[16] Jeff Han, Y. Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, 2005,115~118
[17]木村雅秀,浅川直辉. iPhone带给产品设计师的启示.电子设计应用,2007(12). 75~78
[18]马冬梅.可触摸的桌面电脑.发明与创新, 2008 (5). 26
[19] PMartin Kaltenbrunner, PRoss Bencina. reacTIVision: a computer-vision framework for table-based tangible interaction, In: Proceedings of the 1st international conference on Tangible and embedded interaction, 2007. 69~74
[20]科技部, "863计划信息技术领域2008年度专题课题申请指南," 2008.
[21]马颂德,张正友.计算机视觉——计算理论与算法基础.第一版.北京:科学出版社, 1998. 5~7, 56~59, 72~75
[22] Z. Y. Zhang. A Flexible New Technique for Camera Calibration. IEEE Transactions Pattern Analysis and Machine Intelligence, 1998, 22(11): 1330~1334
[23] M. Lampton. Damping-Undamping Strategies for the Levenberg-Marquardt Nonlinear Least-Squares Method. Computer in Physics Journal, 1997, 11(1): 110~115
[24]章毓晋.图像工程(中册):图像分析.第二版.北京:清华大学出版社, 2005. 61~90
[25]舒添慧.图像分割理论方法及应用: [硕士学位论文].华南理工大学, 1989.
[26]孙惠泉.图论及其应用.北京:科学出版社, 2004. 89~92
[27] Thomas H.Cormen, Charles E.Leiserson, Ronald L.Rivest, et al.算法导论.第二版.北京:机械工业出版社. 2006. 321~419
[28]高纯玲.基于双目立体视觉的手术器械跟踪定位研究:[硕士学位论文].华中科技大学, 2008.
[29]王臣豪,刘富强,田敏.基于Blob的车辆识别及其跟踪算法研究.信号处理, 2005, 21(4): 370~374
[30] The open source library of multi-touch:“TouchLib”, http://nuigroup.com/touchlib/
[31]徐杰,刘鸿飞,于海兰.基于PC/104的图形界面控制软件设计.微计算机信息,2003, 19(5): 19~20
[32] D. Comaniciu, P. Meer. Real-Time Tracking of Non-Rigid Objects using Mean Shift. In: Proceedings of IEEE Conference on Computer Vision and Patter Recognition. Hilton Head Island, South Carolina: IEEE Press, 2000. 142~149
[33] Haiting Zhai, Xiaojuan Wu, Hui Han. Research of a Real-time Hand Tracking Algorithm. IEEE International Conference on Neural Networks and Brains, 2005, 2(1): 1233~1235
[34] R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Transaction of the ASME-Journal of Basic Engineering, 1960, 8(3): 35~45
[35] Paul Dietz, Darren Leigh. DiamondTouch: A Multi-User Touch Technology. ACM Press, 2001,219~226
[36] PHrvoje Benko, PAndrew D. Wilson, PPatrick Baudisch. Precise selection techniques for multi-touch screens. In: Proceedings of the SIGCHI conference on Human Factors in computing systems, 2006, 1263~1272
[37] PTomer Moscovich, PJohn F. Hughes.Jun. Multi-finger cursor techniques. In: Proceedings of Graphics Interface, 2006, 1~7
[38] Florian Echtler, Gudrun Klinker. A multitouch software architecture. In: Proceedings of the 5th Nordic conference on Human-computer interaction, 2008
[39] The Sensor Frame Graphic Manipulator NASA Phase 2 Final Reports, 1985,1~24
[40] Herot, C., Weinzapfel, G. One-Point Touch Input of Vector Information from Computer Displays, Computer Graphics, 1978, 12(3), 210~216
[41] Minsky, M. Manipulating Simulated Objects with Real-World Gestures Using a Force and Position Sensitive Screen, Computer Graphics, 1984, 18(3), 195~203
[2]张毅,覃京燕,李威.基于手语语义学的多点触摸交互系统的用户行为研究.包装工程, 2008, 29(5): 144~145, 156
[3] Wolfeld, Jeffrey A. Real Time Control of a Robot Tactile Sensor: [MSc Thesis]. Philadelphia: Moore School of Electrical Engineering, 1981
[4] Mehta, Nimish. A Flexible Machine Interface: [M.A.Sc. Thesis]. Department of Electrical Engineering, University of Toronto, 1982
[5] Nakatani, L. H., Rohrlich, John A. Soft Machines: A Philosophy of User-Computer Interface Design. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’83), 1983. 12~15
[6] Krueger, Myron, W., Gionfriddo et al. VIDEOPLACE - An Artificial Reality, In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’85), 1985. 35 ~ 40
[7] Leonard Kasday, Bell Labs. Touch position sensitive surface. US. Patent 4484179, 1984.
[8] SK Lee , William Buxton, K. C. Smith. A Multi-Touch Three Dimensional Touch-Sensitive Tablet, ACM Press, 1985. 21~25
[9] William Buxton, Ralph Hill, Peter Rowley. Issues and techniques in touch-sensitive tablet input. ACM Press, 1985. 215~224
[10] McAvinney, P. The Sensor Frame - A Gesture-Based Device for the Manipulation of Graphic Objects. Carnegie-Mellon University, 1986.
[11] Buxton, W., Myers, B. A study in two-handed input. Proceedings of CHI '86, 1986. 321~326
[12] Kabbash, P., Buxton, W., Sellen, A. Two-Handed Input in a Compound Task. Proceedings of CHI'94, 1994:417~423
[13] Buxton, W. Combined keyboard / touch tablet input device. XEROX Disclosure Journal, 1994, 19(2): 109~111
[14] Leganchuk, A., Zhai, S., Buxton, W. Manual and Cognitive Benefits of Two-Handed Input: An Experimental Study. Transactions on Human-Computer Interaction, 1998, 5(4): 326~359
[15] Westerman, Wayne. Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface: [PhD Thesis]. University of Delaware, 1999.
[16] Jeff Han, Y. Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, 2005,115~118
[17]木村雅秀,浅川直辉. iPhone带给产品设计师的启示.电子设计应用,2007(12). 75~78
[18]马冬梅.可触摸的桌面电脑.发明与创新, 2008 (5). 26
[19] PMartin Kaltenbrunner, PRoss Bencina. reacTIVision: a computer-vision framework for table-based tangible interaction, In: Proceedings of the 1st international conference on Tangible and embedded interaction, 2007. 69~74
[20]科技部, "863计划信息技术领域2008年度专题课题申请指南," 2008.
[21]马颂德,张正友.计算机视觉——计算理论与算法基础.第一版.北京:科学出版社, 1998. 5~7, 56~59, 72~75
[22] Z. Y. Zhang. A Flexible New Technique for Camera Calibration. IEEE Transactions Pattern Analysis and Machine Intelligence, 1998, 22(11): 1330~1334
[23] M. Lampton. Damping-Undamping Strategies for the Levenberg-Marquardt Nonlinear Least-Squares Method. Computer in Physics Journal, 1997, 11(1): 110~115
[24]章毓晋.图像工程(中册):图像分析.第二版.北京:清华大学出版社, 2005. 61~90
[25]舒添慧.图像分割理论方法及应用: [硕士学位论文].华南理工大学, 1989.
[26]孙惠泉.图论及其应用.北京:科学出版社, 2004. 89~92
[27] Thomas H.Cormen, Charles E.Leiserson, Ronald L.Rivest, et al.算法导论.第二版.北京:机械工业出版社. 2006. 321~419
[28]高纯玲.基于双目立体视觉的手术器械跟踪定位研究:[硕士学位论文].华中科技大学, 2008.
[29]王臣豪,刘富强,田敏.基于Blob的车辆识别及其跟踪算法研究.信号处理, 2005, 21(4): 370~374
[30] The open source library of multi-touch:“TouchLib”, http://nuigroup.com/touchlib/
[31]徐杰,刘鸿飞,于海兰.基于PC/104的图形界面控制软件设计.微计算机信息,2003, 19(5): 19~20
[32] D. Comaniciu, P. Meer. Real-Time Tracking of Non-Rigid Objects using Mean Shift. In: Proceedings of IEEE Conference on Computer Vision and Patter Recognition. Hilton Head Island, South Carolina: IEEE Press, 2000. 142~149
[33] Haiting Zhai, Xiaojuan Wu, Hui Han. Research of a Real-time Hand Tracking Algorithm. IEEE International Conference on Neural Networks and Brains, 2005, 2(1): 1233~1235
[34] R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Transaction of the ASME-Journal of Basic Engineering, 1960, 8(3): 35~45
[35] Paul Dietz, Darren Leigh. DiamondTouch: A Multi-User Touch Technology. ACM Press, 2001,219~226
[36] PHrvoje Benko, PAndrew D. Wilson, PPatrick Baudisch. Precise selection techniques for multi-touch screens. In: Proceedings of the SIGCHI conference on Human Factors in computing systems, 2006, 1263~1272
[37] PTomer Moscovich, PJohn F. Hughes.Jun. Multi-finger cursor techniques. In: Proceedings of Graphics Interface, 2006, 1~7
[38] Florian Echtler, Gudrun Klinker. A multitouch software architecture. In: Proceedings of the 5th Nordic conference on Human-computer interaction, 2008
[39] The Sensor Frame Graphic Manipulator NASA Phase 2 Final Reports, 1985,1~24
[40] Herot, C., Weinzapfel, G. One-Point Touch Input of Vector Information from Computer Displays, Computer Graphics, 1978, 12(3), 210~216
[41] Minsky, M. Manipulating Simulated Objects with Real-World Gestures Using a Force and Position Sensitive Screen, Computer Graphics, 1984, 18(3), 195~203