基于多点触控的大屏幕人机交互技术研究与应用
|
摘要
随着物联网技术的快速发展,在消防监控和交通管理等很多领域得到了广泛应用。大屏幕作为信息展示的重要设备,正起到愈来愈重要的作用。然而,由于缺乏合理有效的人机交互方式,只能实现大屏幕到用户的单向信息展示,用户无法全面地获得信息支持,直接影响用户对信息的把握与利用。如何让系统中的海量信息被用户通过自然的操控方式全方位、多维度地掌握和利用,已经成为亟待解决的问题。
本文围绕物联网系统中的大屏幕人机交互技术及其实际应用,展开理论和应用研究,提出基于多点触控技术的大屏幕人机交互解决方案,实现了大屏幕信息显示与用户间的双向信息交互,具体研究内容如下：
◆针对多点触控实现成本与技术成熟度的矛盾,以及多点触控在人机交互方面存在的若干问题,提出结合具有多点触控功能的智能终端,在已有的大屏幕设备上实现稳定性好,精度高,并支持多用户交互的大屏幕多点触控方案。
◆结合实际应用需求,针对目前多种多样的大屏幕显示系统和智能终端开发环境,提出基于MvC模式的三层系统架构,使其具有开放性好,可扩展和低耦合性等特点,并设计分阶段的交互协议,以适应各层间的通信需求,从而满足大屏幕与智能终端的多样性需求；
◆针对大屏幕与智能终端在实际应用中的不同交互场景和模式,结合多点触控人机交互的OCGM隐喻,提出多源信息的大屏幕可视化设计和智能终端交互设计,支持多点触控的多种操控方式,满足不同应用场景需求。
◆结合消防信息化系统中的实际应用需求,基于本文提出的系统解决方案与人机交互方法,提出面向消防领域的多源信息展示与人机交互系统的设计方案。
With rapid development of Internet of Things (IOT) and its relative technologies, it has been used in wider fields, such as in fire monitoring, traffic management and so on. Large-scale display is the important equipment to show the huge amounts of information in the IOT system to the users. Without reasonable and effective human-computer interaction (HCI), only one-way information transportation from display to users is available. It makes the displays losing real-time and flexible reaction on user's demand and limits the fully usage of the information. Therefore, the research in HCI of large-scale displays can be great help for the promoting of IOT.
This thesis studies the HCI of large-scale displays in the IOT systems and the practical application. An interaction solution based on multi-touch technique is provided. It establishes a two-way information interaction between display and users. The research objectives and main achievements in the thesis are listed as follows:
◆Based on the contradiction between cost and technology maturity, this thesis presents a large-scale display interaction solution using intelligent terminal with multi-touch to achieve great stability, high precision and support multi-user. It also solves some shortages in HCI of large-scale multi-touch.
◆Targeting on the practical application requirement and multiple development environments for display systems and terminals, a three-layer system architecture based on MVC model is presented. It owns expansibility and low coupling to meet the demand of diversity. With three-phase interaction protocol, terminals can collaborate with display by effective information communication.
◆Aiming at different interaction scenes and models, visualization of information display and interaction of terminal are designed to be applicable to multi-touch control and massive information display.
◆Combining with application requirements from fire information system, multiple source information display and interaction system is designed based on the solution, architecture and interaction of this thesis.
本文围绕物联网系统中的大屏幕人机交互技术及其实际应用,展开理论和应用研究,提出基于多点触控技术的大屏幕人机交互解决方案,实现了大屏幕信息显示与用户间的双向信息交互,具体研究内容如下：
◆针对多点触控实现成本与技术成熟度的矛盾,以及多点触控在人机交互方面存在的若干问题,提出结合具有多点触控功能的智能终端,在已有的大屏幕设备上实现稳定性好,精度高,并支持多用户交互的大屏幕多点触控方案。
◆结合实际应用需求,针对目前多种多样的大屏幕显示系统和智能终端开发环境,提出基于MvC模式的三层系统架构,使其具有开放性好,可扩展和低耦合性等特点,并设计分阶段的交互协议,以适应各层间的通信需求,从而满足大屏幕与智能终端的多样性需求；
◆针对大屏幕与智能终端在实际应用中的不同交互场景和模式,结合多点触控人机交互的OCGM隐喻,提出多源信息的大屏幕可视化设计和智能终端交互设计,支持多点触控的多种操控方式,满足不同应用场景需求。
◆结合消防信息化系统中的实际应用需求,基于本文提出的系统解决方案与人机交互方法,提出面向消防领域的多源信息展示与人机交互系统的设计方案。
With rapid development of Internet of Things (IOT) and its relative technologies, it has been used in wider fields, such as in fire monitoring, traffic management and so on. Large-scale display is the important equipment to show the huge amounts of information in the IOT system to the users. Without reasonable and effective human-computer interaction (HCI), only one-way information transportation from display to users is available. It makes the displays losing real-time and flexible reaction on user's demand and limits the fully usage of the information. Therefore, the research in HCI of large-scale displays can be great help for the promoting of IOT.
This thesis studies the HCI of large-scale displays in the IOT systems and the practical application. An interaction solution based on multi-touch technique is provided. It establishes a two-way information interaction between display and users. The research objectives and main achievements in the thesis are listed as follows:
◆Based on the contradiction between cost and technology maturity, this thesis presents a large-scale display interaction solution using intelligent terminal with multi-touch to achieve great stability, high precision and support multi-user. It also solves some shortages in HCI of large-scale multi-touch.
◆Targeting on the practical application requirement and multiple development environments for display systems and terminals, a three-layer system architecture based on MVC model is presented. It owns expansibility and low coupling to meet the demand of diversity. With three-phase interaction protocol, terminals can collaborate with display by effective information communication.
◆Aiming at different interaction scenes and models, visualization of information display and interaction of terminal are designed to be applicable to multi-touch control and massive information display.
◆Combining with application requirements from fire information system, multiple source information display and interaction system is designed based on the solution, architecture and interaction of this thesis.
引文
[1]. International Telecommunication Union UIT. Internet Reports 2005:The Internet of Things [R/OL]. Tunis, Tunisia:WSIS,2005. [2]. S. Haller, S. Karnouskos, S. Christoph. The Internet of Things in an Enterprise Context [J]. Future Internet- FIS 2008, Berlin 2009,14-28.[3].IBM智慧的地球[EB/OL].: http://www.ibm.com/smarterplanet/cn/zh/index.html?re=sph. [4]. Auto—ID Labs [EB/OL].:http://www.autoidlabs.org,1999.[5].海尔U-home智能家居解决方案[EB/OL].: http://www.haieruhome.com/index-cn.asp. [6]. K. Finkenzeller. RFID Handbook:Fundamentals and Applications in Contactless Smart Cards, Radio Frequency Identification and Near-Field Communication [M]. Wiley,2010.[7].陈红华,李洁等.ETC技术及其发展[J].公路交通科技,2001,18(3).[8]. L. Lin, D.R.Li, et al. Smart Planet Based on Geomatics [J].30th International Conference on Dubrovnik IEEE,2010,1-4. [9]. X. Bi, R. Balakrishnan. Comparing Usage of a Large High-Resolution Display to Single or Dual Desktop Displays for Daily Work[C]. CHI '09 Proceedings of the 27th international conference on Human factors in computing systems, 2009,1005-1014. [10]. G. Wallace, O. J. Anshus, et al. Tools and applications for large-scale display walls [J]. IEEE Computer Graphics and Applications,2005,25(4):24-33. [11]. T. Ni, G. S. Schmidt, et al. A survey of large high-resolution display technologies, techniques, and applications[C]. Proc. IEEE Virtual Reality, 2006,223-236. [12]. A. Khan, G. Fitzmaurice, et al. A Remote Control Interface for Large Display [C]. Proc.17th annual ACM symposium on User interface software and technology,2004,23-31. [13]. Sun et al. Visual input pointing device for interactive display system [P]. USA:7,683,881 B2,2010: [14]. A. Bellucci, A. Malizia, P. Diaz, et al. Human-Display Interaction Technology:Emerging Remote Interfaces for Pervasive Display Environments [J]. IEEE Pervasive Computing,2010,9(2):72-76. [15]. Nichols, S. J. V. New Interfaces at the Touch of a Fingertip [J]. Computer, 2007,40(8):12-15. [16]. M. Weiser. The computer for the 21st century [M]. Scientific American,1995: 272(3):78-89. [17]. C. Pei, H. Guo, et al. Sensors in Smart Phone [C]. Computer and Computing Technologies in Agriculture Ⅳ,2011, vol.345:491-495. [18]. General Packet Radio Service (GPRS) [S/OL]: http://www.gsmworld.com/technology/gprs/intro.shtml. [19]. Wireless Fidelity(WiFi) [S/OL]:http://www.wi-fi.org [20]. R.Want. When cell phones become computers [J]. Pervasive Computing, IEEE,2009,8(2):2-5. [21]. L. Muller. Multi-touch displays:design, applications and performance evaluation [D]. Amsterdam:University of Amsterdam,2008. [22]. Matsushita, N. and J. Rekimoto. HoloWall:designing a finger, hand, body, and object sensitive wall [C]. Proc.10th annual ACM symposium on User interface software and technology.1997,209-210. [23]. J. Y. Han. Low-cost multi-touch sensing through frustrated total internal reflection [C]. Proc.18th annual ACM symposium on User interface software and technology ACM.2005,115-118. [24]. P. Dietz and D. Leigh. DiamondTouch:a multi-user touch technology [C]. Proc.14th annual ACM symposium on User interface software and technology. 2001,219-226. [25]. J. Rekimoto. SmartSkin:an infrastructure for freehand manipulation on interactive surfaces [C]. Proc. SIGCHI conference on Human factors in computing systems:Changing our world, changing ourselves.2002,113-120. [26]. Microsoft Corporation. MS surface [EB/OL]. http://www.microsoft.com/surface/. [27]. R. Hardy, E. Rukzio, et al. Exploring Expressive NFC-based Mobile Phone Interaction with Large Dynamic Displays [C]. First International Workshop on Near Field Communication.2009,36-41. [28]. S. Hartwig, J.-P. Stromann, and P. Resch. Wireless Microservers. Pervasive Computing [J]. IEEE Pervasive Computing,2002, 1(2):58-66. [29]. R. G. J. et al. Touch actuable data input panel assembly [P]. USA:3,673,327, 1972: [30]. L. R. Kasday. Touch position sensitive surface [P]. USA:4,484,179,1984: [31]. J. B. Mallos. Touch position sensitive surface [P]. USA:4,346,376,1982: [32]. R. E. Mueller. Direct television drawing and image manipulation system [P]. USA:3,846,826,1974: [33]. R. M. White. Tactile sensor employing a light conducting element and a resiliently deformable sheet [P]. USA:4,484,179,1987: [34]. B. Shneiderman. Touch screens now offer compelling uses [J]. IEEE Software,1991,8(2):93-94,107. [35]. J. Schoning, P. Brandl, et al. Multi-touch surfaces:A technical guide [R]. Technical Report TUM-I0833, University of Munster. [36]. J. Y. Han. Multi-touch interaction wall [C]. In ACM SIGGRAPH 2006 Emerging technologies,2006,25. [37]. T. Roth. DSI-diffused surface illumination [EB/OL].: http://iad.projects.zhdk.ch/multitouch/. [38]. S. Hodges, S. Izadi, et al. ThinSight:versatile multi-touch sensing for thin form-factor displays [C]. In 20th annual ACM symposium on User interface software and technology,2007,259-268.[39].孟祥旭.人机交互技术：原理与应用[M].北京：清华大学出版社,2006,1-20 [40]. A. van Dam. Post-WIMP user interfaces [J]. Communications of the ACM, 1997,40(2):63-67. [41]. I. E. Sutherland. A head-mounted three dimensional display [C]. In AFIPS '68,1968, Part I. [42]. C. Cruz-Neira, D. J. Sandin, et al. Surround-screen projection-based virtual reality:the design and implementation of the CAVE [C]. In SIGGRAPH '93, 1993,135-142. [43]. D.A. Bowman, E. Kruijff, et al.3D User Interfaces:Theory and Practice [M]. Addison-Wesley/Pearson Education,2005. [44]. R. Jacob, A. Girouard, et al. Reality-Based Interaction:A Framework for Post-WIMP Interfaces [C]. In Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems.2008,201-210. [45]. I. Sutherland. Sketch pad:A man-machine graphical communication system [C]. In DAC '64 Proceedings of the SHARE design automation workshop, 1964,329-346. [46]. D.C. Engelbart. A Conceptual Framework for the Augmentation of Man's Intellect [J]. Vistas In Information Handling, Volume 1, Spartan Books,1963. [47]. B. Stefan, M. Tomitsch, et al. Challenges for Designing the User Experience of Multi-touch Interfaces [C]. In Workshop on Engineering Patterns for Multi-Touch Interfaces,2010. [48]. E. Hoggan, S.A. Brewster and J. Johnston. Investigating the effectiveness of tactile feedback for mobile touchscreens [C]. In SIGCHI conference on Human factors in computing systems,2008,1573-1582. [49]. R. Ballagas, J. Borchers, et al. The smart phone:a ubiquitous input device [J]. Pervasive Computing, IEEE,2006,5(1):70-77. [50]. A. Madhavapeddy et al., Using Camera Phones to Enhance Human-Computer Interaction [C]. Adjunct Proc.6th Int'1 Conf. Ubiquitous Computing,2004. [51]. R. Ballagas, M. Rohs, and J.G. Sheridan. Sweep and Point and Shoot: Phonecam-Based Interactions for Large Public Displays [C]. In CHI 05 Extended Abstracts Human Factors in Computing Systems, ACM Press,2005, 1200-1203. [52]. J.F. Bartlett, Rock'n' Scroll Is Here to Stay [J]. IEEE Computer Graphics and Applications,2000,20(3):40-45. [53]. J. Nichols et al., Personal Universal Controllers:Controlling Complex Appliances with GUIs and Speech [C]. Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 03), ACM Press,2003,624-625. [54]. L. Butts and A. Cockburn. An Evaluation of Mobile Phone Text Input Methods [C], Proc. Conf. Research and Practice in Information Technology Series,3rd Australasian Conf. User Interfaces, Australian Computer Soc., 2002, vol.7:55-59. [55]. R. George and J. Blake. Objects, Containers, Gestures, and Manipulations: Universal Foundational Metaphors of Natural User Interfaces [C]. In CHI 10, 2010. [56]. Wingfield, N. Body in Motion:CES to Showcase Touch Gizmos [N]. Wall Street Journal,2010,1(5). [57]. B. Buxton, CES 2010:NUI with Bill Buxton [EB/OL]. http://channel9.msdn.com/posts/LarryLarsen/CES-2010-NUI-with-Bill-Buxto n/.2010. [58]. J. Hourcade, B. Bederson, et al. Differences in pointing task performance between preschool children and adults using mice [J]. ACM Transactions on Computer-Human Interaction,2004,11(4):357-386. [59]. Vlaming, L., J. Smit, et al. Presenting using two-handed interaction in open space [C]. IEEE Horizontal Interactive Human Computer Systems,2008, 29-32. [60]. T. Tolar, A. Lederberg, et al. The development of the ability to recognize the meaning of iconic signs [J]. Journal of Deaf Studies and Deaf Education,2008, 13:71-86. [61]. K. Szabo. Metaphors and the User Interface [EB/OL]. http://www.katalinszabo.com/metaphor.htm.1995. [62]. A. Gopnik, A.N. Meltzoff. The development of categorization in the second year and its relation to other cognitive and linguistic developments [J]. Child Development,1987,58,1523-1531. [63]. Reenskaug, Trygve. "MVC XEROX PARC 1978-79" [R/OL]. http://heim.ifi.uio.no/-tiygver/themes/mvc/mvc-index.html. [64]. S. Burbeck. Applications programming in Smalltalk-80 (TM):How to use model-view-controller (MVC) [M].1992 [65]. C. Harrison and S. E. Hudson. Providing dynamically changeable physical buttons on a visual display [C]. In 27th international conference on Human factors in computing systems, ACM,2009,299-308. [66]. C. McAdam and S. Brewster. Distal tactile feedback for text entry on tabletop computers [C]. In British Computer Society Conference on Human-Computer Interaction,2009,504-511. [67]. M. Fukumoto. PuyoSheet and PuyoDots:simple techniques for adding button-push feeling to touch panels [C]. In 27th international conference extended abstracts on Human factors in computing systems, ACM,2009. [68]. M. Weiss, R. Jennings, et al. SLAP widgets:bridging the gap between virtual and physical controls on tabletops, ACM,2009,281-490. [69]. Fiebrink, R., D. Morris, et al. Dynamic mapping of physical controls for tabletop groupware [C]. In CHI 2009:Proceedings of the 27th international conference on Human factors in computing systems,2009,471-480. [70]. S.A. Bly, and S.L. Minneman, Commune:A Shared Drawing Surface [C]. In SIGOIS Bulletin, Massachusetts,1990,184-192. [71]. R. T. Smith and W. Piekarski. Public and private workspaces on tabletop displays [C]. In:AUIC '08:Proceedings of the ninth conference on Australasian user interface,2008,51-54.