基于深度图与彩色图像的跑步机游戏交互系统
|
摘要
体感游戏中的运动项目虽然风靡全球,但其运动效果还是远远比不上普通的跑步机。而跑步机游戏系统若能将体感游戏操作和跑步机有机的结合起来,将会给单调的健身运动带来前所未有的娱乐快感,使更多用户能自觉、主动的参与运动中来。
由于具有非接触性和直接性,图像识别已成为目前体感技术的一个研究热点,特别是无标记的识别技术,可让人体不附加任何配件即成为“遥控器”,受到了广泛的关注。经过介绍分析主要的跟踪算法,根据系统需要我们选择了连续自适应均值偏移算法(CAM-Shift)跟踪双手。实验证明,传统的连续自适应均值偏移算法对裸露双手的跟踪效果不尽人意,在复杂背景下基本无法正常工作。对手套的跟踪虽然效果较好,但这违背了系统无标记的设计要求。
为此,在传统连续自适应均值偏移算法的基础上,我们引入了深度信息,并利用深度图从前景分割、初始化和干扰处理几方面对算法进行了改进和扩充。本文主要是对深度图进行阈值分割得到前景目标的二值掩码,屏蔽掉复杂背景以减少非目标区域对搜索结果的干扰影响,同时利用深度图的3D信息初始化搜索窗口。以此为基础,构建了基于深度图和彩色图像的跑步机游戏交互系统。系统使用微软公司的Xbox体感外设Kinect装置获取运动玩家的深度图像和彩色图像序列,并对玩家双手的运动轨迹进行3D跟踪,分析识别其手部动作,实现了非接触、无标记的人机交互。
实验证明系统满足实时要求,对动作反应灵敏准确。
Although the Somatic Game is very popular all over the world, it is still little use for exercise. The treadmill game system will bring people the new fascinating experience of entertainment .And it can make more people do sports more consciously and regularly.
Because of the non-contact and substantivity, recognition from images have become a hotspot in somatic technology. Especially, recognition with non-markers lets human body become remote controller without any accessories and attracts attention widespreadly. After introduced and analyzed the basic tracking algorithm, this thesis chose CAM-Shift algorithm to track bare hands according to the system demands. Then experiments showed that the basic CAM-Shift couldn’t satisfy the system and almost couldn’t work under complex background.
Therefore, this article introduced depth information based on traditional CAM-Shift and developed and extended the algorithm in foreground segmentation, auto-initialization and noise process. This thesis mainly segmented depth map to get the mask off code of the foreground, and made use of the 3D information of the depth maps to initialize the searching window. Then, the thesis built up Interaction System of Treadmill Games based on depth maps and color images. The system realized the man-machine interaction with non-contact and no-markers by getting player’s depth maps and color images with a attachment of Xbox—Kinect from Microsoftm, 3D tracking the player’s bare hands and analyzing and recognizing the movements of the hands. Then the experiments testified this system satisfies the real time request, and the reaction is acute and correct.
由于具有非接触性和直接性,图像识别已成为目前体感技术的一个研究热点,特别是无标记的识别技术,可让人体不附加任何配件即成为“遥控器”,受到了广泛的关注。经过介绍分析主要的跟踪算法,根据系统需要我们选择了连续自适应均值偏移算法(CAM-Shift)跟踪双手。实验证明,传统的连续自适应均值偏移算法对裸露双手的跟踪效果不尽人意,在复杂背景下基本无法正常工作。对手套的跟踪虽然效果较好,但这违背了系统无标记的设计要求。
为此,在传统连续自适应均值偏移算法的基础上,我们引入了深度信息,并利用深度图从前景分割、初始化和干扰处理几方面对算法进行了改进和扩充。本文主要是对深度图进行阈值分割得到前景目标的二值掩码,屏蔽掉复杂背景以减少非目标区域对搜索结果的干扰影响,同时利用深度图的3D信息初始化搜索窗口。以此为基础,构建了基于深度图和彩色图像的跑步机游戏交互系统。系统使用微软公司的Xbox体感外设Kinect装置获取运动玩家的深度图像和彩色图像序列,并对玩家双手的运动轨迹进行3D跟踪,分析识别其手部动作,实现了非接触、无标记的人机交互。
实验证明系统满足实时要求,对动作反应灵敏准确。
Although the Somatic Game is very popular all over the world, it is still little use for exercise. The treadmill game system will bring people the new fascinating experience of entertainment .And it can make more people do sports more consciously and regularly.
Because of the non-contact and substantivity, recognition from images have become a hotspot in somatic technology. Especially, recognition with non-markers lets human body become remote controller without any accessories and attracts attention widespreadly. After introduced and analyzed the basic tracking algorithm, this thesis chose CAM-Shift algorithm to track bare hands according to the system demands. Then experiments showed that the basic CAM-Shift couldn’t satisfy the system and almost couldn’t work under complex background.
Therefore, this article introduced depth information based on traditional CAM-Shift and developed and extended the algorithm in foreground segmentation, auto-initialization and noise process. This thesis mainly segmented depth map to get the mask off code of the foreground, and made use of the 3D information of the depth maps to initialize the searching window. Then, the thesis built up Interaction System of Treadmill Games based on depth maps and color images. The system realized the man-machine interaction with non-contact and no-markers by getting player’s depth maps and color images with a attachment of Xbox—Kinect from Microsoftm, 3D tracking the player’s bare hands and analyzing and recognizing the movements of the hands. Then the experiments testified this system satisfies the real time request, and the reaction is acute and correct.
引文
[1] Falstein N. A Grand Unified Game Theory. In: Miller Freeman. 1999 Game Developers Conference Proceedings. San Fransisco, USA: 1999. 229~239
[2] Minsky M. R-Manipulating simulated objects with real-world gestures using a force and position sensitive screen. AQH Computer Graphics, 1984, 18(3):195~203.
[3] Schl(?)mer Thomas, Popponga Benjamin, Henze Niels et al. Gesture Recognition with a Wii Controller. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. New York, USA: 2008. 11~14
[4]深圳泰山在线科技有限公司.一种目标检测设备及其使用的图像采集装置.中国,实用新型专利, 200920134063, 2010.
[5]深圳泰山在线科技有限公司.一种目标检测方法、系统及立体视觉系统.中国,实用新型专利, 200910107429, 2009.
[6]钱克宠,胡维华.虚拟现实技术在跑步机中的设计与实现.计算机仿真, 2009, (5): 259~261
[7] Mendlovic D, Zalevsky Z, Kiryuschev I et al. Composite Harmonic Filters for Scale. Projection and Shift Invariant Pattern Recognition. Applied Optics, 1995, 34(2):310~316
[8]王江涛.基于视频的目标检测、跟踪及其行为识别研究:[博士学位论文]。南京:南京理工大学, 2007.
[9]王栓,艾海舟,何克忠.基于差分图像的多运动目标的检测和跟踪.中国图象图形学报, 1999, 4(A):470~475
[10] Seki M, Fujiwara H, Sumi K. A Robust Background Subtraction Method for Changing Background. In: IEEE Workshop on Applications of Computer Vision. Palm Springs, USA: 2000. 207~213.
[11] Barron J L, Beauchemin S S, Fleet D J. Performance of Optical Flow Techniques. International Journal of Computer Vision, 1994, 12: 43~47
[12]王新余,张桂林.基于光流的运动目标实时检测方法研究.计算机工程与应用, 2004, 40(1):43~46
[13] Salmond D. Target Tracking: Introduction and Kalman Tracking Filters. In: Proceedings of IEE Target Tracking: Algorithms and Applications. 2001. 2:1/1~1/16
[14] Isard M, Blake A. Condensation-conditional Density Propagation for Visual Tracking. International Journal on Computer Vision, 1998, 29(1):5~28
[15] Fukunaga K, Hostetler L D. The Estimation of the Gradient of a Density Function,with Applications in Pattern Recognition. IEEE Transactions on Information Theory, 1975, 21:32~40
[16] Cheng Yizong. Mean Shift, Mode Seeking, and Clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1995, 17(8):790~799
[17] Bradski G R. Computer Vision Face Tracking for use in a Perceptual User Interface. Intel Technology Journal, 1998, 2nd Quarter.
[18]魏伟波.基于图像的目标自动识别与跟踪技术研究:[博士学位论文].南京:南京理工大学, 2006.
[19]石美红,申亮,龙世忠等.从RGB到HSV色彩空间转换公式的修正.纺织高校基础科学学报, 2008, 21(3):351~356
[20]汤一平,庞成俊.基于全方位视觉传感器的车辆违章检测系统的设计.计算机测量与控制, 2009, 17(6):1045~1048
[21]王辉.基于视觉的实时手势跟踪与识别及其在人机交互中的应用:[硕士学位论文].杭州:浙江大学, 2008.
[22]苏树庆,黄文恺.基于改进型的Camshift的手势定位算法研究.电脑知识与技术, 2010, 6(30):8577~8579
[23] Prime Sense LTD, Tel Aviv. Integrated Processor for 3D Mapping. United States, US 2010/0007717 A1. 2010.
[24] Prime Sense LTD. Method and System for Object Reconstruction. PCT, WO 2007/043036 A1. 2007.
[25] Prime Sense LTD. Depth Mapping Using Projected Patterns. United States, US 2010/0118123 A1. 2010.
[26] Prime Sense LTD. Depth Mapping Using Projected Patterns. United States, US 2008/0240502 A1. 2008.
[27] Avidan S, Shashua A. Trajectory Triangulation: 3D Reconstruction of Moving Points from Amonocular Image Sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(4): 348~357
[28] Leclerc Y G, Bobick A F. The Direct Computation of Height from Shading. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Maui, USA: 1991.552~558
[29] Zhang R, Shah M. Shape from Intensity Gradient. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 1999, 29(3): 318~325
[30] Zhang R, Shah M. Height Recovery from Intensity Gradients. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Seattle, USA: 1994.508~513
[31] Horn B K P. Height and Gradient from Shading. International Journal of Computer Vision, 1990, 5(1):37~76
[32] Bruckstein A M. On Shape from Shading. Computer Vision, Graphics, and Image Processing, 1988, 44(2):139~154
[33] Zhang L, Curless B, Seitz S M. Rapid Shape Acquisition Using Color Structured Light and Multi Pass Dynamic Programming. In: 1st International Symposium on 3D Data Processing Visualization and Transmission. 2002. 24~36
[34] Besl P J. Active, Optical Range Imaging Sensors. Machine Vision and Applications, 1988, 1(2):127~152
[35] Horn E, Kiryati N. Toward Optimal Structured Light Patterns. In: Proceedings of International Conference On Recent Advances in 3D Digital Imaging and Modeling. Ottawa, Canada: 1997.28~35
[36] Asada M, Ichikawa H, Tjuji S. Determining of Surface Properties by Projecting a Stripe Pattern. In: IEEE Proceedings of the International Conference on Pattern Recognition. Paris, France: 1986.86
[37] Asada M, Ichikawa H, Tjuji S. Determining Surface Orientation by Projecting a Stripe Pattern. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988, 10(5): 749~754
[38] Winkelbach S, Wahl F M. Shape from Single Stripe Pattern Illumination. Pattern Recognition, Lecture Notes in Computer Science, 2002, 2449:240~247
[39] Koninckx T P, Gool L Van. Efficient, Active 3D Acquisition, Based on a Pattern-Specific Snake. Pattern Recognition, Lecture Notes in Computer Science, 2002, 2449:557~565
[40] Zigelman G, Kimmel R, Kiryati N. Texure Mapping Using Surface Flattening via Multi-Dimensional Scaling. IEEE Transactions on Visualization and Computer Graphics, 2002, 8(2):198~207
[41]李东晖.数字激光散斑位移测量和跟踪技术研究:[博士学位论文].安徽:中国科学技术大学, 2006.
[42] Dainty J C. Topics in Applied Physics, Volume 9: Laser Speckle and Related Phenomena. Second Enlarged Edition. Berlin and New York: Springer-Verlag, 1975. 254~321.
[43] Mendlovic D, Zalevsky Z, Kiryuschev I, et al. Composite Harmonic Filters for Scale, Projection and Shift Invariant Pattern Recognition. Applied Optics, 1995, 34(2): 310~316
[44]刘曼华,朱秀丽.基于改进CAMShift的多目标自动跟踪.河南大学学报(自然科学版), 2010, 40(5):525~528
[2] Minsky M. R-Manipulating simulated objects with real-world gestures using a force and position sensitive screen. AQH Computer Graphics, 1984, 18(3):195~203.
[3] Schl(?)mer Thomas, Popponga Benjamin, Henze Niels et al. Gesture Recognition with a Wii Controller. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. New York, USA: 2008. 11~14
[4]深圳泰山在线科技有限公司.一种目标检测设备及其使用的图像采集装置.中国,实用新型专利, 200920134063, 2010.
[5]深圳泰山在线科技有限公司.一种目标检测方法、系统及立体视觉系统.中国,实用新型专利, 200910107429, 2009.
[6]钱克宠,胡维华.虚拟现实技术在跑步机中的设计与实现.计算机仿真, 2009, (5): 259~261
[7] Mendlovic D, Zalevsky Z, Kiryuschev I et al. Composite Harmonic Filters for Scale. Projection and Shift Invariant Pattern Recognition. Applied Optics, 1995, 34(2):310~316
[8]王江涛.基于视频的目标检测、跟踪及其行为识别研究:[博士学位论文]。南京:南京理工大学, 2007.
[9]王栓,艾海舟,何克忠.基于差分图像的多运动目标的检测和跟踪.中国图象图形学报, 1999, 4(A):470~475
[10] Seki M, Fujiwara H, Sumi K. A Robust Background Subtraction Method for Changing Background. In: IEEE Workshop on Applications of Computer Vision. Palm Springs, USA: 2000. 207~213.
[11] Barron J L, Beauchemin S S, Fleet D J. Performance of Optical Flow Techniques. International Journal of Computer Vision, 1994, 12: 43~47
[12]王新余,张桂林.基于光流的运动目标实时检测方法研究.计算机工程与应用, 2004, 40(1):43~46
[13] Salmond D. Target Tracking: Introduction and Kalman Tracking Filters. In: Proceedings of IEE Target Tracking: Algorithms and Applications. 2001. 2:1/1~1/16
[14] Isard M, Blake A. Condensation-conditional Density Propagation for Visual Tracking. International Journal on Computer Vision, 1998, 29(1):5~28
[15] Fukunaga K, Hostetler L D. The Estimation of the Gradient of a Density Function,with Applications in Pattern Recognition. IEEE Transactions on Information Theory, 1975, 21:32~40
[16] Cheng Yizong. Mean Shift, Mode Seeking, and Clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1995, 17(8):790~799
[17] Bradski G R. Computer Vision Face Tracking for use in a Perceptual User Interface. Intel Technology Journal, 1998, 2nd Quarter.
[18]魏伟波.基于图像的目标自动识别与跟踪技术研究:[博士学位论文].南京:南京理工大学, 2006.
[19]石美红,申亮,龙世忠等.从RGB到HSV色彩空间转换公式的修正.纺织高校基础科学学报, 2008, 21(3):351~356
[20]汤一平,庞成俊.基于全方位视觉传感器的车辆违章检测系统的设计.计算机测量与控制, 2009, 17(6):1045~1048
[21]王辉.基于视觉的实时手势跟踪与识别及其在人机交互中的应用:[硕士学位论文].杭州:浙江大学, 2008.
[22]苏树庆,黄文恺.基于改进型的Camshift的手势定位算法研究.电脑知识与技术, 2010, 6(30):8577~8579
[23] Prime Sense LTD, Tel Aviv. Integrated Processor for 3D Mapping. United States, US 2010/0007717 A1. 2010.
[24] Prime Sense LTD. Method and System for Object Reconstruction. PCT, WO 2007/043036 A1. 2007.
[25] Prime Sense LTD. Depth Mapping Using Projected Patterns. United States, US 2010/0118123 A1. 2010.
[26] Prime Sense LTD. Depth Mapping Using Projected Patterns. United States, US 2008/0240502 A1. 2008.
[27] Avidan S, Shashua A. Trajectory Triangulation: 3D Reconstruction of Moving Points from Amonocular Image Sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(4): 348~357
[28] Leclerc Y G, Bobick A F. The Direct Computation of Height from Shading. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Maui, USA: 1991.552~558
[29] Zhang R, Shah M. Shape from Intensity Gradient. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 1999, 29(3): 318~325
[30] Zhang R, Shah M. Height Recovery from Intensity Gradients. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Seattle, USA: 1994.508~513
[31] Horn B K P. Height and Gradient from Shading. International Journal of Computer Vision, 1990, 5(1):37~76
[32] Bruckstein A M. On Shape from Shading. Computer Vision, Graphics, and Image Processing, 1988, 44(2):139~154
[33] Zhang L, Curless B, Seitz S M. Rapid Shape Acquisition Using Color Structured Light and Multi Pass Dynamic Programming. In: 1st International Symposium on 3D Data Processing Visualization and Transmission. 2002. 24~36
[34] Besl P J. Active, Optical Range Imaging Sensors. Machine Vision and Applications, 1988, 1(2):127~152
[35] Horn E, Kiryati N. Toward Optimal Structured Light Patterns. In: Proceedings of International Conference On Recent Advances in 3D Digital Imaging and Modeling. Ottawa, Canada: 1997.28~35
[36] Asada M, Ichikawa H, Tjuji S. Determining of Surface Properties by Projecting a Stripe Pattern. In: IEEE Proceedings of the International Conference on Pattern Recognition. Paris, France: 1986.86
[37] Asada M, Ichikawa H, Tjuji S. Determining Surface Orientation by Projecting a Stripe Pattern. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988, 10(5): 749~754
[38] Winkelbach S, Wahl F M. Shape from Single Stripe Pattern Illumination. Pattern Recognition, Lecture Notes in Computer Science, 2002, 2449:240~247
[39] Koninckx T P, Gool L Van. Efficient, Active 3D Acquisition, Based on a Pattern-Specific Snake. Pattern Recognition, Lecture Notes in Computer Science, 2002, 2449:557~565
[40] Zigelman G, Kimmel R, Kiryati N. Texure Mapping Using Surface Flattening via Multi-Dimensional Scaling. IEEE Transactions on Visualization and Computer Graphics, 2002, 8(2):198~207
[41]李东晖.数字激光散斑位移测量和跟踪技术研究:[博士学位论文].安徽:中国科学技术大学, 2006.
[42] Dainty J C. Topics in Applied Physics, Volume 9: Laser Speckle and Related Phenomena. Second Enlarged Edition. Berlin and New York: Springer-Verlag, 1975. 254~321.
[43] Mendlovic D, Zalevsky Z, Kiryuschev I, et al. Composite Harmonic Filters for Scale, Projection and Shift Invariant Pattern Recognition. Applied Optics, 1995, 34(2): 310~316
[44]刘曼华,朱秀丽.基于改进CAMShift的多目标自动跟踪.河南大学学报(自然科学版), 2010, 40(5):525~528