基于计算机视觉的静态手势识别系统
|
摘要
摄像机能够捕捉到用户通过内在自然方式(如眼神、表情、手势和动作等)所表达的信息,因此通过摄像机与计算机进行交互是一种高效自然的人机交互方式,它能使人机对话变得简单,减少人们使用计算机的障碍。特别是最近几年,随着计算机技术的迅猛发展,研究符合人机交流习惯的新颖人机交互技术变得异常活跃,也取得了可喜的进步。这些研究包括人脸识别、面部表情识别、唇读、头部运动跟踪、凝视跟踪、手势识别、以及体势识别等。
手势是一种自然、直观、易于学习的人机交互手段。手势识别按输入设备不同可以分为基于数据手套的手势识别和基于计算机视觉的手势识别。其中基于计算机视觉的手势识别以人手直接作为计算机的输入设备,人机之间的通讯将不再需要中间媒体,用户可以简单地定义一种适当的手势来对周围的机器进行控制。但是由于手势本身具有多样性、多义性以及时间和空间上的差异性等特点,加之人手是复杂变形体以及视觉本身的不适定性,基于视觉的手势识别是一个富有挑战性的、多学科交叉的研究课题。
本文设计实现了一个基于计算机视觉的静态手势识别系统,该系统能够实时地对从摄像头输入的10个常用静态手势进行识别。系统的设计准则一是实时性,二是准确性。在手势建模方面,采用基于表观的手势模型;在手势分析方面,经过手势图像预处理和特征参数提取得到八个手势特征参数;在手势识别方面,采用二次分类(粗分类和细分类)的方法进行识别。
整个系统分三个部分实现。手势图像预处理部分,根据人体的肤色特征从环境中分割出手区域,然后通过图像增强和拉普拉斯边缘提取算法得到手势轮廓;手势特征提取部分,提取了八个手势特征参数,组成特征向量;视频流实时处理部分,使用天敏SDK-2000图像视频采集卡,通过回调函数对摄像头输入的视频流进行计算,提取出单个静态手势图像,并进行实时地识别。
Camera can catch the information naturally expressed by people such as eye movements, expression, gesture and motion. For users, communicatingwith computers by camera is efficient and natural. It can provide a convenient addition to user-computer dialogues and reduce the obstacle to use computer. Recent years espacially, with the development of computer science, research on new human-computer interaction technology become extremely active, and advancement has been achieved. These research includes face recognition, expression recognition, hand gesture regognition, pose recognition and so on.
Hand gesture is a natural and straight human-computer interactionmethod. There are two methods on hand gesture recognition, recognition based on data glove and recogniton based on computer vision. Take hand as the input equipment directly, communication between human and computer will need no more other intermediate media. Users can control the machines around simply sign to it with the hand gesture user itself defines. However, gesture has the characters of multi-mode, multi-meaning and has discrepancy undercertain time and space situation; moreover, human handsare complicated transformed objects and there is visual instability, all of which make gesture recognition based on computer sight become a challengeable multi-subject researchgoal.
This paper realized a static hand gesture recognition system based on computer vision. This system can recognize 10 common static hand gestures inputted from camera at real time. It is a real-time system, so both recognition time and the correctrecognition rate have to be considerd while designing the system. In the aspect of hand gesture modeling, the system adopt hand gesture model basedon apparent; In the aspect of hand gesture analysis, the system picks up eightcharacters through image preprocess and character extraction; In the aspect of recognition, the system adopts two times classifation (rough classification and particular classification).
The system consists of three part. First, preprocession of the originalhand gesture image, in this part hand area is extracted from background through the character of human complexion, and then the system gets the edge through noise smoothing and Laplacian edge extraction; Second, extraxction of hand gesture characters, in this part the system extracts eight characters as an eigenvector; Third, the real-time procession to the Video Capture Card (天敏SDK-2000) , in this part the system processes the real-time data inputted from camera with a callback function and find the single image of static hand gesture.
手势是一种自然、直观、易于学习的人机交互手段。手势识别按输入设备不同可以分为基于数据手套的手势识别和基于计算机视觉的手势识别。其中基于计算机视觉的手势识别以人手直接作为计算机的输入设备,人机之间的通讯将不再需要中间媒体,用户可以简单地定义一种适当的手势来对周围的机器进行控制。但是由于手势本身具有多样性、多义性以及时间和空间上的差异性等特点,加之人手是复杂变形体以及视觉本身的不适定性,基于视觉的手势识别是一个富有挑战性的、多学科交叉的研究课题。
本文设计实现了一个基于计算机视觉的静态手势识别系统,该系统能够实时地对从摄像头输入的10个常用静态手势进行识别。系统的设计准则一是实时性,二是准确性。在手势建模方面,采用基于表观的手势模型;在手势分析方面,经过手势图像预处理和特征参数提取得到八个手势特征参数;在手势识别方面,采用二次分类(粗分类和细分类)的方法进行识别。
整个系统分三个部分实现。手势图像预处理部分,根据人体的肤色特征从环境中分割出手区域,然后通过图像增强和拉普拉斯边缘提取算法得到手势轮廓;手势特征提取部分,提取了八个手势特征参数,组成特征向量;视频流实时处理部分,使用天敏SDK-2000图像视频采集卡,通过回调函数对摄像头输入的视频流进行计算,提取出单个静态手势图像,并进行实时地识别。
Camera can catch the information naturally expressed by people such as eye movements, expression, gesture and motion. For users, communicatingwith computers by camera is efficient and natural. It can provide a convenient addition to user-computer dialogues and reduce the obstacle to use computer. Recent years espacially, with the development of computer science, research on new human-computer interaction technology become extremely active, and advancement has been achieved. These research includes face recognition, expression recognition, hand gesture regognition, pose recognition and so on.
Hand gesture is a natural and straight human-computer interactionmethod. There are two methods on hand gesture recognition, recognition based on data glove and recogniton based on computer vision. Take hand as the input equipment directly, communication between human and computer will need no more other intermediate media. Users can control the machines around simply sign to it with the hand gesture user itself defines. However, gesture has the characters of multi-mode, multi-meaning and has discrepancy undercertain time and space situation; moreover, human handsare complicated transformed objects and there is visual instability, all of which make gesture recognition based on computer sight become a challengeable multi-subject researchgoal.
This paper realized a static hand gesture recognition system based on computer vision. This system can recognize 10 common static hand gestures inputted from camera at real time. It is a real-time system, so both recognition time and the correctrecognition rate have to be considerd while designing the system. In the aspect of hand gesture modeling, the system adopt hand gesture model basedon apparent; In the aspect of hand gesture analysis, the system picks up eightcharacters through image preprocess and character extraction; In the aspect of recognition, the system adopts two times classifation (rough classification and particular classification).
The system consists of three part. First, preprocession of the originalhand gesture image, in this part hand area is extracted from background through the character of human complexion, and then the system gets the edge through noise smoothing and Laplacian edge extraction; Second, extraxction of hand gesture characters, in this part the system extracts eight characters as an eigenvector; Third, the real-time procession to the Video Capture Card (天敏SDK-2000) , in this part the system processes the real-time data inputted from camera with a callback function and find the single image of static hand gesture.
引文
1 M.Flemming,G.Cottrell.Categorization of faces using unsupervised feature extraction.International Joint Conference on Neural Networks,1990(2)
2 T.Kanade.Picture processing by computer complex and recognition of human faces.Technical Report,Kyoto University.Department of Information Science,1973
3 A.O'Toole,A,J.Mistlin,A.J.Chitty.A physical system approach to recognition memory for spatially.transformed faces.Neural Network,1988:179-199
4 T'.Takahashi and F.K.Shino.Hand gesture coding based on experiments using a hand gesture interface device.SIGCHI Bulletin,19.91,23(2):67-73
5 Davis and M.Shah,Visual gesture recognition,In IEEE Proceeding on Vision-Image Signal Processing,April 1994:321-332
6 Starner,T.and Pentland,A.Visual Recognition of American Sign Language Using Hidden Markov Models.Technical Report TR306,Media Lab,MIT,1995
7 Stamer,T.and Pentland,A.Real-time American Sign Language Recognition from Video Using Hidden Markov Models.Technical Report TR375,Media Lab,MIT,1996
8 Kirsti Grobel,Marcell Assam.Isolated sign language recognition using hidden Markov models.In Proceedings of the IEEE International Conference on Systems,Man and Cybernetics,Orlando,FL,1 997:162-167
9 Wen Gao.Enhanced user interface by using hand gesture recognition.Proceedings of IVYCS'95 workshop on software computing,Beijing 1995.
10 吴江琴,高文,陈熙霖.基于数据手套的汉语手指字母的识别.模式识别与人工智能,March,1999,12(1):74-78
11 Jiyong Ma,Wen Gao,Jiangqin Wu and Chunli Wang.A Continuous Chinese Sign Language Recognition System,IEEE.International Conference on Face and Gesture,March,FG'2000:428-433,28-31
12 Wen Gao,Jiyong Ma,Jiangqin Wu and Chunli Wang.Large Vocabulary Sign Language Recognition Based on HMM/ANN/DP.International Journal of Pattern Recognition and Artificial Intelligence,2000,14(5):587-602
13 Wen Gao,Jiyong Ma,Xilin Chen,Shiguan Shan,Wei Zeng,Jie Yan,Hongming Zhang,Jiangqin Wu,Feng Wu,Chunli Wang.HundTalker:A Multimodal Dialog System Using Sign Language and 3-D Virtual I-Iuman.The bird International Conference on Multimodal Interface.Lecture Notes in Computer Science,Beijing Oct.2000:564-571
14 任海兵,祝远新,徐光佑等.连续动态手势的时空表现建模及识别.计算机学报,2000,23(8):824-828.
15 胡友树.手势识别技术综述.中国科技信息.2005,1(2):41-42
16 张良国,吴江琴,高文等.基于Hausdorff距离的手势识别.中国图象图形学报.2002,7(7):1144-1150
17 任海兵,祝远新,徐光祜等.基于视觉手势识别的研究--综述.电子学报.2000,28(2):118-121
18 G.Bradski,Boon-Lock Yeo,Minerva M.Yeung.Gesture for video contentnavigation.SPIE 3656(Proc.of the IS&T/SPIE Conf.on Storage andRetrieval for Image and Video Database Ⅶ),San Jose,California,1999:230-242
19 J.J.Kuch,Vision-based hand modeling and tracking for virtualtelecomferencing and telecollaboration.Proc.IEEE Int'l Conf.ComputerVision,Cambridge,Mass.,1995
20 D.M.Gavrila,L.S.Davis.Towards 3D model-based tracking andrecognition of human movement:a multi-view approach.Proc.Int 'lWorkshop on Automatic Face and Gesture Recognition,Switzerland,1995:272-277
21 J.Lee,T.L.Kunii.Model-based analysis of hand posture.IEEE ComputerGraphics and Applications,Sept.1995:77-86
22 Trevor J.Darrell,Irfan A.Essa,Alex R Pentland.Task-specific gestureanalysis in real-time using interpolated views.IEEE Trans.PAMI,Dec.1996,18(12):1236-1242
23 A.Bobick,J.Davis.Real-time recognition of activity using temporaltemplates.Proc.of Third IEEE Workshop on applications of computer vision,Florida,1996:39-42
24 R.Cipolla,N.J.Hollinghurst.Human-robot interface by pointing withuncalibrated stereo vision.Image and vision computing,Mar.1996,14:171-178
25 Quek F.Unencumbered gestural interaction.IEEE Multimedia,1996:36-47
26 R.Culter,M.Turk.View-based interpretation of real-time optical flow forgesture recognition.Proc.of 3rd Int'l Conf.Automatic Face and GestureRecognition,Japan,1998
27 G.Xu,Y.Zhu,Y.Huang et al.Automatic visual recognition of isolated handgestures with computing spatio-temporal representations.Proc.Of the 1998Symp.on Image,Speech,Signal Processing and Robotics(IS2 SPR'98),1998,Hong Kong,I:49-54
28 T.Stamer,J.Weaver et al.Real-time american sign language recognitionusing desk and wearable computer based video.IEEE Trans.PMAI,1998,20(12):1371-1375
29 David Alan Becker,Sensi.A Real-Time Recognition,Feedback and TrainingSystem for T'ai Chi Gestures.MITMedia Lab,May,1997
30 Foley,J.D.,van Dam,A.Fundamentals of Interactive Computer Grap hics.Reading,MA:Addison-Wesley,1982
31 Gonzalez,R.C.,Woods,R.E.Digital Image Processing.3rd ed,Reading,MA:Addison-Wesley,1992
32 Levkowitz,H..Color Theory and Modeling for Computer Graphics,Visualization,and Multimedia Applications.Boston:Kluwer AcademicPublishers,1997
33 Ledley,S.,Buas,M.,Golab,T..Fundamentals of true-color imageprocessing.In:Proceedings of the 10th International Conference on PatternRecognition.1990:791-795
34 Bajon,J.,Cattoen,M.et al.Real-Time colorimetric transformations used inrobot vision.In:Proceedings of the MICAD.1985:76-86
35 陶霖密,彭振云,徐光祜.人体的肤色特征.软件学报.2001,12(7):1032-1041
36 Rafael C.Gonzalez,Richard E.Woods.数字图像处理.阮秋琦,阮宇智.第二版.电子工业出版社,2003:59-112
37 郭兴伟,葛元,王林泉.基于形状特征的字母手势的分类及识别算法.计算机工程.2004,30(18):130-132,186
38 刘肃亮,周明全,韦智勇.基于VFW的视频应用程序开发.西北大学学报.2003,12(6)
39 张星明.视频图像捕获及运动检测技术的实现.计算机工程.2002,28(8):130-132
401 刘袆玮.Visual C++视频/音频开发实用工程案例精选.人民邮电出版社,2004:11-33
41 郎锐.数字图像处理学Visual C++实现.北京希望电子出版社,2003:27-40
2 T.Kanade.Picture processing by computer complex and recognition of human faces.Technical Report,Kyoto University.Department of Information Science,1973
3 A.O'Toole,A,J.Mistlin,A.J.Chitty.A physical system approach to recognition memory for spatially.transformed faces.Neural Network,1988:179-199
4 T'.Takahashi and F.K.Shino.Hand gesture coding based on experiments using a hand gesture interface device.SIGCHI Bulletin,19.91,23(2):67-73
5 Davis and M.Shah,Visual gesture recognition,In IEEE Proceeding on Vision-Image Signal Processing,April 1994:321-332
6 Starner,T.and Pentland,A.Visual Recognition of American Sign Language Using Hidden Markov Models.Technical Report TR306,Media Lab,MIT,1995
7 Stamer,T.and Pentland,A.Real-time American Sign Language Recognition from Video Using Hidden Markov Models.Technical Report TR375,Media Lab,MIT,1996
8 Kirsti Grobel,Marcell Assam.Isolated sign language recognition using hidden Markov models.In Proceedings of the IEEE International Conference on Systems,Man and Cybernetics,Orlando,FL,1 997:162-167
9 Wen Gao.Enhanced user interface by using hand gesture recognition.Proceedings of IVYCS'95 workshop on software computing,Beijing 1995.
10 吴江琴,高文,陈熙霖.基于数据手套的汉语手指字母的识别.模式识别与人工智能,March,1999,12(1):74-78
11 Jiyong Ma,Wen Gao,Jiangqin Wu and Chunli Wang.A Continuous Chinese Sign Language Recognition System,IEEE.International Conference on Face and Gesture,March,FG'2000:428-433,28-31
12 Wen Gao,Jiyong Ma,Jiangqin Wu and Chunli Wang.Large Vocabulary Sign Language Recognition Based on HMM/ANN/DP.International Journal of Pattern Recognition and Artificial Intelligence,2000,14(5):587-602
13 Wen Gao,Jiyong Ma,Xilin Chen,Shiguan Shan,Wei Zeng,Jie Yan,Hongming Zhang,Jiangqin Wu,Feng Wu,Chunli Wang.HundTalker:A Multimodal Dialog System Using Sign Language and 3-D Virtual I-Iuman.The bird International Conference on Multimodal Interface.Lecture Notes in Computer Science,Beijing Oct.2000:564-571
14 任海兵,祝远新,徐光佑等.连续动态手势的时空表现建模及识别.计算机学报,2000,23(8):824-828.
15 胡友树.手势识别技术综述.中国科技信息.2005,1(2):41-42
16 张良国,吴江琴,高文等.基于Hausdorff距离的手势识别.中国图象图形学报.2002,7(7):1144-1150
17 任海兵,祝远新,徐光祜等.基于视觉手势识别的研究--综述.电子学报.2000,28(2):118-121
18 G.Bradski,Boon-Lock Yeo,Minerva M.Yeung.Gesture for video contentnavigation.SPIE 3656(Proc.of the IS&T/SPIE Conf.on Storage andRetrieval for Image and Video Database Ⅶ),San Jose,California,1999:230-242
19 J.J.Kuch,Vision-based hand modeling and tracking for virtualtelecomferencing and telecollaboration.Proc.IEEE Int'l Conf.ComputerVision,Cambridge,Mass.,1995
20 D.M.Gavrila,L.S.Davis.Towards 3D model-based tracking andrecognition of human movement:a multi-view approach.Proc.Int 'lWorkshop on Automatic Face and Gesture Recognition,Switzerland,1995:272-277
21 J.Lee,T.L.Kunii.Model-based analysis of hand posture.IEEE ComputerGraphics and Applications,Sept.1995:77-86
22 Trevor J.Darrell,Irfan A.Essa,Alex R Pentland.Task-specific gestureanalysis in real-time using interpolated views.IEEE Trans.PAMI,Dec.1996,18(12):1236-1242
23 A.Bobick,J.Davis.Real-time recognition of activity using temporaltemplates.Proc.of Third IEEE Workshop on applications of computer vision,Florida,1996:39-42
24 R.Cipolla,N.J.Hollinghurst.Human-robot interface by pointing withuncalibrated stereo vision.Image and vision computing,Mar.1996,14:171-178
25 Quek F.Unencumbered gestural interaction.IEEE Multimedia,1996:36-47
26 R.Culter,M.Turk.View-based interpretation of real-time optical flow forgesture recognition.Proc.of 3rd Int'l Conf.Automatic Face and GestureRecognition,Japan,1998
27 G.Xu,Y.Zhu,Y.Huang et al.Automatic visual recognition of isolated handgestures with computing spatio-temporal representations.Proc.Of the 1998Symp.on Image,Speech,Signal Processing and Robotics(IS2 SPR'98),1998,Hong Kong,I:49-54
28 T.Stamer,J.Weaver et al.Real-time american sign language recognitionusing desk and wearable computer based video.IEEE Trans.PMAI,1998,20(12):1371-1375
29 David Alan Becker,Sensi.A Real-Time Recognition,Feedback and TrainingSystem for T'ai Chi Gestures.MITMedia Lab,May,1997
30 Foley,J.D.,van Dam,A.Fundamentals of Interactive Computer Grap hics.Reading,MA:Addison-Wesley,1982
31 Gonzalez,R.C.,Woods,R.E.Digital Image Processing.3rd ed,Reading,MA:Addison-Wesley,1992
32 Levkowitz,H..Color Theory and Modeling for Computer Graphics,Visualization,and Multimedia Applications.Boston:Kluwer AcademicPublishers,1997
33 Ledley,S.,Buas,M.,Golab,T..Fundamentals of true-color imageprocessing.In:Proceedings of the 10th International Conference on PatternRecognition.1990:791-795
34 Bajon,J.,Cattoen,M.et al.Real-Time colorimetric transformations used inrobot vision.In:Proceedings of the MICAD.1985:76-86
35 陶霖密,彭振云,徐光祜.人体的肤色特征.软件学报.2001,12(7):1032-1041
36 Rafael C.Gonzalez,Richard E.Woods.数字图像处理.阮秋琦,阮宇智.第二版.电子工业出版社,2003:59-112
37 郭兴伟,葛元,王林泉.基于形状特征的字母手势的分类及识别算法.计算机工程.2004,30(18):130-132,186
38 刘肃亮,周明全,韦智勇.基于VFW的视频应用程序开发.西北大学学报.2003,12(6)
39 张星明.视频图像捕获及运动检测技术的实现.计算机工程.2002,28(8):130-132
401 刘袆玮.Visual C++视频/音频开发实用工程案例精选.人民邮电出版社,2004:11-33
41 郎锐.数字图像处理学Visual C++实现.北京希望电子出版社,2003:27-40