人眼跟踪与视线检测算法研究
|
摘要
计算机视觉作为一个极具挑战性的领域,一直以来都因为其巨大的发展潜力吸引了众多研究者的深入探索和研究。随着计算机性能的不断提高和电子产品的的普及,越来越多的研究人员致力于计算机视觉领域的重要研究方向人眼检测。利用人眼跟踪及人眼的视线检测,可以实现疲劳检测,残疾人交互,视觉游戏等应用。
本文主要研究内容可以分为人眼检测,人眼跟踪和视线检测三个部分,在对相关算法作深入研究的基础上,采用了适合视频图像的人眼检测和实用的视线跟踪方法,并在VC平台上建立了一个人眼视线跟踪系统。
首先,采用了Adaboost人脸检测算法进行人脸检测,并提出了解决平面和深度旋转的人脸的检测算法,能准确实时的检测人脸。本文在人脸检测的基础上,又设计了一些适合于眼睛的矩形特征,利用这些矩形特征训练了Adaboost层叠分类器,用于眼睛检测。由于瞳孔区域是人眼截图中比较稳定的图像信息,在睁眼的时候这部分的特征明显,因此本文采用先定位瞳孔,进而训练眨眼分类器进行眼睛的眨眼状态检测,同时,还精确的检测眼角点,为视线跟踪打下基础。
其次,眼睛的位置确定后可以在此基础上进行人眼跟踪,本文提出了将卡尔曼滤波器引入粒子滤波器中,利用卡尔曼滤波器算法进行采样预测和校正,减少了人眼跟踪中所需的粒子数目,从而达到快速而准确的跟踪目的。
再次,在视线跟踪系统中,该方法中以动点和参考点的差值来计算人眼的注视方向和位置。动点采用瞳孔中心,因为它可以准确反映眼球的变化。参考点采用眼角点,因为眼角点在人脸上是个非常稳定的点,人脸表情的变化基本上不会引起它的位置变化。该方法克服了过去以mark点或普尔钦斑点为参考点的缺点,不需要使用者在脸上做mark点,而且允许人脸在小范围内偏转。实验证明,该方法中自动定位眼角点快速准确,可以很好地解决视线跟踪系统中眼睛相对运动距离的问题。
本文开发的人眼跟踪及人机交互系统,实现了实时人眼检测跟踪和视线检测,并且利用检测到的有意识眨眼信息进行人机交互。本系统对硬件没有过高要求,对一定的光照变化,大幅度侧脸和俯仰变化都具有很好的鲁棒性。该系统的原理与目前主流的视线检测原理不同,具有低成本高效率的特点。
Computer vision as such a challenging field has enormous development potential. It has been attracting many researchers who explore and do intensive study on it. With the development of computer performance and popularity of electronic product, more and more researchers work at the human eye detection which is the important research interest in the fields of computer vision. Eye detection and eye tracking technologies can be used for fatigue detection, human-machine interaction for the disabled, visual game and so on.
The main research content of this paper is divided into three parts, including eye detection,eye tracking and gaze detection,An appropriate method of eye detection for video image and a practical gaze tracking method are used based on the intensive research of related algorithm. And a real-time eye tracking and gaze detection system is constructed in VC.
First, Adaboost face detection algorithm is used to detect face, at the same time, an effective paradigm to cope with big view angle and planar rotated face is proposed. Finally, the face and eyes are detected accurately and in real-time. On the basis of face detection, this paper finds some rectangular features that is suitable with eyes, and with these features the Adaboost cascade classifiers are trained for eye detection.Because the region of the pupil is relatively stable in the eye screenshot image and the pupil’s feature are obvious, so this paper locates the pupil firstly, then, eye blink classifier is trained to detect the blink status of the eyes, at the same time, eye corner is detected accuratly , this is the foundation for gaze tracking.
Second, we can track eyes after finding eyes, the paper brings a Kalman filter into the particle filter, which is used to predict and revise in the sampling stage. This method can reduce the number of particles needed in tracking, and realize the purpose that tracks eyes quickly and exactly.
Finally, this paper proposes a novel method for computing the eye-gaze direction and position in the eye-gaze tracking system. The eye corner is applied as the reference point instead of mark and Purkinje points in traditional ones. The difference of the moving point and the reference one is employed to compute the eye-gaze direction and position in the presented method. The center of iris is explored as the moving point due to that it can accurately reflect the moving state of eye. The eye corner is exploited as the reference point due to that it is the most stable point and relatively insensitive to facial expressions. The proposed method overcomes the shortcomings of conventional ones that adopt the mark and Purkinje as the reference points, and it does not need users to mark the marks on faces and allow the moderate variations of hear pose. Experimental results indicate that the method of the eye corner locating is fast and precise. In this way, the relative distance problem of eye motion arising in the eye-gaze tracking system can be well handled.
The eye tracking and human-machine interaction system of this paper achieves the real-time eye detection and gaze tracking. And with the information of intentional blink we achieve the human-machine interaction. The system has good robustness to a certain illumination change, light side face and head tilt, but it needn’t high-level hardware condition. The method of the system is different from current dominant principle. It has a quality with low cost and high efficiency.
本文主要研究内容可以分为人眼检测,人眼跟踪和视线检测三个部分,在对相关算法作深入研究的基础上,采用了适合视频图像的人眼检测和实用的视线跟踪方法,并在VC平台上建立了一个人眼视线跟踪系统。
首先,采用了Adaboost人脸检测算法进行人脸检测,并提出了解决平面和深度旋转的人脸的检测算法,能准确实时的检测人脸。本文在人脸检测的基础上,又设计了一些适合于眼睛的矩形特征,利用这些矩形特征训练了Adaboost层叠分类器,用于眼睛检测。由于瞳孔区域是人眼截图中比较稳定的图像信息,在睁眼的时候这部分的特征明显,因此本文采用先定位瞳孔,进而训练眨眼分类器进行眼睛的眨眼状态检测,同时,还精确的检测眼角点,为视线跟踪打下基础。
其次,眼睛的位置确定后可以在此基础上进行人眼跟踪,本文提出了将卡尔曼滤波器引入粒子滤波器中,利用卡尔曼滤波器算法进行采样预测和校正,减少了人眼跟踪中所需的粒子数目,从而达到快速而准确的跟踪目的。
再次,在视线跟踪系统中,该方法中以动点和参考点的差值来计算人眼的注视方向和位置。动点采用瞳孔中心,因为它可以准确反映眼球的变化。参考点采用眼角点,因为眼角点在人脸上是个非常稳定的点,人脸表情的变化基本上不会引起它的位置变化。该方法克服了过去以mark点或普尔钦斑点为参考点的缺点,不需要使用者在脸上做mark点,而且允许人脸在小范围内偏转。实验证明,该方法中自动定位眼角点快速准确,可以很好地解决视线跟踪系统中眼睛相对运动距离的问题。
本文开发的人眼跟踪及人机交互系统,实现了实时人眼检测跟踪和视线检测,并且利用检测到的有意识眨眼信息进行人机交互。本系统对硬件没有过高要求,对一定的光照变化,大幅度侧脸和俯仰变化都具有很好的鲁棒性。该系统的原理与目前主流的视线检测原理不同,具有低成本高效率的特点。
Computer vision as such a challenging field has enormous development potential. It has been attracting many researchers who explore and do intensive study on it. With the development of computer performance and popularity of electronic product, more and more researchers work at the human eye detection which is the important research interest in the fields of computer vision. Eye detection and eye tracking technologies can be used for fatigue detection, human-machine interaction for the disabled, visual game and so on.
The main research content of this paper is divided into three parts, including eye detection,eye tracking and gaze detection,An appropriate method of eye detection for video image and a practical gaze tracking method are used based on the intensive research of related algorithm. And a real-time eye tracking and gaze detection system is constructed in VC.
First, Adaboost face detection algorithm is used to detect face, at the same time, an effective paradigm to cope with big view angle and planar rotated face is proposed. Finally, the face and eyes are detected accurately and in real-time. On the basis of face detection, this paper finds some rectangular features that is suitable with eyes, and with these features the Adaboost cascade classifiers are trained for eye detection.Because the region of the pupil is relatively stable in the eye screenshot image and the pupil’s feature are obvious, so this paper locates the pupil firstly, then, eye blink classifier is trained to detect the blink status of the eyes, at the same time, eye corner is detected accuratly , this is the foundation for gaze tracking.
Second, we can track eyes after finding eyes, the paper brings a Kalman filter into the particle filter, which is used to predict and revise in the sampling stage. This method can reduce the number of particles needed in tracking, and realize the purpose that tracks eyes quickly and exactly.
Finally, this paper proposes a novel method for computing the eye-gaze direction and position in the eye-gaze tracking system. The eye corner is applied as the reference point instead of mark and Purkinje points in traditional ones. The difference of the moving point and the reference one is employed to compute the eye-gaze direction and position in the presented method. The center of iris is explored as the moving point due to that it can accurately reflect the moving state of eye. The eye corner is exploited as the reference point due to that it is the most stable point and relatively insensitive to facial expressions. The proposed method overcomes the shortcomings of conventional ones that adopt the mark and Purkinje as the reference points, and it does not need users to mark the marks on faces and allow the moderate variations of hear pose. Experimental results indicate that the method of the eye corner locating is fast and precise. In this way, the relative distance problem of eye motion arising in the eye-gaze tracking system can be well handled.
The eye tracking and human-machine interaction system of this paper achieves the real-time eye detection and gaze tracking. And with the information of intentional blink we achieve the human-machine interaction. The system has good robustness to a certain illumination change, light side face and head tilt, but it needn’t high-level hardware condition. The method of the system is different from current dominant principle. It has a quality with low cost and high efficiency.
引文
[1] Forsyth,D.A. and Ponce,J. Computer vision: A Modern Approach [M].Prentice Hall Professional Technical Reference,2002.
[2]刘伟等.人的视觉-眼动系统的研究[J].人类工效学,2000,6(4):41-44.
[3]时书锋,赵乐军,陈怀琛.眼动测量系统及其芯片设计[J].西安电子科技大学学报(自然科学版),2000,27(1)
[4] Rainer Stiefelhagen, Gaze Tracking for Multimodal Human-Computer Interaction[D].Universitat Karlsruhe,September,1996.
[5] A.H. Gee and R. Cipolla determining the Gaze of Faces in Images[R], CUED/FINFENG/TR 174,Department of Engineering, University of Cambridge,March,1994.
[6]舒梅,董秀成.基于肤色和模板匹配的人眼定位.[J].计算机工程与应用.2009.
[7]张杰,杨晓飞,赵瑞莲.基于Hough变换圆检测的人眼精确定位方法[J].计算机工程与应用. 2005(27): 43~44.
[8] Zhou Z. H, Geng X. Projection functions for eye detection[J]. Pattern Recognition. 2004, 37(5): 1049~1056.
[9] Rizon M, Kawaguchi T. Automatic eye detection using intensity and edge information. Proceedings [J]. IEEE TENCON. 2000, 2: 415~420
[10]徐艳,陈孝威.人脸检测中的眼睛定位算法研究[J].计算机与信息技术. 2006, 9: 24~28.
[11] Zhiwei Z,Qiang J. Eye and gaze tracking for interactive graphic display[J]. Machine Vision and Applications. 2004,15(3):139—148.。
[12] Xinghuas,Guoyong C,Chunxia Z et al. Gaze Estimation of Human Eye[C].6th International Conference on ITS Telecommunications Proceedings,2006:310-313.
[13] Kar-Han T,Kriegman D J,Ahuja N. Appearance based eye gaze estimation[J]. Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision.2002:191-195.
[14] Fujie S,Yamahata T,Kobayashi T. Conversation robot with the function of gaze recognition[C]. Proceedings of 2006 IEEE-RAS International Conference on Humanoid Robots,Humanoids,2006:364-369.
[15] M1yakawa M,Kosugi T.A three-dimensional gazing point detection system for communication analysis[C]. Proceedings of 25th Annual International Conference,EMBS Cancun Mexico,2003:918一921.
[16] J. Fukuda, E. Akutsu and K. Aoki,“An estimation of driver’s drowsiness level using keeping”[J]. JSAE, 1995(16): 185~199.
[17] H. Ueno, M. Kaneda and M. Tasukino,“Development of drowsiness detection system Conference”[C]. 1994, 8: 15~20.
[18] Kazuhiro Tskeshita, Akihiko Uchibori, Yoshiki Mizukami . A Communication system for ALS patients using eye blink International Journal of Applied Electromagnetics and Mechanics 2003[C], 18: 1~3
[19] Arne John Glenstrup and Theo Engell-Nielsen,“Eye Controlled Media: Present and Future State”, Thesis of Bachelor in Informaiton Psychology[C], Psychological Laboratory, University of Copenhagen, Denmark, 1995.
[20] Grauman k, Betke M, Gips J et al. Communication via eye blinks detection and duration analysis in real time[J]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, 1010~1017.
[21] OpenCV中文网站.人脸检测[EB/OL].[2008-12-1]. http://www.opencv.org.cn/index.php/%E9%A6%96%E9%A1%B5
[22] P. Viola and M. J. Jones, Robust real-time face detection[J]. International Journal of Computer Vision, vol. 57, pp. 137-154, May2004.
[23] PAUL V.Robust Real- Time Faee Deteetion [J]. International Journal of Computer Vision. 2004,57(2):137-154.
[24] Froba B,Ernst A. Fast Frontal- View Faee Detection Using a Multi-path Decision Tree[J]. In Proceedings of Audio and Video based Biometric Person Authentication,2003:921-928.
[25] Li SZ, Zhu L, Zhang ZQ, Zhang HJ. Learning to detect multi-view faces in real-time[Z].Washington DC: Proceeding of International Conference on Development and Learning,2002.
[26]张维.眼睛检测与视线跟踪[D].大连:大连理工大学信号与信息处理,2007.
[27]王海涛.基于灰度的人脸检测系统[M].北京:电子工业出版社,2004.
[28] Shen Lansun. Eye Location using gabor transform[J]. measurement and control technology, 2006, 25(6): 27~29.
[29]叶剑波.基于卡尔曼粒子滤波器的人眼跟踪[J].计算机工程. 2006
[30] Doucet A.On Sequential Simulation Monte Carlo Sampling Methods for Bayessian Filtering[J].Statics and Computing,2000,10(3):197-208.
[31] Chen H T,Liu T L.Trust-region Methods for Real-time Tracking[C]. Proc.of Int.Conf.Computer Vision,2001-07:717-722.
[32]孙卓.实时人眼跟踪与眨眼检测.[D].浙江:浙江大学计算机科学与技术学院,2008.28-32.
[33]陈白帆,蔡自兴.基于尺度空间理论的Harris角点检测[N].中南大学报(自然科学版).2005,36(5):pp.751-754.
[34] L.Kitchen and A.Rosenfeld.Gray level Corner detection[J]. Pattern Recognition Letters.1982::pp.95-102.
[35] K.Brunnstrom,T.Lindeberg,and J.O.Eklundh. Active detection and classification of Junctions[C]. Proc.2nd European Conf. Computer Vision LNCS 588. St. Margherita. Italy.1992:pp.701-709.
[36] R.Haralick.Corner detection in color images by multiscale combination of end-stopped cortical cells[C].Proc.IEEE Conf. Computer vision and pattern Recongnition.
[37] W.Fosterner. A feature based correspondence algorithm for image matching.Intl.Arch[J].Photogramn. Remote Sensing26 ,1986:pp.150-166.
[38]高飞.视线跟踪系统研究与设计[D].上海:上海交通大学电子信息学院,2007.
[39] Rein-Lien Hsu. Face Detection in Color images[C]. IEEE Transaction on pattern Analysis and Machine Intelligence.2002 ,34(5):pp.1033～1046.
[40] Milan Sonka,Vaclav Hlavac,Roger Boyle等.图像处理分析与机器视觉[M].北京:人民邮电出版社.2003.
[41]黄乔.视线跟踪技术与视线输入的人机系统研究[D].武汉:武汉理工大学检测技术与自动化装置专业,2008.
[42] Yoshio Matsumotoy,Alexander Zelinsky.An Algorithm for Real-time Stereo VisionImplementation of Head Pose and Gaze Direction Measurement[C] .Proceedings of Fourth International Conference on Automatic Face and Gesture Recognition,2000:pp.499-504.
[43]陈国勇.汽车安全驾驶中的人眼视线估计[D].南京:南京理工大学模式识别与智能控制,2006.
[2]刘伟等.人的视觉-眼动系统的研究[J].人类工效学,2000,6(4):41-44.
[3]时书锋,赵乐军,陈怀琛.眼动测量系统及其芯片设计[J].西安电子科技大学学报(自然科学版),2000,27(1)
[4] Rainer Stiefelhagen, Gaze Tracking for Multimodal Human-Computer Interaction[D].Universitat Karlsruhe,September,1996.
[5] A.H. Gee and R. Cipolla determining the Gaze of Faces in Images[R], CUED/FINFENG/TR 174,Department of Engineering, University of Cambridge,March,1994.
[6]舒梅,董秀成.基于肤色和模板匹配的人眼定位.[J].计算机工程与应用.2009.
[7]张杰,杨晓飞,赵瑞莲.基于Hough变换圆检测的人眼精确定位方法[J].计算机工程与应用. 2005(27): 43~44.
[8] Zhou Z. H, Geng X. Projection functions for eye detection[J]. Pattern Recognition. 2004, 37(5): 1049~1056.
[9] Rizon M, Kawaguchi T. Automatic eye detection using intensity and edge information. Proceedings [J]. IEEE TENCON. 2000, 2: 415~420
[10]徐艳,陈孝威.人脸检测中的眼睛定位算法研究[J].计算机与信息技术. 2006, 9: 24~28.
[11] Zhiwei Z,Qiang J. Eye and gaze tracking for interactive graphic display[J]. Machine Vision and Applications. 2004,15(3):139—148.。
[12] Xinghuas,Guoyong C,Chunxia Z et al. Gaze Estimation of Human Eye[C].6th International Conference on ITS Telecommunications Proceedings,2006:310-313.
[13] Kar-Han T,Kriegman D J,Ahuja N. Appearance based eye gaze estimation[J]. Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision.2002:191-195.
[14] Fujie S,Yamahata T,Kobayashi T. Conversation robot with the function of gaze recognition[C]. Proceedings of 2006 IEEE-RAS International Conference on Humanoid Robots,Humanoids,2006:364-369.
[15] M1yakawa M,Kosugi T.A three-dimensional gazing point detection system for communication analysis[C]. Proceedings of 25th Annual International Conference,EMBS Cancun Mexico,2003:918一921.
[16] J. Fukuda, E. Akutsu and K. Aoki,“An estimation of driver’s drowsiness level using keeping”[J]. JSAE, 1995(16): 185~199.
[17] H. Ueno, M. Kaneda and M. Tasukino,“Development of drowsiness detection system Conference”[C]. 1994, 8: 15~20.
[18] Kazuhiro Tskeshita, Akihiko Uchibori, Yoshiki Mizukami . A Communication system for ALS patients using eye blink International Journal of Applied Electromagnetics and Mechanics 2003[C], 18: 1~3
[19] Arne John Glenstrup and Theo Engell-Nielsen,“Eye Controlled Media: Present and Future State”, Thesis of Bachelor in Informaiton Psychology[C], Psychological Laboratory, University of Copenhagen, Denmark, 1995.
[20] Grauman k, Betke M, Gips J et al. Communication via eye blinks detection and duration analysis in real time[J]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, 1010~1017.
[21] OpenCV中文网站.人脸检测[EB/OL].[2008-12-1]. http://www.opencv.org.cn/index.php/%E9%A6%96%E9%A1%B5
[22] P. Viola and M. J. Jones, Robust real-time face detection[J]. International Journal of Computer Vision, vol. 57, pp. 137-154, May2004.
[23] PAUL V.Robust Real- Time Faee Deteetion [J]. International Journal of Computer Vision. 2004,57(2):137-154.
[24] Froba B,Ernst A. Fast Frontal- View Faee Detection Using a Multi-path Decision Tree[J]. In Proceedings of Audio and Video based Biometric Person Authentication,2003:921-928.
[25] Li SZ, Zhu L, Zhang ZQ, Zhang HJ. Learning to detect multi-view faces in real-time[Z].Washington DC: Proceeding of International Conference on Development and Learning,2002.
[26]张维.眼睛检测与视线跟踪[D].大连:大连理工大学信号与信息处理,2007.
[27]王海涛.基于灰度的人脸检测系统[M].北京:电子工业出版社,2004.
[28] Shen Lansun. Eye Location using gabor transform[J]. measurement and control technology, 2006, 25(6): 27~29.
[29]叶剑波.基于卡尔曼粒子滤波器的人眼跟踪[J].计算机工程. 2006
[30] Doucet A.On Sequential Simulation Monte Carlo Sampling Methods for Bayessian Filtering[J].Statics and Computing,2000,10(3):197-208.
[31] Chen H T,Liu T L.Trust-region Methods for Real-time Tracking[C]. Proc.of Int.Conf.Computer Vision,2001-07:717-722.
[32]孙卓.实时人眼跟踪与眨眼检测.[D].浙江:浙江大学计算机科学与技术学院,2008.28-32.
[33]陈白帆,蔡自兴.基于尺度空间理论的Harris角点检测[N].中南大学报(自然科学版).2005,36(5):pp.751-754.
[34] L.Kitchen and A.Rosenfeld.Gray level Corner detection[J]. Pattern Recognition Letters.1982::pp.95-102.
[35] K.Brunnstrom,T.Lindeberg,and J.O.Eklundh. Active detection and classification of Junctions[C]. Proc.2nd European Conf. Computer Vision LNCS 588. St. Margherita. Italy.1992:pp.701-709.
[36] R.Haralick.Corner detection in color images by multiscale combination of end-stopped cortical cells[C].Proc.IEEE Conf. Computer vision and pattern Recongnition.
[37] W.Fosterner. A feature based correspondence algorithm for image matching.Intl.Arch[J].Photogramn. Remote Sensing26 ,1986:pp.150-166.
[38]高飞.视线跟踪系统研究与设计[D].上海:上海交通大学电子信息学院,2007.
[39] Rein-Lien Hsu. Face Detection in Color images[C]. IEEE Transaction on pattern Analysis and Machine Intelligence.2002 ,34(5):pp.1033～1046.
[40] Milan Sonka,Vaclav Hlavac,Roger Boyle等.图像处理分析与机器视觉[M].北京:人民邮电出版社.2003.
[41]黄乔.视线跟踪技术与视线输入的人机系统研究[D].武汉:武汉理工大学检测技术与自动化装置专业,2008.
[42] Yoshio Matsumotoy,Alexander Zelinsky.An Algorithm for Real-time Stereo VisionImplementation of Head Pose and Gaze Direction Measurement[C] .Proceedings of Fourth International Conference on Automatic Face and Gesture Recognition,2000:pp.499-504.
[43]陈国勇.汽车安全驾驶中的人眼视线估计[D].南京:南京理工大学模式识别与智能控制,2006.