基于双目视觉的乒乓球识别与跟踪问题研究
|
摘要
本文针对仿人机器人打乒乓球的任务,设计并实现了基于双目视觉的机器人外部视觉系统和本体视觉系统对快速运动的乒乓球目标进行识别、定位和跟踪,实时地为机器人提供乒乓球在三维空间中准确的位置信息,为实现仿人机器人打乒乓球奠定了基础。
本文首先设计了基于双目视觉的仿人机器人视觉系统的硬件和软件结构,以及同步控制方式,对视觉系统的基线距和时间同步误差对测量精度的影响进行了分析,并对乒乓球目标的准确、实时的识别进行研究。利用颜色查找表分类方法对图像进行颜色分类,设计了快速的区域搜索算法在颜色分类结果的基础上获取乒乓球的目标区域,利用Canny边缘检测方法在目标区域中提取边缘信息,然后设计快速CHT算法准确计算出乒乓球目标中心的图像坐标。
然后,对乒乓球目标的定位和跟踪问题进行研究。根据摄像机的几何模型和其内、外参数实现对乒乓球目标的定位。并运用了流体力学的方法对乒乓球飞行过程所受阻力进行了分析,在此基础之上建立乒乓球的运动模型,利用卡尔曼滤波方法实现了对乒乓球目标的跟踪,以获得图像中的感兴趣区域(ROI),增强视觉系统对乒乓球识别的实时性和鲁棒性。
最后,提出了基于特征区域角点的摄像机外参实时标定方法,使得本体视觉系统能够在线精确地获得其摄像机的外部参数,实现了在其跟随仿人机器人运动时对乒乓球目标位置信息实时、准确的获取。
This dissertation designs and realizes an external vision system and an on-board vision system based on binocular vision to recognize, locate and track the fast moving ping-pong aiming at the task of playing ping-pong by the humanoid robot. The precise three dimensional localization information of the ping-pong is provided to the humanoid robot by the vision systems in real-time, which lays the foundation for successfully playing ping-pong by the robot.
Firstly, the dissertation designs the framework and the synchronization method of the hardware and software of the vision systems of the humanoid robot based on binocular vision. And the effects caused by the distance of the baseline and the time error of the synchronization are analyzed. The precise and real-time recognition of the ping-pong is also studied in this thesis. The color look-up table method is used to classify the color image. Then a fast area searching algorithm is designed to obtain the goal area of the ping-pong based on the color classification result. In the goal area, Canny edge detector is used to detect the edge information, and then a fast CHT algorithm is designed to calculate the image coordinate of the center of the ping-pong accurately.
Secondly, the localization and tracking problems are also studied in this dissertation. The localization of the ping-pong is realized after the geometric camera model and the camera intrinsic and extrinsic parameters are known. And the air resistance to the ping-pong during the flying of it is analyzed using the tools of hydromechanics. Then the motion model of the ping-pong is founded to apply Kalman Filter to track the ping-pong. The region of interest (ROI) can be obtained with the tracking of ping-pong, which improves the real-time and robust performance of vision systems in recognizing the ping-pong.
Finally, the dissertation proposed a real-time calibration method for camera extrinsic parameters based on corners of characteristic areas, which qualifies the on-board vision system to obtain the camera extrinsic parameters of it on-line in high precision. Then the precise three dimensional localization information of the ping-pong is obtained by the on-board vision system in real-time even when the vision system is moving with the humanoid robot.
本文首先设计了基于双目视觉的仿人机器人视觉系统的硬件和软件结构,以及同步控制方式,对视觉系统的基线距和时间同步误差对测量精度的影响进行了分析,并对乒乓球目标的准确、实时的识别进行研究。利用颜色查找表分类方法对图像进行颜色分类,设计了快速的区域搜索算法在颜色分类结果的基础上获取乒乓球的目标区域,利用Canny边缘检测方法在目标区域中提取边缘信息,然后设计快速CHT算法准确计算出乒乓球目标中心的图像坐标。
然后,对乒乓球目标的定位和跟踪问题进行研究。根据摄像机的几何模型和其内、外参数实现对乒乓球目标的定位。并运用了流体力学的方法对乒乓球飞行过程所受阻力进行了分析,在此基础之上建立乒乓球的运动模型,利用卡尔曼滤波方法实现了对乒乓球目标的跟踪,以获得图像中的感兴趣区域(ROI),增强视觉系统对乒乓球识别的实时性和鲁棒性。
最后,提出了基于特征区域角点的摄像机外参实时标定方法,使得本体视觉系统能够在线精确地获得其摄像机的外部参数,实现了在其跟随仿人机器人运动时对乒乓球目标位置信息实时、准确的获取。
This dissertation designs and realizes an external vision system and an on-board vision system based on binocular vision to recognize, locate and track the fast moving ping-pong aiming at the task of playing ping-pong by the humanoid robot. The precise three dimensional localization information of the ping-pong is provided to the humanoid robot by the vision systems in real-time, which lays the foundation for successfully playing ping-pong by the robot.
Firstly, the dissertation designs the framework and the synchronization method of the hardware and software of the vision systems of the humanoid robot based on binocular vision. And the effects caused by the distance of the baseline and the time error of the synchronization are analyzed. The precise and real-time recognition of the ping-pong is also studied in this thesis. The color look-up table method is used to classify the color image. Then a fast area searching algorithm is designed to obtain the goal area of the ping-pong based on the color classification result. In the goal area, Canny edge detector is used to detect the edge information, and then a fast CHT algorithm is designed to calculate the image coordinate of the center of the ping-pong accurately.
Secondly, the localization and tracking problems are also studied in this dissertation. The localization of the ping-pong is realized after the geometric camera model and the camera intrinsic and extrinsic parameters are known. And the air resistance to the ping-pong during the flying of it is analyzed using the tools of hydromechanics. Then the motion model of the ping-pong is founded to apply Kalman Filter to track the ping-pong. The region of interest (ROI) can be obtained with the tracking of ping-pong, which improves the real-time and robust performance of vision systems in recognizing the ping-pong.
Finally, the dissertation proposed a real-time calibration method for camera extrinsic parameters based on corners of characteristic areas, which qualifies the on-board vision system to obtain the camera extrinsic parameters of it on-line in high precision. Then the precise three dimensional localization information of the ping-pong is obtained by the on-board vision system in real-time even when the vision system is moving with the humanoid robot.
引文
[1] Zhengtao Zhang, De Xu, Junzhi Yu. Research and Latest Development of Ping-Pong Robot Player. Proceedings of the 7th World Congress on Intelligent Control and Automation. Jun. 2008: 4881-4886.
[2] R. L. Andersson. A low latency 60 Hz Stereo Vision System for real-time visual control. Proceedings of the 5th IEEE International Symposium on Intelligent Control, Sept. 1990, 1: 165-170.
[3] R. L. Andersson. Understanding and Applying A Robot Ping-pong Player’s Expert Controller. Proceedings of IEEE International Conference on Robotics and Automation, 1989, 3: 1284-1289.
[4] M. Takeuchi, F. Miyazaki, M. Matsushima, et al. Dynamic Dexterity for the Performance of“Wall-Bouncing”Tasks. Proceedings of IEEE International Conference on Robotics and Automation, 2002, 2: 1559-1564.
[5] F. Miyazaki, M. Takeuchi, M. Matsushima, et al. Realization of the Table Tennis Task Based on Virtual Targets. Proceedings of IEEE International Conference on Robotics and Automation, 2002, 4: 3844-3849.
[6] M. Matsushima, T. Hashimoto, M. Takeuchi, et al. A Learning Approach to Robotic Table Tennis. IEEE Transactions on Robotics, Aug. 2005, 21(4): 767-771.
[7] L. Acosta, J.J. Rodrigo, J.A. Mendez, et al. Ping-pong player prototype. IEEE Robotics & Automation Magazine, Dec. 2003, 10(4): 44-52.
[8]杜森森.基于PC的乒乓球机器人的机器视觉的程序设计[硕士学位论文].浙江:浙江大学, 2006.
[9]洪永潮.基于PC的七自由度乒乓球机器人伺服控制系统的研究[硕士学位论文].浙江:浙江大学, 2006.
[10]张正涛,徐德.基于智能摄像机的高速视觉系统及其目标跟踪算法研究.机器人, May, 2009, 31(3): 229-234.
[11] Huimin Lu, Hui Zhang, Shaowu Yang, et al. Vision-based Ball Recognition for Soccer Robots without Color Classification. Proceedings of the 2009 IEEE International Conference on Information and Automation, June, 2009: 916-921.
[12]刘斐,卢惠民,郑志强.基于线性分类器的混合空间查找表颜色分类方法,中国图象图形学报, 2008, 13(1): 104-108.
[13] J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern analysis and Machine Intelligence, 1986, 8 (6): 679-698.
[14] R.O. Duda and P.E. Hart. Use of the Hough Transform to Detect Lines and Curver in Pictures. Communications of the ACM, Jan. 1972, 15(1): 11-15.
[15] C. Kimme, D. Ballard, and J. Sklansky. Finding circles by an array of accumulators. Communications of the ACM, Feb. 1975, 18(2): 120-122.
[16] L. Minor and J. Sklansky. Detection and segmentation of blobs in infrared images. IEEE Transactions on Systems, Man and Cybernetics, Mar. 1981, 11(3): 194-201.
[17] D. Scaramuzza, S. Pagnottelli and P. Valigi. Ball Detection and Predictive Ball Following Based on a Stereoscopic Vision System. Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Apr. 2005: 1561-1566.
[18] T. Atherton and D. Kerbyson. Size invariant circle detection. Image and Vision Computing, Sept. 1999, 17(11): 795-803.
[19] T. D’Orazio, N Ancona, G. Cicirelli, et al. A Ball Detection Algorithm for Real Soccer Image Sequences. Proceedings of the 16th International Conference on Pattern Recognition, Aug. 2002: 210-213.
[20]韩崇昭,朱洪艳,段战胜等著.多源信息融合.清华大学出版社, 2006.
[21]侯志强,韩崇昭.视觉跟踪技术综述.自动化学报, 2006, 32(4): 603-617.
[22] Dawei Liang, Yang Liu, Qingming, et al. A Scheme for Ball Detection and Tracking in Broadcast Soccer Video. PCM 2005, LNCS 3767, Springer-Verlag, Berlin/Heidelberg, 2005: 864-875.
[23] M. Isard and A. Blake. Condensation-Conditional Density Propagation for Visual Tracking. International Journal of Computer Vision, 1998, 29(1): 5-28.
[24] Xiao-Feng Tong, Han-Qing Lu, Qing-Shan Liu. An effective and Fast Soccer Ball Detection and Tracking Method. Proceedings of the International Conference on Pattern Recognition, 2004, 4: 795-798.
[25]邱茂林,马颂德,李毅.计算机视觉中摄像机定标综述.自动化学报, 2000, 26(1): 43-55.
[26] R.Y. Tsai. A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology Using Off-the-shelf TV Cameras and Lenses. IEEE Journal of Robotics and Automation, 1987, RA-3(4): 323-344.
[27] O.D. Faugeras,S. Maybank. Motion From Point Matches: Multiplicity of Solutions. International Journal of Computer Vision, 1990, 4: 225-246.
[28] Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Transaction on Pattern Analysis and Machine Intelligence, Nov. 2000, 22(11): 1330-1334.
[29] Abdel Aziz Y I, Karara H M. Direct Linear Transformation Into Object Space Coordinates in Close Range Photogrammetry. Proceedings of Close Range Photogrammetry, 1971: 1-18.
[30] W. Faig. Calibration of Close Range Photogrammetric Systems: Mathematical Formulation. Photogrammetric Engineering, 1975, 41(12): 1479-1486.
[31] B.K.P. Horn. Closed-form Solution of Absolute Orientation Using Unit Quaternion. Journal of the Optical Society of America A, Apr. 1987, 4(4): 629-642.
[32] R. Mukundan , K.R. Ramakrishnan. A Quaternion Solution to the Pose Determination Problem for Rendezvous and Docking Simulations. Mathematics and Computers in Simulation, 1995, 39: 143-153.
[33] N. K. Philip, M. R. Ananthasayanam. Relative Position and Attitude Estimation and Control Schemes for the Final Phase of An Autonomous Docking Mission of Spacecraft. Acta Astronautica, 2003, 52: 511-522.
[34]张世杰,曹喜滨,陈雪芹.航天器相对位姿参数光学测量解析算法.航空学报, Mar. 2005, 26(2): 214-218.
[35]张广军.机器视觉.北京:科学出版社,2005.
[36]侯捷. Win32多线程设计.北京:科学出版社,2005.
[37]卢惠民.机器人全向视觉系统自定位方法研究[硕士学位论文].湖南:国防科技大学, 2005.
[38]刘斐.应用于足球机器人的彩色全向视觉关键技术研究[博士学位论文].湖南:国防科技大学, 2007.
[39]刘伟. RoboCup中型组机器人全景视觉系统设计与实现[硕士学位论文].湖南:国防科技大学, 2004.
[40] Milan Sonka, Vaclav Hlavac, Roger Boyle.艾海舟,武勃等译.图像处理、分析与机器视觉.北京:人民邮电出版社,2003.
[41] D.C. Brown. Close-Range Camera Calibration. Photogrammetric Engineering, 1971, 37(8): 855-866.
[42] G. Wei and S. Ma. Implicit and Explicit Camera Calibration: Theory and Experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994, 16(5): 469-480.
[43] J. Heikkil? and O. Silvén. A Four-step Camera Calibration Procedure with Implicit Image Correction. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1997: 1106-1112.
[44] J.Y. Bouguet. Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/, Jul. 2008.
[45]周江华,苗育红,王明海.姿态运动的Rodrigues参数描述.宇航学报, 2004, 25(5): 514-519.
[46] David A. Forsyth, Jean Ponce.林学訚.王宏等译.计算机视觉:一种现代方法.北京:电子工业出版社,2004.
[47]秦永元,张洪钺,汪叔华.卡尔曼滤波与组合导航原理.西安:西北工业大学出版社,1998.
[48]郑永令,贾起民,方小敏.力学(第二版).北京:高等教育出版社,2002.
[49]贾月梅,赵秋霞,赵广慧.流体力学.北京:国防工业出版社,2006.
[50] S.M. Smith and J.M. Brady. SUSAN: A New Approach to Low Level Image Processing. International Journal of Computer Vision, May 1997, 23(1):45-78.
[51] R.I. Hartley. In Defense of the Eight-Point Algorithm. IEEE Transaction on Pattern Analysis and Machine Intelligence. Jun. 1997, 19(6): 580-593.
[2] R. L. Andersson. A low latency 60 Hz Stereo Vision System for real-time visual control. Proceedings of the 5th IEEE International Symposium on Intelligent Control, Sept. 1990, 1: 165-170.
[3] R. L. Andersson. Understanding and Applying A Robot Ping-pong Player’s Expert Controller. Proceedings of IEEE International Conference on Robotics and Automation, 1989, 3: 1284-1289.
[4] M. Takeuchi, F. Miyazaki, M. Matsushima, et al. Dynamic Dexterity for the Performance of“Wall-Bouncing”Tasks. Proceedings of IEEE International Conference on Robotics and Automation, 2002, 2: 1559-1564.
[5] F. Miyazaki, M. Takeuchi, M. Matsushima, et al. Realization of the Table Tennis Task Based on Virtual Targets. Proceedings of IEEE International Conference on Robotics and Automation, 2002, 4: 3844-3849.
[6] M. Matsushima, T. Hashimoto, M. Takeuchi, et al. A Learning Approach to Robotic Table Tennis. IEEE Transactions on Robotics, Aug. 2005, 21(4): 767-771.
[7] L. Acosta, J.J. Rodrigo, J.A. Mendez, et al. Ping-pong player prototype. IEEE Robotics & Automation Magazine, Dec. 2003, 10(4): 44-52.
[8]杜森森.基于PC的乒乓球机器人的机器视觉的程序设计[硕士学位论文].浙江:浙江大学, 2006.
[9]洪永潮.基于PC的七自由度乒乓球机器人伺服控制系统的研究[硕士学位论文].浙江:浙江大学, 2006.
[10]张正涛,徐德.基于智能摄像机的高速视觉系统及其目标跟踪算法研究.机器人, May, 2009, 31(3): 229-234.
[11] Huimin Lu, Hui Zhang, Shaowu Yang, et al. Vision-based Ball Recognition for Soccer Robots without Color Classification. Proceedings of the 2009 IEEE International Conference on Information and Automation, June, 2009: 916-921.
[12]刘斐,卢惠民,郑志强.基于线性分类器的混合空间查找表颜色分类方法,中国图象图形学报, 2008, 13(1): 104-108.
[13] J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern analysis and Machine Intelligence, 1986, 8 (6): 679-698.
[14] R.O. Duda and P.E. Hart. Use of the Hough Transform to Detect Lines and Curver in Pictures. Communications of the ACM, Jan. 1972, 15(1): 11-15.
[15] C. Kimme, D. Ballard, and J. Sklansky. Finding circles by an array of accumulators. Communications of the ACM, Feb. 1975, 18(2): 120-122.
[16] L. Minor and J. Sklansky. Detection and segmentation of blobs in infrared images. IEEE Transactions on Systems, Man and Cybernetics, Mar. 1981, 11(3): 194-201.
[17] D. Scaramuzza, S. Pagnottelli and P. Valigi. Ball Detection and Predictive Ball Following Based on a Stereoscopic Vision System. Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Apr. 2005: 1561-1566.
[18] T. Atherton and D. Kerbyson. Size invariant circle detection. Image and Vision Computing, Sept. 1999, 17(11): 795-803.
[19] T. D’Orazio, N Ancona, G. Cicirelli, et al. A Ball Detection Algorithm for Real Soccer Image Sequences. Proceedings of the 16th International Conference on Pattern Recognition, Aug. 2002: 210-213.
[20]韩崇昭,朱洪艳,段战胜等著.多源信息融合.清华大学出版社, 2006.
[21]侯志强,韩崇昭.视觉跟踪技术综述.自动化学报, 2006, 32(4): 603-617.
[22] Dawei Liang, Yang Liu, Qingming, et al. A Scheme for Ball Detection and Tracking in Broadcast Soccer Video. PCM 2005, LNCS 3767, Springer-Verlag, Berlin/Heidelberg, 2005: 864-875.
[23] M. Isard and A. Blake. Condensation-Conditional Density Propagation for Visual Tracking. International Journal of Computer Vision, 1998, 29(1): 5-28.
[24] Xiao-Feng Tong, Han-Qing Lu, Qing-Shan Liu. An effective and Fast Soccer Ball Detection and Tracking Method. Proceedings of the International Conference on Pattern Recognition, 2004, 4: 795-798.
[25]邱茂林,马颂德,李毅.计算机视觉中摄像机定标综述.自动化学报, 2000, 26(1): 43-55.
[26] R.Y. Tsai. A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology Using Off-the-shelf TV Cameras and Lenses. IEEE Journal of Robotics and Automation, 1987, RA-3(4): 323-344.
[27] O.D. Faugeras,S. Maybank. Motion From Point Matches: Multiplicity of Solutions. International Journal of Computer Vision, 1990, 4: 225-246.
[28] Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Transaction on Pattern Analysis and Machine Intelligence, Nov. 2000, 22(11): 1330-1334.
[29] Abdel Aziz Y I, Karara H M. Direct Linear Transformation Into Object Space Coordinates in Close Range Photogrammetry. Proceedings of Close Range Photogrammetry, 1971: 1-18.
[30] W. Faig. Calibration of Close Range Photogrammetric Systems: Mathematical Formulation. Photogrammetric Engineering, 1975, 41(12): 1479-1486.
[31] B.K.P. Horn. Closed-form Solution of Absolute Orientation Using Unit Quaternion. Journal of the Optical Society of America A, Apr. 1987, 4(4): 629-642.
[32] R. Mukundan , K.R. Ramakrishnan. A Quaternion Solution to the Pose Determination Problem for Rendezvous and Docking Simulations. Mathematics and Computers in Simulation, 1995, 39: 143-153.
[33] N. K. Philip, M. R. Ananthasayanam. Relative Position and Attitude Estimation and Control Schemes for the Final Phase of An Autonomous Docking Mission of Spacecraft. Acta Astronautica, 2003, 52: 511-522.
[34]张世杰,曹喜滨,陈雪芹.航天器相对位姿参数光学测量解析算法.航空学报, Mar. 2005, 26(2): 214-218.
[35]张广军.机器视觉.北京:科学出版社,2005.
[36]侯捷. Win32多线程设计.北京:科学出版社,2005.
[37]卢惠民.机器人全向视觉系统自定位方法研究[硕士学位论文].湖南:国防科技大学, 2005.
[38]刘斐.应用于足球机器人的彩色全向视觉关键技术研究[博士学位论文].湖南:国防科技大学, 2007.
[39]刘伟. RoboCup中型组机器人全景视觉系统设计与实现[硕士学位论文].湖南:国防科技大学, 2004.
[40] Milan Sonka, Vaclav Hlavac, Roger Boyle.艾海舟,武勃等译.图像处理、分析与机器视觉.北京:人民邮电出版社,2003.
[41] D.C. Brown. Close-Range Camera Calibration. Photogrammetric Engineering, 1971, 37(8): 855-866.
[42] G. Wei and S. Ma. Implicit and Explicit Camera Calibration: Theory and Experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994, 16(5): 469-480.
[43] J. Heikkil? and O. Silvén. A Four-step Camera Calibration Procedure with Implicit Image Correction. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1997: 1106-1112.
[44] J.Y. Bouguet. Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/, Jul. 2008.
[45]周江华,苗育红,王明海.姿态运动的Rodrigues参数描述.宇航学报, 2004, 25(5): 514-519.
[46] David A. Forsyth, Jean Ponce.林学訚.王宏等译.计算机视觉:一种现代方法.北京:电子工业出版社,2004.
[47]秦永元,张洪钺,汪叔华.卡尔曼滤波与组合导航原理.西安:西北工业大学出版社,1998.
[48]郑永令,贾起民,方小敏.力学(第二版).北京:高等教育出版社,2002.
[49]贾月梅,赵秋霞,赵广慧.流体力学.北京:国防工业出版社,2006.
[50] S.M. Smith and J.M. Brady. SUSAN: A New Approach to Low Level Image Processing. International Journal of Computer Vision, May 1997, 23(1):45-78.
[51] R.I. Hartley. In Defense of the Eight-Point Algorithm. IEEE Transaction on Pattern Analysis and Machine Intelligence. Jun. 1997, 19(6): 580-593.