吊装机器人肢体动作指令识别技术研究
|
摘要
鉴于Kinect相机进行肢体识别监控距离有限,提出使用网络大变焦摄像头、构建CNN-BP融合网络进行肢体动作识别,并以9组机器人吊装指令为例进行训练和识别。首先,基于OpenPose提取18个骨架节点坐标,生成RGB骨架图和骨架向量;然后,采用迁移学习方法对RGB骨架图使用InceptionV3网络提取图像深层抽象特征,并对训练数据集采用旋转、平移、缩放和仿射多种数据增强方式,以扩充训练数据,防止过拟合;再将提取的骨架向量使用BP神经网络提取点线面等浅层特征;最后对InceptionV3网络和BP神经网络输出进行融合,并使用Softmax求解器得到肢体识别结果。将肢体识别结果输入机器人辅助吊装控制系统,建立双重验证控制方法,完成机器人辅助吊装操作。实验结果表明,该方法保证了模型运行的精度和时效性,实时识别精度达0. 99以上,大大提升了远距离人机交互能力。
In view of the limited monitoring distance of Kinect for limb recognition,the large zoom network camera was used and CNN-BP fusion network for human behavior recognition was constructed,and the nine groups of robot lifting instructions were trained and identified. Firstly,totally 18 skeleton nodes were extracted based on OpenPose to generate RGB skeleton map and skeleton vector. Then,using the migration learning method,the InceptionV3 network was used to extract the deep abstract features of the image,and the training data set was rotated,translated,scaled and affine. A variety of data enhancement methods were used to extend the training data to prevent overfitting; and then the extracted skeleton vector was extracted from the shallow layer features such as the point line surface using BP neural network; the InceptionV3 network and the BP neural network output were merged and obtained by using the Softmax solver to obtain limb classification results. Finally,the result of limb recognition was input into the robot auxiliary hoisting control system,and the double verification control mode was established to complete the robot auxiliary hoisting operation. The test results showed that the method ensured the timeliness of the model operation,and the real-time recognition accuracy reached 0. 99,which greatly improved the long-distance human-computer interaction capability.
In view of the limited monitoring distance of Kinect for limb recognition,the large zoom network camera was used and CNN-BP fusion network for human behavior recognition was constructed,and the nine groups of robot lifting instructions were trained and identified. Firstly,totally 18 skeleton nodes were extracted based on OpenPose to generate RGB skeleton map and skeleton vector. Then,using the migration learning method,the InceptionV3 network was used to extract the deep abstract features of the image,and the training data set was rotated,translated,scaled and affine. A variety of data enhancement methods were used to extend the training data to prevent overfitting; and then the extracted skeleton vector was extracted from the shallow layer features such as the point line surface using BP neural network; the InceptionV3 network and the BP neural network output were merged and obtained by using the Softmax solver to obtain limb classification results. Finally,the result of limb recognition was input into the robot auxiliary hoisting control system,and the double verification control mode was established to complete the robot auxiliary hoisting operation. The test results showed that the method ensured the timeliness of the model operation,and the real-time recognition accuracy reached 0. 99,which greatly improved the long-distance human-computer interaction capability.
引文
[1] PATRICK R,WOHLFROMM L,TRACHT K. Implementation of virtual reality systems for simulation of human-robot collaboration[J]. Procedia Manufacturing,2018,19:164-170.
[2] GHABRI S,OUARDA W,ALIMI A M. Towards human behavior recognition based on spatio temporal features and support vector machines[C]∥International Conference on Machine Vision,2017.
[3]黄凯.人体行为识别研究及其在水电站视频监控中的应用[D].杭州:浙江工业大学,2013.HUANG Kai. Research on human behavior recognition and application in video monitoring of hydropower station[D].Hangzhou:Zhejiang University of Technology,2013.(in Chinese)
[4]满君丰,李倩倩,温向兵.视频监控中可变人体行为的识别[J].东南大学学报(自然科学版),2011,41(3):492-497.MAN Junfeng,LI Qianqian,WEN Xiangbing. Recognition for changable human behaviors in video surveillance[J]. Journal of Southeast University(Natural Science Edition),2011,41(3):492-497.(in Chinese)
[5]何杰.视频监控中的人体异常行为识别方法研究[D].重庆:重庆大学,2014.HE Jie. Research on human abnormal activity recognition in video surveillance[D]. Chongqing:Chongqing University,2014.(in Chinese)
[6] GOWSIKHAA D,ABIRAMI S,BASKARAN R. Automated human behavior analysis from surveillance videos:a survey[J].Artificial Intelligence Review,2014,42(4):747-765.
[7] BAXTER R H,ROBERTSON N M,LANE D M. Human behaviour recognition in data-scarce domains[J]. Pattern Recognition,2015,48(8):2377-2393.
[8] LAO W W,HAN J J,WITH D P P. Automatic video-based human motion analyzer for consumer surveillance system[J]. IEEE Transactions on Consumer Electronics,2009,55(2):591-598.
[9]倪涛,赵泳嘉,张红彦,等.基于Kinect动态手势识别的机械臂实时位姿控制系统[J/OL].农业机械学报,2017,48(10):417-423.NI Tao,ZHAO Yongjia,ZHANG Hongyan,et al. Real-time mechanical arm position and pose control system by dynamic hand gesture recognition based on Kinect device[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2017,48(10):417-423. http:∥www. j-csam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20171053&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298. 2017. 10. 053.(in Chinese)
[10]权龙哲,李成林,冯正阳,等.体感操控多臂棚室机器人作业决策规划算法研究[J/OL].农业机械学报,2017,48(3):14-23.QUAN Longzhe,LI Chenglin,FENG Zhengyang,et al. Algorithm of works'decision for three arms robot in greenhouse based on control with motion sensing technology[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2017,48(3):14-23. http:∥www. jcsam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20170302&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298. 2017. 03. 002.(in Chinese)
[11]唐新星,倪涛,何丽鹏,等.基于立体视觉的遥操作工程机器人自主作业系统[J/OL].农业机械学报,2012,43(10):224-228.TANG Xinxing,NI Tao,HE Lipeng,et al. Autonomous task control system of construction tele-robot based on stereo vision[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2012,43(10):224-228. http:∥www. j-csam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20121040&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298.2012. 10. 040.(in Chinese)
[12]马淼,李贻斌.基于多级图像序列和卷积神经网络的人体行为识别[J].吉林大学学报(工学版),2017,47(4):1244-1252.MA Miao,LI Yibin. Multi-level image sequences and convolutional neural networks based human action recognition method[J]. Journal of Jilin University(Engineering and Technology Edition),2017,47(4):1244-1252.(in Chinese)
[13] DU Y,WANG W,WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]∥CVPR,2015.
[14] ZHU W,LAN C,XING J,et al. Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks[C]∥The 30th AAAI Conference on Artificial Intelligence(AAAI-16),2016.
[15] SHAHROUDY A,LIU J,NG T T,et al. NTU RGB+D:a large scale dataset for 3D human activity analysis[C]∥CVPR,2016.
[16] LIU J,SHAHROUDY A,XU D,et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[C]∥European Conference on Computer Vision(ECCV),2016.
[17] WU Z,JIANG Y G,WANG X,et al. Multi-stream multi-class fusion of deep networks for video classification[C]∥ACM on Multimedia Conference,ACM,2016.
[18] SHI Y,TIAN Y,WANG Y,et al. Sequential deep trajectory descriptor for action recognition with three-stream CNN[J].IEEE Transactions on Multimedia,2016,19(7):1510-1520.
[19] WANG J,YANG Y,MAO J,et al. CNN-RNN:a unified framework for multi-label image classification[C]∥CVPR,2016.
[20] FAN Y,LU X,LI D,et al. Video-based emotion recognition using CNN-RNN and C3D hybrid networks[C]∥International Conference on Multimodal Interaction,ACM,2016.
[21] LIANG G,HONG H,XIE W,et al. Combining convolutional neural network with recursive neural network for blood cell image classification[J]. IEEE Access,2018,99(6):36188-36197.
[22] VESAL S,RAVIKUMAR N,DAVARI A A,et al. Classification of breast cancer histology images using transfer learning[C]∥International Conference Image Analysis&Recognition. Springer,Cham,2018.
[23] RAJAGOPAL A K,SUBRAMANIAN R,RICCI E,et al. Exploring transfer learning approaches for head pose classification from multi-view surveillance images[J]. International Journal of Computer Vision,2014,109(1-2):146-167.
[2] GHABRI S,OUARDA W,ALIMI A M. Towards human behavior recognition based on spatio temporal features and support vector machines[C]∥International Conference on Machine Vision,2017.
[3]黄凯.人体行为识别研究及其在水电站视频监控中的应用[D].杭州:浙江工业大学,2013.HUANG Kai. Research on human behavior recognition and application in video monitoring of hydropower station[D].Hangzhou:Zhejiang University of Technology,2013.(in Chinese)
[4]满君丰,李倩倩,温向兵.视频监控中可变人体行为的识别[J].东南大学学报(自然科学版),2011,41(3):492-497.MAN Junfeng,LI Qianqian,WEN Xiangbing. Recognition for changable human behaviors in video surveillance[J]. Journal of Southeast University(Natural Science Edition),2011,41(3):492-497.(in Chinese)
[5]何杰.视频监控中的人体异常行为识别方法研究[D].重庆:重庆大学,2014.HE Jie. Research on human abnormal activity recognition in video surveillance[D]. Chongqing:Chongqing University,2014.(in Chinese)
[6] GOWSIKHAA D,ABIRAMI S,BASKARAN R. Automated human behavior analysis from surveillance videos:a survey[J].Artificial Intelligence Review,2014,42(4):747-765.
[7] BAXTER R H,ROBERTSON N M,LANE D M. Human behaviour recognition in data-scarce domains[J]. Pattern Recognition,2015,48(8):2377-2393.
[8] LAO W W,HAN J J,WITH D P P. Automatic video-based human motion analyzer for consumer surveillance system[J]. IEEE Transactions on Consumer Electronics,2009,55(2):591-598.
[9]倪涛,赵泳嘉,张红彦,等.基于Kinect动态手势识别的机械臂实时位姿控制系统[J/OL].农业机械学报,2017,48(10):417-423.NI Tao,ZHAO Yongjia,ZHANG Hongyan,et al. Real-time mechanical arm position and pose control system by dynamic hand gesture recognition based on Kinect device[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2017,48(10):417-423. http:∥www. j-csam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20171053&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298. 2017. 10. 053.(in Chinese)
[10]权龙哲,李成林,冯正阳,等.体感操控多臂棚室机器人作业决策规划算法研究[J/OL].农业机械学报,2017,48(3):14-23.QUAN Longzhe,LI Chenglin,FENG Zhengyang,et al. Algorithm of works'decision for three arms robot in greenhouse based on control with motion sensing technology[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2017,48(3):14-23. http:∥www. jcsam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20170302&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298. 2017. 03. 002.(in Chinese)
[11]唐新星,倪涛,何丽鹏,等.基于立体视觉的遥操作工程机器人自主作业系统[J/OL].农业机械学报,2012,43(10):224-228.TANG Xinxing,NI Tao,HE Lipeng,et al. Autonomous task control system of construction tele-robot based on stereo vision[J/OL]. Transactions of the Chinese Society for Agricultural Machinery,2012,43(10):224-228. http:∥www. j-csam. org/jcsam/ch/reader/view_abstract. aspx? flag=1&file_no=20121040&journal_id=jcsam. DOI:10. 6041/j. issn. 1000-1298.2012. 10. 040.(in Chinese)
[12]马淼,李贻斌.基于多级图像序列和卷积神经网络的人体行为识别[J].吉林大学学报(工学版),2017,47(4):1244-1252.MA Miao,LI Yibin. Multi-level image sequences and convolutional neural networks based human action recognition method[J]. Journal of Jilin University(Engineering and Technology Edition),2017,47(4):1244-1252.(in Chinese)
[13] DU Y,WANG W,WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]∥CVPR,2015.
[14] ZHU W,LAN C,XING J,et al. Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks[C]∥The 30th AAAI Conference on Artificial Intelligence(AAAI-16),2016.
[15] SHAHROUDY A,LIU J,NG T T,et al. NTU RGB+D:a large scale dataset for 3D human activity analysis[C]∥CVPR,2016.
[16] LIU J,SHAHROUDY A,XU D,et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[C]∥European Conference on Computer Vision(ECCV),2016.
[17] WU Z,JIANG Y G,WANG X,et al. Multi-stream multi-class fusion of deep networks for video classification[C]∥ACM on Multimedia Conference,ACM,2016.
[18] SHI Y,TIAN Y,WANG Y,et al. Sequential deep trajectory descriptor for action recognition with three-stream CNN[J].IEEE Transactions on Multimedia,2016,19(7):1510-1520.
[19] WANG J,YANG Y,MAO J,et al. CNN-RNN:a unified framework for multi-label image classification[C]∥CVPR,2016.
[20] FAN Y,LU X,LI D,et al. Video-based emotion recognition using CNN-RNN and C3D hybrid networks[C]∥International Conference on Multimodal Interaction,ACM,2016.
[21] LIANG G,HONG H,XIE W,et al. Combining convolutional neural network with recursive neural network for blood cell image classification[J]. IEEE Access,2018,99(6):36188-36197.
[22] VESAL S,RAVIKUMAR N,DAVARI A A,et al. Classification of breast cancer histology images using transfer learning[C]∥International Conference Image Analysis&Recognition. Springer,Cham,2018.
[23] RAJAGOPAL A K,SUBRAMANIAN R,RICCI E,et al. Exploring transfer learning approaches for head pose classification from multi-view surveillance images[J]. International Journal of Computer Vision,2014,109(1-2):146-167.