Research on unified recognition model and algorithm for multi-modal gestures
|
摘要
In gesture recognition, static gestures, dynamic gestures and trajectory gestures are collectively known as multi-modal gestures. To solve the existing problem in different recognition methods for different modal gestures, a unified recognition algorithm is proposed. The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect. Through the preprocessing of the multi-source heterogeneous data, all hand gestures were considered as curves while solving hand shaking, and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition. In this way, complex gesture recognition was transformed into the problem of a simple comparison of curves similarities. The main innovations: 1) Aiming at solving the problem of multi-modal gesture recognition, an unified recognition model and a new algorithm is proposed; 2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved. By testing 50 kinds of gestures, the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.
In gesture recognition, static gestures, dynamic gestures and trajectory gestures are collectively known as multi-modal gestures. To solve the existing problem in different recognition methods for different modal gestures, a unified recognition algorithm is proposed. The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect. Through the preprocessing of the multi-source heterogeneous data, all hand gestures were considered as curves while solving hand shaking, and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition. In this way, complex gesture recognition was transformed into the problem of a simple comparison of curves similarities. The main innovations: 1) Aiming at solving the problem of multi-modal gesture recognition, an unified recognition model and a new algorithm is proposed; 2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved. By testing 50 kinds of gestures, the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.
In gesture recognition, static gestures, dynamic gestures and trajectory gestures are collectively known as multi-modal gestures. To solve the existing problem in different recognition methods for different modal gestures, a unified recognition algorithm is proposed. The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect. Through the preprocessing of the multi-source heterogeneous data, all hand gestures were considered as curves while solving hand shaking, and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition. In this way, complex gesture recognition was transformed into the problem of a simple comparison of curves similarities. The main innovations: 1) Aiming at solving the problem of multi-modal gesture recognition, an unified recognition model and a new algorithm is proposed; 2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved. By testing 50 kinds of gestures, the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.
引文
1.Liu X.Research of dynamic gesture recognition based on the kinect depth image.Shenyang,LN,China:Northeastern University,2014:77p (in Chinese)
2.Lee C,Xu Y.Online,interactive learning of gestures for human/robot interfaces.Proceedings of IEEE International Conference on Robotics and Automation,Apr 22-28,1996,Minneapolis,MN,USA:IEEE,2002(4):2982-2987
3.Nayan M K,Manalee D S.Recognition of sign language alphabets and numbers based on hand kinematics using a data glove.Procedia Computer Science,2018(133):55-62
4.Kazuma I,Masato S,Hirokazu T.Development of finger motion skill learning support system based on data gloves.Procedia Computer Science,2014,35(6):1307-1314
5.Dutta D,Modak S,Kumar A,et al.Bayesian network aided grasp and grip efficiency estimation using a smart data glove for post-stroke diagnosis.Biocybernetics and Biomedical Engineering,2017,37(1):44-58
6.Yao Y,Zhang L J,Qiao W B.Hand part labeling and gesture recognition from RGB-D data.Journal of Computer-Aided Design and Computer Graphics,2013,25(12):1810-1816 (in Chinese)
7.Li H B,Ding L J,et al.Static three-dimensional gesture recognition based on kinect skeleton data.Computer Applications and Software,2015,32(9):161-165 (in Chinese)
8.Batabyal T,Chattopadhyay T,Mukherjee D P.Action recognition using joint coordinates of 3D skeleton data.IEEE International Conference on Image Processing,Sep 27-30,2015,Quebec City,Canada:IEEE,2015:4107-4111
9.Zhu H M,Pun C M.Human action recognition with skeletal information from depth camera.IEEE International Conference on Information and Automation,July 26-31,2014,Hulun Buir,NM,China:IEEE,2014:1082-1085
10.Ding I J,Chang C W.Feature design scheme for kinect-based DTW human gesture recognition.Multimedia Tools and Applications,2016,75(16):9669-9684
11.Wu D,Pigou L,Kindermans P J,et al.Deep dynamic neural networks for multimodal gesture segmentation and recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(8):1583-1597
12.Yang X W,Feng Z Q,Huang Z Z,et al.A gesture recognition algorithm using hausdorff-like distance template matching based on the main direction of gesture.Applied Mechanics and Materials,2015,713-715(3):2156-2159
13.Singha J,Laskar R H.Hand gesture recognition using two-level speed normalization,feature selection and classifier fusion.Multimedia Systems,2017,23(4):499-514
14.Lee W P,Kaoli C,Huang J Y.A smart TV system with body-gesture control,tag-based rating and context-aware recommendation.Knowledge-Based Systems,2014,56(3):167-178
15.Ghaleb F F M,Youness E A,Elmezain M,et al.Vision-based hand gesture spotting and recognition using CRF and SVM.Journal of Software Engineering and Applications,2015,8(7):313-323
16.Dias T,Variz M,Jorge P,et al.Gesture interaction system for social web applications on smart TVs.Proceedings of the 10th Conference on Open Research Areas in Information Retrieval,May 15-17,2013,Lisbon,Portugal,2013:225-226
17.Kim Y C,Jiang J S.Remote gesture control using HMM classifier based on electric potential sensors.International Journal of Multimedia and Ubiquitous Engineering,2015,10(2):413-420
18.Wang L,Liu G X,Duan H Y.Dynamic and combined gestures recognition based on multi-feature fusion in a complex environment.The Journal of China Universities of Posts and Telecommunications,2015,22(2):81-88
19.Shehu V,Dika A.Curve similarity measurement algorithms for automatic gesture detection systems.Proceedings of IEEE 35th International Convention MIPRO,May 21-25,2012,Opatija,Croatia:IEEE,2012:973-976
20.Shen L,Huo S C,Chen M S,et al.Multi-touch gesture recognition algorithm of vehicle electronic devices-based on Bezier curve optimization strategy.2017 32nd Youth Academic Annual Conference of Chinese Association of Automation (YAC),May 19-21,2017,Hefei,AH,China:IEEE,2017:720-723
21.Wu J X,Cheng J.Bayesian co-boosting for multi-modal gesture recognition.The Journal of Machine Learning Research,2017,15(1):3013-3036
22.Liang B,Zheng L.Multi-modal gesture recognition using skeletal joints and motion trail model.Workshop at the European Conference on Computer Vision,Sep 6-12,2014,Zurich,Switzerland,2014:623-628
23.Jia X Y,Xu C S,Bai X.The invention and way of thinking on least squares.Journal of Northwest University (Natural Science Edition),2006,36(3):507-511 (in Chinese)
24.Yang F,Feng X,Ruan L,et al.Correlation study of water tree and VLF tanδ based on pearson correlation coefficient.High Voltage Apparatus,2014,50(6):21-31 (in Chinese)
2.Lee C,Xu Y.Online,interactive learning of gestures for human/robot interfaces.Proceedings of IEEE International Conference on Robotics and Automation,Apr 22-28,1996,Minneapolis,MN,USA:IEEE,2002(4):2982-2987
3.Nayan M K,Manalee D S.Recognition of sign language alphabets and numbers based on hand kinematics using a data glove.Procedia Computer Science,2018(133):55-62
4.Kazuma I,Masato S,Hirokazu T.Development of finger motion skill learning support system based on data gloves.Procedia Computer Science,2014,35(6):1307-1314
5.Dutta D,Modak S,Kumar A,et al.Bayesian network aided grasp and grip efficiency estimation using a smart data glove for post-stroke diagnosis.Biocybernetics and Biomedical Engineering,2017,37(1):44-58
6.Yao Y,Zhang L J,Qiao W B.Hand part labeling and gesture recognition from RGB-D data.Journal of Computer-Aided Design and Computer Graphics,2013,25(12):1810-1816 (in Chinese)
7.Li H B,Ding L J,et al.Static three-dimensional gesture recognition based on kinect skeleton data.Computer Applications and Software,2015,32(9):161-165 (in Chinese)
8.Batabyal T,Chattopadhyay T,Mukherjee D P.Action recognition using joint coordinates of 3D skeleton data.IEEE International Conference on Image Processing,Sep 27-30,2015,Quebec City,Canada:IEEE,2015:4107-4111
9.Zhu H M,Pun C M.Human action recognition with skeletal information from depth camera.IEEE International Conference on Information and Automation,July 26-31,2014,Hulun Buir,NM,China:IEEE,2014:1082-1085
10.Ding I J,Chang C W.Feature design scheme for kinect-based DTW human gesture recognition.Multimedia Tools and Applications,2016,75(16):9669-9684
11.Wu D,Pigou L,Kindermans P J,et al.Deep dynamic neural networks for multimodal gesture segmentation and recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(8):1583-1597
12.Yang X W,Feng Z Q,Huang Z Z,et al.A gesture recognition algorithm using hausdorff-like distance template matching based on the main direction of gesture.Applied Mechanics and Materials,2015,713-715(3):2156-2159
13.Singha J,Laskar R H.Hand gesture recognition using two-level speed normalization,feature selection and classifier fusion.Multimedia Systems,2017,23(4):499-514
14.Lee W P,Kaoli C,Huang J Y.A smart TV system with body-gesture control,tag-based rating and context-aware recommendation.Knowledge-Based Systems,2014,56(3):167-178
15.Ghaleb F F M,Youness E A,Elmezain M,et al.Vision-based hand gesture spotting and recognition using CRF and SVM.Journal of Software Engineering and Applications,2015,8(7):313-323
16.Dias T,Variz M,Jorge P,et al.Gesture interaction system for social web applications on smart TVs.Proceedings of the 10th Conference on Open Research Areas in Information Retrieval,May 15-17,2013,Lisbon,Portugal,2013:225-226
17.Kim Y C,Jiang J S.Remote gesture control using HMM classifier based on electric potential sensors.International Journal of Multimedia and Ubiquitous Engineering,2015,10(2):413-420
18.Wang L,Liu G X,Duan H Y.Dynamic and combined gestures recognition based on multi-feature fusion in a complex environment.The Journal of China Universities of Posts and Telecommunications,2015,22(2):81-88
19.Shehu V,Dika A.Curve similarity measurement algorithms for automatic gesture detection systems.Proceedings of IEEE 35th International Convention MIPRO,May 21-25,2012,Opatija,Croatia:IEEE,2012:973-976
20.Shen L,Huo S C,Chen M S,et al.Multi-touch gesture recognition algorithm of vehicle electronic devices-based on Bezier curve optimization strategy.2017 32nd Youth Academic Annual Conference of Chinese Association of Automation (YAC),May 19-21,2017,Hefei,AH,China:IEEE,2017:720-723
21.Wu J X,Cheng J.Bayesian co-boosting for multi-modal gesture recognition.The Journal of Machine Learning Research,2017,15(1):3013-3036
22.Liang B,Zheng L.Multi-modal gesture recognition using skeletal joints and motion trail model.Workshop at the European Conference on Computer Vision,Sep 6-12,2014,Zurich,Switzerland,2014:623-628
23.Jia X Y,Xu C S,Bai X.The invention and way of thinking on least squares.Journal of Northwest University (Natural Science Edition),2006,36(3):507-511 (in Chinese)
24.Yang F,Feng X,Ruan L,et al.Correlation study of water tree and VLF tanδ based on pearson correlation coefficient.High Voltage Apparatus,2014,50(6):21-31 (in Chinese)