基于彩色-深度视频和CLDS的手语识别
|
摘要
提出一种基于彩色-深度视频和复线性动态系统(Complex linear dynamic system,CLDS)的手语识别方法,可以保证时序建模数据与原始数据严格对应,准确刻画手语特征,从而显著提高分类精度。利用深度视频补偿RGB视频中的缺失信息,提取手语视频运动边界直方图(Motion boundary histogram,MBH)特征,得到每种行为的特征矩阵。对特征矩阵进行CLDS时序建模,输出能唯一表示该类手语视频的描述符M=(A,C),然后利用子空间角度计算各模型之间的相似度;通过改进的K最近邻(K-nearest neighbors,KNN)算法得到最终分类结果。在中国手语数据集(Chinese sign language,CSL)上的实验表明,本文方法与现有的手语识别方法相比,具有更高的识别率。
This paper proposes a sign language recognition method based on color-depth videos and complex linear dynamic system(CLDS),which ensures that the time series modeling data can strictly correspond to the original data and accurately characterize the sign language features. Thus the classification precision is improved significantly. The depth videos are used to compensate the missing information of RGB videos,and the motion boundary histogram(MBH) features are extracted from the sign language videos to obtain the feature matrix of each behavior. The feature matrixes are modelled using CLDS method with output of the descriptor M=(A,C) which can uniquely represent the sign language video.Then the similarities between the models are calculated utilizing the subspace angles and the improved KNN algorithm is presented to achieve the final classification result. Experiments on the Chinese sign language dataset (CSL) show that the proposed sign language recognition approach has higher precision than many existing methods.
This paper proposes a sign language recognition method based on color-depth videos and complex linear dynamic system(CLDS),which ensures that the time series modeling data can strictly correspond to the original data and accurately characterize the sign language features. Thus the classification precision is improved significantly. The depth videos are used to compensate the missing information of RGB videos,and the motion boundary histogram(MBH) features are extracted from the sign language videos to obtain the feature matrix of each behavior. The feature matrixes are modelled using CLDS method with output of the descriptor M=(A,C) which can uniquely represent the sign language video.Then the similarities between the models are calculated utilizing the subspace angles and the improved KNN algorithm is presented to achieve the final classification result. Experiments on the Chinese sign language dataset (CSL) show that the proposed sign language recognition approach has higher precision than many existing methods.
引文
[1]Suk H I,Sin B K,Lee S W.Hand gesture recognition based on dynamic Bayesian network framework[J].Pattern Recognition,2010,43(9):3059-3072.
[2]Joshi A,Ghosh S,Betke M,et al.Personalizing gesture recognition using hierarchical bayesian neural networks[C]//IEEEConference on Computer Vision&Pattern Recognition.[S.l.]:IEEE Computer Society,2017:455-464.
[3]蔡军,李晓娟,张毅,等.改进的DAGSVM手势识别方法[J].华中科技大学学报:自然科学版,2013,41(5):86-89.Cai Jun,Li Xiaojuan,Zhang Yi,et al.Improved DAGSVM hand gesture recognition approach[J].Journal of Huazhong University of Science and Technology:Natural Science Edition,2013,41(5):86-89.
[4]张毅,张烁,罗元.基于视角不变的三维手势轨迹识别[J].电子科技大学学报,2014,43(1):60-65.Zhang Yi,Zhang Shuo,Luo Yuan.View-invariant 3D hand trajectory-based recognition[J].Journal of University of Electronic Science and Technology of China,2014,43(1):60-65.
[5]Wang H,Chai X,Zhou Y,et al.Fast sign language recognition benefited from low rank approximation[C]//IEEEInternational Conference and Workshops on Automatic Face and Gesture Recognition.[S.l.]:IEEE,2015:1-6.
[6]Wang H,Chai X,Chen X.Sparse observation(SO)alignment for sign language recognition[J].Neurocomputing,2016,175:674-685.
[7]Huang J,Zhou W,Li H,et al.Attention based 3D-CNNs for large-vocabulary sign language recognition[J].IEEETransactions on Circuits and Systems for Video Technology,2018,1:1.
[8]Cui R,Liu H,Zhang C.Recurrent convolutional neural networks for continuous sign language recognition by staged optimization[C]//IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2017:1610-1618.
[9]Wu D,Pigou L,Kindermans P J,et al.Deep dynamic neural networks for multimodal gesture segmentation and recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2016,38(8):1583-1597.
[10]Chan A B,Vasconcelos N.Probabilistic kernels for the classification of auto-regressive visual processes[C]//IEEE Computer Society Conference on Computer Vision&Pattern Recognition.[S.l.]:IEEE,2005:846-851.
[11]Vishwanathan S V N,Smola A J,Vidal R.Binet-Cauchy kernels on dynamical systems and its application to the analysis of dynamic scenes[J].International Journal of Computer Vision,2007,73(1):95-119.
[12]Li L,Prakash B A.Time series clustering:Complex is simpler![C]//ICML.Pittsburgh:[s.n.],2011:185-192.
[13]Schuldt C,Laptev I,Caputo B.Recognizing human actions:A local SVM approach[C]//Proc of the 17th Int Conf on Pattern Recognition.Piscataway,NJ:IEEE,2004:1051-4651.
[14]Kovashka A,Grauman K.Learning a hierarchy of discriminative space-time neighborhood features for human action recognition[C]//Computer Vision&Pattern Recognition.[S.l.]:IEEE,2010:2046-2053.
[15]刘砚秋,王修晖.基于圆弧扫描线的手势特征提取和实时手势识别[J].数据采集与处理,2016,31(1):184-189.Liu Yanqiu,Wang Xiuhui.Gesture feature extraction and recognition based on circular scan lines[J].Journal of Data Acquisition and Processing,2016,31(1):184-189.
[16]Huang J,Zhou W,Zhang Q,et al.Video-based sign language recognition without temporal segmentation[C]//32nd AAAIConference on Artificial Intelligence.[S.l.]:AAAI,2018:2257-2264.
[17]Oneata D,Verbeek J,Schmid C.Action and event recognition with fisher vectors on a compact feature set[C]//IEEEInternational Conference on Computer Vision(ICCV).[S.l.]:IEEE,2013:1817-1824.
[18]Laptev I.On space-time interest points[J].International Journal of Computer Vision,2005,64(2/3):107-123..
[19]Wang H,Kl?ser A,Schmid C,et al.Action recognition by dense trajectories[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2011:3169-3176.
[20]Tang A,Lu K,Wang Y,et al.A real-time hand posture recognition system using deep neural networks[J].ACMTransactions on Intelligent Systems and Technology,2015,6(2):1-23.
[2]Joshi A,Ghosh S,Betke M,et al.Personalizing gesture recognition using hierarchical bayesian neural networks[C]//IEEEConference on Computer Vision&Pattern Recognition.[S.l.]:IEEE Computer Society,2017:455-464.
[3]蔡军,李晓娟,张毅,等.改进的DAGSVM手势识别方法[J].华中科技大学学报:自然科学版,2013,41(5):86-89.Cai Jun,Li Xiaojuan,Zhang Yi,et al.Improved DAGSVM hand gesture recognition approach[J].Journal of Huazhong University of Science and Technology:Natural Science Edition,2013,41(5):86-89.
[4]张毅,张烁,罗元.基于视角不变的三维手势轨迹识别[J].电子科技大学学报,2014,43(1):60-65.Zhang Yi,Zhang Shuo,Luo Yuan.View-invariant 3D hand trajectory-based recognition[J].Journal of University of Electronic Science and Technology of China,2014,43(1):60-65.
[5]Wang H,Chai X,Zhou Y,et al.Fast sign language recognition benefited from low rank approximation[C]//IEEEInternational Conference and Workshops on Automatic Face and Gesture Recognition.[S.l.]:IEEE,2015:1-6.
[6]Wang H,Chai X,Chen X.Sparse observation(SO)alignment for sign language recognition[J].Neurocomputing,2016,175:674-685.
[7]Huang J,Zhou W,Li H,et al.Attention based 3D-CNNs for large-vocabulary sign language recognition[J].IEEETransactions on Circuits and Systems for Video Technology,2018,1:1.
[8]Cui R,Liu H,Zhang C.Recurrent convolutional neural networks for continuous sign language recognition by staged optimization[C]//IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2017:1610-1618.
[9]Wu D,Pigou L,Kindermans P J,et al.Deep dynamic neural networks for multimodal gesture segmentation and recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2016,38(8):1583-1597.
[10]Chan A B,Vasconcelos N.Probabilistic kernels for the classification of auto-regressive visual processes[C]//IEEE Computer Society Conference on Computer Vision&Pattern Recognition.[S.l.]:IEEE,2005:846-851.
[11]Vishwanathan S V N,Smola A J,Vidal R.Binet-Cauchy kernels on dynamical systems and its application to the analysis of dynamic scenes[J].International Journal of Computer Vision,2007,73(1):95-119.
[12]Li L,Prakash B A.Time series clustering:Complex is simpler![C]//ICML.Pittsburgh:[s.n.],2011:185-192.
[13]Schuldt C,Laptev I,Caputo B.Recognizing human actions:A local SVM approach[C]//Proc of the 17th Int Conf on Pattern Recognition.Piscataway,NJ:IEEE,2004:1051-4651.
[14]Kovashka A,Grauman K.Learning a hierarchy of discriminative space-time neighborhood features for human action recognition[C]//Computer Vision&Pattern Recognition.[S.l.]:IEEE,2010:2046-2053.
[15]刘砚秋,王修晖.基于圆弧扫描线的手势特征提取和实时手势识别[J].数据采集与处理,2016,31(1):184-189.Liu Yanqiu,Wang Xiuhui.Gesture feature extraction and recognition based on circular scan lines[J].Journal of Data Acquisition and Processing,2016,31(1):184-189.
[16]Huang J,Zhou W,Zhang Q,et al.Video-based sign language recognition without temporal segmentation[C]//32nd AAAIConference on Artificial Intelligence.[S.l.]:AAAI,2018:2257-2264.
[17]Oneata D,Verbeek J,Schmid C.Action and event recognition with fisher vectors on a compact feature set[C]//IEEEInternational Conference on Computer Vision(ICCV).[S.l.]:IEEE,2013:1817-1824.
[18]Laptev I.On space-time interest points[J].International Journal of Computer Vision,2005,64(2/3):107-123..
[19]Wang H,Kl?ser A,Schmid C,et al.Action recognition by dense trajectories[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2011:3169-3176.
[20]Tang A,Lu K,Wang Y,et al.A real-time hand posture recognition system using deep neural networks[J].ACMTransactions on Intelligent Systems and Technology,2015,6(2):1-23.