时空特征局部保持的运动视频关键帧提取
|
摘要
为提高运动视频关键帧的运动表达能力和压缩率,提出柔性姿态估计和时空特征嵌入结合的运动视频关键帧提取技术。首先,利用人体动作的时间连续性保持建立具有时间约束限制的柔性部件铰接人体(ST-FMP)模型,通过非确定性人体部位动作连续性约束,采用N-best算法估计单帧图像中的人体姿态参数;接着,采用人体部位的相对位置和运动方向描述人体运动特征,通过拉普拉斯分值法实施数据降维,获得局部拓扑结构表达能力强的判别性人体运动特征向量;最后,采用迭代自组织数据分析技术(ISODATA)算法动态地确定关键帧。在健美操动作视频关键帧提取实验中,ST-FMP模型将柔性混合铰接人体模型(FMP)的非确定性人体部位的识别准确率提高约15个百分点,取得了81%的关键帧提取准确率,优于KFE和运动块的关键帧算法。所提算法对人体运动特征和人体姿态敏感,适用于运动视频批注审阅。
To improve the motion expression and compression rate of the motion video key frames, a dynamic video frame extraction technique based on flexible pose estimation and spatial-temporal feature embedding was proposed. Firstly, a SpatialTemporal feature embedded Flexible Mixture-of-Parts articulated human model( ST-FMP) was designed by preserving the spatial-temporal features of body parts, and the N-best algorithm was adopted with spatial-temporal locally preserving of uncertain body parts to estimate the body configuration in a single frame based on ST-FMP. Then, the relative position and motion direction of the human body were used to describe the characteristics of the human body motion. The Laplacian scoring algorithm was used to implement dimensionality reduction to obtain the discriminant human motion feature vector with local topological structure. Finally, the ISODATA( Iterative Self-Organizing Data Analysis Technique) algorithm was used to dynamically determine the key frames. In the key frame extraction experiment on aerobics video, compared to articulated human model with Flexible Mixture-of-Parts( FMP) and motion block, the accuracy of uncertain body parts by using ST-FMP was 15 percentage points higher than that by using FMP, achieved 81%, which was higher than that by using Key Frames Extraction based on prior knowledge( KFE) and key frame extraction based on motion blocks. The experimental results on key frame extraction for calisthenics video show that the proposed approach is sensitive to motion feature selection and human pose configuration, and it can be used for sports video annotation.
To improve the motion expression and compression rate of the motion video key frames, a dynamic video frame extraction technique based on flexible pose estimation and spatial-temporal feature embedding was proposed. Firstly, a SpatialTemporal feature embedded Flexible Mixture-of-Parts articulated human model( ST-FMP) was designed by preserving the spatial-temporal features of body parts, and the N-best algorithm was adopted with spatial-temporal locally preserving of uncertain body parts to estimate the body configuration in a single frame based on ST-FMP. Then, the relative position and motion direction of the human body were used to describe the characteristics of the human body motion. The Laplacian scoring algorithm was used to implement dimensionality reduction to obtain the discriminant human motion feature vector with local topological structure. Finally, the ISODATA( Iterative Self-Organizing Data Analysis Technique) algorithm was used to dynamically determine the key frames. In the key frame extraction experiment on aerobics video, compared to articulated human model with Flexible Mixture-of-Parts( FMP) and motion block, the accuracy of uncertain body parts by using ST-FMP was 15 percentage points higher than that by using FMP, achieved 81%, which was higher than that by using Key Frames Extraction based on prior knowledge( KFE) and key frame extraction based on motion blocks. The experimental results on key frame extraction for calisthenics video show that the proposed approach is sensitive to motion feature selection and human pose configuration, and it can be used for sports video annotation.
引文
[1]ZHOU M.Chinese university students'acceptance of MOOCs:a self-determination perspective[J].Computers&Education,2016,92/93:194-203.
[2]LEHMANN R,SEITZ A,BOSSE H M,et al.Student perceptions of a video-based blended learning approach for improving pediatric physical examination skills[J].Annals of Anatomy-Anatomischer Anzeiger,2016,208:179-182.
[3]PANG Y J.Techniques for enhancing hybrid learning of physical education[C]//International Conference on Hybrid Learning,LNCS6248.Berlin:Springer,2010:94-105.
[4]JI S,XU W,YANG M,et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231.
[5]姬晓飞,左鑫孟.基于关键帧特征库统计特征的双人交互行为识别[J].计算机应用,2016,36(8):2287-2291.(JI X F,ZUO XM.Human interaction recognition based on statistical features of key frame feature library[J].Journal of Computer Applications,2016,36(8):2287-2291.)
[6]HANNANE R,ELBOUSHAKI A,AFDEL K,et al.An efficient method for video shot boundary detection and keyframe extraction using SIFT-point distribution histogram[J].International Journal of Multimedia Information Retrieval,2016,5(2):89-104.
[7]王少华,狄岚,梁久祯.基于核与局部信息的多维度模糊聚类图像分割算法[J].计算机应用,2015,35(11):3227-3231.(WANGS H,DI L,LIANG J Z.Multi-dimensional fuzzy clustering image segmentation algorithm based on kernel metric and local information[J].Journal of Computer Applications,2015,35(11):3227-3231.)
[8]JANWE M N J,BHOYAR K K.Video key-frame extraction using unsupervised clustering and mutual comparison[J].International Journal of Image Processing,2016,10(2):73-84.
[9]庞亚俊.基于先验的动作视频关键帧提取[J].河南理工大学学报(自然科学版),2016,35(6):862-868.(PANG Y J.Key frames extraction of motion video based on prior knowledge[J].Journal of Henan Polytechnic University(Natural Science),2016,35(6):862-868.)
[10]YANG Y,RAMANAN D.Articulated human detection with flexible mixtures of parts[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2013,35(12):2878-2890.
[11]胡琼,秦磊,黄庆明.基于视觉的人体动作识别综述[J].计算机学报,2013,36(12):2512-2524.(HU Q,QIN L,HUANG Q M.A survey on visual human action recognition[J].Chinese Journal of Computers,2013,36(12):2512-2524.)
[12]PARK D,RAMANAN D.N-best maximal decoders for part models[C]//Proceedings of the 2011 International Conference on Computer Vision.Washington,DC:IEEE Computer Society,2011:2627-2634.
[13]SUN D,ROTH S,BLACK M J.Secrets of optical flow estimation and their principles[C]//Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE,2010:2432-2439.
[14]FU Y.Human Activity Recognition and Prediction[M].Berlin:Springer,2016:462-471.
[15]HE X,CAI D,NIYOGI P.Laplacian score for feature selection[C]//Proceedings of the 18th International Conference on Neural Information Processing Systems.Cambridge,MA:MIT Press,2005:507-514.
[16]BALL G H,HALL J.ISODATA,a novel method of data analysis and pattern classification[R].Springfield:NTIS,1965.
[17]应锐,蔡瑾,冯辉,等.基于运动块及关键帧的人体动作识别[J].复旦学报(自然科学版),2014,53(6):815-822.(YING R,CAI J,FENG H,et al.Human action recognition based on motion blocks and key frames[J].Journal of Fudan University(Natural Science),2014,53(6):815-822.)
[18]XIA G,SUN H,NIU X,et al.Keyframe extraction for human motion capture data based on joint kernel sparse representation[J].IEEE Transactions on Industrial Electronics,2017,64(2):1589-1599.
[19]LIU Z,ZHU J,BU J,et al.A survey of human pose estimation[J].Journal of Visual Communication&Image Representation,2015,32(C):10-19.
[2]LEHMANN R,SEITZ A,BOSSE H M,et al.Student perceptions of a video-based blended learning approach for improving pediatric physical examination skills[J].Annals of Anatomy-Anatomischer Anzeiger,2016,208:179-182.
[3]PANG Y J.Techniques for enhancing hybrid learning of physical education[C]//International Conference on Hybrid Learning,LNCS6248.Berlin:Springer,2010:94-105.
[4]JI S,XU W,YANG M,et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231.
[5]姬晓飞,左鑫孟.基于关键帧特征库统计特征的双人交互行为识别[J].计算机应用,2016,36(8):2287-2291.(JI X F,ZUO XM.Human interaction recognition based on statistical features of key frame feature library[J].Journal of Computer Applications,2016,36(8):2287-2291.)
[6]HANNANE R,ELBOUSHAKI A,AFDEL K,et al.An efficient method for video shot boundary detection and keyframe extraction using SIFT-point distribution histogram[J].International Journal of Multimedia Information Retrieval,2016,5(2):89-104.
[7]王少华,狄岚,梁久祯.基于核与局部信息的多维度模糊聚类图像分割算法[J].计算机应用,2015,35(11):3227-3231.(WANGS H,DI L,LIANG J Z.Multi-dimensional fuzzy clustering image segmentation algorithm based on kernel metric and local information[J].Journal of Computer Applications,2015,35(11):3227-3231.)
[8]JANWE M N J,BHOYAR K K.Video key-frame extraction using unsupervised clustering and mutual comparison[J].International Journal of Image Processing,2016,10(2):73-84.
[9]庞亚俊.基于先验的动作视频关键帧提取[J].河南理工大学学报(自然科学版),2016,35(6):862-868.(PANG Y J.Key frames extraction of motion video based on prior knowledge[J].Journal of Henan Polytechnic University(Natural Science),2016,35(6):862-868.)
[10]YANG Y,RAMANAN D.Articulated human detection with flexible mixtures of parts[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2013,35(12):2878-2890.
[11]胡琼,秦磊,黄庆明.基于视觉的人体动作识别综述[J].计算机学报,2013,36(12):2512-2524.(HU Q,QIN L,HUANG Q M.A survey on visual human action recognition[J].Chinese Journal of Computers,2013,36(12):2512-2524.)
[12]PARK D,RAMANAN D.N-best maximal decoders for part models[C]//Proceedings of the 2011 International Conference on Computer Vision.Washington,DC:IEEE Computer Society,2011:2627-2634.
[13]SUN D,ROTH S,BLACK M J.Secrets of optical flow estimation and their principles[C]//Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE,2010:2432-2439.
[14]FU Y.Human Activity Recognition and Prediction[M].Berlin:Springer,2016:462-471.
[15]HE X,CAI D,NIYOGI P.Laplacian score for feature selection[C]//Proceedings of the 18th International Conference on Neural Information Processing Systems.Cambridge,MA:MIT Press,2005:507-514.
[16]BALL G H,HALL J.ISODATA,a novel method of data analysis and pattern classification[R].Springfield:NTIS,1965.
[17]应锐,蔡瑾,冯辉,等.基于运动块及关键帧的人体动作识别[J].复旦学报(自然科学版),2014,53(6):815-822.(YING R,CAI J,FENG H,et al.Human action recognition based on motion blocks and key frames[J].Journal of Fudan University(Natural Science),2014,53(6):815-822.)
[18]XIA G,SUN H,NIU X,et al.Keyframe extraction for human motion capture data based on joint kernel sparse representation[J].IEEE Transactions on Industrial Electronics,2017,64(2):1589-1599.
[19]LIU Z,ZHU J,BU J,et al.A survey of human pose estimation[J].Journal of Visual Communication&Image Representation,2015,32(C):10-19.