网球比赛视频分析的若干技术研究
|
摘要
随着视频数据的飞速膨胀,如何找到所需的信息,已经成为一个急需解决的问题。本文面向体育视频领域,研究了基于内容的视频分析和检索技术。其目的就是通过对视频内容进行计算机处理、分析和理解,建立结构和索引以方便用户获取,具有重要的理论意义和应用前景。
本文以网球比赛视频为对象,主要对下面几个问题进行了讨论,包括镜头边界检测、关键帧选取、特征提取,以及镜头分类、音频类型识别、网球视频感兴趣事件的检测。
镜头边界检测是进行视频处理的第一步,本文在总结了现有镜头边界检测方法的基础上,采用了一种基于动态阈值的镜头突变检测算法,实现了对突变镜头的检测;采用一种基于灰度级平均的方法,实现了对渐变镜头的检测。
在对每个镜头选取关键帧的基础上,本文提出了一种基于K均值聚类结合关键帧主色率统计的镜头分类算法,完成了对比赛镜头的筛选,并在此基础上,采用一种基于场地颜色的场地类型检测方法,实现了网球比赛场地类型的检测。
对于镜头分类得到的比赛镜头,提取出音频流,采用SVM决策树的方法,实现镜头中有意义的音频段的识别。然后,综合音视域的分析结果,实现网球比赛精彩事件的检测。
最后,本文以Visual C++6.0和Matlab 7.0为开发平台,实现了一个网球视频自动分析原型系统。实验表明,本文采用的网球视频语义分析算法具有令人满意的效果。
With the rapid increase of the amount of digital videos, the problem how to find useful information has become much urgent. This paper focuses on content-based video analysis and retrieval technology in sports video domain. The aim is to process, analyze and understand video content with computer to construct index and structure for facilitating user's access, which is provided with important academic appeals and commercial potentials.
This paper takes tennis video as research object, and discusses several problems in the process of content-based video retrieval, including shot boundary detection, key frame selection, feature extraction and shot classification, audio type classification, interesting events detecting.
Shot boundary detection is the first step of video processing, starting from a generalization of methods of shot boundary detection, this paper analysised an algorithm for detecting abrupt video shot boundaries based on method of adaptive threshold, and an algorithm for gradual shot boundaries detecting using the mean gray level (MGL) of image.
Based on abstracting typical frames of video shots, this paper proposed a shot classification method which is based on K-means cluster processing combined with main color rate computing. After that, this paper used a algorithm for detecting court class based on court color.
For play shots, important auditory features including both ball hitting and cheer are detected by using SVM. Afterwards, audio features are applied into play shots for interesting events detecting in tennis video.
Finally, This paper implements a prototype system for content-based tennis video anlysis by Visual C++ 6.0 and Matlab 7.0, and the experiments have demonstrate that all these methods are effective.
本文以网球比赛视频为对象,主要对下面几个问题进行了讨论,包括镜头边界检测、关键帧选取、特征提取,以及镜头分类、音频类型识别、网球视频感兴趣事件的检测。
镜头边界检测是进行视频处理的第一步,本文在总结了现有镜头边界检测方法的基础上,采用了一种基于动态阈值的镜头突变检测算法,实现了对突变镜头的检测;采用一种基于灰度级平均的方法,实现了对渐变镜头的检测。
在对每个镜头选取关键帧的基础上,本文提出了一种基于K均值聚类结合关键帧主色率统计的镜头分类算法,完成了对比赛镜头的筛选,并在此基础上,采用一种基于场地颜色的场地类型检测方法,实现了网球比赛场地类型的检测。
对于镜头分类得到的比赛镜头,提取出音频流,采用SVM决策树的方法,实现镜头中有意义的音频段的识别。然后,综合音视域的分析结果,实现网球比赛精彩事件的检测。
最后,本文以Visual C++6.0和Matlab 7.0为开发平台,实现了一个网球视频自动分析原型系统。实验表明,本文采用的网球视频语义分析算法具有令人满意的效果。
With the rapid increase of the amount of digital videos, the problem how to find useful information has become much urgent. This paper focuses on content-based video analysis and retrieval technology in sports video domain. The aim is to process, analyze and understand video content with computer to construct index and structure for facilitating user's access, which is provided with important academic appeals and commercial potentials.
This paper takes tennis video as research object, and discusses several problems in the process of content-based video retrieval, including shot boundary detection, key frame selection, feature extraction and shot classification, audio type classification, interesting events detecting.
Shot boundary detection is the first step of video processing, starting from a generalization of methods of shot boundary detection, this paper analysised an algorithm for detecting abrupt video shot boundaries based on method of adaptive threshold, and an algorithm for gradual shot boundaries detecting using the mean gray level (MGL) of image.
Based on abstracting typical frames of video shots, this paper proposed a shot classification method which is based on K-means cluster processing combined with main color rate computing. After that, this paper used a algorithm for detecting court class based on court color.
For play shots, important auditory features including both ball hitting and cheer are detected by using SVM. Afterwards, audio features are applied into play shots for interesting events detecting in tennis video.
Finally, This paper implements a prototype system for content-based tennis video anlysis by Visual C++ 6.0 and Matlab 7.0, and the experiments have demonstrate that all these methods are effective.
引文
[1]章毓晋.基于内容的视频信息检索.科学出版社,2003.
[2]章毓晋.图像理解与计算机视觉。清华大学出版社,2000。
[3]庄越挺,潘云鹤,吴飞.网上多媒体信息分析与检索.清华大学出版社,2002.
[4]王扉.体育视频的内容分析技术研究.博士学位论文。北京:中国科学院研究生院,2005.
[5]赵亚琴.基于内容的视频片段检索技术与研究.博士学位论文。南京:南京理工大学,2006.
[6]S.W.Smoliar,H.J.Zhang.Content-Based Video Indexing and Retrieval.IEEE Multimedia,vol.1,no.2,pp.62-72,1994.
[7]D.Yow,et al.Analysis and presentation of soccer highlights from digital video.In Proceedings Asian Conference on Computer Vision,1995.
[8]Y.H.Gong,L.T.Sin,et al。Automatic parsing of soccer programs.In Proceedings IEEE International Conference Multimedia Computer System,1995:167-174.
[9]L.Xie,S.F.Chang,A.Divakaran,H.Sun.Structure analysis of soccer video with hidden Markov models.In Proceedings IEEE International Conference Acoustics,Speech,and Signal Processing,2002.
[10]A.Ekin,A.M.Tekalp.Shot type classification by dominant color for sports video segmentation and summarization.Acoustics,Speech,and Signal Processing,2003.Proceedings.(ICASSP'03).2003 IEEE International Conference on,6-10 April 2003vol.3:173-176.
[11]Ohno Y,Miurs J,Shirai Y.Tracking players and a ball in soccer games.Multisensor Fusion and Integration for Intelligent Systems,1999.MFI'99.Proceedings.1999 IEEE/SICE/RSJ International Conference on,15-18 Aug.1999:147-152.
[12]Ohno Y,Miura J,Shirai Y.Tracking players and estimation of the 3D position of a ball in soccer games.Pattern Recognition,2000.Proceedings.15th International Conference on,Volume 1,3-7 Sept.2000:145-148 vol.1.
[13]Jianyun Chen,Yunhao Li,Songyang Lao,Lingda Wu,A BSU-based sports video content analysis framework for content0based streams filtering.Computer Networks and Mobile Computing,2003.ICCNMC 2003.2003 International Conference on,20-23 Oct.2003:446-449
[14]W.S.Zhou,et al.Online knowledge and rule-based video classification system for video indexing and dissemination.Information System,2002(27):559-586.
[15]Y.Rui,A.Gupta,and A.Acero.Automatically extracting highlights for TV baseball programs.In Proceedings ACM Multimedia,2000.
[16]D.Zhang and S.F.Chang.Event Detection in Baseball Video Using Superimposed Caption Recognition.Proceedings of ACM International Conference on Multimedia,December 2002.
[17]R.Leonardi and P.Migliorati.Semantic indexing of multimedia documents.IEEE Multimedia,2002,9(2),44-51.
[18]G.S.Pingali,Y.Jean,and I.Carlbom.Real time tracking for enhanced tennis broadcasts.Proc.IEEE Comp.Vision Patt.Reco(CVPR),1998:260-265.
[19]M.etkovic,Z.Zivkovic,W.Jonker.Recognizing strokes in tennis videos using Hidden Markov Models.In:Proceedings of IASTED International Conference on Visualization,Imaging and Image Processing,Spain,2001.
[20]LY.Xing,QX.Ye,WG.Zhang.A scheme for racquet sports video analysis with the combination of audio-visual information.In:Proceedings of International Conference on Visual Communications and Image Processing,Vol.5960,2005.
[21]http://www.jdl.ac.cn/en/project/mrhomepage/index.htm.
[22]http://www.sharptechnologyventures.com/tech/himpact.php.
[23]http://www.goalgle.com/.
[24]Zhang h j,Wu jianhua,Zhong di.An integrated system for content-based video retrieval and browsing pattern recognition.1997,30(4):643-657.
[25]A.nagasaka and Y.tanaka.Atutomatic video indexing and full-video search for object appearances,second working conference on visual database systems,IFIP WG2.6,october 1991.119-133.
[26]Arman F,Hsu A,Chiu M Y.Image processing on compressed video data for large video databases.ACM multimedia,1993,267-272.
[27]Patel Nilesh V,Sethi Ishawr K.Video shot detection and characterization for video databases.Pattern recognition,1997,30(4):583-592.
[28]王东辉,朱森良,吴春明.一种用于自动视频分段的WIPE转换检测和模式识别方法.计算机研究与发展,2002,39(2),247-253。
[29]Zhang Y J,Yao Y R,He Y,1998b.Color image segmentation based on HSI model.High Technology Letters,4(1):28-31.
[30]李国辉,柳伟,曹利华.一种基于颜色特征的图像检索方法.中国图像图形学报,Vol.4,No.3,1999-03:248-251.
[31]A.nagasaka,Y.tanaka.Atutomatic video indexing and full-video search for object appearances,second working conference on visual database systems,IFIP WG2.6,october 1991.119-133.
[32]张继东,陈都。基于内容的视频检索技术。电视技术,2002(8):17-19,23.
[33]Shahraray B,Gibbon D C.Automatic Generation of Pictorial Transcripts of Video retrieval and browsing.Pattern Recognition,1997(30):643-648.
[34]Zhang H J,Wu jianhua,Zhong di.An integrated system for content-based video retrieval and browsing pattern recognition.1997,30(4):643-657.
[35]Wolf Wayne.Key frame selection by motion analysis.In Proc.of IEEE Int.Conf.On Acoustics,Speech and Signal Processing,ICASSP,Atlanta,1996,7-10.
[36]P O Gresle,T S Huang.Gisting of Video Documents:A Key Frames Selection Algorithm Using Relative Activity Measure,in The 2nd Int.Conf.On Visual Information System,1997.
[37]Zhuang Y T,Rui Y,Huang T S,et al.Adaptive key frame extraction using unsupervised clustering[C].Chicago:Proceedings of the 1998 IEEE International Conference on Image Processing,1998.884-887.
[38]A Hanjalic,H Zhang.An Intergrated Scheme for Automated Video Abstraction Based on Unsupervised Cluster Validity Analysis,IEEE Transactions on Circuits and Systems for Video Technology,1999,9(8).
[39]Stricker M,Orengo M.Similarity of color images.SPIE Storage and Retrieval for Image and Video Databases Ⅲ,Feb.1995,2185:381-392.
[40]万旺根等.音频信息检索研究现状与发展趋势.上海大学学报,Vol.13,N0.4,2007.
[41]V.Vapnik.The nature of statistical learning theory.New York:Springer,1995.
[42]T.M.Cover.Geometrical and statistical properties of systems and linear inequalities with applications in patter recognition.IEEE Trans.On Electronic computers,1965(3):326-334.
[43]白亮.音频分类与分割技术研究.硕士学位论文.长沙:国防科技大学.2004.
[44]H.Pan,P.van Beek,and M.I.Sezan.Detection of Slow-Motion Replay Segments in Sports Video for Highlights Generation.Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing,2001.
[45]H.Pan,B.Li,and M.I.Sezan.Automatic Detection of Replay Segments in Broadcast Sports Programs by Detection of Logos in Scene Transitions.Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing,2002.
[46]Y.F.Ma,and H.J.Zhang.A Model of Motion Attention for Video Skimming. Proceeding of IEEE International Conference on Image Processing,Rochester,New York,September,2002.
[47]Y.F.Ma,L.Lu,H.J.Zhang,Mo Li.A User Attention Model for Video Summarization.Proceedings of ACM International Conference on Multimedia,December 2002.
[48]A,Hanjalic.Generic Approach to Highlights Extraction from a Sport Video.Proceedings of IEEE International Conference on Image Processing,2003.
[49]陈世举.篮球比赛视频分析关键技术研究。硕士学位论文.北京:北京工业大学,2006.
[50]Plataniotix K N,Venetsanopoulos A No Color Image Processing and Application.Berlin,Germany:Springer-Verlag,2000:25-32,260-275.
[51]Guo G D,Li S Z.Content-based audio classification and retrieval by support vector machines[J].IEEE Trans on Neural Networks,2003,14(1):209-215.
[52]Z Liu,J Huang,Y Wang,T Chen.Audio feature extraction and analysis for scence classification.IEEE Signal Processing Society 1997 Workshop on Multimedia Signal Processing.
[53]J.Foote.Content-based retrieval of music and audio.In:C.C.J.Kuo et al.(eds)Multimedia Storage and Archiving Systems Ⅱ,Proc.of SPIE,volume 3229,pp.138-147,1997.
[54]E.Wold,T.Blum,D.Keslar.Content-based classification,search,and retrieval of audio,IEEE Multimedia,Fall,1996,pp.27-36.
[55]Feiten B,Frank R,Ungvary To Organization of Sounds with Neural Nets.In:Proceedings of the 1991 International Computer Music Conference,International Computer Music Associatiom.San Francisco,1991,441-444.
[56]Andreas Rauder,Elias Pampalk.Using PsychoAcoustic Models and Self-Organizing Maps to Create a Hierarchical Structuring of Music by Sound Similarity.IRCAM,2002.
[57]卢坚,陈毅松,孙正兴,张福炎.语音/音乐自动分类中的特征分析.计算机辅助设计与图形学学报,Vol.14,No.3,2003.
[2]章毓晋.图像理解与计算机视觉。清华大学出版社,2000。
[3]庄越挺,潘云鹤,吴飞.网上多媒体信息分析与检索.清华大学出版社,2002.
[4]王扉.体育视频的内容分析技术研究.博士学位论文。北京:中国科学院研究生院,2005.
[5]赵亚琴.基于内容的视频片段检索技术与研究.博士学位论文。南京:南京理工大学,2006.
[6]S.W.Smoliar,H.J.Zhang.Content-Based Video Indexing and Retrieval.IEEE Multimedia,vol.1,no.2,pp.62-72,1994.
[7]D.Yow,et al.Analysis and presentation of soccer highlights from digital video.In Proceedings Asian Conference on Computer Vision,1995.
[8]Y.H.Gong,L.T.Sin,et al。Automatic parsing of soccer programs.In Proceedings IEEE International Conference Multimedia Computer System,1995:167-174.
[9]L.Xie,S.F.Chang,A.Divakaran,H.Sun.Structure analysis of soccer video with hidden Markov models.In Proceedings IEEE International Conference Acoustics,Speech,and Signal Processing,2002.
[10]A.Ekin,A.M.Tekalp.Shot type classification by dominant color for sports video segmentation and summarization.Acoustics,Speech,and Signal Processing,2003.Proceedings.(ICASSP'03).2003 IEEE International Conference on,6-10 April 2003vol.3:173-176.
[11]Ohno Y,Miurs J,Shirai Y.Tracking players and a ball in soccer games.Multisensor Fusion and Integration for Intelligent Systems,1999.MFI'99.Proceedings.1999 IEEE/SICE/RSJ International Conference on,15-18 Aug.1999:147-152.
[12]Ohno Y,Miura J,Shirai Y.Tracking players and estimation of the 3D position of a ball in soccer games.Pattern Recognition,2000.Proceedings.15th International Conference on,Volume 1,3-7 Sept.2000:145-148 vol.1.
[13]Jianyun Chen,Yunhao Li,Songyang Lao,Lingda Wu,A BSU-based sports video content analysis framework for content0based streams filtering.Computer Networks and Mobile Computing,2003.ICCNMC 2003.2003 International Conference on,20-23 Oct.2003:446-449
[14]W.S.Zhou,et al.Online knowledge and rule-based video classification system for video indexing and dissemination.Information System,2002(27):559-586.
[15]Y.Rui,A.Gupta,and A.Acero.Automatically extracting highlights for TV baseball programs.In Proceedings ACM Multimedia,2000.
[16]D.Zhang and S.F.Chang.Event Detection in Baseball Video Using Superimposed Caption Recognition.Proceedings of ACM International Conference on Multimedia,December 2002.
[17]R.Leonardi and P.Migliorati.Semantic indexing of multimedia documents.IEEE Multimedia,2002,9(2),44-51.
[18]G.S.Pingali,Y.Jean,and I.Carlbom.Real time tracking for enhanced tennis broadcasts.Proc.IEEE Comp.Vision Patt.Reco(CVPR),1998:260-265.
[19]M.etkovic,Z.Zivkovic,W.Jonker.Recognizing strokes in tennis videos using Hidden Markov Models.In:Proceedings of IASTED International Conference on Visualization,Imaging and Image Processing,Spain,2001.
[20]LY.Xing,QX.Ye,WG.Zhang.A scheme for racquet sports video analysis with the combination of audio-visual information.In:Proceedings of International Conference on Visual Communications and Image Processing,Vol.5960,2005.
[21]http://www.jdl.ac.cn/en/project/mrhomepage/index.htm.
[22]http://www.sharptechnologyventures.com/tech/himpact.php.
[23]http://www.goalgle.com/.
[24]Zhang h j,Wu jianhua,Zhong di.An integrated system for content-based video retrieval and browsing pattern recognition.1997,30(4):643-657.
[25]A.nagasaka and Y.tanaka.Atutomatic video indexing and full-video search for object appearances,second working conference on visual database systems,IFIP WG2.6,october 1991.119-133.
[26]Arman F,Hsu A,Chiu M Y.Image processing on compressed video data for large video databases.ACM multimedia,1993,267-272.
[27]Patel Nilesh V,Sethi Ishawr K.Video shot detection and characterization for video databases.Pattern recognition,1997,30(4):583-592.
[28]王东辉,朱森良,吴春明.一种用于自动视频分段的WIPE转换检测和模式识别方法.计算机研究与发展,2002,39(2),247-253。
[29]Zhang Y J,Yao Y R,He Y,1998b.Color image segmentation based on HSI model.High Technology Letters,4(1):28-31.
[30]李国辉,柳伟,曹利华.一种基于颜色特征的图像检索方法.中国图像图形学报,Vol.4,No.3,1999-03:248-251.
[31]A.nagasaka,Y.tanaka.Atutomatic video indexing and full-video search for object appearances,second working conference on visual database systems,IFIP WG2.6,october 1991.119-133.
[32]张继东,陈都。基于内容的视频检索技术。电视技术,2002(8):17-19,23.
[33]Shahraray B,Gibbon D C.Automatic Generation of Pictorial Transcripts of Video retrieval and browsing.Pattern Recognition,1997(30):643-648.
[34]Zhang H J,Wu jianhua,Zhong di.An integrated system for content-based video retrieval and browsing pattern recognition.1997,30(4):643-657.
[35]Wolf Wayne.Key frame selection by motion analysis.In Proc.of IEEE Int.Conf.On Acoustics,Speech and Signal Processing,ICASSP,Atlanta,1996,7-10.
[36]P O Gresle,T S Huang.Gisting of Video Documents:A Key Frames Selection Algorithm Using Relative Activity Measure,in The 2nd Int.Conf.On Visual Information System,1997.
[37]Zhuang Y T,Rui Y,Huang T S,et al.Adaptive key frame extraction using unsupervised clustering[C].Chicago:Proceedings of the 1998 IEEE International Conference on Image Processing,1998.884-887.
[38]A Hanjalic,H Zhang.An Intergrated Scheme for Automated Video Abstraction Based on Unsupervised Cluster Validity Analysis,IEEE Transactions on Circuits and Systems for Video Technology,1999,9(8).
[39]Stricker M,Orengo M.Similarity of color images.SPIE Storage and Retrieval for Image and Video Databases Ⅲ,Feb.1995,2185:381-392.
[40]万旺根等.音频信息检索研究现状与发展趋势.上海大学学报,Vol.13,N0.4,2007.
[41]V.Vapnik.The nature of statistical learning theory.New York:Springer,1995.
[42]T.M.Cover.Geometrical and statistical properties of systems and linear inequalities with applications in patter recognition.IEEE Trans.On Electronic computers,1965(3):326-334.
[43]白亮.音频分类与分割技术研究.硕士学位论文.长沙:国防科技大学.2004.
[44]H.Pan,P.van Beek,and M.I.Sezan.Detection of Slow-Motion Replay Segments in Sports Video for Highlights Generation.Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing,2001.
[45]H.Pan,B.Li,and M.I.Sezan.Automatic Detection of Replay Segments in Broadcast Sports Programs by Detection of Logos in Scene Transitions.Proceedings of IEEE International Conference on Acoustics,Speech,and Signal Processing,2002.
[46]Y.F.Ma,and H.J.Zhang.A Model of Motion Attention for Video Skimming. Proceeding of IEEE International Conference on Image Processing,Rochester,New York,September,2002.
[47]Y.F.Ma,L.Lu,H.J.Zhang,Mo Li.A User Attention Model for Video Summarization.Proceedings of ACM International Conference on Multimedia,December 2002.
[48]A,Hanjalic.Generic Approach to Highlights Extraction from a Sport Video.Proceedings of IEEE International Conference on Image Processing,2003.
[49]陈世举.篮球比赛视频分析关键技术研究。硕士学位论文.北京:北京工业大学,2006.
[50]Plataniotix K N,Venetsanopoulos A No Color Image Processing and Application.Berlin,Germany:Springer-Verlag,2000:25-32,260-275.
[51]Guo G D,Li S Z.Content-based audio classification and retrieval by support vector machines[J].IEEE Trans on Neural Networks,2003,14(1):209-215.
[52]Z Liu,J Huang,Y Wang,T Chen.Audio feature extraction and analysis for scence classification.IEEE Signal Processing Society 1997 Workshop on Multimedia Signal Processing.
[53]J.Foote.Content-based retrieval of music and audio.In:C.C.J.Kuo et al.(eds)Multimedia Storage and Archiving Systems Ⅱ,Proc.of SPIE,volume 3229,pp.138-147,1997.
[54]E.Wold,T.Blum,D.Keslar.Content-based classification,search,and retrieval of audio,IEEE Multimedia,Fall,1996,pp.27-36.
[55]Feiten B,Frank R,Ungvary To Organization of Sounds with Neural Nets.In:Proceedings of the 1991 International Computer Music Conference,International Computer Music Associatiom.San Francisco,1991,441-444.
[56]Andreas Rauder,Elias Pampalk.Using PsychoAcoustic Models and Self-Organizing Maps to Create a Hierarchical Structuring of Music by Sound Similarity.IRCAM,2002.
[57]卢坚,陈毅松,孙正兴,张福炎.语音/音乐自动分类中的特征分析.计算机辅助设计与图形学学报,Vol.14,No.3,2003.