跨媒体知识图谱构建中多模态数据语义相关性研究
|
摘要
[目的/意义]跨媒体知识图谱是解决跨媒体检索的重要方法之一。对多模态数据语义相关性研究,为跨媒体知识图谱的构建提供了一定的理论基础和发展方向。[方法/过程]以构建基于跨媒体的知识图谱为出发点,通过深入剖析知识图谱的内涵与构建技术,提出一种基于跨媒体数据内容的语义相关性分析模型。该模型充分利用媒体对象的高层语义的语义标签信息,将多媒态文档中的同模态对象提取出来,从而挖掘不同媒体内容间的语义关系。[结果/结论]实证结果表明,模型能够较为准确的发现不同模态数据对象间的语义相关性并将其关联起来。文章的研究对跨媒体知识图谱构建过程中实体的有效抽取及关系确立有一定的指导和帮助作用。
[Purpose/significance]Cross-media knowledge graph is one of the important methods to solve cross-media retrieval problems.The research on the semantic correlation of multimodal data provides certain theoretical basis and development direction for the construction of cross-media knowledge graph.[Method/process]This paper starts with the construction of cross-media knowledge graph,and constructs a semantic correlation analysis model based on cross-media data content by deeply analyzing the connotation and construction technology of knowledge graph.The model makes full use of the high-level semantic tag information of media objects to extract the same modal targets in multimedia documents,so as to explore the semantic relationship between different media content.[Result/conclusion]The empirical results show that the model can accurately discover the semantic correlation between multimodal data and correlate them.The paper also offers guidance and help for the effective extraction of entities and the establishment of entity relationship in the process of constructing cross-media knowledge graph.
[Purpose/significance]Cross-media knowledge graph is one of the important methods to solve cross-media retrieval problems.The research on the semantic correlation of multimodal data provides certain theoretical basis and development direction for the construction of cross-media knowledge graph.[Method/process]This paper starts with the construction of cross-media knowledge graph,and constructs a semantic correlation analysis model based on cross-media data content by deeply analyzing the connotation and construction technology of knowledge graph.The model makes full use of the high-level semantic tag information of media objects to extract the same modal targets in multimedia documents,so as to explore the semantic relationship between different media content.[Result/conclusion]The empirical results show that the model can accurately discover the semantic correlation between multimodal data and correlate them.The paper also offers guidance and help for the effective extraction of entities and the establishment of entity relationship in the process of constructing cross-media knowledge graph.
引文
[1]吴飞,庄越挺.互联网跨媒体分析与检索:理论与算法[J].计算机辅助设计与图形学学报,2010,22(1):1-9.
[2]张洋,谢卓力.基于多源网络学术信息聚合的知识图谱构建研究[J].图书情报工作,2014,58(22):84-94.
[3]FANG Q,XU C,SANG J,et al.Folksonomy-based visual ontology construction and its applications[J].IEEE Transactions on Multimedia,2016,18(4):702-713.
[4]ZHU Y,ZHANG C,RC,et al.Building a large-scale multimodal knowledge base system for answering visual queries[J].Computer Science,2015.
[5]刘峤,李杨,段宏,等.知识图谱构建技术综述[J].计算机研究与发展,2016,53(3):582-600.
[6]漆桂林,高桓,吴天星.知识图谱研究进展[J].情报工程,2017,3(1):4-25.
[7]ETZIONI O,CAFARELLA M,DOWNEY D,et al.Unsupervised named-entity extraction from the Web:An experimental study[J].Artificial Intelligence,2005,165(1):91-134.
[8]谭鹏许,张来顺.采用树自动机推理技术的信息抽取方法[J].计算机工程与应用,2010,46(16):153-156.
[9]LING Xiao,WELD D S.Fine-grained entity recognition[C]//Proc of the 26th conf on Association for the advancement of Artificial Intelligence.Menlo Park,CA:AAAI,2012:94-100
[10]赵军,刘康,周光有,等.开放式文本信息抽取[J].中文信息学报,2011,25(6):98-111.
[11]CARLSON A,BETTERIDGE J,KISIEL B,et al.Toward an architecture for never-ending language learning[C]//TwentyFourth AAAI Conference on Artificial Intelligence.AAAI Press,2010:1306-1313.
[12]CILIBRASI R L,VITNYI P M B.The Google Similarity Distance[J].IEEE Transactions on Knowledge&Data Engineering,2004,19(3):370-383.
[13]PENG Yuxin,et al.Cross-media analysis and reasoning:advances and directions[J].Frontiers of Information Technology&Electronic Engineering,2017,18(1):44-57.
[14]FENG F,WANG X,Li R.Cross-modal retrieval with correspondence autoencoder[C]//ACM MM.ACM,2014:7-16.
[15]ZHANG H,YUAN J,GAO X,et al.Boosting cross-media retrieval via visual-auditory feature analysis and relevance feedback[C]//ACM International Conference on Multimedia.ACM,2014:953-956.
[16]明均仁,何超.基于语义关联挖掘的数字图书馆跨媒体检索方法研究[J].图书情报工作,2013(7):101-105.
[17]李爱明.数字图书馆中基于语义关联挖掘的跨媒体检索研究:模型设计与实验分析[J].情报科学,2014(1):85-88.
[18]彭欣.基于深度学习的数字图书馆跨媒体语义检索方法研究[J].情报探索,2018,1(2):16-19.
[19]LU T,JIN Y,SU F,et al.Content-oriented multimedia document understanding through cross-media correlation[J].Multimedia Tools&Applications,2015,74(18):8105-8135.
[20]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.
[21]魏晗,李弼程,张瑞杰,等.图像语义提取方法研究[J].现代电子技术,2011,34(24):103-106.
[22]FOOTE J T.Content-based retrieval of music and audio[J].Proceedings of SPIE-The International Society for Optical Engineering,1997,3229:138-147.
[23]THOMAS H.Unsupervised learning by probabilistic latent semantic analysis[J].Machine Learning,2001,42(1-2):177-196.
[24]林婉霞.基于多模态融合和传播的跨模态信息检索算法[D].南京:南京大学,2012.
[25]李志欣,施智平,刘曦,等.建模连续视觉特征的图像语义标注方法[J].计算机辅助设计与图形学学报,2010,22(8):1412-1420.
[26]百度图片.百度[EB/OL].[2018-10-20].http://image.baidu.com/.
[27]音效素材.站长之家[EB/OL].[2018-10-20].http://sc.chinaz.com/yinxiao.
[2]张洋,谢卓力.基于多源网络学术信息聚合的知识图谱构建研究[J].图书情报工作,2014,58(22):84-94.
[3]FANG Q,XU C,SANG J,et al.Folksonomy-based visual ontology construction and its applications[J].IEEE Transactions on Multimedia,2016,18(4):702-713.
[4]ZHU Y,ZHANG C,RC,et al.Building a large-scale multimodal knowledge base system for answering visual queries[J].Computer Science,2015.
[5]刘峤,李杨,段宏,等.知识图谱构建技术综述[J].计算机研究与发展,2016,53(3):582-600.
[6]漆桂林,高桓,吴天星.知识图谱研究进展[J].情报工程,2017,3(1):4-25.
[7]ETZIONI O,CAFARELLA M,DOWNEY D,et al.Unsupervised named-entity extraction from the Web:An experimental study[J].Artificial Intelligence,2005,165(1):91-134.
[8]谭鹏许,张来顺.采用树自动机推理技术的信息抽取方法[J].计算机工程与应用,2010,46(16):153-156.
[9]LING Xiao,WELD D S.Fine-grained entity recognition[C]//Proc of the 26th conf on Association for the advancement of Artificial Intelligence.Menlo Park,CA:AAAI,2012:94-100
[10]赵军,刘康,周光有,等.开放式文本信息抽取[J].中文信息学报,2011,25(6):98-111.
[11]CARLSON A,BETTERIDGE J,KISIEL B,et al.Toward an architecture for never-ending language learning[C]//TwentyFourth AAAI Conference on Artificial Intelligence.AAAI Press,2010:1306-1313.
[12]CILIBRASI R L,VITNYI P M B.The Google Similarity Distance[J].IEEE Transactions on Knowledge&Data Engineering,2004,19(3):370-383.
[13]PENG Yuxin,et al.Cross-media analysis and reasoning:advances and directions[J].Frontiers of Information Technology&Electronic Engineering,2017,18(1):44-57.
[14]FENG F,WANG X,Li R.Cross-modal retrieval with correspondence autoencoder[C]//ACM MM.ACM,2014:7-16.
[15]ZHANG H,YUAN J,GAO X,et al.Boosting cross-media retrieval via visual-auditory feature analysis and relevance feedback[C]//ACM International Conference on Multimedia.ACM,2014:953-956.
[16]明均仁,何超.基于语义关联挖掘的数字图书馆跨媒体检索方法研究[J].图书情报工作,2013(7):101-105.
[17]李爱明.数字图书馆中基于语义关联挖掘的跨媒体检索研究:模型设计与实验分析[J].情报科学,2014(1):85-88.
[18]彭欣.基于深度学习的数字图书馆跨媒体语义检索方法研究[J].情报探索,2018,1(2):16-19.
[19]LU T,JIN Y,SU F,et al.Content-oriented multimedia document understanding through cross-media correlation[J].Multimedia Tools&Applications,2015,74(18):8105-8135.
[20]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.
[21]魏晗,李弼程,张瑞杰,等.图像语义提取方法研究[J].现代电子技术,2011,34(24):103-106.
[22]FOOTE J T.Content-based retrieval of music and audio[J].Proceedings of SPIE-The International Society for Optical Engineering,1997,3229:138-147.
[23]THOMAS H.Unsupervised learning by probabilistic latent semantic analysis[J].Machine Learning,2001,42(1-2):177-196.
[24]林婉霞.基于多模态融合和传播的跨模态信息检索算法[D].南京:南京大学,2012.
[25]李志欣,施智平,刘曦,等.建模连续视觉特征的图像语义标注方法[J].计算机辅助设计与图形学学报,2010,22(8):1412-1420.
[26]百度图片.百度[EB/OL].[2018-10-20].http://image.baidu.com/.
[27]音效素材.站长之家[EB/OL].[2018-10-20].http://sc.chinaz.com/yinxiao.