基于相关反馈的图像语义检索技术
|
摘要
随着多媒体、网络、数据库等技术的不断发展,图像逐渐成为一种非常重要的媒体表现手段。如何从海量图像数据中检索出所需的资源也就成为当前研究热点之一。
基于内容的图像检索技术近年来得到长足的发展,相关反馈技术的加入使得图像检索技术向前跨越了一大步。然而,基于内容的图像检索与反馈技术仍然不能完全体现出高层语义的特征,如何缩小这个“语义鸿沟”,如何结合语义和低层特征得到更好的检索效果,这也是目前乃至未来数年内研究的难点。
本文主要致力于图像语义检索方法的研究,着重研究了相关反馈技术,提出了一种基于语义矩阵相关反馈的图像语义检索方法,同时利用感兴趣区域提出了一种结合语义和低层特征进行相关反馈的图像语义检索方法。主要研究成果如下:
1)利用语义关键字和图像构成的语义矩阵完成了对图像的一个初步的语义提取及检索。首先将图像按照颜色特征进行量化,分块后获得图像的颜色语义,映射到图像的语义关键字上,再经过基于语义矩阵的反馈技术进行了检索。实验结果表明,该方法能够进行初步的语义提取,并能够对某些类的图像获得较好的检索效果。
2)提出了一种基于感兴趣区域及相关反馈技术的图像语义检索方法。利用感兴趣区域,结合语义和低层特征,通过提取感兴趣区域的低层特征及语义信息,减少了图像内容上的冗余信息,改进了相关反馈方法,减少了反馈次数,检索效率有所改观。
3)设计实现了一个图像检索原型系统。对基于语义矩阵相关反馈的语义检索和基于感兴趣区域相关反馈的语义检索进行了验证。
With the constant development of multimedia technology, network technology and database technology, image is becoming a very important means of media representation. How to retrieve the needed resources from the mass of data is accordingly becoming one of the current research hotspots.
Content-based image retrieval technology has made rapid progress in recent years, and it took a big step forward for image retrieval technology that relevance feedback technology went into this technology. However, content-based image retrieval and relevance feedback can’t still fully reflect the features of high-level semantics, how to narrow the "semantic gap", and how to combine the semantic with the low-level features for better retrieval results, which are the current problems as well as the next few years.
The dissertation put the focus on the research of semantic-based image retrieval as well as relevance feedback technology. It proposed a semantic extraction algorithm based on a semantic matrix, using feedback technology and the semantic concept, and it also proposed another relevance feedback algorithm based on the combination of semantic and region of interest(ROI), which improved the efficiency of the semantic retrieval. The main research are as follows:
(1) The dissertation proposed an initial extraction and semantic retrieval method using the semantic keywords and semantic matrix which constituted by images. First of all, we quantified the image by the color, accordingly, we got the color of every block, which was mapped to the semantic keyword, and then we extracted semantic meaning of images through the feedback technology based on semantic matrix, finally, we retrieved the image from the image groups using this method. The result showed that this method could initially extract the semantic of images, and it could obtain better retrieval results for the certain types of images.
(2) The dissertation proposed a means of semantic-based image retrieval based on relevance feedback and ROI. We added the element of ROI to the presentation of low-level features, and took advantage of the relation of semantic features to ROI, consequently, we improved the feedback process, and appropriately reduced the number of feedback, and the retrieval efficiency could be improved.
(3) The dissertation designed an image retrieval system. The system fulfilled the two means above.
基于内容的图像检索技术近年来得到长足的发展,相关反馈技术的加入使得图像检索技术向前跨越了一大步。然而,基于内容的图像检索与反馈技术仍然不能完全体现出高层语义的特征,如何缩小这个“语义鸿沟”,如何结合语义和低层特征得到更好的检索效果,这也是目前乃至未来数年内研究的难点。
本文主要致力于图像语义检索方法的研究,着重研究了相关反馈技术,提出了一种基于语义矩阵相关反馈的图像语义检索方法,同时利用感兴趣区域提出了一种结合语义和低层特征进行相关反馈的图像语义检索方法。主要研究成果如下:
1)利用语义关键字和图像构成的语义矩阵完成了对图像的一个初步的语义提取及检索。首先将图像按照颜色特征进行量化,分块后获得图像的颜色语义,映射到图像的语义关键字上,再经过基于语义矩阵的反馈技术进行了检索。实验结果表明,该方法能够进行初步的语义提取,并能够对某些类的图像获得较好的检索效果。
2)提出了一种基于感兴趣区域及相关反馈技术的图像语义检索方法。利用感兴趣区域,结合语义和低层特征,通过提取感兴趣区域的低层特征及语义信息,减少了图像内容上的冗余信息,改进了相关反馈方法,减少了反馈次数,检索效率有所改观。
3)设计实现了一个图像检索原型系统。对基于语义矩阵相关反馈的语义检索和基于感兴趣区域相关反馈的语义检索进行了验证。
With the constant development of multimedia technology, network technology and database technology, image is becoming a very important means of media representation. How to retrieve the needed resources from the mass of data is accordingly becoming one of the current research hotspots.
Content-based image retrieval technology has made rapid progress in recent years, and it took a big step forward for image retrieval technology that relevance feedback technology went into this technology. However, content-based image retrieval and relevance feedback can’t still fully reflect the features of high-level semantics, how to narrow the "semantic gap", and how to combine the semantic with the low-level features for better retrieval results, which are the current problems as well as the next few years.
The dissertation put the focus on the research of semantic-based image retrieval as well as relevance feedback technology. It proposed a semantic extraction algorithm based on a semantic matrix, using feedback technology and the semantic concept, and it also proposed another relevance feedback algorithm based on the combination of semantic and region of interest(ROI), which improved the efficiency of the semantic retrieval. The main research are as follows:
(1) The dissertation proposed an initial extraction and semantic retrieval method using the semantic keywords and semantic matrix which constituted by images. First of all, we quantified the image by the color, accordingly, we got the color of every block, which was mapped to the semantic keyword, and then we extracted semantic meaning of images through the feedback technology based on semantic matrix, finally, we retrieved the image from the image groups using this method. The result showed that this method could initially extract the semantic of images, and it could obtain better retrieval results for the certain types of images.
(2) The dissertation proposed a means of semantic-based image retrieval based on relevance feedback and ROI. We added the element of ROI to the presentation of low-level features, and took advantage of the relation of semantic features to ROI, consequently, we improved the feedback process, and appropriately reduced the number of feedback, and the retrieval efficiency could be improved.
(3) The dissertation designed an image retrieval system. The system fulfilled the two means above.
引文
[1] Rui Y,Huang T S,Ortega. et al.Relevance feedback:A power tool for interactive content-based image retrieval. IEEE Transactions on Circuits and Video Technology,1998,8(5):1~13
[2] Ishikawa Y,Subramanya R,Faloutsos C.Mindreader: Query databases through multiple examples.Proceedings of the 24th VLDB conference,1998:218~227
[3] Rui Y,Huang T S ,Mehrotra S.Relevance Feedback:a Powerful Tool in Interactive Content-based Image Retrieval [J].IEEE Trans,1998,8(5):644-655
[4] Cox I J,Miller T S,Minka T P ,et al.The Bayesian Image Retrieval System, PicHunter, Theory, Implementation,and Psychophysical Experiments. IEEE Trans on Image Processing,2000,9(1)
[5] Wu H,Lu H,Ma S D.The Role of Sample Distribution in Relevance Feedback for Content-Based Image Retrieval.In:Proceedings of IEEE International Conference on Multimedia and Expo,Lausanne,Switzerland,2002
[6]王崇骏,杨育彬,陈世福.基于高层语义的图像检索算法.软件学报,2004(10)
[7] Qian F,Li M J,Zhang L,Zhang H J,Zhang B.Gaussian Mixture Model for Relevance Feedback in Image Retrieval[J].IEEE conf. on ICME,2002:356-366
[8] Wang J Z and Li J.Learning-based Linguistic Indexing of Pictures with 2-D MHMMs [J].ACM conference on Multimedia,2002:45-67
[9] Zhang L,Lin F Z ,Zhang B.Support Vector Machine Learning for Image Retrieval [J].IEEE Conference on Image Processing,2001:15-26
[10] Tong S,Chang E.Support Vector Machine Active Learning for Image Retrieval [J].Proc.of ACM Multimedia,Ottawa, Canada,2001:145-156
[11] Jing Li,Nigel Allinson,Dacheng Tao,Xuelong Li.Multitraining Support Vector Machine for Image Retrieval. IEEE Transactions on Image Processing,Vol. 15, No. 11,November,2006
[12] Dacheng Tao, Xiaoou Tang, Xuelong Li, and Xindong Wu.Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval.,IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol 28,No 7,July 2006
[13]朱兴全,张宏江,刘文印,吴立德.iFind:一个结合语义和视觉特征的图像相关反馈检索系统.计算机学报,2002,25(7):681-688
[14]余卫宇,曹燕,余英林,基于语义的图像物体提取的新方法.微计算机信息(管控一体化),2007,23(7-3):211-213
[15]黄元元,郭丽,杨静宇.基于主色调匹配的图像检索方法.计算机工程,2002,Vol.28,No.6
[16] M Stricker and M Orengo.Similarity of color images.SPIE Storage and Retrieval for Image and Video Databases III,Feb 1995,Vo12 185:381-392
[17] G Pass and R Zabih. Histogram refinement for content-based image retrieval.IEEE Work shop on Application of Computer Vision,1996:96-102
[18] H Tamura,S Mori and T Yamawaki.Texture features corresponding to visualperception.IEEE Trans On Systems,Man and Cybernetics,June 1978,Vol Smc-8,No6
[19] Calvin C Gotlieb and Herbert E Kreyszig. Texture descriptors based on co-occurrence matrices. Comput Vision and Image Proc,1990,51:70-86
[20] Cross G R, Jain A K. Markov random field texture models. IEEE-PAMI1983, 5(1): 25-39
[21] T Chang and C C Jay Kuo.Texture analysis and classification withtree-structured wavelet transform.IEEE Trans On Image Processing,October 1993,Vol 2, No4:429-441
[22] M K Hu. Visual pattern recognition by moment invariants.In:J K Aggarwal, R O Duda and A Rosenfeld.Computer Methodsin image analysis.Los Angles,CA:IEEE computer Society,1997
[23] L Yang and F Algregtsen. Fast computation of invariant geometric moments: A new method giving correct results. Proc IEEE Int Conf On Image Processing,1994
[24] Deepak Kapur,Y N Lakshman and Tushar Saxena. Computing invariants using elimination methods In Proc IEEE Int Conf on Image roc,1995
[25] Yueting Zhuang, Intelligent multimedia information analysis and retrieval with application stovisual design.PhD thesis,Zhejiang University,1998
[26]邬浩,潘云鹤,庄越挺,杨宇艇.基于对象形状的图像查询技术.软件学报,1998年5月,Vo19,No5:343-349
[27] A Pentland and R.W.Picard and S Sclaroff. Photobook:Content-based manipulation of image databases.Int J Comput Vis,1996,18(3):233-254
[28] Esther M Arkin,L Chew,D Huttenlocher,K Kedem and J Mitchell. An efficienly computable metric for comparing polygonal shapes.IEEE Trans Patt Recog And Mach Intell, March 1991,13(3)
[29] Gene C H Chuang and C C Jay Koo.Wavelet descriptor of planar curves: Theory and applications. IEEE Trans Image Proc, Jan 1996,5(l):56-70
[30] Hermes Tetal.Image retrieval for information systems in Storage and Retrieval for Image and Video Databases III[C].Proc SPIE,1995,24(20):394-405
[31] Voorhees E.Using WordNet to Disambiguate Word Senses for Text Retrieval[C].in Proc16th Annual ACM SIGIR Conference on Research and Development in Information Retrieval Pittsburgh,1993,1(1):171-180
[32] Zhuang Y,Mehrotra S,and Huang T S.A multimedia information retrieval modelbased on semantic and visual content[C].In Proceedings of the 5th International ICYCS Conference,1999,6(1):123-129
[33] Hongming Zhang,Wen Gao,Xilin Chen,Debin Zhao.Object detection usingspatial histogram features[J].Journal of Image andVisionComputing,2006,24(4):327-341
[34] YongHong Tian. Context-Based Statistical Relational Learning. AI Communications, 2006, 19(3):291-293
[35] Cavazza M,Green R J and Palmer I J.Multimedia Semantic Features and Image Content Description[C].In Proceedings of the 1998 MultiMedia Modeling, 1998,3(5):91-97
[36] Reguirements Group.MPEG-7 requirements document[C]. ISO/IECJTC1/SC29/ WG11 MPEG99/N4035, 2001,1(1):1-120
[37] Paek S,Chang S F,Puri A,Huang Q,Smith J R,Li C S,et al.Object-based multimedia content description schemes and applications for MPEG-7[C].Signal Processing Image Communication,2000,16(2):235-269
[38] Reguirements Group.MPEG-7 DDL development document V.2[C].ISO/IEC JTC1/SC29/WG11 MPEG99/N2997 Melbourne Australia,1999,10(1):1-130
[39] Kherfi M L,Ziou D,Bernardi A.Learning from negative example in relevance feedback for content-based image retrieval[C].In:Proceed-ings of 16th International Conference on Pattern Recognition,2002:2:933-936
[40] Sheng-Rong Gong,Zhao-Hui Wang,Jian-Min Zhao.A learning strategy in CBIR system design[C].In:Proceedings of International Conference on Machine Learning and Cybermetics,2002:2:754-756
[41] Z.Su, H.J.Zhang, S.P.Ma. relevance feedback using a bayesian classifier in content-based image retrieval,In SPIE,2000
[42] Peng-Yeng Yin,Bir Bhanu,Fellow,IEEE,Kuang-Cheng Chang,and Anlei Dong.Integrating Relevance Feedback Techniques for Image Retrieval Using Reinforcement Learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005,27(10):1536-1551
[43] Sudhakara Reddy P,Raju S.Bapi,Chakravarthy Bhagvati,B.L.Deekshatulu. Concept Pre-digestion Method for Image Relevance Reinforcement Learning.Proceedings of the International Conference on Computing:Theory and Applications(ICCTA’07).IEEE,2007
[44]王南,赵捧未,窦永香,秦春秀,赵飞.图像语义检索中的反馈噪声及其抑制算法研究,2007,10:42-46
[45] H.Moravec.Toward automatic visual obstacle avoidance, Proc.of 5th Int.Joint /conf.on Artificial Intelligence.1977,584
[46] B.Moghaddam, H.Bienman, and D.Margaritis. Defining image content with multiple regions-of-interest.IEEE Workshop on Content-Based Access of Imageand Video Libraries.1999,6
[47] H Tamura, S Mori and T Yamawaki. Texture features corresponding to visual perception. IEEE Trans On Systems, Man and Cybernetics, June 1978,Vol Smc-8, No 6
[48] Jie Luo. Content-based sub-image retrieval using relevance feedback. Technical Report, Dept.of Computer Science, University of Alberta Edmonton, Alberta, Canada,July 2004
[49]李洁,丁颖.语义网、语义网格和语义网络,计算机与现代化,2007,7.
[50]沈玉利,郭雷,耿苑.一种新型图像检索语义网络构建方法.计算机应用研究,2005,148-150,156.
[2] Ishikawa Y,Subramanya R,Faloutsos C.Mindreader: Query databases through multiple examples.Proceedings of the 24th VLDB conference,1998:218~227
[3] Rui Y,Huang T S ,Mehrotra S.Relevance Feedback:a Powerful Tool in Interactive Content-based Image Retrieval [J].IEEE Trans,1998,8(5):644-655
[4] Cox I J,Miller T S,Minka T P ,et al.The Bayesian Image Retrieval System, PicHunter, Theory, Implementation,and Psychophysical Experiments. IEEE Trans on Image Processing,2000,9(1)
[5] Wu H,Lu H,Ma S D.The Role of Sample Distribution in Relevance Feedback for Content-Based Image Retrieval.In:Proceedings of IEEE International Conference on Multimedia and Expo,Lausanne,Switzerland,2002
[6]王崇骏,杨育彬,陈世福.基于高层语义的图像检索算法.软件学报,2004(10)
[7] Qian F,Li M J,Zhang L,Zhang H J,Zhang B.Gaussian Mixture Model for Relevance Feedback in Image Retrieval[J].IEEE conf. on ICME,2002:356-366
[8] Wang J Z and Li J.Learning-based Linguistic Indexing of Pictures with 2-D MHMMs [J].ACM conference on Multimedia,2002:45-67
[9] Zhang L,Lin F Z ,Zhang B.Support Vector Machine Learning for Image Retrieval [J].IEEE Conference on Image Processing,2001:15-26
[10] Tong S,Chang E.Support Vector Machine Active Learning for Image Retrieval [J].Proc.of ACM Multimedia,Ottawa, Canada,2001:145-156
[11] Jing Li,Nigel Allinson,Dacheng Tao,Xuelong Li.Multitraining Support Vector Machine for Image Retrieval. IEEE Transactions on Image Processing,Vol. 15, No. 11,November,2006
[12] Dacheng Tao, Xiaoou Tang, Xuelong Li, and Xindong Wu.Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval.,IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol 28,No 7,July 2006
[13]朱兴全,张宏江,刘文印,吴立德.iFind:一个结合语义和视觉特征的图像相关反馈检索系统.计算机学报,2002,25(7):681-688
[14]余卫宇,曹燕,余英林,基于语义的图像物体提取的新方法.微计算机信息(管控一体化),2007,23(7-3):211-213
[15]黄元元,郭丽,杨静宇.基于主色调匹配的图像检索方法.计算机工程,2002,Vol.28,No.6
[16] M Stricker and M Orengo.Similarity of color images.SPIE Storage and Retrieval for Image and Video Databases III,Feb 1995,Vo12 185:381-392
[17] G Pass and R Zabih. Histogram refinement for content-based image retrieval.IEEE Work shop on Application of Computer Vision,1996:96-102
[18] H Tamura,S Mori and T Yamawaki.Texture features corresponding to visualperception.IEEE Trans On Systems,Man and Cybernetics,June 1978,Vol Smc-8,No6
[19] Calvin C Gotlieb and Herbert E Kreyszig. Texture descriptors based on co-occurrence matrices. Comput Vision and Image Proc,1990,51:70-86
[20] Cross G R, Jain A K. Markov random field texture models. IEEE-PAMI1983, 5(1): 25-39
[21] T Chang and C C Jay Kuo.Texture analysis and classification withtree-structured wavelet transform.IEEE Trans On Image Processing,October 1993,Vol 2, No4:429-441
[22] M K Hu. Visual pattern recognition by moment invariants.In:J K Aggarwal, R O Duda and A Rosenfeld.Computer Methodsin image analysis.Los Angles,CA:IEEE computer Society,1997
[23] L Yang and F Algregtsen. Fast computation of invariant geometric moments: A new method giving correct results. Proc IEEE Int Conf On Image Processing,1994
[24] Deepak Kapur,Y N Lakshman and Tushar Saxena. Computing invariants using elimination methods In Proc IEEE Int Conf on Image roc,1995
[25] Yueting Zhuang, Intelligent multimedia information analysis and retrieval with application stovisual design.PhD thesis,Zhejiang University,1998
[26]邬浩,潘云鹤,庄越挺,杨宇艇.基于对象形状的图像查询技术.软件学报,1998年5月,Vo19,No5:343-349
[27] A Pentland and R.W.Picard and S Sclaroff. Photobook:Content-based manipulation of image databases.Int J Comput Vis,1996,18(3):233-254
[28] Esther M Arkin,L Chew,D Huttenlocher,K Kedem and J Mitchell. An efficienly computable metric for comparing polygonal shapes.IEEE Trans Patt Recog And Mach Intell, March 1991,13(3)
[29] Gene C H Chuang and C C Jay Koo.Wavelet descriptor of planar curves: Theory and applications. IEEE Trans Image Proc, Jan 1996,5(l):56-70
[30] Hermes Tetal.Image retrieval for information systems in Storage and Retrieval for Image and Video Databases III[C].Proc SPIE,1995,24(20):394-405
[31] Voorhees E.Using WordNet to Disambiguate Word Senses for Text Retrieval[C].in Proc16th Annual ACM SIGIR Conference on Research and Development in Information Retrieval Pittsburgh,1993,1(1):171-180
[32] Zhuang Y,Mehrotra S,and Huang T S.A multimedia information retrieval modelbased on semantic and visual content[C].In Proceedings of the 5th International ICYCS Conference,1999,6(1):123-129
[33] Hongming Zhang,Wen Gao,Xilin Chen,Debin Zhao.Object detection usingspatial histogram features[J].Journal of Image andVisionComputing,2006,24(4):327-341
[34] YongHong Tian. Context-Based Statistical Relational Learning. AI Communications, 2006, 19(3):291-293
[35] Cavazza M,Green R J and Palmer I J.Multimedia Semantic Features and Image Content Description[C].In Proceedings of the 1998 MultiMedia Modeling, 1998,3(5):91-97
[36] Reguirements Group.MPEG-7 requirements document[C]. ISO/IECJTC1/SC29/ WG11 MPEG99/N4035, 2001,1(1):1-120
[37] Paek S,Chang S F,Puri A,Huang Q,Smith J R,Li C S,et al.Object-based multimedia content description schemes and applications for MPEG-7[C].Signal Processing Image Communication,2000,16(2):235-269
[38] Reguirements Group.MPEG-7 DDL development document V.2[C].ISO/IEC JTC1/SC29/WG11 MPEG99/N2997 Melbourne Australia,1999,10(1):1-130
[39] Kherfi M L,Ziou D,Bernardi A.Learning from negative example in relevance feedback for content-based image retrieval[C].In:Proceed-ings of 16th International Conference on Pattern Recognition,2002:2:933-936
[40] Sheng-Rong Gong,Zhao-Hui Wang,Jian-Min Zhao.A learning strategy in CBIR system design[C].In:Proceedings of International Conference on Machine Learning and Cybermetics,2002:2:754-756
[41] Z.Su, H.J.Zhang, S.P.Ma. relevance feedback using a bayesian classifier in content-based image retrieval,In SPIE,2000
[42] Peng-Yeng Yin,Bir Bhanu,Fellow,IEEE,Kuang-Cheng Chang,and Anlei Dong.Integrating Relevance Feedback Techniques for Image Retrieval Using Reinforcement Learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005,27(10):1536-1551
[43] Sudhakara Reddy P,Raju S.Bapi,Chakravarthy Bhagvati,B.L.Deekshatulu. Concept Pre-digestion Method for Image Relevance Reinforcement Learning.Proceedings of the International Conference on Computing:Theory and Applications(ICCTA’07).IEEE,2007
[44]王南,赵捧未,窦永香,秦春秀,赵飞.图像语义检索中的反馈噪声及其抑制算法研究,2007,10:42-46
[45] H.Moravec.Toward automatic visual obstacle avoidance, Proc.of 5th Int.Joint /conf.on Artificial Intelligence.1977,584
[46] B.Moghaddam, H.Bienman, and D.Margaritis. Defining image content with multiple regions-of-interest.IEEE Workshop on Content-Based Access of Imageand Video Libraries.1999,6
[47] H Tamura, S Mori and T Yamawaki. Texture features corresponding to visual perception. IEEE Trans On Systems, Man and Cybernetics, June 1978,Vol Smc-8, No 6
[48] Jie Luo. Content-based sub-image retrieval using relevance feedback. Technical Report, Dept.of Computer Science, University of Alberta Edmonton, Alberta, Canada,July 2004
[49]李洁,丁颖.语义网、语义网格和语义网络,计算机与现代化,2007,7.
[50]沈玉利,郭雷,耿苑.一种新型图像检索语义网络构建方法.计算机应用研究,2005,148-150,156.