基于手绘轮廓图的移动端图像检索
|
摘要
针对传统利用图像特征信息进行图像检索中难以从语义层次上理解图像相似性的问题,基于深度学习框架,提出一种结合类别分类和精确特征匹配的基于手绘轮廓图的移动端图像检索方法.首先在预处理阶段建立具有输入层、隐藏层以及Softmax输出层的神经网络分类模型,并利用训练数据集对模型进行训练,使其不断优化网络结构权值,实现输入图像的分类预测并提取分类图像标签;然后利用VGG16模型与ResNet50模型分别提取各个分类图像集下的精确特征,得到精确特征向量;最后将归一化并经组合后的特征向量与各个分类图像标签建立映射关系,实现移动端图像检索.采用移动端-服务器架构,用户在移动端输入手绘轮廓图后,系统进行自动预处理并与图像服务器实现交互,图像服务器进行分类预测和精确特征匹配得到检索结果,移动端展示最终检索结果.基于Keras深度学习开发框架,结合VGG16模型与ResNet50模型,实验结果表明,该方法能够根据手绘轮廓图高效、便捷地检索得到目标图像.
Traditional low level features based image retrieval techniques usually have some difficulties on understanding the image similarity from the high level semantic information. To overcome this issue, under the deep-learning framework, a novel hand-sketching contour based image retrieval method on mobile devices is presented in this paper by combining image classification and exact retrieval steps. Firstly, a neural network of image classification is built including the input layer, the hidden layer and the Softmax output layer, which would be trained by image dataset. It will tell which class the input contour image belongs to after training and gets the classification label. Secondly, the VGG16 model and ResNet50 model can be loaded, by which the exact image features of each class can be extracted. Finally, a map between the combinational feature vectors and the image classification labels can be built for the purpose of image retrieval on mobile devices. Based on the C/S structure, the proposed image retrieval system would exchange data with server automatically after mobile device got the contours of input hand-sketching images. And according to the feature index and network model, the server would return the retrieval results. Using the VGG16 model and ResNet50 model loaded with Keras framework, our approach can retrieve images generated by hand-sketching contours efficiently and conveniently.
Traditional low level features based image retrieval techniques usually have some difficulties on understanding the image similarity from the high level semantic information. To overcome this issue, under the deep-learning framework, a novel hand-sketching contour based image retrieval method on mobile devices is presented in this paper by combining image classification and exact retrieval steps. Firstly, a neural network of image classification is built including the input layer, the hidden layer and the Softmax output layer, which would be trained by image dataset. It will tell which class the input contour image belongs to after training and gets the classification label. Secondly, the VGG16 model and ResNet50 model can be loaded, by which the exact image features of each class can be extracted. Finally, a map between the combinational feature vectors and the image classification labels can be built for the purpose of image retrieval on mobile devices. Based on the C/S structure, the proposed image retrieval system would exchange data with server automatically after mobile device got the contours of input hand-sketching images. And according to the feature index and network model, the server would return the retrieval results. Using the VGG16 model and ResNet50 model loaded with Keras framework, our approach can retrieve images generated by hand-sketching contours efficiently and conveniently.
引文
[1]Rui Y,Huang T S,Chang S F.Image retrieval:current techniques,promising directions,and open issues[J].Journal of Visual Communication and Image Representation,1999,10(1):39-62
[2]Gordo A,Almazan J,Revaud J,et al.End-to-end learning of deep visual representations for image retrieval[J].International Journal of Computer Vision,2017,124(2):237-254
[3]Zheng L,Yang Y,Tian Q.SIFT meets CNN:a decade survey of instance retrieval[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(5):1224-1244
[4]Lowe D G.Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision.Los Alamitos:IEEEComputer Society Press,1999:1150-1157
[5]Bay H,Ess A,Tuytelaars T,et al.Speeded-up robust features(SURF)[J].Computer Vision and Image Understanding,2008,110(3):346-359
[6]Oliva A,Torralba A.Modeling the shape of the scene:a holistic representation of the spatial envelope[J].International Journal of Computer Vision,2001,42(3):145-175
[7]Tian Hong,Yang Shugang.Color image retrieval algorithm based on significant bit-planes[J].Journal of Computer-Aided Design&Computer Graphics,2010,22(2):279-285(in Chinese)(田宏,杨树刚.基于重要位平面的真彩色图像检索算法[J].计算机辅助设计与图形学学报,2010,22(2):279-285)
[8]Chen Tianding.Color indexing by wavelet-based image salient point extraction algorithm[J].Journal of Computer-Aided Design&Computer Graphics,2003,15(6):706-710(in Chinese)(陈添丁.小波图像色彩索引的凸点提取算法[J].计算机辅助设计与图形学学报,2003,15(6):706-710)
[9]Chen Jiazhou,Hu Wenwen,Miao Yongwei,et al.Pattern analysis and digitization modeling of papercutting textures[J].Journal of Computer-Aided Design&Computer Graphics,2016,28(9):1465-1475(in Chinese)(陈佳舟,胡文文,缪永伟,等.剪纸图案的构造模式分析和数字化建模[J].计算机辅助设计与图形学学报,2016,28(9):1465-1475)
[10]Goswami G,Ratha N,Agarwal A,et al.Unravelling robustness of deep learning based face recognition against adversarial attacks[OL].[2018-06-20].https://arxiv.org/pdf/1803.00401.pdf
[11]Xiao X F,Jin L W,Yang Y F,et al.Building fast and compact convolutional neural networks for offline handwritten Chinese character recognition[J].Pattern Recognition,2017,72:72-81
[12]Hu W,Huang Y Y,Wei L,et al.Deep convolutional neural networks for hyperspectral image classification[J].Journal of Sensors,2015,2015:Article No.258619
[13]Seuret M,Alberti M,Liwicki M,et al.PCA-initialized deep neural networks applied to document image analysis[C]//Proceedings of 14th IAPR IEEE International Conference on Document Analysis and Recognition.Los Alamitos:IEEEComputer Society Press,2017,1:877-882
[14]Perez Lara C,Lux M,Mejia-Lavalle M.Toward improving content-based image retrieval systems by means of text detection[C]//Proceedings of the IEEE International Conference on Mechatronics,Electronics and Automotive Engineering.Los Alamitos:IEEE Computer Society Press,2014:50-53
[15]Makadia A,Pavlovic V,Kumar S.Baselines for image annotation[J].International Journal of Computer Vision,2010,90(1):88-105
[16]Miller G A.Introduction to word net:an on-line lexical database[J].International Journal of Lexicography,1990,3(4):235-244
[17]K?l?n?D,Alpkocak A.An expansion and reranking approach for annotation-based image retrieval from Web[J].Expert Systems with Applications,2011,38(10):13121-13127
[18]Pentland A P,Picard R W,Scarloff S.Photobook:tools for content-based manipulation of image databases[C]//Proceedings of Storage and Retrieval for Image and Video Databases II.Bellingham:Society of Photo-Optical Instrumentation Engineers,1994,2185:34-48
[19]Smith J R,Chang S F.VisualSEEk:a fully automated content-based image query system[C]//Proceedings of the 4th ACM International Conference on Multimedia.New York:ACM Press,1996:87-98
[20]Smeulders A W M,Worring M,Santini S,et al.Content based image retrieval at the end of the early years[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2000,22(12):1349-1380
[21]Li X R,Uricchio T,Ballan L,et al.Socializing the semantic gap:a comparative survey on image tag assignment,refinement,and retrieval[J].ACM Computing Surveys,2016,49(1):Article No.14
[22]Cao Y,Wang H,Wang C H,et al.MindFinder:interactive sketch-based image search on millions of images[C]//Proceedings of the 18th ACM International Conference on Multimedia.New York:ACM Press,2010:1605-1608
[23]Cao Y,Wang C H,Zhang L Q,et al.Edgel index for large-scale sketch-based image search[C]//Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2011:761-768
[24]Chen T,Cheng M M,Tan P,et al.Sketch2Photo:internet image montage[J].ACM Transactions on Graphics,2009,28(5):Article No.124
[25]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[OL].[2018-06-20].https://arxiv.org/pdf/1409.1556.pdf
[26]He K M,Zhang X Y,Ren S Q,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEEComputer Society Press,2016:770-778
[27]Deng J,Dong W,Socher R,et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2009:248-255
[28]Chatzichristofis S A,Boutalis Y S.CEDD:color and edge directivity descriptor:a compact descriptor for image indexing and retrieval[C]//Proceedings of the International Conference on Computer Vision Systems.Heidelberg:Springer,2008,5008:312-322
[29]Bosch A,Zisserman A,Munoz X.Image classification using random forests and ferns[C]//Proceedings of the 11th IEEEInternational Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2007:1-8
[2]Gordo A,Almazan J,Revaud J,et al.End-to-end learning of deep visual representations for image retrieval[J].International Journal of Computer Vision,2017,124(2):237-254
[3]Zheng L,Yang Y,Tian Q.SIFT meets CNN:a decade survey of instance retrieval[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(5):1224-1244
[4]Lowe D G.Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision.Los Alamitos:IEEEComputer Society Press,1999:1150-1157
[5]Bay H,Ess A,Tuytelaars T,et al.Speeded-up robust features(SURF)[J].Computer Vision and Image Understanding,2008,110(3):346-359
[6]Oliva A,Torralba A.Modeling the shape of the scene:a holistic representation of the spatial envelope[J].International Journal of Computer Vision,2001,42(3):145-175
[7]Tian Hong,Yang Shugang.Color image retrieval algorithm based on significant bit-planes[J].Journal of Computer-Aided Design&Computer Graphics,2010,22(2):279-285(in Chinese)(田宏,杨树刚.基于重要位平面的真彩色图像检索算法[J].计算机辅助设计与图形学学报,2010,22(2):279-285)
[8]Chen Tianding.Color indexing by wavelet-based image salient point extraction algorithm[J].Journal of Computer-Aided Design&Computer Graphics,2003,15(6):706-710(in Chinese)(陈添丁.小波图像色彩索引的凸点提取算法[J].计算机辅助设计与图形学学报,2003,15(6):706-710)
[9]Chen Jiazhou,Hu Wenwen,Miao Yongwei,et al.Pattern analysis and digitization modeling of papercutting textures[J].Journal of Computer-Aided Design&Computer Graphics,2016,28(9):1465-1475(in Chinese)(陈佳舟,胡文文,缪永伟,等.剪纸图案的构造模式分析和数字化建模[J].计算机辅助设计与图形学学报,2016,28(9):1465-1475)
[10]Goswami G,Ratha N,Agarwal A,et al.Unravelling robustness of deep learning based face recognition against adversarial attacks[OL].[2018-06-20].https://arxiv.org/pdf/1803.00401.pdf
[11]Xiao X F,Jin L W,Yang Y F,et al.Building fast and compact convolutional neural networks for offline handwritten Chinese character recognition[J].Pattern Recognition,2017,72:72-81
[12]Hu W,Huang Y Y,Wei L,et al.Deep convolutional neural networks for hyperspectral image classification[J].Journal of Sensors,2015,2015:Article No.258619
[13]Seuret M,Alberti M,Liwicki M,et al.PCA-initialized deep neural networks applied to document image analysis[C]//Proceedings of 14th IAPR IEEE International Conference on Document Analysis and Recognition.Los Alamitos:IEEEComputer Society Press,2017,1:877-882
[14]Perez Lara C,Lux M,Mejia-Lavalle M.Toward improving content-based image retrieval systems by means of text detection[C]//Proceedings of the IEEE International Conference on Mechatronics,Electronics and Automotive Engineering.Los Alamitos:IEEE Computer Society Press,2014:50-53
[15]Makadia A,Pavlovic V,Kumar S.Baselines for image annotation[J].International Journal of Computer Vision,2010,90(1):88-105
[16]Miller G A.Introduction to word net:an on-line lexical database[J].International Journal of Lexicography,1990,3(4):235-244
[17]K?l?n?D,Alpkocak A.An expansion and reranking approach for annotation-based image retrieval from Web[J].Expert Systems with Applications,2011,38(10):13121-13127
[18]Pentland A P,Picard R W,Scarloff S.Photobook:tools for content-based manipulation of image databases[C]//Proceedings of Storage and Retrieval for Image and Video Databases II.Bellingham:Society of Photo-Optical Instrumentation Engineers,1994,2185:34-48
[19]Smith J R,Chang S F.VisualSEEk:a fully automated content-based image query system[C]//Proceedings of the 4th ACM International Conference on Multimedia.New York:ACM Press,1996:87-98
[20]Smeulders A W M,Worring M,Santini S,et al.Content based image retrieval at the end of the early years[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2000,22(12):1349-1380
[21]Li X R,Uricchio T,Ballan L,et al.Socializing the semantic gap:a comparative survey on image tag assignment,refinement,and retrieval[J].ACM Computing Surveys,2016,49(1):Article No.14
[22]Cao Y,Wang H,Wang C H,et al.MindFinder:interactive sketch-based image search on millions of images[C]//Proceedings of the 18th ACM International Conference on Multimedia.New York:ACM Press,2010:1605-1608
[23]Cao Y,Wang C H,Zhang L Q,et al.Edgel index for large-scale sketch-based image search[C]//Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2011:761-768
[24]Chen T,Cheng M M,Tan P,et al.Sketch2Photo:internet image montage[J].ACM Transactions on Graphics,2009,28(5):Article No.124
[25]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[OL].[2018-06-20].https://arxiv.org/pdf/1409.1556.pdf
[26]He K M,Zhang X Y,Ren S Q,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEEComputer Society Press,2016:770-778
[27]Deng J,Dong W,Socher R,et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2009:248-255
[28]Chatzichristofis S A,Boutalis Y S.CEDD:color and edge directivity descriptor:a compact descriptor for image indexing and retrieval[C]//Proceedings of the International Conference on Computer Vision Systems.Heidelberg:Springer,2008,5008:312-322
[29]Bosch A,Zisserman A,Munoz X.Image classification using random forests and ferns[C]//Proceedings of the 11th IEEEInternational Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2007:1-8