深度卷积特征表达的多模态遥感影像模板匹配方法
|
摘要
多模态遥感影像间(光学、红外、SAR等)存在显著的非线性辐射差异,传统方法难以有效地提取影像间的共有特征,匹配效果不佳。鉴于此,本文将深度学习方法引入影像匹配中,提出了一种基于Siamese网络提取多模态影像共有特征的匹配方法。首先通过去除Siamese网络中的池化层和抽取特征来优化该网络,保持特征信息的完整性和位置精度,使其可有效地提取多模态影像间的共有特征,然后采用模板匹配策略,实现多模态遥感影像高精度匹配。通过利用多组多模态遥感影像进行试验,结果表明,本文方法的匹配正确率和匹配精度都优于传统的模板匹配方法。
Due to significant non-linear radiometric differences between multimodal remote sensing images(e.g., optical, infrared, and SAR), traditional methods cannot efficiently extract common features between such images, and are vulnerable for image matching. To address that, the deep learning technique is introduced into the present study to design a matching method based on Siamese network, which aims to extract common features between multimodal images. The network is first optimized by removing the pooling layer and extracting the feature layer from Siamese network to maintain the integrity and positional accuracy of the feature information, making it possible the effective extraction of common features between multimodal images. Then, the template matching strategy is adopted to achieve high-precision matching of multimodal images. The proposed method is evaluated by using multiple multimodal remote sensing images. The results show that the proposed method outperforms traditional template-matching methods in both the matching correct ratio and matching accuracy.
Due to significant non-linear radiometric differences between multimodal remote sensing images(e.g., optical, infrared, and SAR), traditional methods cannot efficiently extract common features between such images, and are vulnerable for image matching. To address that, the deep learning technique is introduced into the present study to design a matching method based on Siamese network, which aims to extract common features between multimodal images. The network is first optimized by removing the pooling layer and extracting the feature layer from Siamese network to maintain the integrity and positional accuracy of the feature information, making it possible the effective extraction of common features between multimodal images. Then, the template matching strategy is adopted to achieve high-precision matching of multimodal images. The proposed method is evaluated by using multiple multimodal remote sensing images. The results show that the proposed method outperforms traditional template-matching methods in both the matching correct ratio and matching accuracy.
引文
[1]闫利,王紫琦,叶志云.顾及灰度和梯度信息的多模态影像配准算法[J].测绘学报,2018,47(1):71-81.DOI:10.11947/j.AGCS.2018.20170368.YAN Li,WANG Ziqi,YE Zhiyun.Multimodal image registration algorithm considering grayscale and gradient information[J].Acta Geodaetica et Cartographica Sinica,2018,47(1):71-81.DOI:10.11947/j.AGCS.2018.20170368.
[2]YE Yuanxin,SHAN Jie,BRUZZONE L,et al.Robust registration of multimodal remote sensing images based on structural similarity[J].IEEE Transactions on Geoscience and Remote Sensing,2017,55(5):2941-2958.
[3]ZITOVB,FLUSSER J.Image registration methods:a survey[J].Image and Vision Computing,2003,21(11):977-1000.
[4]余先川,吕中华,胡丹.遥感图像配准技术综述[J].光学精密工程,2013,21(11):2960-2972.YU Xianchuan,LZhonghua,HU Dan.Review of remote sensing image registration techniques[J].Optics and Precision Engineering,2013,21(11):2960-2972.
[5]MORAVEC H P.Towards automatic visual obstacle avoidance[C]∥Proceedings of the 5th International Joint Conference on Artificial Intelligence.Massachusetts US:MIT,1977.
[6]HARRIS C,STEPHENS M.A combined corner and edge detector[C]∥Proceedings of the 4th Alvey Vision Conference.Manchester,UK:Organising Committee AVC,1988:147-151.
[7]ZIOU D,TABBONE S.Edge detection techniques:an overview[J].International Journal of Pattern Recognition and Image Analysis,1998(8):537-559.
[8]ZHANG Dengrong,YU Le,CAI Zhigang.Automatic registration for ASAR and TM images based on region features[C]∥Proceedings of SPIE 6752,Geoinformatics 2007:Remotely Sensed Data and Information.Nanjing:SPIE,2007:6752.
[9]张宝尚,田铮,延伟东.基于分割区域的SAR图像配准方法研究[J].工程数学学报,2011,28(1):7-14.ZHANG Baoshang,TIAN Zheng,YAN Weidong.An SAR image registration algorithm based on segmentationderived regions[J].Chinese Journal of Engineering Mathematics,2011,28(1):7-14.
[10]LOWE D G.Object recognition from local scale-invariant features[C]∥Proceeding of the 7th International Conference on Computer Vision.Kerkyra,Corfu,Greece:IEEE,1999:1150-1157.
[11]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.
[12]BELONGIE S,MALIK J,PUZICHA J.Shape matching and object recognition using shape contexts[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2002,24(4):509-522.
[13]SURI S,REINARTZ P.Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J].IEEE Transactions on Geoscience and Remote Sensing,2010,48(2):939-949.
[14]MARTINEZ A,GARCIA-CONSUEGRA J,ABAD F.Acorrelation-symbolic approach to automatic remotely sensed image rectification[C]∥Proceedings of IEEE 1999International Geoscience and Remote Sensing Symposium.Hamburg,Germany:IEEE,1999:336-338.
[15]JOHNSON K,COLE-RHODES A,ZAVORIN I,et al.Mutual information as a similarity measure for remote sensing image registration[C]∥Proceedings of SPIE4383,Geo-Spatial Image and Data ExploitationⅡ.Orlando,FL,United States:SPIE,2001:51-61.
[16]YE Yuanxin,SHAN Jie,HAO Siyuan,et al.A local phase based invariant feature for remote sensing image matching[J].ISPRS Journal of Photogrammetry and Remote Sensing,2018(142):205-221.
[17]ZAGORUYKO S,KOMODAKIS N.Learning to compare image patches via convolutional neural networks[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEE,2015:4353-4361.
[18]范大昭,董杨,张永生.卫星影像匹配的深度卷积神经网络方法[J].测绘学报,2018,47(6):844-853.DOI:10.11947/j.AGCS.2018.20170627.FAN Dazhao,DONG Yang,ZHANG Yongsheng.Satellite image matching method based on deep convolution neural network[J].Acta Geodaetica et Cartographica Sinica,2018,47(6):844-853.DOI:10.11947/j.AGCS.2018.20170627.
[19]HE Haiqing,CHEN Min,CHEN Ting,et al.Matching of remote sensing images with complex background variations via Siamese convolutional neural network[J].Remote Sensing,2018,10(3):355.
[20]BROMLEY J,GUYON I,LECUN Y,et al.Signature verification using a“Siamese”time delay neural network[C]∥Proceedings of the 6th International Conference on Neural Information Processing Systems.Denver,Colorado:Morgan Kaufmann Publishers Inc,1993.
[21]CHOPRA S,HADSELL R,LECUN Y.Learning a similarity metric discriminatively,with Application to Face Verification∥Proceeding of 2005IEEE Computer Society Conference on Computer Vision and Pattern Recognition.San Diego,CA:IEEE,2005.
[22]许慧敏.基于深度学习U-Net模型的高分辨率遥感影像分类方法研究[D].成都:西南交通大学,2018.XU Huimin.Method research of high resolution remote sensing imagery classification based on U-Net model of deep learning[D].Chengdu:Southwest Jiaotong University,2018.
[23]周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.ZHOU Feiyan,JIN Linpeng,DONG Jun.Review of convolutional neural network[J].Chinese Journal of Computers,2017,40(6):1229-1251.
[24]HADSELL R,CHOPRA S,LECUN Y.Dimensionality reduction by learning an invariant mapping[C]∥Proceedins of 2006IEEE Computer Society Conference on Computer Vision and Pattern Recognition.New York:IEEE,2006.
[25]TRAPPEY A J C,HSU F C,TRAPPEY C V,et al.Development of a patent document classification and search platform using a back-propagation network[J].Expert Systems with Applications,2006,31(4):755-765.
[26]YE Yuanxin,SHAN Jie.A local descriptor based registration method for multispectral remote sensing images with nonlinear intensity differences[J].ISPRS Journal of Photogrammetry and Remote Sensing,2014(90):83-95.
[27]林善明,朱小艳,周建华,等.一种结合互信息和模板匹配的配准方法[J].计算机工程,2010,36(14):198-200.LIN Shanming,ZHU Xiaoyan,ZHOU Jianhua,et al.Registration method with mutual information and template matching[J].Computer Engineering,2010,36(14):198-200.
[2]YE Yuanxin,SHAN Jie,BRUZZONE L,et al.Robust registration of multimodal remote sensing images based on structural similarity[J].IEEE Transactions on Geoscience and Remote Sensing,2017,55(5):2941-2958.
[3]ZITOVB,FLUSSER J.Image registration methods:a survey[J].Image and Vision Computing,2003,21(11):977-1000.
[4]余先川,吕中华,胡丹.遥感图像配准技术综述[J].光学精密工程,2013,21(11):2960-2972.YU Xianchuan,LZhonghua,HU Dan.Review of remote sensing image registration techniques[J].Optics and Precision Engineering,2013,21(11):2960-2972.
[5]MORAVEC H P.Towards automatic visual obstacle avoidance[C]∥Proceedings of the 5th International Joint Conference on Artificial Intelligence.Massachusetts US:MIT,1977.
[6]HARRIS C,STEPHENS M.A combined corner and edge detector[C]∥Proceedings of the 4th Alvey Vision Conference.Manchester,UK:Organising Committee AVC,1988:147-151.
[7]ZIOU D,TABBONE S.Edge detection techniques:an overview[J].International Journal of Pattern Recognition and Image Analysis,1998(8):537-559.
[8]ZHANG Dengrong,YU Le,CAI Zhigang.Automatic registration for ASAR and TM images based on region features[C]∥Proceedings of SPIE 6752,Geoinformatics 2007:Remotely Sensed Data and Information.Nanjing:SPIE,2007:6752.
[9]张宝尚,田铮,延伟东.基于分割区域的SAR图像配准方法研究[J].工程数学学报,2011,28(1):7-14.ZHANG Baoshang,TIAN Zheng,YAN Weidong.An SAR image registration algorithm based on segmentationderived regions[J].Chinese Journal of Engineering Mathematics,2011,28(1):7-14.
[10]LOWE D G.Object recognition from local scale-invariant features[C]∥Proceeding of the 7th International Conference on Computer Vision.Kerkyra,Corfu,Greece:IEEE,1999:1150-1157.
[11]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.
[12]BELONGIE S,MALIK J,PUZICHA J.Shape matching and object recognition using shape contexts[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2002,24(4):509-522.
[13]SURI S,REINARTZ P.Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J].IEEE Transactions on Geoscience and Remote Sensing,2010,48(2):939-949.
[14]MARTINEZ A,GARCIA-CONSUEGRA J,ABAD F.Acorrelation-symbolic approach to automatic remotely sensed image rectification[C]∥Proceedings of IEEE 1999International Geoscience and Remote Sensing Symposium.Hamburg,Germany:IEEE,1999:336-338.
[15]JOHNSON K,COLE-RHODES A,ZAVORIN I,et al.Mutual information as a similarity measure for remote sensing image registration[C]∥Proceedings of SPIE4383,Geo-Spatial Image and Data ExploitationⅡ.Orlando,FL,United States:SPIE,2001:51-61.
[16]YE Yuanxin,SHAN Jie,HAO Siyuan,et al.A local phase based invariant feature for remote sensing image matching[J].ISPRS Journal of Photogrammetry and Remote Sensing,2018(142):205-221.
[17]ZAGORUYKO S,KOMODAKIS N.Learning to compare image patches via convolutional neural networks[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEE,2015:4353-4361.
[18]范大昭,董杨,张永生.卫星影像匹配的深度卷积神经网络方法[J].测绘学报,2018,47(6):844-853.DOI:10.11947/j.AGCS.2018.20170627.FAN Dazhao,DONG Yang,ZHANG Yongsheng.Satellite image matching method based on deep convolution neural network[J].Acta Geodaetica et Cartographica Sinica,2018,47(6):844-853.DOI:10.11947/j.AGCS.2018.20170627.
[19]HE Haiqing,CHEN Min,CHEN Ting,et al.Matching of remote sensing images with complex background variations via Siamese convolutional neural network[J].Remote Sensing,2018,10(3):355.
[20]BROMLEY J,GUYON I,LECUN Y,et al.Signature verification using a“Siamese”time delay neural network[C]∥Proceedings of the 6th International Conference on Neural Information Processing Systems.Denver,Colorado:Morgan Kaufmann Publishers Inc,1993.
[21]CHOPRA S,HADSELL R,LECUN Y.Learning a similarity metric discriminatively,with Application to Face Verification∥Proceeding of 2005IEEE Computer Society Conference on Computer Vision and Pattern Recognition.San Diego,CA:IEEE,2005.
[22]许慧敏.基于深度学习U-Net模型的高分辨率遥感影像分类方法研究[D].成都:西南交通大学,2018.XU Huimin.Method research of high resolution remote sensing imagery classification based on U-Net model of deep learning[D].Chengdu:Southwest Jiaotong University,2018.
[23]周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.ZHOU Feiyan,JIN Linpeng,DONG Jun.Review of convolutional neural network[J].Chinese Journal of Computers,2017,40(6):1229-1251.
[24]HADSELL R,CHOPRA S,LECUN Y.Dimensionality reduction by learning an invariant mapping[C]∥Proceedins of 2006IEEE Computer Society Conference on Computer Vision and Pattern Recognition.New York:IEEE,2006.
[25]TRAPPEY A J C,HSU F C,TRAPPEY C V,et al.Development of a patent document classification and search platform using a back-propagation network[J].Expert Systems with Applications,2006,31(4):755-765.
[26]YE Yuanxin,SHAN Jie.A local descriptor based registration method for multispectral remote sensing images with nonlinear intensity differences[J].ISPRS Journal of Photogrammetry and Remote Sensing,2014(90):83-95.
[27]林善明,朱小艳,周建华,等.一种结合互信息和模板匹配的配准方法[J].计算机工程,2010,36(14):198-200.LIN Shanming,ZHU Xiaoyan,ZHOU Jianhua,et al.Registration method with mutual information and template matching[J].Computer Engineering,2010,36(14):198-200.