模态间匹配学习的高光谱和激光雷达联合分类
Joint classification of hyperspectral and LiDAR data based on inter-modality match learning
- 2024年28卷第1期 页码：154-167
纸质出版日期： 2024-01-07
DOI： 10.11834/jrs.20232635
扫 描 看 全 文
浏览全部资源
扫码关注微信
摘要
如何有效地提取和融合不同模态的特征是高光谱图像和激光雷达数据联合分类的关键。近年来，得益于深度学习强大的特征学习能力，其在高光谱图像和激光雷达数据联合分类领域受到了越来越多的关注。然而，现有的深度学习模型大多基于监督学习的模式，分类性能依赖标注样本的数量和质量。为此，本文提出了一种基于模态间匹配学习的联合分类方法，充分利用未标注样本的信息，减少对标注信息的依赖性。具体而言，本文首先通过高光谱图像和激光雷达数据之间的匹配关系和KMeans聚类算法，构造模态匹配标签。然后，利用该标签训练含有多个卷积层的匹配学习网络。该网络由两个并行分支构成，每个分支负责提取单个模态的特征。最后，以该网络为基础，构造高光谱图像和激光雷达数据联合分类模型。该模型的参数由匹配学习网络进行初始化，因而只需要少量标注样本进行微调即可达到理想的分类效果。为了验证本文方法的有效性，在Houston和MUUFL两个常用的高光谱图像和激光雷达数据联合分类数据集上进行了大量的实验。实验结果表明，与已有的分类模型相比，本文方法能够获得更高的分类性能。
Abstract
Several excellent models for joint classification of hyperspectral image and LiDAR data
which were designed on the basis of supervised learning methods such as convolutional neural networks
have been developed in recent years. Their classification performance depends largely on the quantity and quality of training samples. However
when the distribution of ground objects becomes increasingly complex and the resolutions of remote sensing images grow increasingly high
obtaining high-quality labels only with limited cost and manpower is difficult. Therefore
numerous scholars have made efforts to learn features directly from unlabeled samples. For instance
the theory of autoencoder was applied to multimodal joint classification
achieving satisfactory performance. Methods based on the reconstruction idea reduce the dependence on labeled information to a certain extent
but several problems that must be settled still exist. For example
these methods pay more attention to data reconstruction but fail to guarantee that the extracted features have sufficient discriminant capability
thus affecting the performance of joint classification.
This paper proposes an effective model named Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning to address the aforementioned issue. Different from feature extraction models based on reconstruction idea
the proposed model tends to compare the matching relationship between samples from different modalities
thereby enhancing the discriminative capability of features. Specifically
this model comprises inter-modality matching learning and multimodal joint classification networks. The former is prone to identify matching of the input patch pairs of hyperspectral image and LiDAR data; therefore
reasonable construction of matching labels is essential. Thus
spatial positions of center pixels in cropped patches and KMeans clustering methods are employed. These constructed labels and patch pairs are combined to train the network. Notably
this process does not use manual labeled information and can directly extract features from abundant unlabeled samples. Furthermore
in the joint classification stage
the structure and trained parameters of matching learning network are transferred
and a small number of manually labeled training samples are then used to finetune the model parameters.
Extensive experiments were conducted on two widely used datasets
namely Houston and MUUFL
to verify the effectiveness of the proposed model. These experiments include comparison experiments with several state-of-the-art models
hyperparameter analysis experiments
and ablation studies. Take the first experiment as an example. Compared with other models
such as CNN
EMFNet
AE_H
AE_HL
CAE_H
CAE_HL
IP-CNN
and PToP CNN
the proposed model can achieve higher performance on both datasets with OAs of 88.39% and 81.46%
respectively. Overall
the proposed model reduces the dependence on manually labeled data and improves the joint classification accuracy in the case of limited training samples. A superior model structure and additional testing datasets will be explored in the future to make further improvements.
关键词
遥感图像高光谱图像激光雷达数据深度学习匹配学习联合分类
Keywords
remote sensing imagehyperspectral imageLiDAR datadeep learningmatch learningjoint classification
references
Abbasi A N and He M Y. 2019. Convolutional neural network with PCA and batch normalization for hyperspectral image classification//2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE: 959-962 [DOI: 10.1109/IGARSS.2019.8899329http://dx.doi.org/10.1109/IGARSS.2019.8899329]
Cao Q, Ma A L, Zhong Y F, Zhao J, Zhao B and Zhang L P. 2019. Urban classification by multi-feature fusion of hyperspectral image and LiDAR data. Journal of Remote Sensing, 23(5): 892-903
曹琼, 马爱龙, 钟燕飞, 赵济, 赵贝, 张良培. 2019. 高光谱-LiDAR多级融合城区地表覆盖分类. 遥感学报, 23(5): 892-903 [DOI: 10.11834/jrs.20197512http://dx.doi.org/10.11834/jrs.20197512]
Chen Y S, Jiang H L, Li C Y, Jia X P and Ghamisi P. 2016. Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, 54(10): 6232-6251 [DOI: 10.1109/TGRS.2016.2584107http://dx.doi.org/10.1109/TGRS.2016.2584107]
Chen Y S, Li C Y, Ghamisi P, Jia X P and Gu Y F. 2017. Deep fusion of remote sensing data for accurate classification. IEEE Geoscience and Remote Sensing Letters, 14(8): 1253-1257 [DOI: 10.1109/LGRS.2017.2704625http://dx.doi.org/10.1109/LGRS.2017.2704625]
Chen Y S, Lin Z H, Zhao X, Wang G and Gu Y F. 2014. Deep learning-based classification of hyperspectral data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6): 2094-2107 [DOI: 10.1109/JSTARS.2014.2329330http://dx.doi.org/10.1109/JSTARS.2014.2329330]
Dalponte M, Bruzzone L and Gianelle D. 2008. Fusion of hyperspectral and LIDAR remote sensing data for classification of complex forest areas. IEEE Transactions on Geoscience and Remote Sensing, 46(5): 1416-1427 [DOI: 10.1109/TGRS.2008.916480http://dx.doi.org/10.1109/TGRS.2008.916480]
Debes C, Merentitis A, Heremans R, Hahn J, Frangiadakis N, Van Kasteren T, Liao W Z, Bellens R, Pizurica A, Gautama S, Philips W, Prasad S, Du Q and Pacifici F. 2014. Hyperspectral and LiDAR data fusion: outcome of the 2013 GRSS data fusion contest. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6): 2405-2418 [DOI: 10.1109/JSTARS.2014.2305441http://dx.doi.org/10.1109/JSTARS.2014.2305441]
Gader P, Zare A, Close R, Aitken J and Tuell G. 2013. MUUFL Gulfport Hyperspectral and Lidar Airborne Data Set. University of Florida
Ge C R, Du Q, Sun W W, Wang K Y, Li J J and Li Y S. 2021. Deep residual network-based fusion framework for hyperspectral and LiDAR data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 2458-2472 [DOI: 10.1109/JSTARS.2021.3054392http://dx.doi.org/10.1109/JSTARS.2021.3054392]
Hang R L, Li Z, Ghamisi P, Hong D F, Xia G Y and Liu Q S. 2020. Classification of hyperspectral and LiDAR Data using coupled CNNs. IEEE Transactions on Geoscience and Remote Sensing, 58(7): 4939-4950 [DOI: 10.1109/TGRS.2020.2969024http://dx.doi.org/10.1109/TGRS.2020.2969024]
He X, Chen Y S and Lin Z H. 2021. Spatial-spectral transformer for hyperspectral image classification. Remote Sensing, 13(3): 498 [DOI: 10.3390/rs13030498http://dx.doi.org/10.3390/rs13030498]
Hong D F, Gao L R, Hang R L, Zhang B and Chanussot J. 2022. Deep encoder-decoder networks for classification of hyperspectral and LiDAR data. IEEE Geoscience and Remote Sensing Letters, 19: 5500205 [DOI: 10.1109/LGRS.2020.3017414http://dx.doi.org/10.1109/LGRS.2020.3017414]
Hong D F, Gao L R, Yokoya N, Yao J, Chanussot J, Du Q and Zhang B. 2021. More diverse means better: multimodal deep learning meets remote-sensing imagery classification. IEEE Transactions on Geoscience and Remote Sensing, 59(5): 4340-4354 [DOI: 10.1109/TGRS.2020.3016820http://dx.doi.org/10.1109/TGRS.2020.3016820]
Huang R and Zhu J T. 2013. Using Random Forest to integrate lidar data and hyperspectral imagery for land cover classification//2013 IEEE International Geoscience and Remote Sensing Symposium. Melbourne: IEEE: 3978-3981 [DOI: 10.1109/IGARSS.2013.6723704http://dx.doi.org/10.1109/IGARSS.2013.6723704]
Jahan F, Zhou J, Awrangjeb M and Gao Y S. 2020. Inverse coefficient of variation feature and multilevel fusion technique for hyperspectral and LiDAR data classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13: 367-381 [DOI: 10.1109/JSTARS.2019.2962659http://dx.doi.org/10.1109/JSTARS.2019.2962659]
Kemker R and Kanan C. 2017. Self-taught feature learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(5): 2693-2705 [DOI: 10.1109/TGRS.2017.2651639http://dx.doi.org/10.1109/TGRS.2017.2651639]
Li C X, Hang R L and Rasti B. 2021. EMFNet: enhanced multisource fusion network for land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 4381-4389 [DOI: 10.1109/JSTARS.2021.3073719http://dx.doi.org/10.1109/JSTARS.2021.3073719]
Li H, Ghamisi P, Soergel U and Zhu X X. 2018. Hyperspectral and LiDAR fusion using deep three-stream convolutional neural networks. Remote Sensing, 10(10): 1649 [DOI: 10.3390/rs10101649http://dx.doi.org/10.3390/rs10101649]
Liu Q C, Xiao L, Yang J X and Wei Z H. 2021. CNN-enhanced graph convolutional network with pixel- and superpixel-level feature fusion for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 59(10): 8657-8671 [DOI: 10.1109/TGRS.2020.3037361http://dx.doi.org/10.1109/TGRS.2020.3037361]
Liu W Q, Wang C, Zang Y, Hu Q, Yu S S and Lai B Q. 2022. A survey on information extraction technology based on remote sensing big data. Big Data Research, 8(2): 28-57
刘伟权, 王程, 臧彧, 胡倩, 于尚书, 赖柏锜. 2022. 基于遥感大数据的信息提取技术综述. 大数据, 8(2): 28-57 [DOI: 10.11959/j.issn.2096-0271.2022014http://dx.doi.org/10.11959/j.issn.2096-0271.2022014]
Ni L, Gao L R, Li S S, Li J and Zhang B. 2014. Edge-constrained Markov random field classification by integrating hyperspectral image with LiDAR data over urban areas. Journal of Applied Remote Sensing, 8(1): 085089[DOI: 10.1117/1.JRS.8.085089http://dx.doi.org/10.1117/1.JRS.8.085089]
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z M, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J J and Chintala S. 2019. PyTorch: an imperative style, high-performance deep learning library//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc.: 8026-8037
Prabha K A and Visalakshi N K. 2014. Improved particle swarm optimization based K-means clustering//2014 International Conference on Intelligent Computing Applications. Coimbatore: IEEE: 59-63 [DOI: 10.1109/ICICA.2014.21http://dx.doi.org/10.1109/ICICA.2014.21]
Qu Y, Baghbaderani R K, Li W, Gao L R, Zhang Y X and Qi H R. 2021. Physically constrained transfer learning through shared abundance space for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10455-10472 [DOI: 10.1109/TGRS.2020.3045790http://dx.doi.org/10.1109/TGRS.2020.3045790]
Tong Q X, Zhang B and Zhang L F. 2016. Current progress of hyperspectral remote sensing in China. Journal of Remote Sensing, 20(5): 689-707
童庆禧, 张兵, 张立福. 2016. 中国高光谱遥感的前沿进展. 遥感学报, 20(5): 689-707 [DOI: 10.11834/jrs.20166264http://dx.doi.org/10.11834/jrs.20166264]
Wang X H, Feng Y N, Song R X, Mu Z H and Song C M. 2022. Multi-attentive hierarchical dense fusion net for fusion classification of hyperspectral and LiDAR data. Information Fusion, 82: 1-18 [DOI: 10.1016/j.inffus.2021.12.008http://dx.doi.org/10.1016/j.inffus.2021.12.008].
Xia J S, Ghamisi P, Yokoya N and Iwasaki A. 2018. Random forest ensembles and extended multiextinction profiles for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 56(1): 202-216 [DOI: 10.1109/TGRS.2017.2744662http://dx.doi.org/10.1109/TGRS.2017.2744662]
Xu X D, Li W, Ran Q, Du Q, Gao L R and Zhang B. 2018. Multisource remote sensing data classification based on convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 56(2): 937-949 [DOI: 10.1109/TGRS.2017.2756851http://dx.doi.org/10.1109/TGRS.2017.2756851]
Zhang L P and Shen H F. 2016. Progress and future of remote sensing data fusion. Journal of Remote Sensing, 20(5): 1050-1061
张良培, 沈焕锋. 2016. 遥感数据融合的进展与前瞻. 遥感学报, 20(5): 1050-1061 [DOI: 10.11834/jrs.20166243http://dx.doi.org/10.11834/jrs.20166243]
Zhang M M, Li W, Du Q, Gao L R and Zhang B. 2020. Feature extraction for classification of hyperspectral and LiDAR data using Patch-to-Patch CNN. IEEE Transactions on Cybernetics, 50(1): 100-111 [DOI: 10.1109/TCYB.2018.2864670http://dx.doi.org/10.1109/TCYB.2018.2864670]
Zhang M M, Li W, Tao R, Li H C and Du Q. 2022. Information fusion for classification of hyperspectral and LiDAR data using IP-CNN. IEEE Transactions on Geoscience and Remote Sensing, 60: 5506812 [DOI: 10.1109/TGRS.2021.3093334http://dx.doi.org/10.1109/TGRS.2021.3093334]
Zhao W D, Li S S, Li A, Zhang B and Chen J. 2021. Deep fusion of hyperspectral images and multi-source remote sensing data for classification with convolutional neural network. National Remote Sensing Bulletin, 25(7): 1489-1502
赵伍迪, 李山山, 李安, 张兵, 陈俊. 2021. 结合深度学习的高光谱与多源遥感数据融合分类. 遥感学报, 25(7): 1489-1502 [DOI: 10.11834/jrs.20219117http://dx.doi.org/10.11834/jrs.20219117]
文章被引用时，请邮件提醒。
提交
相关作者
暂无数据
相关机构
南京理工大学计算机科学与工程学院
电子信息系统复杂电磁环境效应国家重点实验室
国防科技大学 电子科学学院
武汉大学 电子信息学院
武汉大学 宇航科学与技术研究院
选择翻译语言
- 地址：北京市海淀区北四环西路19号 《遥感学报》编辑部 邮编：100190
- 联系电话：010-58887052 Email：nrsb@aircas.ac.cn
- 技术支持由北京北大方正电子有限公司提供 京ICP备20021838号-9
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies
0
回到顶部