地学知识图谱引导的遥感影像语义分割

李彦胜; 武康; 欧阳松; 杨坤; 李和平; 张永军

doi:10.11834/jrs.20231110

遥感智能解译 | 浏览量 : 0 下载量: 827 CSCD: 2

R-PDF
PDF
导出
分享
收藏
专辑

地学知识图谱引导的遥感影像语义分割
Geographic knowledge graph-guided remote sensing image semantic segmentation
2024年28卷第2期页码：455-469
纸质出版日期： 2024-02-07 ，
DOI： 10.11834/jrs.20231110
稿件说明：

移动端阅览

李彦胜，武康，欧阳松，杨坤，李和平，张永军.2024.地学知识图谱引导的遥感影像语义分割.遥感学报，28（2）： 455-469

Li Y S，Wu K，Ouyang S，Yang K， Li H P and Zhang Y J. 2024. Geographic knowledge graph-guided remote sensing image semantic segmentation. National Remote Sensing Bulletin， 28（2）：455-469
李彦胜，武康，欧阳松，杨坤，李和平，张永军.2024.地学知识图谱引导的遥感影像语义分割.遥感学报，28（2）： 455-469 DOI： 10.11834/jrs.20231110.

Li Y S，Wu K，Ouyang S，Yang K， Li H P and Zhang Y J. 2024. Geographic knowledge graph-guided remote sensing image semantic segmentation. National Remote Sensing Bulletin， 28（2）：455-469 DOI： 10.11834/jrs.20231110.

摘要

尽管深度语义分割网络有效提升了遥感影像语义分割性能，但其效果远未达到人类领域专家的目视解译水平。原因是人类视觉系统在进行遥感影像解译时，往往会综合运用视觉特征、语义信息和先验知识。然而，深度语义分割网络本质上是数据驱动的面向像素级损失反向优化的分类方法。这种基于像素级优化的深度语义分割网络，一方面受限于像素空间尺度，缺乏整体性目标线索挖掘；另一方面难以跨越结构化数据和非结构化知识之间的鸿沟，无法充分利用地学先验知识和空间语义信息。针对以上两方面的问题，本文提出了地学知识图谱引导的遥感影像深度语义分割方法，运用从地学知识图谱中抽取的地物目标语义信息和地学先验知识构建实体级连通约束和实体间共生约束，引导深度语义分割网络训练。其中，实体级连通约束以连通域实体而非像素单元计算损失，得到实体级别的特征表示，使得分割结果更具整体性，边界模糊和随机噪声现象得到抑制。实体间共生约束将共生条件概率量化的空间共生知识嵌入到数据驱动的深度语义分割网络中，实现空间语义信息和地学先验知识对实体空间分布的约束引导和自动优化。验证结果表明，在实体级连通约束和实体间共生约束的引导下，深度语义分割网络可以完成对实体级特征的学习并根据空间共生知识自动优化地物实体的空间分布，有效改善了遥感影像语义分割性能。

Abstract

Although the Deep Semantic Segmentation Network (DSSN) has notably enhanced remote-sensing image semantic segmentation

it still falls short of human experts’ visual interpretation. Unlike DSSN’s data-driven

pixel-level optimization

human experts rely on visual features

semantic insight

and prior knowledge for remote-sensing image interpretation. DSSN’s pixel-level approach is constrained by spatial scale

lacking comprehensive target inference and struggling to bridge structured data and unstructured knowledge. In response to the two issues above

this paper proposes a geographic knowledge graph-guided deep semantic segmentation network for remote-sensing imagery. We use the ground-object semantic information and geoscience prior knowledge extracted from the geographic knowledge graph to construct loss constraints

thereby autonomously guiding the training process of DSSN.

The essence of our approach lies in the intricately crafted design of loss constraints. These loss constraints include the entity-level connectivity constraint and the inter-entity symbiosis constraint. The former calculates the loss in the unit of connected domain entities instead of pixels to achieve overall constraints on the entity. The latter embeds the spatial symbiosis knowledge quantified by the symbiosis conditional probability into the data-driven DSSN to constrain the spatial distribution of segmented entities. The entity-level connectivity constraint guides DSSN to autonomously learn entity-level feature representations during training. Accordingly

the segmentation results become more holistic and suppresses blurry boundaries and random noise. The inter-entity symbiosis constraint adjusts the spatial distribution of entities according to the spatial semantic information and the prior geoscience knowledge. This adjustment realizes the automatic optimization of the spatial distribution of segmented entities.

Extensive experiments show that under the guidance of the entity-level connectivity constraint and the inter-entity symbiosis constraint

DSSN can complete the learning of entity-level features. It can also automatically optimize the spatial distribution of ground objects based on spatial symbiosis knowledge

thereby effectively improving the performance of remote-sensing image semantic segmentation.

Our novel geographic knowledge graph-guided approach to deep semantic segmentation in remote-sensing imagery has successfully addressed the challenges posed by DSSN’s pixel-level optimization. By incorporating entity-level connectivity and inter-entity symbiosis constraints

we have enabled DSSN to autonomously learn comprehensive feature representations and optimize spatial distribution. The resulting improvements in semantic segmentation performance showcase the potential of merging domain-specific knowledge with data-driven techniques

bridging the gap between automated methods and human interpretation in remote-sensing image analysis.

关键词

地学知识图谱深度语义分割网络实体级连通约束空间共生知识约束地学知识嵌入优化

Keywords

geographic knowledge graphdeep semantic segmentation networkentity-level connectivity constraintspatial symbiosis knowledge constraintgeographic knowledge embedding optimization

references

Achanta R, Shaji A, Smith K, Lucchi A, Fua P and Süsstrunk S. 2012. SLIC Superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11): 2274-2282 [DOI: 10.1109/TPAMI.2012.120http://dx.doi.org/10.1109/TPAMI.2012.120]

Alirezaie M, Längkvist M, Sioutis M and Loutfi A. 2019. Semantic referee: a neural-symbolic framework for enhancing geospatial semantic segmentation. Semantic Web, 10(5): 863-880 [DOI: 10.3233/SW-190362http://dx.doi.org/10.3233/SW-190362]

Arvor D, Belgiu M, Falomir Z, Mougenot I and Durieux L. 2019. Ontologies to interpret remote sensing images: why do we need them? GIScience and Remote Sensing, 56(6): 911-939 [DOI: 10.1080/15481603.2019.1587890http://dx.doi.org/10.1080/15481603.2019.1587890]

Badrinarayanan V, Kendall A and Cipolla R. 2017. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12): 2481-2495 [DOI: 10.1109/TPAMI.2016.2644615http://dx.doi.org/10.1109/TPAMI.2016.2644615]

Camps-Valls G, Tuia D, Bruzzone L and Benediktsson J A. 2014. Advances in hyperspectral image classification: earth monitoring with statistical learning methods. IEEE Signal Processing Magazine, 31(1): 45-54 [DOI: 10.1109/MSP.2013.2279179http://dx.doi.org/10.1109/MSP.2013.2279179]

Chen L C, Zhu Y K, Papandreou G, Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 833-851 [DOI: 10.1007/978-3-030-01234-2_49http://dx.doi.org/10.1007/978-3-030-01234-2_49]

Dash T, Chitlangia S, Ahuja A and Srinivasan A. 2022. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports, 12(1): 1040 [DOI: 10.1038/s41598-021-04590-0http://dx.doi.org/10.1038/s41598-021-04590-0]

Demir I, Koperski K, Lindenbaum D, Pang G, Huang J, Basu S, Hughes F, Tuia D and Raska R. 2018. DeepGlobe 2018: a challenge to parse the earth through satellite images//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City: IEEE: 172-17209 [DOI: 10.1109/CVPRW.2018.00031http://dx.doi.org/10.1109/CVPRW.2018.00031]

Fan J, Yu W Z, Wu W and Shen Y. 2017. Knowledge-guided fitting method for sparse time series remote sensing data. Journal of Remote Sensing, 21(5): 749-756

范菁, 余维泽, 吴炜, 沈瑛. 2017. 知识引导的稀疏时间序列遥感数据拟合. 遥感学报, 21(5): 749-756 [DOI: 10.11834/jrs.20176434http://dx.doi.org/10.11834/jrs.20176434]

Goodenough D G, Goldberg M, Plunkett G and Zelek J. 1987. An expert system for remote- sensing. IEEE Transactions on Geoscience and Remote Sensing, GE-25(3): 349-359 [DOI: 10.1109/TGRS.1987.289805http://dx.doi.org/10.1109/TGRS.1987.289805]

Gu H Y, Li H T, Yan L, Liu Z J, Blaschke T and Soergel U. 2017. An object-based semantic classification method for high resolution remote sensing imagery using ontology. Remote Sensing, 9(4): 329 [DOI: 10.3390/rs9040329http://dx.doi.org/10.3390/rs9040329]

Gui R, Xu X, Dong H, Song C and Pu F L. 2016. Individual building extraction from TerraSAR-X images based on ontological semantic analysis. Remote Sensing, 8(9): 708 [DOI: 10.3390/rs8090708http://dx.doi.org/10.3390/rs8090708]

He K M, Gkioxari G, Dollár P and Girshick R. 2018. Mask R-CNN//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 2980-2988 [DOI: 10.1109/ICCV.2017.322http://dx.doi.org/10.1109/ICCV.2017.322]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE: 770-778 [DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]

Hogan A, Blomqvist E, Cochez M, D’Amato C, de Melo G, Gutierrez C, Kirrane S, Gayo J E L, Navigli R, Neumaier S, Ngomo A C N, Polleres A, Rashid S M, Rula A, Schmelzeisen L, Sequeda J, Staab S and Zimmermann A. 2022. Knowledge graphs. ACM Computing Surveys, 54(4): 71 [DOI: 10.1145/3447772http://dx.doi.org/10.1145/3447772]

Li F S, Li X J, Chen W T, Dong Y S, Li Y K and Wang L Z, 2022. Automatic lithology classification based on deep features using dual polarization SAR images. Earth Science, 47(11): 4267-4279

李发森, 李显巨, 陈伟涛, 董玉森, 李雨柯, 王力哲. 2022. 基于深度特征的双极化SAR遥感图像岩性自动分类. 地球科学, 47(11): 4267-4279 [DOI: 10.3799/dqkx.2022.129http://dx.doi.org/10.3799/dqkx.2022.129]

Li J, Wu Z C, Sheng Q H, Wang B, Hu Z W, Zheng S B, Camps-Valls G and Molinier M. 2022. A hybrid generative adversarial network for weakly-supervised cloud detection in multispectral images. Remote Sensing of Environment, 280: 113197 [DOI: 10.1016/j.rse.2022.113197http://dx.doi.org/10.1016/j.rse.2022.113197]

Li N, Zhu X F, Pan Y Z and Zhan P. 2018. Optimized SVM based on artificial bee colony algorithm for remote sensing image classification. Journal of Remote Sensing, 22(4): 559-569

李楠, 朱秀芳, 潘耀忠, 詹培. 2018. 人工蜂群算法优化的SVM遥感影像分类. 遥感学报, 22(4): 559-569 [DOI: 10.11834/jrs.20187176http://dx.doi.org/10.11834/jrs.20187176]

Li Y S, Chen W, Huang X, Gao Z, Li S W, He T and Zhang Y J. 2023. MFVNet: a deep adaptive fusion network with multiple field-of-views for remote sensing image semantic segmentation. Science China Information Sciences, 66(4): 140305 [DOI: 10.1007/s11432-022-3599-yhttp://dx.doi.org/10.1007/s11432-022-3599-y]

Li Y S, Chen W, Zhang Y J, Tao C, Xiao R and Tan Y H. 2020. Accurate cloud detection in high-resolution remote sensing imagery by weakly supervised deep learning. Remote Sensing of Environment, 250: 112045 [DOI: 10.1016/j.rse.2020.112045http://dx.doi.org/10.1016/j.rse.2020.112045]

Li Y S, Kong D Y, Zhang Y J, Tan Y H and Chen L. 2021a. Robust deep alignment network with remote sensing knowledge graph for zero-shot and generalized zero-shot remote sensing image scene classification. ISPRS Journal of Photogrammetry and Remote Sensing, 179: 145-158 [DOI: 10.1016/j.isprsjprs.2021.08.001http://dx.doi.org/10.1016/j.isprsjprs.2021.08.001]

Li Y S, Shi T, Zhang Y J, Chen W, Wang Z B and Li H. 2021b. Learning deep semantic segmentation network under multiple weakly-supervised constraints for cross-domain remote sensing image semantic segmentation. ISPRS Journal of Photogrammetry and Remote Sensing, 175: 20-33 [DOI: 10.1016/j.isprsjprs.2021.02.009http://dx.doi.org/10.1016/j.isprsjprs.2021.02.009]

Li Y S and Zhang Y J. 2022. A new paradigm of remote sensing image interpretation by coupling knowledge graph and deep learning. Geomatics and Information Science of Wuhan University, 47(8): 1176-1190

李彦胜, 张永军. 2022. 耦合知识图谱和深度学习的新一代遥感影像解译范式. 武汉大学学报(信息科学版), 47(8): 1176-1190 [DOI: 10.13203/j.whugis20210652http://dx.doi.org/10.13203/j.whugis20210652]

Li Y S, Zhang Y J, Huang X, Zhu H and Ma J Y. 2018. Large-scale remote sensing image retrieval by deep hashing neural networks. IEEE Transactions on Geoscience and Remote Sensing, 56(2): 950-965 [DOI: 10.1109/TGRS.2017.2756911http://dx.doi.org/10.1109/TGRS.2017.2756911]

Li Y S, Zhou Y H, Zhang Y J, Zhong L H, Wang J and Chen J D. 2022. DKDFN: domain knowledge-guided deep collaborative fusion network for multimodal unitemporal remote sensing land cover classification. ISPRS Journal of Photogrammetry and Remote Sensing, 186: 170-189 [DOI: 10.1016/j.isprsjprs.2022.02.013http://dx.doi.org/10.1016/j.isprsjprs.2022.02.013]

Liu B, Du S H, Du S J and Zhang X Y. 2020. Incorporating deep features into GEOBIA paradigm for remote sensing imagery classification: a patch-based approach. Remote Sensing, 12(18): 3007 [DOI: 10.3390/rs12183007http://dx.doi.org/10.3390/rs12183007]

Long J, Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE: 3431-3440 [DOI: 10.1109/CVPR.2015.7298965http://dx.doi.org/10.1109/CVPR.2015.7298965]

Ma L, Liu Y, Zhang X L, Ye Y X, Yin G F and Johnson B A. 2019. Deep learning in remote sensing applications: a meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing, 152: 166-177 [DOI: 10.1016/j.isprsjprs.2019.04.015http://dx.doi.org/10.1016/j.isprsjprs.2019.04.015]

Ouyang S and Li Y S. 2021. Combining deep semantic segmentation network and graph convolutional neural network for semantic segmentation of remote sensing imagery. Remote Sensing, 13(1): 119 [DOI: 10.3390/rs13010119http://dx.doi.org/10.3390/rs13010119]

Ouyang S B, Chen W T, Li X J, Dong Y S and Wang L Z. 2022. Geomorphological scene classification dataset of high-resolution remote sensing imagery in vegetation-covered areas. National Remote Sensing Bulletin, 26(4): 606-619

欧阳淑冰, 陈伟涛, 李显巨, 董玉森, 王力哲. 2022. 植被覆盖区高精度遥感地貌场景分类数据集. 遥感学报, 26(4): 606-619 [DOI: 10.11834/jrs.20221385http://dx.doi.org/10.11834/jrs.20221385]

Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer- Assisted Intervention. Munich: Springer: 234-241 [DOI: 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28]

Shao Z F, Yang K and Zhou W X. 2018. Performance evaluation of single-label and multi-label remote sensing image retrieval using a dense labeling dataset. Remote Sensing, 10(6): 964 [DOI: 10.3390/rs10060964http://dx.doi.org/10.3390/rs10060964]

Wang Z H, Yang X M and Zhou C H. 2021. Geographic knowledge graph for remote sensing big data. Journal of Geo-information Science, 23(1): 16-28

王志华, 杨晓梅, 周成虎. 2021. 面向遥感大数据的地学知识图谱构想. 地球信息科学学报, 23(1): 16-28 [DOI: 10.12082/dqxxkx.2021.200632http://dx.doi.org/10.12082/dqxxkx.2021.200632]

Wu R, Luo W F and Chen J H. 2022. Knowledge graph embedding spectral unmixing. National Remote Sensing Bulletin, 26(12): 1-17

吴瑞, 罗文斐, 陈江浩. 2022. 知识图谱嵌入的光谱解混算法. 遥感学报, 26(12): 1-17 [DOI: 10.11834/jrs.20222253http://dx.doi.org/10.11834/jrs.20222253]

Wu X, Shi Z W and Zou Z X. 2021. A geographic information-driven method and a new large scale dataset for remote sensing cloud/snow detection. ISPRS Journal of Photogrammetry and Remote Sensing, 174: 87-104 [DOI: 10.1016/j.isprsjprs.2021.01.023http://dx.doi.org/10.1016/j.isprsjprs.2021.01.023]

Xiao C J, Li Y, Zhang H Q and Chen J. 2020. Semantic segmentation of remote sensing image based on deep fusion networks and conditional random field. Journal of Remote Sensing, 24(3): 254-264

肖春姣, 李宇, 张洪群, 陈俊. 2020. 深度融合网结合条件随机场的遥感图像语义分割. 遥感学报, 24(3): 254-264 [DOI: 10.11834/jrs.20208298http://dx.doi.org/10.11834/jrs.20208298]

Xie E Z, Wang W H, Yu Z D, Anandkumar A, Álvarez J M and Luo P. 2021. SegFormer: simple and efficient design for semantic segmentation with transformers. arXiv: 2105.15203.[DOI: 10.48550/arXiv.2105.15203http://dx.doi.org/10.48550/arXiv.2105.15203]

Xu J C, Xiong Z X and Bhattacharyya S P. 2023. PIDNet: a real-time semantic segmentation network inspired by PID controllers//Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE: 19529-19539 [DOI: 10.1109/CVPR52729.2023.01871http://dx.doi.org/10.1109/CVPR52729.2023.01871]

Xu Z X, Wu K, Huang L, Wang Q M and Ren P. 2022. Cloudy image arithmetic: a cloudy scene synthesis paradigm with an application to deep-learning-based thin cloud removal. IEEE Transactions on Geoscience and Remote Sensing, 60: 5612616 [DOI: 10.1109/TGRS.2021.3122253http://dx.doi.org/10.1109/TGRS.2021.3122253]

Zhang H Y, Zhou C H, Lv G N, Wu Z F, Lu F, Wang J F, Yue T X, Luo J C, Ge Y and Qin C Z. 2020. The connotation and inheritance of Geo-information Tupu. Journal of Geo-information Science, 22(4): 653-661

张洪岩, 周成虎, 闾国年, 吴志峰, 陆锋, 王劲峰, 岳天祥, 骆剑承, 葛咏, 秦承志. 2020. 试论地学信息图谱思想的内涵与传承. 地球信息科学学报, 22(4): 653-661 [DOI: 10.12082/dqxxkx.2020.200167http://dx.doi.org/10.12082/dqxxkx.2020.200167]

Zhang Y J, Wang F, Li Y S, Ouyang S, Wei D, Liu X J, Kong D Y, Chen R X and Zhang B. 2023. Remote sensing knowledge graph construction and its application in typical scenarios. National Remote Sensing Bulletin, 27(2): 249-266

张永军, 王飞, 李彦胜, 欧阳松, 魏东, 刘晓建, 孔德宇, 陈瑞贤, 张斌. 2023. 遥感知识图谱创建及其典型场景应用技术. 遥感学报, 27(2): 249-266 [DOI: 10.11834/jrs.20210469http://dx.doi.org/10.11834/jrs.20210469]

Zhao H S, Shi J P, Qi X J, Wang X G and Jia J Y. 2017. Pyramid scene parsing network//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 6230-6239 [DOI: 10.1109/CVPR.2017.660http://dx.doi.org/10.1109/CVPR.2017.660]

Zhu X X, Tuia D, Mou L C, Xia G S, Zhang L P, Xu F and Fraundorfer F. 2017. Deep learning in remote sensing: a comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4): 8-36 [DOI: 10.1109/MGRS.2017.2762307http://dx.doi.org/10.1109/MGRS.2017.2762307]

文章被引用时，请邮件提醒。

提交

暂无数据