分级监督范式指导下的遥感图像超分辨率方法

李明锴; 徐其志

doi:10.11834/jrs.20243274

卫星信息智能处理技术 | 浏览量 : 0 下载量: 1373 CSCD: 1

R-PDF
PDF
导出
分享
收藏
专辑

分级监督范式指导下的遥感图像超分辨率方法
Remote sensing image super-resolution guided by multi-level supervision paradigm
2024年28卷第7期页码：1746-1759
收稿：2023-07-06，

纸质出版：2024-07-07
DOI： 10.11834/jrs.20243274
稿件说明：

移动端阅览

李明锴，徐其志.2024.分级监督范式指导下的遥感图像超分辨率方法.遥感学报，28（7）： 1746-1759 DOI： 10.11834/jrs.20243274.

Li M K and Xu Q Z. 2024. Remote sensing image super-resolution guided by multi-level supervision paradigm. National Remote Sensing Bulletin， 28（7）：1746-1759 DOI： 10.11834/jrs.20243274.

摘要

超分辨率技术可提升遥感图像空间分辨率，为基于遥感图像的目标检测、场景分类等任务提供更加清晰的数据集，具有广泛的应用价值。然而，现有基于深度学习的超分辨率方法存在监督次数不足的问题，导致超分辨率重建图像中易出现细节损失和伪细节。针对这一问题，本文提出基于分级监督的遥感图像超分辨率方法（MSSR）。首先，提出了一个分级监督网络架构，通过引入多级真值图像作为监督，为超分辨率过程提供充足的图像细节恢复指引，进而减少超分辨率结果中细节损失和伪细节的出现。其次，为了便于构建级数可变、超分辨率倍数可变的分级监督网络，设计了一个轻量化的、超分辨率倍数可灵活调整的同构超分辨率模块（BSRC）。各级BSRC的网络结构基本相同，便于迁移网络权重，缩短训练时间。最后，针对分级网络超分辨率倍数一定时，网络级数及各级分辨率倍数有多种组合方式的问题，对比多种分级方式下的超分辨率结果，给出最佳网络分级方式。此外，构建了一个包含世界各地复杂细节地面场景的遥感图像数据集（RSSRD）。在该数据集和UCMerced、AID两个公开数据集上进行超分辨率实验，实验结果显示本文方法优于现有常用超分辨率方法。

Abstract

Super-resolution improves the spatial resolution of remote sensing images

providing detailed information for various satellite applications. However

existing methods often generate pseudo-detail and lose true detail in reconstructed images due to insufficient supervision images. To address this issue

a progressive super-resolution method based on multilevel supervision structure (MSSR) was proposed.

First

the MSSR introduced ground truth images as guides

which reduced the loss of true detail and mitigated the appearance of pseudo-detail in an output image. The MSSR network consists of several basic super-resolution components (BSRCs) and multilevel supervision. The overall super-resolution scale factor of the MSSR network can be set flexibly. The BSRCs can be increased or decreased

similar to building blocks

and BSRCs decrease with decreasing overall super-resolution scale factor and increase with increasing overall super-resolution scale factor. The scale factor of each BSRC is determined by the overall scale factor and the number of BSRCs. Second

a scale-factor-adjustable and lightweight basic super-resolution component was designed to enable the construction of multilevel supervision networks with different number of BSRCs and different scale factors

such as building blocks. The BSRC consists of a multiscale feature extraction module

a global feature extraction module and an image reconstruction module. Given that the scale factor of each BSRC should have a degree of flexibility

the network structure of each BSRC is the same except for the image reconstruction module. This approach shortens the overall training time of the network. Finally

a method of dividing super-resolution overall scale factor was proposed

and the effects of different number and different scale factors of BSRCs on the performance of multilevel supervision network were explored. For the super-resolution process with a certain scale factor

we need a method to divide the overall scale factor into each BSRC. The number of supervision increases with the number of BSRCs

and the super-resolution ill-posedness is reduced. In addition

the total number of network layers and the number of computations increase. We determined the optimal number of BSRCs and their respective super-resolution scale factors by comparing the super-resolution effects of multiple BSRC combinations. Additionally

a new remote sensing dataset containing worldwide scenes was constructed for the super-resolution task in this paper.

To adequately train and test the proposed super-resolution method and existing methods

we used our datasets and two existing super-resolution datasets: the UCMerced and AID datasets. We compared our method with the state-of-the-art methods: VDSR

SRGAN

RDN

RCAN

DRN

and TransENet. The experiment results on three datasets demonstrated that our MSSR network outperformed these methods.

The progressive network and multilevel supervision structure can effectively suppress super-resolution ill-posedness and reduce pseudo-detail and detail loss in super-resolution results. By further analyzing the experimental results

we found that multilevel supervision has greater performance gains in super-resolution tasks with a larger scale factor than other methods. We speculated that the multilevel supervision network exhibit improved performance in super-resolution tasks at 4× magnification. In future research

we will explore multilevel supervision networks in super-resolution tasks at large magnifications (i.e.

5× and 8×).

关键词

Keywords

references

Arun P V , Buddhiraju K M , Porwal A and Chanussot J . 2020 . CNN based spectral super-resolution of remote sensing images . Signal Processing , 169 : 107394 [ DOI: 10.1016/j.sigpro.2019.107394 http://dx.doi.org/10.1016/j.sigpro.2019.107394 ]

Chen H and Luo B . 2021 . Multi-angle remote sensing images super-resolution reconstruction using dynamic upsampling filter deep network . Geomatics and Information Science of Wuhan University , 46 ( 11 ): 1716 - 1726

陈行 , 罗斌 . 2021 . 利用动态上采样滤波深度网络进行多角度遥感影像超分辨率重建 . 武汉大学学报 (信息科学版) , 46 ( 11 ): 1716 - 1726 [ DOI: 10.13203/j.whugis20200651 http://dx.doi.org/10.13203/j.whugis20200651 ]

Cheng G and Han J W . 2016 . A survey on object detection in optical remote sensing images . ISPRS Journal of Photogrammetry and Remote Sensing , 117 : 11 - 28 [ DOI: 10.1016/j.isprsjprs.2016.03.014 http://dx.doi.org/10.1016/j.isprsjprs.2016.03.014 ]

Cheng G , Han J W and Lu X Q . 2017 . Remote sensing image scene classification: benchmark and state of the art . Proceedings of the IEEE , 105 ( 10 ): 1865 - 1883 [ DOI: 10.1109/JPROC.2017.2675998 http://dx.doi.org/10.1109/JPROC.2017.2675998 ]

Deng Z P , Sun H , Zhou S L , Zhao J P , Lei L and Zou H X . 2018 . Multi-scale object detection in remote sensing imagery with convolutional neural networks . ISPRS Journal of Photogrammetry and Remote Sensing , 145 : 3 - 22 [ DOI: 10.1016/j.isprsjprs.2018.04.003 http://dx.doi.org/10.1016/j.isprsjprs.2018.04.003 ]

Diakogiannis F I , Waldner F , Caccetta P and Wu C . 2020 . ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data . ISPRS Journal of Photogrammetry and Remote Sensing , 162 : 94 - 114 [ DOI: 10.1016/j.isprsjprs.2020.01.013 http://dx.doi.org/10.1016/j.isprsjprs.2020.01.013 ]

Dong C , Loy C C , He K M and Tang X O . 2014 . Learning a deep convolutional network for image super-resolution // 13th European Conference on Computer Vision . Zurich : Springer: 184 - 199 [ DOI: 10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13 ]

Dong C , Loy C C and Tang X O . 2016 . Accelerating the super-resolution convolutional neural network // 14th European Conference on Computer Vision . Amsterdam : Springer: 391 - 407 [ DOI: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]

Gong Y F , Liao P Y , Zhang X D , Zhang L F , Chen G Z , Zhu K , Tan X L and Lv Z Y . 2021 . Enlighten-GAN for super resolution reconstruction in mid-resolution remote sensing images . Remote Sensing , 13 ( 6 ): 1104 [ DOI: 10.3390/rs13061104 http://dx.doi.org/10.3390/rs13061104 ]

Guo Y , Chen J , Wang J D , Chen Q , Cao J Z , Deng Z S , Xu Y W and Tan M K . 2020 . Closed-loop matters: dual regression networks for single image super-resolution // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE: 5406 - 5415 [ DOI: 10.1109/CVPR42600.2020.00545 http://dx.doi.org/10.1109/CVPR42600.2020.00545 ]

Hu J F , Huang T Z , Deng L J , Dou H X , Hong D F and Vivone G . 2022 . Fusformer: a transformer-based fusion network for hyperspectral image super-resolution . IEEE Geoscience and Remote Sensing Letters , 19 : 6012305 [ DOI: 10.1109/LGRS.2022.3194257 http://dx.doi.org/10.1109/LGRS.2022.3194257 ]

Im J , Park H and Takeuchi W . 2019 . Advances in remote sensing-based disaster monitoring and assessment . Remote Sensing , 11 ( 18 ): 2181 [ DOI: 10.3390/rs11182181 http://dx.doi.org/10.3390/rs11182181 ]

Jiang K , Wang Z Y , Yi P , Wang G C , Lu T and Jiang J J . 2019 . Edge-enhanced GAN for remote sensing image superresolution . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 8 ): 5799 - 5812 [ DOI: 10.1109/TGRS.2019.2902431 http://dx.doi.org/10.1109/TGRS.2019.2902431 ]

Johnson J , Alahi A and Fei-Fei L . 2016 . Perceptual losses for real-time style transfer and super-resolution // 14th European Conference on Computer Vision . Amsterdam : Springer: 694 - 711 [ DOI: 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43 ]

Kampffmeyer M , Salberg A B and Jenssen R . 2016 . Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Las Vegas : IEEE: 680 - 688 [ DOI: 10.1109/CVPRW.2016.90 http://dx.doi.org/10.1109/CVPRW.2016.90 ]

Kim J , Lee J K and Lee K M . 2016 . Accurate image super-resolution using very deep convolutional networks // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE: 1646 - 1654 [ DOI: 10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]

Ledig C , Theis L , Huszár F , Caballero J , Cunningham A , Acosta A , Aitken A , Tejani A , Totz J , Wang Z H and Shi W Z . 2017 . Photo-realistic single image super-resolution using a generative adversarial network // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu : IEEE: 105 - 114 [ DOI: 10.1109/CVPR.2017.19 http://dx.doi.org/10.1109/CVPR.2017.19 ]

Lei S , Shi Z W and Mo W J . 2022 . Transformer-based multistage enhancement for remote sensing image super-resolution . IEEE Transactions on Geoscience and Remote Sensing , 60 : 5615611 [ DOI: 10.1109/TGRS.2021.3136190 http://dx.doi.org/10.1109/TGRS.2021.3136190 ]

Li K , Wan G , Cheng G , Meng L Q and Han J W . 2020 . Object detection in optical remote sensing images: a survey and a new benchmark . ISPRS Journal of Photogrammetry and Remote Sensing , 159 : 296 - 307 [ DOI: 10.1016/j.isprsjprs.2019.11.023 http://dx.doi.org/10.1016/j.isprsjprs.2019.11.023 ]

Liang J Y , Cao J Z , Sun G L , Zhang K , Van Gool L and Timofte R . 2021 . SwinIR: image restoration using swin transformer // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops . Montreal : IEEE: 1833 - 1844 [ DOI: 10.1109/ICCVW54120.2021.00210 http://dx.doi.org/10.1109/ICCVW54120.2021.00210 ]

Lim B , Son S , Kim H , Nah S and Lee K M . 2017 . Enhanced deep residual networks for single image super-resolution // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu : IEEE: 1132 - 1140 [ DOI: 10.1109/CVPRW.2017.151 http://dx.doi.org/10.1109/CVPRW.2017.151 ]

Liu Z , Lin Y T , Cao Y , Hu H , Wei Y X , Zhang Z , Lin S and Guo B N . 2021 . Swin transformer: hierarchical vision transformer using shifted windows // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 9992 - 10002 [ DOI: 10.1109/ICCV48922.2021.00986 http://dx.doi.org/10.1109/ICCV48922.2021.00986 ]

Lu X Q , Zheng X T and Yuan Y . 2017 . Remote sensing scene classification by unsupervised representation learning . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 9 ): 5148 - 5157 [ DOI: 10.1109/TGRS.2017.2702596 http://dx.doi.org/10.1109/TGRS.2017.2702596 ]

Ma J Q , Liang Z T and Zhang L . 2022 . A text attention network for spatial deformation robust scene text image super-resolution // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans : IEEE: 5901 - 5910 [ DOI: 10.1109/CVPR52688.2022.00582 http://dx.doi.org/10.1109/CVPR52688.2022.00582 ]

Mei Y Q , Fan Y C and Zhou Y Q . 2021 . Image super-resolution with non-local sparse attention // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville : IEEE: 3516 - 3525 [ DOI: 10.1109/CVPR46437.2021.00352 http://dx.doi.org/10.1109/CVPR46437.2021.00352 ]

Shi W Z , Caballero J , Huszár F , Totz J , Aitken A P , Bishop R , Rueckert D and Wang Z H . 2016 . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE: 1874 - 1883 [ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]

Tang X T , Yang X , Li F , Ma J and Liang L . Super-resolution reconstruction of hyper-temporal remote sensing images based on self-attention . National Remote Sensing Bulletin (唐晓天, 杨雪, 李峰, 马骏, 梁亮. 基于自注意力的超时相遥感图像超分辨率重建. 遥感学报 [DOI: 10.11834/jrs.20221825]

Tian C W , Yuan Y X , Zhang S C , Lin C W , Zuo W M and Zhang D . 2022 . Image super-resolution with an enhanced group convolutional neural network . Neural Networks , 153 : 373 - 385 [ DOI: 10.1016/j.neunet.2022.06.009 http://dx.doi.org/10.1016/j.neunet.2022.06.009 ]

Twumasi N Y D , Shao Z F and Orhan A . 2019 . Remote sensing and GIS methods in urban disaster monitoring and management–an overview . International Journal of Trend in Scientific Research and Development , 3 ( 4 ): 918 - 926 [ DOI: 10.31142/ijtsrd23976 http://dx.doi.org/10.31142/ijtsrd23976 ]

Voinov S , Schwarz E , Krause D and Berg M . 2018 . Processing framework to support maritime surveillance applications based on optical remote sensing images // Proceedings Volume 10773, Sixth International Conference on Remote Sensing and Geoinformation of the Environment . Paphos : SPIE: 268 - 273 [ DOI: 10.1117/12.2326058 http://dx.doi.org/10.1117/12.2326058 ]

Wang P , Yao H Y and Zhang G . 2021 . Super-resolution flood inundation mapping for multispectral image based on super-pixel scale . National Remote Sensing Bulletin , 25 ( 2 ): 641 - 652

王鹏 , 姚红雨 , 张弓 . 2021 . 超像元尺度下的多光谱图像超分辨率洪水淹没制图 . 遥感学报 , 25 ( 2 ): 641 - 652 [ DOI: 10.11834/jrs.20210199 http://dx.doi.org/10.11834/jrs.20210199 ]

Wang Q , Zhang X D , Chen G Z , Dai F , Gong Y F and Zhu K . 2018 . Change detection based on Faster R-CNN for high-resolution remote sensing images . Remote Sensing Letters , 9 ( 10 ): 923 - 932 [ DOI: 10.1080/2150704X.2018.1492172 http://dx.doi.org/10.1080/2150704X.2018.1492172 ]

Wang T F , Xie J X , Sun W X , Yan Q and Chen Q F . 2021 . Dual-camera super-resolution with aligned attention modules // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 1981 - 1990 [ DOI: 10.1109/ICCV48922.2021.00201 http://dx.doi.org/10.1109/ICCV48922.2021.00201 ]

Xia G S , Hu J W , Hu F , Shi B G , Bai X , Zhong Y F , Zhang L P and Lu X Q . 2017 . AID: a benchmark data set for performance evaluation of aerial scene classification . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 7 ): 3965 - 3981 [ DOI: 10.1109/TGRS.2017.2685945 http://dx.doi.org/10.1109/TGRS.2017.2685945 ]

Xie J , He N J , Fang L Y and Plaza A . 2019 . Scale-free convolutional neural network for remote sensing scene classification . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 9 ): 6916 - 6928 [ DOI: 10.1109/TGRS.2019.2909695 http://dx.doi.org/10.1109/TGRS.2019.2909695 ]

Xu Y Y , Luo W , Hu A N , Xie Z , Xie X J and Tao L F . 2022 . TE-SAGAN: an improved generative adversarial network for remote sensing super-resolution images . Remote Sensing , 14 ( 10 ): 2425 [ DOI: 10.3390/rs14102425 http://dx.doi.org/10.3390/rs14102425 ]

Yang Y and Newsam S . 2010 . Bag-of-visual-words and spatial extensions for land-use classification // Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems . San Jose : ACM: 270 - 279 [ DOI: 10.1145/1869790.1869829 http://dx.doi.org/10.1145/1869790.1869829 ]

Ye P . 2022 . Remote sensing approaches for meteorological disaster monitoring: recent achievements and new challenges . International Journal of Environmental Research and Public Health , 19 ( 6 ): 3701 [ DOI: 10.3390/ijerph19063701 http://dx.doi.org/10.3390/ijerph19063701 ]

Yuan X H , Shi J F and Gu L C . 2021 . A review of deep learning methods for semantic segmentation of remote sensing imagery . Expert Systems with Applications , 169 : 114417 [ DOI: 10.1016/j.eswa.2020.114417 http://dx.doi.org/10.1016/j.eswa.2020.114417 ]

Zhang K , Liang J Y , Van Gool L and Timofte R . 2021 . Designing a practical degradation model for deep blind image super-resolution // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 4771 - 4780 [ DOI: 10.1109/ICCV48922.2021.00475 http://dx.doi.org/10.1109/ICCV48922.2021.00475 ]

Zhang Y , Tian Y , Kong Y , Zhong B and Fu Y . 2018a . Residual dense network for image super-resolution . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 2472 - 2481

Zhang S M , Wu R Z , Xu K Y , Wang J M and Sun W W . 2019 . R-CNN-based ship detection from high resolution remote sensing imagery . Remote Sensing , 11 ( 6 ): 631 [ DOI: 10.3390/rs11060631 http://dx.doi.org/10.3390/rs11060631 ]

Zhang Y , Li K , Li K , Wang L , Zhong B and Fu Y . 2018b . Image super-resolution using very deep residual channel attention networks/ /Proceedings of the European conference on computer vision (ECCV). 286 - 301

Zhen J N , Jiang X P , Zhao D M , Wang J J , Miao J and Wu G F . 2022 . Retrieving canopy nitrogen content of mangrove forests from Sentinel-2 super-resolution reconstruction data . National Remote Sensing Bulletin , 26 ( 6 ): 1206 - 1219

甄佳宁 , 蒋侠朋 , 赵德梅 , 王俊杰 , 苗菁 , 邬国锋 . 2022 . 利用Sentinel-2影像超分辨率重建的红树林冠层氮含量反演 . 遥感学报 , 26 ( 6 ): 1206 - 1219 [ DOI: 10.11834/jrs.20221461 http://dx.doi.org/10.11834/jrs.20221461 ]

文章被引用时，请邮件提醒。

提交

基于类脑脉冲神经网络的遥感图像检测算法

模态间匹配学习的高光谱和激光雷达联合分类

改进CenterNet在遥感图像目标检测中的应用

遥感图像倾斜边界框目标检测研究进展与展望

基于深度学习的高分四号卫星图像超分辨率重建