A lightweight deep convolutional network with inverted residuals for matching optical and SAR images

Due to the differences in radiation and geometric characteristics of optical and synthetic aperture radar (SAR) images, there is still a huge challenge for accurate matching. In this paper, we propose a patch-matching network (PM-Net) to improve the matching performance of optical and SAR images. First, a multi-level keypoints detector (MKD) with fused high-level and low-level features is presented to extract more robust keypoints from optical and SAR images. Second, we use a two-channel network structure to improve the image patch matching performance. Benefiting from this design, the proposed method can directly learn the similarity between optical and SAR image patches without manually designing features and descriptors. Finally, the MKD and two-channel net-work are trained separately on GL3D and QXS-SAROPT data sets, and the PM-Net is tested on multiple pairs of optical and SAR images. The experimental results demonstrate that the proposed method outperforms four advanced image matching networks on qualitative and quantitative assessments. The quantitative experiment results show that using our method correct matching points numbers are increased by more than 1.15 times, the value of F1-measure is raised by an average of 7.4% and the root mean squared error (RMSE) is reduced by more than 15.3%. The advantages of MKD and the two-channel network are also verified through ablation experiments.

The KAZE algorithm has been popularly used to match the remote sensing images. However, it has limited performance when its standard form is directly applied to match the synthetic aperture radar (SAR) images containing significant multiplicative speckle noise. In this paper, an effective SAR image matching algorithm is proposed, which is a combination of KAZE, phase congruency, and speckle noise removing anisotropic diffusion. In this algorithm, the ratio-based Phase Congruency (PC) information is used to eliminate the erroneous features, which improves the repeatability rate of the KAZE features. In addition, the scale layers of the input SAR images are constructed by speckle noise removing anisotropic diffusion method which significantly reduces the effect of noise. The proposed method can improve the repeatability of the extracted KAZE features. Moreover, it can give better recall and precision values than the state-of-the-art methods. It gives the root mean square error (RMSE) values in the range of 0.881 to 1.104 pixels in SAR image matching. Experiments on multi-temporal SAR images demonstrate the applicability of the proposed method.

Abstract. Complex geometric distortions and nonlinear radiation differences between optical and synthetic aperture radar (SAR) images present challenges for the matching of sufficient and evenly distributed corresponding points. To address this problem, this paper proposes a deep convolutional network based on an attention mechanism for matching optical and SAR images. In order to obtain robust feature points, we employ phase consistency instead of image intensity and gradient information for feature detection. A deep convolutional network (DCN) is designed to extract high-level semantic features between optical and SAR images, providing robustness to geometric distortion and nonlinear radiation changes. Notably, incorporating multiple inverted residual structures in the DCN facilitates efficient extraction of local and global features, promoting feature reuse, and reducing the loss of key features. Furthermore, a dense feature fusion module based on coordinate attention is designed, focusing on the spatial positional information of effective features, integrating key features into deep descriptors to enhance the robustness of deep descriptors to nonlinear radiometric differences. A coarse-to-fine strategy is then employed to enhance accuracy by eliminating mismatches. Experimental results demonstrate that the proposed network performs better than the manually designed descriptors-based methods and the stateof- the-art deep learning networks in both matching effectiveness and accuracy. Specifically, the number of matches achieved is approximately 2 times greater than that of other methods, with a 10% improvement in F-measure.

In the field of remote sensing, cloud cover severely reduces the quality of satellite observations of the earth. Due to the complete absence of information in cloud-covered regions, cloud removal with a single optical image is an ill-posed problem. Since the synthetic aperture radar (SAR) can effectively penetrate clouds, fusing SAR and optical remote sensing images will effectively alleviate this problem. However, existing SAR-based optical cloud removal methods fail to effectively leverage the global information provided by the SAR image, resulting in limited performance gains. In this paper, we introduce a novel cloud removal method named the Multi-Level SAR-Guided Contextual Attention Network (MSGCA-Net). MSGCA-Net is designed with a multi-level architecture that integrates a SAR-Guided Contextual Attention (SGCA) module to fuse the dependable global contextual information from SAR images with the local features of optical images effectively. In the module of SGCA, the SAR image provides reliable global contextual information and genuine structure of cloud-covered regions, while the optical image provides the local feature information. The proposed model can efficiently extract and fuse global and local contextual information in SAR and optical images. We trained and evaluated the performance of the model on both simulated and real-world datasets. Both qualitative and quantitative experimental evaluation demonstrated that the proposed method can yield high quality cloud-free images and outperform state-of-the-art cloud removal methods.

Thanks to the availability of large-scale data, deep Convolutional Neural Networks (CNNs) have witnessed success in various applications of computer vision. However, the performance of CNNs on Synthetic Aperture Radar (SAR) image classification is unsatisfactory due to the lack of well-labeled SAR data, as well as the differences in imaging mechanisms between SAR images and optical images. Therefore, this paper addresses the problem of SAR image classification by employing the Generative Adversarial Network (GAN) to produce more labeled SAR data. We propose special GANs for generating SAR images to be used in the training process. First, we incorporate the quadratic operation into the GAN, extending the convolution to make the discriminator better represent the SAR data; second, the statistical characteristics of SAR images are integrated into the GAN to make its value function more reasonable; finally, two types of parallel connected GANs are designed, one of which we call PWGAN, combining the Deep Convolutional GAN (DCGAN) and Wasserstein GAN with Gradient Penalty (WGAN-GP) together in the structure, and the other, which we call CNN-PGAN, applying a pre-trained CNN as a discriminator to the parallel GAN. Both PWGAN and CNN-PGAN consist of a number of discriminators and generators according to the number of target categories. Experimental results on the TerraSAR-X single polarization dataset demonstrate the effectiveness of the proposed method.

The usage of a hypersonic platform for remote sensing application has promising prospects, especially for hypersonic platform-borne synthetic aperture radar (SAR) imaging. However, the high-speed of hypersonic platform will lead to extreme friction between the platform and air, which will cause the ionization of air. The ionized gas forms the plasma sheath wrapped around the hypersonic platform. The plasma sheath will severely affect the propagation of SAR signal and further affect the SAR imaging. Therefore, hypersonic platform-borne SAR imaging should be studied from a physical perspective. In this paper, hypersonic platform-borne SAR imaging under plasma sheath is analyzed. The SAR signal propagation in plasma sheath is computed using scatter matrix method. The proposed SAR signal model is verified by using a ground experiment system. Moreover, the effect of attenuation caused by plasma sheath on SAR imaging is studied under different SAR parameters and plasma sheath. The result shows that attenuation caused by plasma sheath will degrade the SAR imaging quality and even cause the point and area targets to be submerged into the noise. The real SAR images under plasma sheath also illustrate this phenomenon. Furthermore, by studying imaging results under different SAR and plasma parameters, it can be concluded that the severe degradation of SAR imaging quality appears at condition of high plasma sheath electron density and low SAR carrier frequency. The work in this paper will be beneficial for the study of hypersonic platform-borne SAR imaging and design of hypersonic SAR imaging systems in the future.

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

Compressed sensing (CS) is currently utilized in the application of Synthetic Aperture Radar (SAR) imaging. Unfortunately, the CS SAR imaging performance degrades as SNR decreases. However, there exists a denoising convolutional neural network (DnCNN) based on 2D convolution (DnCNN-2D) in the field of image denoising. Traditional DnCNN-2D is not directly applicable to SAR imaging. Thus, a DnCNN based on 1D convolution (DnCNN-1D) for signal denoising is proposed and employed to denoise the SAR signal. The denoised SAR signal is subsequently utilized for CS SAR imaging. Each convolutional layer of DnCNN-1D has a fixed number and fixed size of filters and activation functions. The residual learning strategy and batch normalization are adopted in the DnCNN-1D. The segmental signal-to-noise ratio (SEGSNR) and the normalized mean square error (NMSE) are selected to evaluate signal denoising and SAR imaging performance, respectively. The experimental results indicate that imaging performance is significantly improved after denoising the SAR signal by the proposed method.

Convolutional neural networks (CNNs) have been applied to learn spatial features for high-resolution (HR) synthetic aperture radar (SAR) image classification. However, there has been little work on integrating the unique statistical distributions of SAR images which can reveal physical properties of terrain objects, into CNNs in a supervised feature learning framework. To address this problem, a novel end-to-end supervised classification method is proposed for HR SAR images by considering both spatial context and statistical features. First, to extract more effective spatial features from SAR images, a new deep spatial context encoder network (DSCEN) is proposed, which is a lightweight structure and can be effectively trained with a small number of samples. Meanwhile, to enhance the diversity of statistics, the nonstationary joint statistical model (NS-JSM) is adopted to form the global statistical features. Specifically, SAR images are transformed into the Gabor wavelet domain and the produced multi-subbands magnitudes and phases are modeled by the log-normal and uniform distribution. The covariance matrix is further utilized to capture the inter-scale and intra-scale nonstationary correlation between the statistical subbands and make the joint statistical features more compact and distinguishable. Considering complementary advantages, a feature fusion network (Fusion-Net) base on group compression and smooth normalization is constructed to embed the statistical features into the spatial features and optimize the fusion feature representation. As a result, our model can learn the discriminative features and improve the final classification performance. Experiments on four HR SAR images validate the superiority of the proposed method over other related algorithms.

ABSTRACTAutomatic Target Recognition (ATR) based on Synthetic Aperture Radar (SAR) images plays a key role in military applications. However, there are difficulties with this traditional recognition method. Principally, it is a challenge to design robust features and classifiers for different SAR images. Although Convolutional Neural Networks (CNNs) are very successful in many image classification tasks, building a deep network with limited labeled data remains a problem. The topologies of CNNs like the fully connected structure will lead to redundant parameters and the negligence of channel-wise information flow. A novel CNNs approach, called Group Squeeze Excitation Sparsely Connected Convolutional Networks (GSESCNNs), is therefore proposed as a solution. The group squeeze excitation performs dynamic channel-wise feature recalibration with less parameters than squeeze excitation. Sparsely connected convolutional networks are a more efficient way to operate the concatenation of feature maps from different layers. Experimental results on Moving and Stationary Target Acquisition and Recognition (MSTAR) SAR images, demonstrate that this approach achieves, at 99.79%, the best prediction accuracy, outperforming the most common skip connection models, such as Residual Networks and Densely Connected Convolutional Networks, as well as other methods reported in the MSTAR dataset.

The existing Synthetic Aperture Radar (SAR) image target detection methods based on convolutional neural networks (CNNs) have achieved remarkable performance, but these methods require a large number of target-level labeled training samples to train the network. Moreover, some clutter is very similar to targets in SAR images with complex scenes, making the target detection task very difficult. Therefore, a SAR target detection network based on a semi-supervised learning and attention mechanism is proposed in this paper. Since the image-level label simply marks whether the image contains the target of interest or not, which is easier to be labeled than the target-level label, the proposed method uses a small number of target-level labeled training samples and a large number of image-level labeled training samples to train the network with a semi-supervised learning algorithm. The proposed network consists of a detection branch and a scene recognition branch with a feature extraction module and an attention module shared between these two branches. The feature extraction module can extract the deep features of the input SAR images, and the attention module can guide the network to focus on the target of interest while suppressing the clutter. During the semi-supervised learning process, the target-level labeled training samples will pass through the detection branch, while the image-level labeled training samples will pass through the scene recognition branch. During the test process, considering the help of global scene information in SAR images for detection, a novel coarse-to-fine detection procedure is proposed. After the coarse scene recognition determining whether the input SAR image contains the target of interest or not, the fine target detection is performed on the image that may contain the target. The experimental results based on the measured SAR dataset demonstrate that the proposed method can achieve better performance than the existing methods.

In general, synthetic aperture radar (SAR) imaging and image processing are two sequential steps in SAR image processing. Due to the large size of SAR images, most image processing algorithms require image segmentation before processing. However, the existence of speckle noise in SAR images, as well as poor contrast and the uneven distribution of gray values in the same target, make SAR images difficult to segment. In order to facilitate the subsequent processing of SAR images, this paper proposes a new method that combines the back-projection algorithm (BPA) and a first-order gradient operator to enhance the edges of SAR images to overcome image segmentation problems. For complex-valued signals, the gradient operator was applied directly to the imaging process. The experimental results of simulated images and real images validate our proposed method. For the simulated scene, the supervised image segmentation evaluation indexes of our method have more than 1.18%, 11.2% and 11.72% improvement on probabilistic Rand index (PRI), variability index (VI), and global consistency error (GCE). The proposed imaging method will make SAR image segmentation and related applications easier.

Transformer-based methods have demonstrated their potential capabilities in ship detection of synthetic aperture radar (SAR) images. However, their exclusive reliance on global context information hinders the precise localization of small ships, leading to suboptimal detection performance. In this work, a global-to-local detection transformer (GL-DETR) framework is proposed to enhance the detection accuracy of small ships in SAR images. In GL-DETR, the decoder is constituted of a global layer and a carefully designed local layer. Within the global layer, object queries fully interact with global context information. Furthermore, a local interaction attention (LIA) module is designed in the local layer, with the purpose of refining and enriching the object query features through local multiscale region of interest (ROI) information. Additionally, a multiscale information enhancement (MIE) module is introduced to enhance the high-frequency information of small ships through Gaussian filtering. Numerical results demonstrate that GL-DETR outperforms baseline by 3.71% and 4.89% on classical SAR datasets LS-SSDD and HRSID, respectively. These results demonstrate the effectiveness and superiority of the proposed strategy.

This paper proposes a novel method on Synthetic Aperture Radar (SAR) Image denoising, which is based on context modeling combined with multiscale orthogonal bandlet coefficients. For the main influence of SAR image is multiplicative speckle noise, the logarithm uniform transform is used to convert it to additive noise. In this paper, the multiscale orthogonal bandlet transform is applied to noisy SAR Image. The contextual model is used to compute contextual values of bandlet coefficients on each scale. Therefore, bandlet coefficients are reconstructed into some parts by contextual values. The spatially adaptive soft-threshold (T) is to denoise SAR image in each part. Experimental results demonstrate that this method outperforms the methods of Ehance-Lee (E-lee), contextual model based on wavelet domain (Wavelet-context), and Hidden Markov Tree model bases on wavelet domain (Wavelet-hmt) on there indexes: Floating index (FI), Standard deviation and Edge saving index of vertical direction, by using a real SAR image. However, the method can not only reduce the speckle noise but also preserve edges and high frequency information, and gets considerable visual effect on SAR Image denoising.

Ship detection from synthetic aperture radar (SAR) images is a hot topic, but the difficulty in collecting labeled SAR images may hinder the development of deep-learning-based detection methods. Inspired by the idea of domain adaptation, in this article, we propose a hierarchical similarity alignment neural network (HSANet) for ship detection in SAR images, which is a domain adaptive (DA) approach with optical remote sensing images as training samples. The kernel target of HSANet is to mine and align both the global structure and the local instance information between SAR and optical images, where two modules, structural alignment module (SAM) and prototype alignment module (PAM), are designed to, respectively, conduct two hierarchies of alignment process. In general, SAM attempts to extract the global structure similarity which exists in image-level feature representation, while PAM tends to extract the local shape similarity which is instance-level representation. To be specific, SAM is developed by Fourier-based feature alignment, which tries to describe the similar structural relationship between optical and SAR images. Meanwhile, PAM is proposed based on the conjoint confidence analysis where the instance-level ship representations of the source and target domains is aligned. SAM and PAM work together to construct a hierarchical domain adaptation network for SAR ship detection. Experiments on several public datasets may indicate the effectiveness of the proposed method.