M-ReDet: A mamba-based method for remote sensing ship object detection and fine-grained recognition.

Semantic segmentation is a fundamental task in remote sensing image processing. The large appearance variations of ground objects make this task quite challenging. Recently, deep convolutional neural networks (DCNNs) have shown outstanding performance in this task. A common strategy of these methods (e.g., SegNet) for performance improvement is to combine the feature maps learned at different DCNN layers. However, such a combination is usually implemented via feature map summation or concatenation, indicating that the features are considered indiscriminately. In fact, features at different positions contribute differently to the final performance. It is advantageous to automatically select adaptive features when merging different-layer feature maps. To achieve this goal, we propose a gated convolutional neural network to fulfill this task. Specifically, we explore the relationship between the information entropy of the feature maps and the label-error map, and then a gate mechanism is embedded to integrate the feature maps more effectively. The gate is implemented by the entropy maps, which are generated to assign adaptive weights to different feature maps as their relative importance. Generally, the entropy maps, i.e., the gates, guide the network to focus on the highly-uncertain pixels, where detailed information from lower layers is required to improve the separability of these pixels. The selected features are finally combined to feed into the classifier layer, which predicts the semantic label of each pixel. The proposed method achieves competitive segmentation accuracy on the public ISPRS 2D Semantic Labeling benchmark, which is challenging for segmentation by only using the RGB images.

In the complex and changeable actual scene, the identification of rice diseases generally has the problem of small and dense targets. The current method mainly introduces the spatial regularization loss into the Fast R-CNN framework, and integrates the detection and technical tasks into a model. Due to the limitations of this method in the actual scene, a rice disease recognition method based on improved Mask R-CNN is proposed in this paper. This proposed method mainly improves the feature fusion process of the feature pyramid, changes the original top-down path to bottom-up to retain more spatial position information in the shallow feature map, and adds multi-scale dilation convolution in the feature fusion process to increase the receptive field of the feature map and keep the resolution unchanged, which can avoid the loss of features in the down-sampling process; the dataset was established and labeled by using the photographs of rice bacterial blight. The proposed algorithm is compared with the Mask R-CNN algorithm. When the IOU is 0.6, 0.7 and 0.8, the accuracy is increased by 0.46, 0.66 and 0.48 respectively. Experimental results show that the algorithm can get more accurate results on the established dataset.

Traffic forecasting with graph spatial–temporal position recurrent network

The cross-view geo-localization task aims to accurately retrieve a location using images captured from different platforms, such as satellites and drones, which is particularly challenging due to a large variation in viewpoint. Current methods mainly focus on rigid strategies like partitioning or sorting local features, which may be ill-suited to accommodate the variance of viewpoint and distance scale in different camera perspectives. To address these issues, we propose a novel method called query-driven feature learning (QDFL) to query viewpoint-invariant feature vectors autonomously. Our method incorporates an adaptive query embedding unit (AQEU) and a feature fusion unit (FFU). AQEU adjusts feature map and implements a coarse query process to extract the contextual clues. FFU further refines feature map, fusing it at spatial and channel dimensions, tending to withdraw more fine-grained features. Subsequently, AQEU executes a fine query on salient landmarks in the fused feature map, enhancing the minutia descriptive power of query vectors. Additionally, we employ parameter-efficient transfer learning manner by integrating tunable adapters into the frozen pre-trained backbone, maintaining feature representation capabilities of foundation models while enabling seamless adaptation to cross-view geo-localization task. Extensive experiments show that our method achieves state-of-the-art performances on two well-known datasets, University-1652 and SUES-200. Moreover, our method exhibits an excellent generalizability compared with current state-of-the-art methods in cross-dataset experiments. The code is available at https://github.com/Shuyu-Hu/QDFL.

Pansharpening technology plays a crucial role in remote sensing image processing by integrating low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This process addresses the limitations of satellite sensors, which cannot directly capture HRMS images. Despite significant developments achieved by deep learning-based pansharpening methods over traditional approaches, most existing techniques either fail to account for the modal differences between LRMS and PAN images, relying on direct concatenation, or use similar network structures to extract spectral and spatial information. Additionally, many methods neglect the extraction of common features between LRMS and PAN images and lack network architectures specifically designed to extract spectral features. To address these limitations, this study proposed a novel three-branch pansharpening network that leverages both spatial and frequency domain interactions, resulting in improved spectral and spatial fidelity in the fusion outputs. The proposed method was validated on three datasets, including IKONOS, WorldView-3 (WV3), and WorldView-4 (WV4). The results demonstrate that the proposed method surpasses several leading techniques, achieving superior performance in both visual quality and quantitative metrics.

There are many approaches to fuse panchromatic (PAN) and multispectral (MS) images for classification, mainly including sharpening-then-classification methods, classification-then-sharpening methods, and segmentation-then-classification methods. The generalized Chinese restaurant franchise (gCRF) is a segmentation-then-classification-like method to fuse very high resolution (VHR) PAN and MS images for classification, which has the limitation the same as that of the general segmentation-then-classification methods that segmentation errors will affect the subsequent classification. The problems of gCRF are that during the segmentation step, the spatial coherence in the image plane is deficient and the global clusters without spatial position information are used for segmentation, which may lead to undersegmented and disconnected regions in the segmentation results and decrease classification accuracy. In this paper, we propose an improved model, which overcomes the problems of the gCRF during the segmentation step, to increase the classification accuracy by the following two ways: 1) building the spatial coherence in the image plane by introducing neighborhood information of superpixels to construct the subimages and 2) using localized clusters with spatial location information instead of global clusters to measure the similarity between superpixels and segments. The experimental results show that the problems of undersegmentation and disconnected segments are both alleviated, resulting in better classification results in terms of the visual and quantitative aspects.

Recent studies show a significant growth in semantic segmentation. However, many semantic segmentation models still have a large number of parameters, making them unsuitable for resource-constrained embedded devices. To address this issue, we propose an efficient Shared Feature Reuse Segmentation (SFRSeg) model containing several novelties: a new yet effective shared-branch multiple sub-encoders design, a context mining module and a semantic aggregating module for better context granularity. In particular, our shared-branch approach improves the entire feature hierarchy by sharing the spatial and context knowledge in both shallow and deep branches. After every shared point in each sub-encoder, a proposed cascading context mining (CCM) module is deployed to filter out the noisy spatial details from the feature maps and provides a diverse size of receptive fields for capturing the latent context between multi-scale geometric shapes in the scene. To overcome the gradient vanishing issue at the early stage, we reduce the number of layers in the first sub-encoder and employ a unique multiple sub-encoders design which reprocesses the rich global feature maps through multiple sub-encoders for better feature refinement. Later, the rich semantic features generated by the efficient sub-encoders at different levels are fused by the proposed Hybrid Path Attention Semantic Aggregation (HPA-SA) module that effectively reduces the semantic gap between feature maps at different levels and alleviate the well-known boundary degeneration effect. To make it computationally efficient for resource-constrained embedded devices, a series of lightweight methods such as a lightweight encoder, a squeeze-and-excitation design, separable convolution filters, channel reduction (CR) are carefully exploited. With an exceptional performance on Cityscapes (70.6% test mIoU) and CamVid (74.7% test mIoU) data sets, the proposed model is shown to be superior over existing light real-time semantic segmentation models whilst having only 1.6 million parameters.

Accurate segmentation of medical images is crucial for disease diagnosis and understanding disease changes. Deep learning methods, utilizing encoder-decoder structures, have demonstrated cutting-edge performance in various medical image segmentation tasks. However, the pooling operation in the encoding stage results in feature loss, which makes the network lack the ability to fuse multi-scale information at different levels, hinders its effective perception of multi-scale information, and leads to poor segmentation performance. Drawing inspiration from the U-shaped network, this study introduces a multi-branch feature hybrid attention and adaptive receptive field network (MFHARFNet) for medical image segmentation. Building upon the encoder-decoder framework, we initially devise a multi-branch feature hybrid attention module (MFHAM) to seamlessly integrate feature maps of varying scales, capturing both fine-grained features and coarse-grained semantics across the entire scale. Furthermore, we redesign the skip connection to amalgamate feature information from different branches in the encoder stage and efficiently transmit it to the decoder, providing the decoder with global context feature maps at different levels. Finally, the adaptive receptive field (ARF) module is introduced in the decoder feature reconstruction stage to adapt and focus on related fields, ensuring the model’s adaptation to different segmentation target features, and achieving different weights for the output of different convolution kernels to improve segmentation performance. We comprehensively evaluate our method on medical image segmentation tasks, by using four public datasets across CT and MRI. Remarkably, MFHARFNet method consistently outperforms other state-of-the-art methods, exceeding UNet by 2.1%, 0.9%, 6.6% and 1.0% on Dice on ATLAS, LiTs, BraTs2019 and Spine and intervertebral disc datasets, respectively. In addition, MFHARFNet minimizes network parameters and computational complexity as much as possible. The source codes are in https://github.com/OneHundred99/MFHARFNet.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

Learning subtle and distinctive features is the key to fine-grained object recognition. Previous approaches use attention mechanisms to localize discriminatory regions and learn fine-grained details, which often treat each object part attention independently while neglecting the correlations among them and may lead them attentive to the same distinctive features. In this paper, we proposed a novel fine-grained visual recognition method to solve this problem by adversarial complementary attention (ACA) strategy and hierarchical bilinear pooling. Our method learns an attention-based classification branch through an attention module, and then train a counterpart classifier branch with adversarial complementary attention parts by erasing attentive regions in the first network branch from the feature maps. To capture different attention features interactions, a hierarchical bilinear pooling framework is used. Our method can be trained end-to-end without any bounding box/part annotations. Extensive experiments indicate that our approach is efficient and achieves the state-of-the-art performance on widely used fine-grained classification datasets, including CUB-200-2011, Stanford Cars, and FGVC-Aircraft datasets.

Hyperspectral image (HSI) classification has been extensively applied for analyzing remotely sensed images. HSI data consist of multiple bands that provide abundant spatial information. Convolutional neural networks (CNNs) have emerged as powerful deep learning methods for processing visual data. In recent work, CNNs have shown impressive results in HSI classification. In this paper, we propose a hierarchical neural network architecture called feature extraction with hybrid spectral CNN (FE-HybridSN) to extract superior spectral–spatial features. FE-HybridSN effectively captures more spectral–spatial information while reducing computational complexity. Considering the prevalent issue of class imbalance in experimental datasets (IP, UP, SV) and real-world hyperspectral datasets, we apply the focal loss to mitigate these problems. The focal loss reconstructs the loss function and facilitates effective achievement of the aforementioned goals. We propose a framework (FEHN-FL) that combines FE-HybridSN and the focal loss for HSI classification and then conduct extensive HSI classification experiments using three remote sensing datasets: Indian Pines (IP), University of Pavia (UP), and Salinas Scene (SV). Using cross-entropy loss as a baseline, we assess the hyperspectral classification performance of various backbone networks and examine the influence of different spatial sizes on classification accuracy. After incorporating focal loss as our loss function, we not only compare the classification performance of the FE-HybridSN backbone network under different loss functions but also evaluate their convergence rates during training. The proposed classification framework demonstrates satisfactory performance compared to state-of-the-art end-to-end deep-learning-based methods, such as 2D-CNN, 3D-CNN, etc.

Learning how to capture long-range dependencies and restore spatial information of down-sampled feature maps are the basis of the encoder-decoder structure networks in medical image segmentation. U-Net based methods use feature fusion to alleviate these two problems, but the global feature extraction ability and spatial information recovery ability of U-Net are still insufficient. In this paper, we propose a Global Feature Reconstruction (GFR) module to efficiently capture global context features and a Local Feature Reconstruction (LFR) module to dynamically up-sample features, respectively. For the GFR module, we first extract the global features with category representation from the feature map, then use the different level global features to reconstruct features at each location. The GFR module establishes a connection for each pair of feature elements in the entire space from a global perspective and transfers semantic information from the deep layers to the shallow layers. For the LFR module, we use low-level feature maps to guide the up-sampling process of high-level feature maps. Specifically, we use local neighborhoods to reconstruct features to achieve the transfer of spatial information. Based on the encoder-decoder architecture, we propose a Global and Local Feature Reconstruction Network (GLFRNet), in which the GFR modules are applied as skip connections and the LFR modules constitute the decoder path. The proposed GLFRNet is applied to four different medical image segmentation tasks and achieves state-of-the-art performance.

Change detection (CD) is an essential and challenging task in remote sensing image processing. Its performance relies heavily on the exploitation of spatial image information and the extraction of change semantic information. Although some deep feature-based methods have been successfully applied to change detection, most of them use plain encoders to extract the original image features. The plain encoders often have the below disadvantages: (i) the lack of semantic information leads to lower discrimination of shallow features, and (ii) the successive down-sampling leads to less accurate spatial localization of deep features. These problems affect the performance of the network in complex scenes and are particularly detrimental to the detection of small objects and object edges. In this paper, we propose a full-scale feature fusion siamese network (F3SNet), which on one hand enhances the spatial localization of deep features by densely connecting raw image features from shallow to deep layers, and on the other hand, complements the changing semantics of shallow features by densely connecting the concatenated feature maps from deep to shallow layers. In addition, a full-scale classifier is proposed for aggregating feature maps at different scales of the decoder. The full-scale classifier in nature is a variant of full-scale deep supervision, which generates prediction maps at all scales of the decoder and then combines them for the final classification. Experimental results show that our method significantly outperforms other state-of-the-art (SOTA) CD methods, and is particularly beneficial for detecting small objects and object edges. On the LEVIR-CD dataset, our method achieves an F1-score of 0.905 using only 0.966M number of parameters and 3.24 GFLOPs.

Left frontal eye field remembers “where” but not “what”

The traditional pulse coupled neural network model only considers the spatial position information between neurons, but neglects sensitivity of human visual perception to luminance change, especially when the brightness is high or low, the visual sensitivity of the human eye is relatively low, and in a certain range of brightness, the visual sensitivity of the human eye is relatively high, so it is easy to have false segmentation. In order to solve this problem of the traditional pulse coupled neural network model, this paper redescribes the connection weights between neurons by using the human eye luminance weight factor, and obtains the new pulse coupled neural network model based on visual perception. Through the simulation experiment of life images, experimental results show that the proposed algorithm is better than the traditional pulse coupled neural network model.