Query-Driven Feature Learning for Cross-View Geo-Localization

Advanced driver assistance systems (ADAS) mainly consist of three components: environmental perception, decision planning, and motion control. As a fundamental component of the ADAS environmental perception system, 3D object detection enables vehicles to avoid obstacles and ensure driving safety only through accurate and real‐time prediction and localization of three‐dimensional targets such as vehicles and pedestrians in road scenes. Therefore, to improve both the real‐time performance and accuracy of 3D object detection, we propose a lightweight depth prediction‐based 3D object detection model with multi‐scale fusion—SPWS‐Transformer. First, to enhance the model's accuracy, we propose a feature extraction network incorporating multi‐scale feature fusion and depth prediction. By designing a multi‐scale feature fusion module, we effectively combine multi‐scale semantic and fine‐grained information from feature maps of different scales to enhance the network's feature extraction capability. To capture spatial information from the feature maps, we apply convolution, group normalization, and nonlinear activation operations on the fused feature maps to generate depth feature maps. Both the fused feature maps and depth feature maps serve as inputs for subsequent network stages. To further improve accuracy, we leverage the long‐range modelling advantages of Transformers by designing a feature enhancement encoder to strengthen the representation capability of depth feature maps. We incorporate a dilated encoder to perform positional encoding on depth feature maps and utilize multi‐head self‐attention mechanisms to capture contextual relationships within the input scene, thereby enhancing the detection capability of the 3D object detection network. Then, to improve real‐time performance, we design a decoder structure with scale‐aware attention. By predefining masks of different scales, we adaptively learn a scale‐aware filter using depth and visual features to enhance object queries. Finally, on the KITTI dataset, the improved algorithm achieves an AP of 24.66% for the car category, with more significant improvements in detection accuracy under the ‘hard’ difficulty level. The model achieves an inference time of 24 ms.

To remove image haze and make haze image scene clear, we proposed an image dehazing network based on multi-scale feature extraction (MSFNet) in this paper. The MSFNet first directly performs feature extraction on hazy images with three different resolutions to obtain fine feature maps and concatenates them with the rough feature maps extracted in the downsampling process for fusing and obtaining richer image information. Then, the fused feature maps are put into a network module composed of ResNeXt building blocks for network learning. Next, the feature maps extracted by upsampling are sequentially concatenated with the feature maps learned by the ResNeXt module for obtaining the residual image. Finally, the learned residual image is added to the input hazy image to obtain the image dehazing result. The experimental results on the SOTS dataset show that the MSFNet improves effectiveness of image dehazing.KeywordsImage dehazingConvolutional neural networksMulti-scale feature

We propose a learning framework named Feature Fusion Learning (FFL) that efficiently trains a powerful classifier through a fusion module which combines the feature maps generated from parallel neural networks and generates meaningful feature maps. Specifically, we train a number of parallel neural networks as sub-networks, then we combine the feature maps from each sub-network using a fusion module to create a more meaningful feature map. The fused feature map is passed into the fused classifier for overall classification. Unlike existing feature fusion methods, in our framework, an ensemble of sub-network classifiers transfers its knowledge to the fused classifier and then the fused classifier delivers its knowledge back to each subnetwork, mutually teaching one another in an online-knowledge distillation manner. This mutually teaching system not only improves the performance of the fused classifier but also obtains performance gain in each sub-network. Moreover, our model is more beneficial than other alternative methods because different types of network can be used for each sub-network. We have performed a variety of experiments on multiple datasets such as CIFAR-10, CIFAR-100 and ImageNet and proved that our method is more effective than other alternative methods in terms of performances of both sub-networks and the fused classifier, and the aspect of generating meaningful feature maps.

3D human pose estimation (3D-HPE) has emerged as a prominent research area with diverse applications. This work focuses on enhancing the accuracy of 3D-HPE by proposing a two-stage model with a multi-feature fusion approach. The proposed model utilizes convolutional kernels of different sizes to extract feature maps with diverse resolutions and dimensions. These feature maps, along with the 2D coordinates of key joint points from the input frame, are fused in the first stage. In the second stage, the fused feature map is combined with the feature points of 2D key joints to jointly predict the key joints in 3D space. Experimental evaluations demonstrate the superiority of the proposed model over representative methods. It achieves significant improvements of 9.47% and 8.55% in average MPJPE and average P-MPJPE, respectively, which are critical metrics for evaluating pose estimation accuracy. The proposed two-stage model with multi-feature fusion offers a comprehensive and accurate approach to 3D-HPE. It outperforms existing methods and showcases its effectiveness in capturing the intricate details of human poses. The results validate the significance of the proposed model in advancing the field of 3D-HPE.

Effectively capturing the multi-scale, multi-channel, and boundary features of target images is beneficial for improving the accuracy of salient object detection. A multichannel feature enhancement module (MFEM) is constructed by concatenating two feature maps along the channel, performing convolution activation, dot product addition, and convolution operations. An edge feature enhancement module (EFEM) is constructed by performing an inverse attention mechanism and Gaussian-Laplace transform on the predicted map, channel-wise concatenation, and convolution operations with the feature submap. Multi-scale features are extracted using a SwinTransformer encoder and enhanced using the MFEM. The enhanced features are then fused using a cross-attention mechanism for multi-level features. The fused feature map, the high-frequency feature map obtained by Gaussian-Laplace transforming the predicted map, and the external edge features are input into the EFEM to obtain the salient object detection results. This results in a novel salient object detection model with multi-scale, multi-channel, and boundary feature enhancement: MfENet. Experimental results show that the Mean Absolute Error (MAE), S-measure and F-measure accuracies of our method on the HKU-IS dataset are 0.018,0.924 and 0.959 respectively. These results demonstrate the effectiveness of our method. Furthermore, this work highlights the importance of multi-scale and boundary awareness in enhancing salient object detection, and its contribution to the technical field relevant to the theme of this conference.

In this study, a video deblurring approach using transformer-based deep neural networks is proposed, which consists of feature extraction module, feature alignment module, temporal fusion transformer, and reconstruction transformer. Feature extraction module is employed to extract frequency and spatial domain features of video frames. Feature alignment module is employed to bi-directionally propagate feature maps and utilize optical flow to guide the alignment of feature maps. Temporal fusion transformer is employed to build correspondences across neighboring frames and fully utilize sharp details in neighboring frames. The fused feature map is fed into reconstruction transformer to gradually reconstruct fine-grained details and generate the residual frame. Finally, each deblurred frame is obtained by adding the residual frame to the corresponding input frame. Based on the experimental results obtained in this study, in terms of two objective performance metrics and subjective evaluation, the performance of the proposed approach is better than those of four comparison approaches.

Semantic segmentation of high-resolution optical remote sensing images is an important but challenging task. To solve the problem that many semantic segmentation networks fail to efficiently utilize global and local context information to improve the segmentation performance, this paper proposes a semantic segmentation network based on sparse self-attention (SDANet) to model the global context dependencies. Specifically, the feature maps are first divided into four regions in spatial and channel dimensions, respectively, and the divided feature maps are rearranged to form new regions. Second, the position and channel self-attention operations are performed on the rearranged regions. Third, the feature maps are restored to the original combination and the position together with channel self-attention operations are performed again to obtain the output feature maps. Finally, semantic segmentation is completed based on the output feature maps. Extensive experiments conducted on the ISPRS Vaihingen dataset demonstrate that the proposed method is superior to self-attention-based DANet, CCNet, and other general semantic segmentation networks, such as FCN, Deeplabv3+, HRNet, etc.

Fusing two-stream convolutional neural networks for RGB-T object tracking

Traditional tooth image analysis methods primarily focus on feature extraction from individual images, often overlooking critical tooth shape and position information. This paper presents a novel computer-aided diagnosis method, Collaborative learning with Mask Region-based Convolutional Neural Network (Co-Mask R-CNN), designed to enhance tooth image analysis by leveraging the integration of complementary information. First, image enhancement is employed to generate an edge-enhanced tooth edge image. Then, a collaborative learning strategy combined with Mask R-CNN is introduced, where the original and edge images are input simultaneously, and a two-stream encoder extracts feature maps from complementary images. By utilizing an attention mechanism, the output features from the two branches are dynamically fused, quantifying the relative importance of the two complementary images at different spatial positions. Finally, the fused feature map is utilized for tooth instance segmentation. Extensive experiments are conducted using a proprietary dataset to evaluate the effectiveness of Co-Mask R-CNN, and the results are compared against those of an alternative segmentation network. The results demonstrate that Co-Mask R-CNN outperforms the other networks in terms of both segmentation accuracy and robustness. Consequently, this method holds considerable promise for providing medical professionals with precise tooth segmentation results, establishing a reliable foundation for subsequent tooth disease diagnosis and treatment.

In order to solve the problems of low efficiency, small scale, and data reliability in the traditional manual counting method, a method using mask region convolutional neural network (Mask R-CNN) to automatically calculate attendance was proposed. In order to extract deeper image information, the feature extraction network is designed as ResNet101, and feature map fusion is performed on multi-level feature maps. In order to make up for the lack of recognition of the objects whose body parts are occluded, the Mask R-CNN algorithm is used for the second recognition. The experimental results on the self-built classroom monitoring screenshot data set show that compared with the method of directly using the Mask R-CNN algorithm for recognition, the secondary recognition method can identify more targets and improve the accuracy of identifying the number of people.

The existing learning-based image feature point extraction methods ignore the contextual information in the image, and perform poorly in repetitive and weak texture scenes such as forest scenes, which brings great difficulties to the 3D reconstruction task of forest scenes. In order to solve the above problems, this paper proposes a new self-supervised image feature point extraction method with attention mechanism. First, the input image is processed by an encoder and encoded into a feature map containing multi-scale information. Then, the feature map is fed into the dual attention mechanism module, which includes a position attention module and a channel attention module, so as to obtain the image feature correlation in the spatial dimension and the channel dimension respectively, and obtain the enhanced feature map. Finally, the enhanced feature map is used to train the detector and descriptor respectively. Experiments show that our method has achieved some improvement on both the public dataset Hpatches and the virtual dataset Tartan Air, especially in the weakly textured forest scene.

Ship object detection and fine-grained recognition of remote sensing images are hot topics in remote sensing image processing, with applications in fishing vessel operation command, merchant ship navigation route planning, and other fields. In order to improve the detection accuracy for different types of remote sensing ship objects, this paper proposes a ship object perception and feature refinement method based on the improved ReDet, called Mamba-ReDet (M-ReDet). First, this paper designs a ship object fine-grained feature extraction backbone (Mamba-ReResNet, M-ReResNet), which selects and reconstructs the unique features of different types of ship objects through the Mamba's selective memory to improve the algorithm's ability to extract fine-grained features. Secondly, the M-ReDet consists of the Ship Object Perception Module (SOPM) and the Ship Feature Refinement Module (SFRM), which can extract the ship's spatial position information from the feature map, fuse different scales of spatial position information and use this information to refine the fine-grained features to improve the detection accuracy of the algorithm for different categories of ships. Finally, we use the KFIoU and Focal Loss as the regression loss and classification loss of the algorithm to improve the accuracy of the training. The experimental results show that the mAP0.5 of the M-ReDet algorithm on the FAIR1M(ship) and DOTAv1.0 visible light (RGB) remote sensing image datasets are 43.29% and 82.09%, respectively, which is 2.78% and 3.34% higher than that of the ReDet.

Since the non-specificity of acute bilirubin encephalopathy (ABE), accurate classification based on structural MRI is intractable. Due to the complexity of the diagnosis, multi-modality fusion has been widely studied in recent years. The most current medical image classification researches only fuse image data of different modalities. Phenotypic features that may carry useful information are usually excluded from the model. In the paper, a multi-modal fusion strategy for classifying ABE was proposed, which combined the different modalities of MRI with clinical phenotypic data. The baseline consists of three individual paths for training different MRI modalities i.e., T1, T2, and T2-flair. The feature maps from different paths were concatenated to form multi-modality image features. The phenotypic inputs were encoded into a two-dimensional vector to prevent the loss of information. The Text-CNN was applied as the feature extractor of the clinical phenotype. The extracted text feature map will be concatenated with the multi-modality image feature map along the channel dimension. The obtained MRI-phenotypic feature map is sent to the fully connected layer. We trained/tested (80%/20%) the approach on a database containing 800 patients data. Each sample is composed of three modalities 3D brain MRI and its corresponding clinical phenotype data. Different comparative experiments were designed to explore the fusion strategy. The results demonstrate that the proposal achieves an accuracy of 0.78, a sensitivity of 0.46, and a specificity of 0.99, which outperforms the model using MRI or clinical phenotype as input alone. Our work suggests the fusion of clinical phenotype data and image data can improve the performance of ABE classification.

Aiming at the low efficiency of monitoring forest fire by manual inspection and sensor acquisition, and being easily affected by the environment, a forest fire detection algorithm PL-YOLO based on YOLOv5 is proposed. In order to enhance the ability of model feature extraction and effectively fuse feature maps at different scales, this paper proposes two feature extraction and fusion modules. First, we design the Res2-C3 Block to extract more detailed features, reduce the repetitive semantic information in the network and expand the differences between features. Second, the EUASPP module is constructed to achieve effective fusion and stacking of feature maps of different sampling rates at the same level at the cost of adding a small amount of operations. Finally, the loss function is replaced. The PL- YOLO achieved a precision of 89.65% and a mAP of 87.78% on the dataset we construct. The experimental results show that the proposed algorithm achieves a good detection effect on smoke and flame targets in fire scenes, and it is possible to deploy it on a robot platform to realize real-time automatic forest fire monitoring.

Aiming at the low detection rate and high false alarm in small ship detection in SAR images, we propose a small-scale ship detection algorithm based on convolutional neural network in this paper. First, we redesign the feature extraction network according to the characters of ship targets in SAR images. The modified network can enrich the spatial and semantics information of small ships. Then, we propose the Path Argumentation Fusion Network (PAFN) to improve the fusion of different feature maps. PAFN uses bottom-up and top-down ways to fuse more location information and semantic information. Both these two optimizations can enhance the detection for small ships. We evaluate our model based on the open SAR-Ship-Dataset and Gaofen-3 SAR images. The experiment results show that our method has excellent performance for small ship detection compared with other deep learning models. Our model improves AP by 6.5% and has higher detection efficiency compared with the baseline YOLOv3 model.