YOLO-GSD-seg: YOLO for Guide Rail Surface Defect Segmentation and Detection

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Object detection (OD) in unmanned aerial vehicle (UAV) images faces many challenges, with diverse-scale objects and small objects being particularly prominent issues. To alleviate these challenges, we propose a novel multiscale feature learning and feature fusion network under the guidance of deformable convolution. First, a deformable convolution-guided feature learning (DCGFL) block is designed in the backbone to extract more effective multiscale features. The DCGFL block leverages the adaptability of deformable convolution to the shapes and scales of objects, akin to spatial attention. Moreover, it also employs channel attention to identify important feature maps. Hence, the proposed backbone possesses the functionality of spatial attention and channel attention. Second, in the neck, we devise a simple generalized feature pyramid network (SimpleGFPN) with several deformable convolution-guided feature fusion (DCGFF) blocks to fuse multiscale features. The proposed neck has cross-layer and cross-scale pathways, facilitating effective information exchange and fusion between shallow spatial and deep semantic features. Third, the SIoU loss is used to better model the bounding box regression loss. Finally, experimental results on the VisDrone2021 and UAVDT datasets show that the proposed method outperforms the compared OD methods. In terms of mean average precision, we obtain 37.8% on VisDrone2021 and 18.5% on UAVDT.

Unmanned aerial vehicles (UAVs) have been widely used in post-disaster search and rescue operations, object tracking, and other tasks. Therefore, the autonomous perception of UAVs based on computer vision has become a research hotspot in recent years. However, UAV images include dense objects, small objects, and arbitrary object directions, which bring about significant challenges to existing object detection methods. To alleviate these issues, we propose a global-local feature enhanced network (GLF-Net). Considering the difficulty of processing UAV images with complex scenes and dense objects, we designed a backbone based on an involution and self-attention that can extract effective features from complex objects. A multiscale feature fusion module is also proposed to address the presence of numerous small objects in UAV images through multiscale object detection and feature fusion. To accurately detect rotated objects, a rotated regional proposal network was designed based on the midpoint offset representation, which can apply a rotated box to determine the real direction and contour of an object. GLF-Net achieves a state-of-the-art detection accuracy (86.52% mAP) on our created RO-UAV dataset, while achieving 96.95% and 97% mAP on the public datasets HRSC2016 and UCAS-AOU, respectively. The experimental results demonstrate that our method achieves a high detection accuracy and generalization, which can meet the practical requirements of UAVs under various complex scenarios.

Automatic modulation classification (AMC), as a key technology of cognitive radio, has become a focal point of research. However, most deep learning-based AMC methods require an extensive number of labeled signals to acquire a comprehensive understanding of modulation types, placing substantial pressure on signal acquisition and labeling. To solve this issue, we propose a few-shot AMC (FSAMC) method to facilitate rapid generalization and recognition with limited data, namely multi-scale feature fusion and distribution similarity network (MS2F-DS). Firstly, we design a multi-scale feature fusion (MS2F) model, which aims to extract features with varying fields of view and boost feature fusion, enabling the derivation of contextual information from the signal. Furthermore, we introduce a distribution similarity (DS) classifier to address the insufficient measurement of current similarity measurement functions by considering both micro and macro perspectives of vectors, further increasing intra-class compactness and inter-class separability. Finally, extensive experiments were conducted on 3-way 1, 3, and 5-shot FSAMC tasks using public datasets RML2016.10a and RML2016.10b, and the results demonstrated the effectiveness of our method.

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

In recent decades, remote sensing object counting has attracted increasing attention from academia and industry due to its potential benefits in urban traffic, public safety, and road planning. However, this issue is becoming a challenge for computer vision because of various technical barriers, such as large-scale variation, complex background interference, and nonuniform density distribution. Recent results show hopeful prospects for object counting using convolutional neural networks (CNNs), but most existing CNN-based methods draw on larger and more complex architectures, which leads to a huge computational and storage burdens, severely limiting their application in real-world scenarios. In this article, a lightweight multiscale feature fusion network for remote sensing object counting, named LMSFFNet, is presented to achieve a better balance between the running speed of the network and the counting accuracy. Specifically, in the encoding process, we select a MobileViT module as the backbone of the network to reduce the numbers of network parameters and computing cost. In return, a cascade structure of the channel–spatial attention mechanisms compensates for the weaker feature extraction ability of the lightweight network. In the decoding process, a lightweight multiscale context fusion module (LMCFM) as a multiscale feature fusion module is developed to solve the problem that the number of parameters increases with the expansion of the object scale when extracting multiscale features. In addition, a lightweight counting scale pooling module (LCSPM) is used to mine the subtle features of the target object. Two kinds of typical object counting experiments, namely, experiments on remote sensing benchmarks (RSOC dataset) and crowd benchmarks (ShanghaiTech, UCF-QNRF, and UCF_CC_50 datasets), show the effectiveness of the proposed method.

Spatiotemporal fusion is an effective and cost-effective method to obtain both high temporal resolution and high spatial resolution images. However, existing methods do not sufficiently extract the deeper features of the image, resulting in fused images which do not recover good topographic detail and poor fusion quality. In order to obtain higher quality spatiotemporal fusion images, a novel spatiotemporal fusion method based on deep learning is proposed in this paper. The method combines an attention mechanism and a multiscale feature fusion network to design a network that more scientifically explores deeper features of the image for different input image characteristics. Specifically, a multiscale feature fusion module is introduced into the spatiotemporal fusion task and combined with an efficient spatial-channel attention module to improve the capture of spatial and channel information while obtaining more effective information. In addition, we design a new edge loss function and incorporate it into the compound loss function, which helps to generate fused images with richer edge information. In terms of both index performance and image details, our proposed model has excellent results on both datasets compared with the current mainstream spatiotemporal fusion methods.

This paper proposes a multi-source data fusion algorithm for rail surface defect detection in both camera-based rail inspection images and ultrasound B-scan images. First, we design a rail surface segmentation algorithm based on image bilateral filtering, Sobel edge detection, and rail surface edge detection to extract the rail surface area. Second, we build a camera and ultrasound data fusion (CUFuse) model for rail surface defect detection, including two main networks: multi-source data feature extraction and multi-scale feature fusion networks. The multi-source data feature extraction network consists of two BoTNet 50 networks as feature extraction networks to extract five stages of features in camera-based images and ultrasound B-scan images. The multi-scale feature fusion network consists of five feature fusion modules to fuse the feature information output by the multi-source data feature extraction network. Finally, we use the CUFuse model to detect the rail surface defect dataset, and output five rail surface state types, including Light, Moderate, Severe, Normal, and Joint. The results show that the accuracy of the CUFuse model is 96.97%, which can accomplish the task of rail surface defect detection on railway sites.

Deep learning-based surface defect segmentation approaches have shown promising performance in recent years. However, segmenting defects with complex shapes, large variations in size, and weakly textured defects with indistinct characteristics still poses significant challenges. In this article, a novel semantic guidance and texture priors based dual-branch surface defect segmentation network (SGTP-Net) is proposed for those issues. Firstly, we construct a feature extraction network combines semantic and texture branches. The semantic branch establishes global contextual relationships, while the texture branch captures local features of defects, this dual-branch ensured the network to extract features from various complex defects. Secondly, we design a feature fusion strategy based on semantic guidance and texture priors. The semantic information is used to guides the output of texture branch. After that, the guided texture information provides valuable edge texture priors for each layers output in semantic branch. The two branches mutually guide each other for improving ability of weak textures feature extraction. Finally, we run our method on the NEU-Seg, MT-Defect and MSD datasets to conduct a comprehensive comparison with some state-of-the-art general object segmentation models and specialized surface defect segmentation methods. The experimental results show that our SGTP-Net performs well in surface defect detection, offering excellent semantic segmentation accuracy and exhibiting good stability and robustness in detecting various surface defects.

The detection of defects is indispensable in industrial production. Surface defects have different scales. Both minimal flaws and significant scratches may appear on the same product. The standard method uses a multi-scale feature fusion network, introducing many parameters that may reduce the inference speed. In actual industrial production scenarios, inference speed and accuracy play an equally important role. Therefore we propose an algorithm to effectively improve the detection speed while improving the detection accuracy. The model proposed in this paper called “YOLO with lightweight feature fusion network(LFF-YOLO).” First, we use ShuffleNetv2 as a feature extraction network to reduce the number of parameters. Then, to improve the efficiency of multi-scale feature fusion, we propose the lightweight feature pyramid network (LFPN). Considering that the fixed receptive field is difficult to adapt to the defects of different scales, it may lead to the difficulty of model convergence and seriously affect the detection performance. Therefore, we propose the adaptive receptive field feature extraction (ARFFE) module, which weights the multi-receptive field channels to generate multi-receptive field information. In addition, focal loss is used to solve the problem of imbalance between positive and negative samples. Finally, we conducted experiments on NEU-DET (79.23% mAP), Peking University printed circuit board defect dataset (93.31% mAP),and GC10-DET (59.78% mAP), respectively. Extensive experiments show that our proposed method achieves optimal detection speed compared with the prevailing methods, and the detection accuracy of our method is also highly competitive.

The detection of rail surface defects is an important tool to ensure the safe operation of rail transit. Due to the complex diversity of track surface defect features and the small size of the defect area, it is difficult to obtain satisfying detection results by traditional machine vision methods. The existing deep learning-based methods have the problems of large model sizes, excessive parameters, low accuracy and slow speed. Therefore, this paper proposes a new method based on an improved YOLOv4 (You Only Look Once, YOLO) for railway surface defect detection. In this method, MobileNetv3 is used as the backbone network of YOLOv4 to extract image features, and at the same time, deep separable convolution is applied on the PANet layer in YOLOv4, which realizes the lightweight network and real-time detection of the railway surface. The test results show that, compared with YOLOv4, the study can reduce the amount of the parameters by 78.04%, speed up the detection by 10.36 frames per second and decrease the model volume by 78%. Compared with other methods, the proposed method can achieve a higher detection accuracy, making it suitable for the fast and accurate detection of railway surface defects.

A multi-scale semantic feature fusion method for remote sensing crop classification

To address the problem of inaccurate representation, inadequate fusion and unsmooth results in video-based single person three-dimensional human pose estimation, a multi-scale spatial-temporal feature fusion method is proposed. Firstly, the joint, limb and upper/lower body tokens were defined in spatial domain to represent the spatial multi-scale features of human body using positional embeddings. Secondly, the spatial multi-scale feature fusion module was constructed based on self-attention mechanism and multilayer perceptron to fuse joint, limb and upper/lower body features, obtaining initial pose feature sequence. Lastly, the temporal multi-scale encoding was established for temporal feature fusion to acquire final pose feature sequence, and optimize the generation of refined three-dimensional human pose through temporal decoding. Experimental results on Human3.6M dataset show that the mean per joint position error and joint velocity errors are 33.6 and 2.4 respectively, which reduce by 2.3% and 4.0%. The proposed method can improve three-dimensional human pose estimation accuracy and generate precise and smooth results while reducing computational cost. Furthermore, experimental results on HumanEva-I dataset show that the proposed method also has a certain degree of generalization ability.

Online knowledge distillation conducts knowledge transfer among all student models to alleviate the reliance on pre-trained models. However, existing online methods rely heavily on the prediction distributions and neglect the further exploration of the representational knowledge. In this paper, we propose a novel Multi-scale Feature Extraction and Fusion method (MFEF) for online knowledge distillation, which comprises three key components: Multi-scale Feature Extraction, Dual-attention and Feature Fusion, towards generating more informative feature maps for distillation. The multiscale feature extraction exploiting divide-and-concatenate in channel dimension is proposed to improve the multi-scale representation ability of feature maps. To obtain more accurate information, we design a dual-attention to strengthen the important channel and spatial regions adaptively. Moreover, we aggregate and fuse the former processed feature maps via feature fusion to assist the training of student models. Extensive experiments on CIF AR-10, CIF AR-100, and CINIC-10 show that MFEF transfers more beneficial representational knowledge for distillation and outperforms alternative methods among various network architectures