A Lightweight Multiscale Feature Fusion Network for Remote Sensing Object Counting

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Unmanned aerial vehicles (UAVs) have been widely used in post-disaster search and rescue operations, object tracking, and other tasks. Therefore, the autonomous perception of UAVs based on computer vision has become a research hotspot in recent years. However, UAV images include dense objects, small objects, and arbitrary object directions, which bring about significant challenges to existing object detection methods. To alleviate these issues, we propose a global-local feature enhanced network (GLF-Net). Considering the difficulty of processing UAV images with complex scenes and dense objects, we designed a backbone based on an involution and self-attention that can extract effective features from complex objects. A multiscale feature fusion module is also proposed to address the presence of numerous small objects in UAV images through multiscale object detection and feature fusion. To accurately detect rotated objects, a rotated regional proposal network was designed based on the midpoint offset representation, which can apply a rotated box to determine the real direction and contour of an object. GLF-Net achieves a state-of-the-art detection accuracy (86.52% mAP) on our created RO-UAV dataset, while achieving 96.95% and 97% mAP on the public datasets HRSC2016 and UCAS-AOU, respectively. The experimental results demonstrate that our method achieves a high detection accuracy and generalization, which can meet the practical requirements of UAVs under various complex scenarios.

A multi-scale semantic feature fusion method for remote sensing crop classification

In general, most of the existing convolutional neural network (CNN)-based deep-learning models suffer from spatial-information loss and inadequate feature-representation issues. This is due to their inability to capture multiscale-context information and the exclusion of semantic information throughout the pooling operations. In the early layers of a CNN, the network encodes simple semantic representations, such as edges and corners, while, in the latter part of the CNN, the network encodes more complex semantic features, such as complex geometric shapes. Theoretically, it is better for a CNN to extract features from different levels of semantic representation because tasks such as classification and segmentation work better when both simple and complex feature maps are utilized. Hence, it is also crucial to embed multiscale capability throughout the network so that the various scales of the features can be optimally captured to represent the intended task. Multiscale representation enables the network to fuse low-level and high-level features from a restricted receptive field to enhance the deep-model performance. The main novelty of this review is the comprehensive novel taxonomy of multiscale-deep-learning methods, which includes details of several architectures and their strengths that have been implemented in the existing works. Predominantly, multiscale approaches in deep-learning networks can be classed into two categories: multiscale feature learning and multiscale feature fusion. Multiscale feature learning refers to the method of deriving feature maps by examining kernels over several sizes to collect a larger range of relevant features and predict the input images’ spatial mapping. Multiscale feature fusion uses features with different resolutions to find patterns over short and long distances, without a deep network. Additionally, several examples of the techniques are also discussed according to their applications in satellite imagery, medical imaging, agriculture, and industrial and manufacturing systems.

To address the challenges of accurately extracting features from elongated scratches, irregular defects, and small-scale surface flaws on high-precision linear guide rails, this paper proposes a novel instance segmentation algorithm tailored for guide rail surface defect detection. The algorithm integrates the YOLOv8 instance segmentation framework with deformable convolutional networks and multi-scale feature fusion to enhance defect feature extraction and segmentation performance. A dedicated guide rail surface Defect (GSD) segmentation dataset is constructed to support model training and evaluation. In the backbone, the DCNv3 module is incorporated to strengthen the extraction of elongated and irregular defect features while simultaneously reducing model parameters. In the feature fusion network, a multi-scale feature fusion module and a triple-feature encoding module are introduced to jointly capture global contextual information and preserve fine-grained local defect details. Furthermore, a Channel and Position Attention Module (CPAM) is employed to integrate global and local features, improving the model’s sensitivity to channel and positional cues of small-target defects and thereby enhancing segmentation accuracy. Experimental results show that, compared with the original YOLOv8n-Seg, the proposed method achieves improvements of 3.9% and 3.8% in Box and Mask mAP50, while maintaining a real-time inference speed of 148 FPS. Additional evaluations on the public MSD dataset further demonstrate the model’s strong versatility and robustness.

Spatiotemporal fusion is an effective and cost-effective method to obtain both high temporal resolution and high spatial resolution images. However, existing methods do not sufficiently extract the deeper features of the image, resulting in fused images which do not recover good topographic detail and poor fusion quality. In order to obtain higher quality spatiotemporal fusion images, a novel spatiotemporal fusion method based on deep learning is proposed in this paper. The method combines an attention mechanism and a multiscale feature fusion network to design a network that more scientifically explores deeper features of the image for different input image characteristics. Specifically, a multiscale feature fusion module is introduced into the spatiotemporal fusion task and combined with an efficient spatial-channel attention module to improve the capture of spatial and channel information while obtaining more effective information. In addition, we design a new edge loss function and incorporate it into the compound loss function, which helps to generate fused images with richer edge information. In terms of both index performance and image details, our proposed model has excellent results on both datasets compared with the current mainstream spatiotemporal fusion methods.

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

Bacterial wound infection poses a major challenge in trauma care and can lead to severe complications such as sepsis and organ failure. Therefore, rapid and accurate identification of the pathogen, along with targeted intervention, is of vital importance for improving treatment outcomes and reducing risks. However, current detection methods are still constrained by procedural complexity and long processing times. In this study, a hyperspectral imaging (HSI) acquisition system for bacterial analysis and a multi-scale dual-domain feature fusion transformer (MDF2Former) were developed for classifying wound bacteria. MDF2Former integrates three modules: a multi-scale feature enhancement and fusion module that generates tokens with multi-scale discriminative representations, a spatial-spectral dual-branch attention module that strengthens joint feature modeling, and a frequency and spatial-spectral domain encoding module that captures global and local interactions among tokens through a hierarchical stacking structure, thereby enabling more efficient feature learning. Extensive experiments on our self-constructed HSI dataset of typical wound bacteria demonstrate that MDF2Former achieved outstanding performance across five metrics: Accuracy (91.94%), Precision (92.26%), Recall (91.94%), F1-score (92.01%), and Kappa coefficient (90.73%), surpassing all comparative models. These results have verified the effectiveness of combining HSI with deep learning for bacterial identification, and have highlighted its potential in assisting in the identification of bacterial species and making personalized treatment decisions for wound infections.

The sEMG-based gesture recognition has great potential in human-computer interaction. However, the current approaches are far from optimal. In this paper, a novel convolutional neural network which combines the multi-scale kernal and feature fusion (MKFF-CNN) is proposed. This model could extract multi-scale features and make full use of these feature maps. One dataset called “gForce dataset” is recorded in this work, comprising 13 able-bodied participants. To verify the effectiveness of the proposed model, evaluations on both the gForce dataset and the Ninapro DB6 are conducted. The experimental results demonstrate that, MKFF-CNN achieves an accuracy of 97.65% on the gForce dataset, which is 1.39% higher than that of the subnetwork without early fusion, and at least 6.54% higher than that of the subnetwork with single-scale convolution kernel. On the Ninapro DB6, MKFF-CNN achieves 98.52% accuracy, which is 1.3% higher than the state-of-the-art works, showing the superiority of the proposed model.

To address the problem of inaccurate representation, inadequate fusion and unsmooth results in video-based single person three-dimensional human pose estimation, a multi-scale spatial-temporal feature fusion method is proposed. Firstly, the joint, limb and upper/lower body tokens were defined in spatial domain to represent the spatial multi-scale features of human body using positional embeddings. Secondly, the spatial multi-scale feature fusion module was constructed based on self-attention mechanism and multilayer perceptron to fuse joint, limb and upper/lower body features, obtaining initial pose feature sequence. Lastly, the temporal multi-scale encoding was established for temporal feature fusion to acquire final pose feature sequence, and optimize the generation of refined three-dimensional human pose through temporal decoding. Experimental results on Human3.6M dataset show that the mean per joint position error and joint velocity errors are 33.6 and 2.4 respectively, which reduce by 2.3% and 4.0%. The proposed method can improve three-dimensional human pose estimation accuracy and generate precise and smooth results while reducing computational cost. Furthermore, experimental results on HumanEva-I dataset show that the proposed method also has a certain degree of generalization ability.

Aiming at the problem that the relationship between geometric features and semantic features is ignored in the point cloud data downsampling process, which leads to inaccurate segmentation of object boundaries and structural details, this paper proposes a 3D point cloud semantic segmentation network based on multi-scale dense nested type. Firstly, a dense nested network architecture is constructed by nesting multiple multi-scale feature fusion modules to fuse multi-scale features of different directions between encoder-decoder paths, so as to effectively propagate local ge-ometric context information and enhance the ability of cross-scale information interaction. Secondly, a local feature aggregation unit is constructed in the multi-scale feature fusion module, which strengthens the structural awareness within the local point set based on graph convolution and attention mechanism, and promotes the complementarity of local geometric features and abstract semantic information. Then, the cross-layer multi-loss supervision module is combined to further optimize the multi-scale feature propagation, which makes the network training more stable and improves the point cloud segmentation accuracy. Finally, this paper verified the proposed network on the S3DIS dataset. The experimental results show that the proposed network has the mean intersection over Union of 71.2% and the overall accuracy of 88.7%, which is 1.2 and 0.7 percentage points higher than RandLA-Net, respectively, which proves that the proposed network can effectively improve the accuracy of 3D point cloud semantic segmentation.

Automatic methods that count the number of living bodies is extremely valuable for modern farms. Recently, there are an increasing number of studies conducted on imagery based automatic object counting. However, these studies suffer from extreme overlap of objects, scene perspective, the size of instances and etc. Moreover, real-time performance needs to be met in many application scenarios. In this study, we propose a video imagery based pipeline for real-time accurate object counting on smart farms. Firstly, we introduce the concept - overlap degree of a frame (ODF). Next, an end-to-end frame filtering algorithm based on ODF is established. Then, we propose an object detection and counting algorithm based on Faster region-based convolutional neural networks (Faster RCNN). Finally, the counting results are fused to accurately obtain the number of detected objects. The experimental results demonstrate that our proposed method can automatically and accurately count the living bodies on farms in real-time. In our experiments, the average speed of detection is 40 frames per second, and the total difference is 1.69%. Therefore, our method is considered to be a suitable tool to conduct automatic object counting on smart farms.

High-density object counting in surveillance scenes is challenging mainly due to the drastic variation of object scales. The prevalence of deep learning has largely boosted the object counting accuracy on several benchmark datasets. However, does the global counts really count? Armed with this question we dive into the predicted density map whose summation over the whole regions reports the global counts for more in-depth analysis. We observe that the object density map generated by most existing methods usually lacks of local consistency, i.e., counting errors in local regions exist unexpectedly even though the global count seems to well match with the ground-truth. Towards this problem, in this paper we propose a constrained multi-stage Convolutional Neural Networks (CNNs) to jointly pursue locally consistent density map from two aspects. Different from most existing methods that mainly rely on the multi-column architectures of plain CNNs, we exploit a stacking formulation of plain CNNs. Benefited from the internal multi-stage learning process, the feature map could be repeatedly refined, allowing the density map to approach the ground-truth density distribution. For further refinement of the density map, we also propose a grid loss function. With finer local-region-based supervisions, the underlying model is constrained to generate locally consistent density values to minimize the training errors considering both the global and local counts accuracy. Experiments on two widely-tested object counting benchmarks with overall significant results compared with state-of-the-art methods demonstrate the effectiveness of our approach.

Deep convolutional neural networks have demonstrated significant performance improvements in face super-resolution tasks. However, many deep learning-based approaches tend to overlook the inherent structural information and feature correlation across different scales in face images, making the accurate recovery of face structure in low-resolution cases challenging. To address this, this paper proposes a method that fuses multi-scale features while preserving the facial structure. It introduces a novel multi-scale residual block (MSRB) to reconstruct key facial parts and structures from spatial and channel dimensions, and utilizes pyramid attention (PA) to exploit non-local self-similarity, improving the details of the reconstructed face. Feature Enhancement Modules (FEM) are employed in the upscale stage to refine and enhance current features using multi-scale features from previous stages. The experimental results on CelebA, Helen and LFW datasets provide evidence that our method achieves superior quantitative metrics compared to the baseline, the Peak Signal-to-Noise Ratio (PSNR) outperforms the baseline by 0.282 dB, 0.343 dB, and 0.336 dB. Furthermore, our method demonstrates improved visual performance on two additional no-reference datasets, Widerface and Webface.

IntroductionThe cigar leaves moisture content (CLMC) is a critical parameter for controlling curing barn conditions. Along with the continuous advancement of deep learning (DL) technologies, convolutional neural networks (CNN) have provided a way of thinking for the non-destructive estimation of CLMC during the air-curing process. Nevertheless, relying merely on single-perspective imaging makes it difficult to comprehensively capture the complementary morphological features of the front and back sides of cigar leaves during the air-curing process.MethodsThis study constructed a dual-view image dataset covering the air-curing process, and proposes a regression framework named CADFFNet (channel attention weight-based dual-branch feature fusion network) for the non-destructive estimation of CLMC during the curing process based on dual-view RGB images. Firstly, the model utilizes two independent and parallel ResNet as its backbone structure to capture the heterogeneous features of dual-view images. Secondly, the Dual Efficient Channel Attention (DECA) module is introduced to dynamically adjust the channel attention weights of the features, thereby facilitating interaction between the two branches. Lastly, a Multi-scale convolutional feature fusion (MSCFF) module is designed for the deep fusion of features from the front and back images to aggregate multi-scale features for robust regression.ResultsOn five-fold cross-validation, CADFFNet attains R2 of 0.974±0.007 and mean absolute error (MAE) of 3.80±0.37%. On an independent cross-region, cross-variety testing set, it maintains strong generalization (R2=0.899, MAE=5.82%), compared with the classic CNN models ResNet18, GoogLeNet, VGG19Net, DenseNet121, and MobileNetV2, its R2 value has increased by 0.047, 0.041, 0.055, 0.098, and 0.090 respectively.DiscussionGenerally, the proposed CADFFNet offers an efficient and convenient method for non-destructive detection of CLMC, providing a theoretical basis for automating the air-curing process. It also provides a new perspective for moisture content prediction during the drying process of other crops, such as tea, asparagus, and mushrooms.