A Review on Multiscale-Deep-Learning Applications.

A multi-scale semantic feature fusion method for remote sensing crop classification

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

Convolutional neural network (CNN) has led to significant progress in object detection. In order to detect the objects in various sizes, the object detectors often exploit the hierarchy of the multi-scale feature maps called feature pyramid, which is readily obtained by the CNN architecture. However, such feature maps do not fully consider the supplementary effect of contextual information on semantics. In this work, we proposed a feature fusion method of residual attention based on the SSD benchmark network call Improved SSD to make full use of context information to improve the characterization ability of feature maps. Besides, we use the residual attention mechanism to reinforce the key features to further improve the detector performance. The experiment result on benchmark dataset PASCAL VOC shows that the map of the proposed method with input image sizes of 300×300 and 512×512 is 78.8% and 80.7%.

The sEMG-based gesture recognition has great potential in human-computer interaction. However, the current approaches are far from optimal. In this paper, a novel convolutional neural network which combines the multi-scale kernal and feature fusion (MKFF-CNN) is proposed. This model could extract multi-scale features and make full use of these feature maps. One dataset called “gForce dataset” is recorded in this work, comprising 13 able-bodied participants. To verify the effectiveness of the proposed model, evaluations on both the gForce dataset and the Ninapro DB6 are conducted. The experimental results demonstrate that, MKFF-CNN achieves an accuracy of 97.65% on the gForce dataset, which is 1.39% higher than that of the subnetwork without early fusion, and at least 6.54% higher than that of the subnetwork with single-scale convolution kernel. On the Ninapro DB6, MKFF-CNN achieves 98.52% accuracy, which is 1.3% higher than the state-of-the-art works, showing the superiority of the proposed model.

The traditional convolutional neural network (CNN) only focuses on the features of the last layer of the network, ignoring those of other layers. The detailed information of the shallow network can improve the accuracy of classification to some extent. With the deepening of the network, the problems such as gradient disappearance and explosion are obvious. In order to alleviate this problem, a method for residual network image classification with multi-scale feature fusion is proposed in the paper. First of all, the Random Image Cropping and Patching (RICAP), a data augmentation method, is adopted to cut and splice the new training samples from the training set, based on which feature maps of different sizes are obtained by residual modules. Secondly, after reducing the dimension of high-level feature maps, all the feature maps with the same dimension are processed by the multi-scale fusion. Finally, these feature vectors are input into the Softmax classifier for training and classification, and the method of learning rate decay is used in the training process. The experiment results indicate that this method has a good performance on image classification based on the public datasets of MNIST and CIFAR-10.KeywordsMulti-scale feature fusionResidual networkData augmentationImage classification

Object detection (OD) in unmanned aerial vehicle (UAV) images faces many challenges, with diverse-scale objects and small objects being particularly prominent issues. To alleviate these challenges, we propose a novel multiscale feature learning and feature fusion network under the guidance of deformable convolution. First, a deformable convolution-guided feature learning (DCGFL) block is designed in the backbone to extract more effective multiscale features. The DCGFL block leverages the adaptability of deformable convolution to the shapes and scales of objects, akin to spatial attention. Moreover, it also employs channel attention to identify important feature maps. Hence, the proposed backbone possesses the functionality of spatial attention and channel attention. Second, in the neck, we devise a simple generalized feature pyramid network (SimpleGFPN) with several deformable convolution-guided feature fusion (DCGFF) blocks to fuse multiscale features. The proposed neck has cross-layer and cross-scale pathways, facilitating effective information exchange and fusion between shallow spatial and deep semantic features. Third, the SIoU loss is used to better model the bounding box regression loss. Finally, experimental results on the VisDrone2021 and UAVDT datasets show that the proposed method outperforms the compared OD methods. In terms of mean average precision, we obtain 37.8% on VisDrone2021 and 18.5% on UAVDT.

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Radio imaging has become instrumental in examining internal anomalies and body organs. The medical expert manually delineates the images in routine practice and reaches the decision. However, manual intervention leads to errors due to human limitations, and the decision becomes operator-dependent. Therefore, automatic liver segmentation from computed tomography (CT) images plays a decisive role in detecting hepatic anomalies, treatment planning, liver transplantation, liver cancer treatment, and post-treatment assessment. Recently deep learning (DL) techniques have proven competence in medical image segmentation. The proposed method is DL-based multiscale feature fusion with channel-wise attention network (MFCA-Net) designed using the computationally efficient Res2Net (R2N) layer, which has the ability to enhance the receptive field of convolutional neural network (CNN) and extract the multiscale information at a more granular level.Further, we reconstructed the multiscale low-level features and fused them with high-level features that enhance the semantic details in the features. In addition, we employed the channel-wise attention mechanism that renovates the features by modelling the interdependencies between the channels and focusing on the prominent features. Also, we altered the low-level features by fusing the renovated features that augment the contextual information, which simplifies the network learning potential. The efficacy of the proposed network was demonstrated on the CHAOS challenge test dataset, where the network attained the DICE of 96.67 ± 0.72%. Thus, the proposed MFCA-Net is computationally efficient, and liver segmentation performance is comparable with state-of-the-art methods.KeywordsMultiscale featuresFeature fusionChannel attentionLiver segmentationCNNCT images

To ensure the safety of vehicle travel, the maintenance of road infrastructure has become increasingly critical, with efficient and accurate detection techniques for road cracks emerging as a key research focus in the industry. The development of deep learning technologies has shown tremendous potential in improving the efficiency of road crack detection. While convolutional neural networks have proven effective in most semantic segmentation tasks, overcoming their limitations in road crack segmentation remains a challenge. To address this, this paper proposes a novel road crack segmentation network that leverages the powerful spatial feature modeling capabilities of Swin Transformer and the Encoder–Decoder architecture of DeepLabv3+. Additionally, the incorporation of a multi-scale coding module and attention mechanism enhances the network’s ability to densely fuse multi-scale features and expand the receptive field, thereby improving the integration of information from feature maps. Performance comparisons with current mainstream semantic segmentation models on crack datasets demonstrate that the proposed model achieves the best results, with an MIoU of 81.06%, Precision of 79.95%, and F1-score of 77.56%. The experimental results further highlight the model’s superior ability in identifying complex and irregular cracks and extracting contours, providing guidance for future applications in this field.

A hybrid approach for EEG motor imagery classification using adaptive margin disparity and knowledge transfer in convolutional neural networks.

Deep convolutional neural networks have demonstrated significant performance improvements in face super-resolution tasks. However, many deep learning-based approaches tend to overlook the inherent structural information and feature correlation across different scales in face images, making the accurate recovery of face structure in low-resolution cases challenging. To address this, this paper proposes a method that fuses multi-scale features while preserving the facial structure. It introduces a novel multi-scale residual block (MSRB) to reconstruct key facial parts and structures from spatial and channel dimensions, and utilizes pyramid attention (PA) to exploit non-local self-similarity, improving the details of the reconstructed face. Feature Enhancement Modules (FEM) are employed in the upscale stage to refine and enhance current features using multi-scale features from previous stages. The experimental results on CelebA, Helen and LFW datasets provide evidence that our method achieves superior quantitative metrics compared to the baseline, the Peak Signal-to-Noise Ratio (PSNR) outperforms the baseline by 0.282 dB, 0.343 dB, and 0.336 dB. Furthermore, our method demonstrates improved visual performance on two additional no-reference datasets, Widerface and Webface.

In the field of surface defect detection, the scale difference of product surface defects is often huge. The existing defect detection methods based on Convolutional Neural Networks (CNNs) are more inclined to express macro and abstract features, and the ability to express local and small defects is insufficient, resulting in an imbalance of feature expression capabilities. In this paper, a Multi-Scale Feature Learning Network (MSF-Net) based on Dual Module Feature (DMF) extractor is proposed. DMF extractor is mainly composed of optimized Concatenated Rectified Linear Units (CReLUs) and optimized Inception feature extraction modules, which increases the diversity of feature receptive fields while reducing the amount of calculation; the feature maps of the middle layer with different sizes of receptive fields are merged to increase the richness of the receptive fields of the last layer of feature maps; the residual shortcut connections, batch normalization layer and average pooling layer are used to replace the fully connected layer to improve training efficiency, and make the multi-scale feature learning ability more balanced at the same time. Two representative multi-scale defect data sets are used for experiments, and the experimental results verify the advancement and effectiveness of the proposed MSF-Net in the detection of surface defects with multi-scale features.

Image dehazing is critical for visual enhancement and a wide range of computer vision applications. Despite significant advancements, challenges remain in preserving fine details and adapting to diverse, non-uniformly degraded scenes. To address these issues, we propose a novel image dehazing method that introduces a contrastive learning framework, enhanced by the InfoNCE loss, to improve model robustness. In this framework, hazy images are treated as negative samples and their clear counterparts as positive samples. By optimizing the InfoNCE loss, the model is trained to maximize the similarity between positive pairs and minimize that between negative pairs, thereby improving its ability to distinguish haze artifacts from intrinsic scene features and better preserving the structural integrity of images. In addition to contrastive learning, our method integrates a multi-scale dynamic feature fusion with a hybrid attention mechanism. Specifically, we introduce dynamically adjustable frequency band filters and refine the hybrid attention module to more effectively capture fine-grained, cross-scale image details. Extensive experiments on the RESIDE-6K and RS-Haze datasets demonstrate that our approach outperforms most existing methods, offering a promising solution for practical image dehazing applications.

The performance of underwater target detection algorithms is affected by poor imaging quality in underwater environments. Due to the arithmetic power limitation of underwater devices, existing deep learning networks are unable to provide efficient detection processes with high detection accuracy. Lightweight CNN models have been actively applied for underwater environment detection, yet their lite feature fusion networks cannot provide effective fusion effects and reduce the detection accuracy. In this paper, a lightweight algorithm based on multi-scale feature fusion was proposed, with the model parameters greatly reduced, improving the target detection accuracy. The forward propagation memory overhead is reduced by using multi-scale shared convolutional kernels and pooling operations to co-construct the query matrix in the Tansformer encoding stage. Then, the feature fusion path is optimized in order to enhance the connection of multi-scale features. A multiscale feature adaptive fusion strategy is used to enhance the detection performance and reduce the dependence on the complex feature extraction network. The feature extraction network is also reparameterized to simplify the operation. Using the UPRC offshore dataset for validation, the study results have demonstrated that the statistical mAP metrics validate the detection accuracy. Compared with SSD, RetinaNet and YOLOv5-s improved by 13%, 8.6%, and 0.8%, while the number of parameters decreased by 76.09%, 89.74%, and 87.67%. In addition, compared with the YOLOv5-lite model algorithm with the same parameter volume, the mAP is improved by 3.8%, which verifies the accuracy and efficiency of the algorithm in this paper.

Multi-scale deep feature fusion for automated classification of macular pathologies from OCT images