A Lightweight Neural Network Combining Dilated Convolution and Depthwise Separable Convolution

Convolutional neural networks (CNNs) are becoming increasingly popular in modern remote sensing image processing tasks and exhibit outstanding capability for hyperspectral image (HSI) classification. However, for the existing CNN-based HSI-classification methods, most of them only consider single-scale feature extraction, which may neglect some important fine information and cannot guarantee to capture optimal spatial features. Moreover, many state-of-the-art methods have a huge number of network parameters needed to be tuned, which will cause high computational cost. To address the aforementioned two issues, a novel multiscale residual network (MSRN) is proposed for HSI classification. Specifically, the proposed MSRN introduces depthwise separable convolution (DSC) and replaces the ordinary depthwise convolution in DSC with mixed depthwise convolution (MDConv), which mixes up multiple kernel sizes in a single depthwise convolution operation. The DSC with mixed depthwise convolution (MDSConv) can not only explore features at different scales from each feature map but also greatly reduce learnable parameters in the network. In addition, a multiscale residual block (MRB) is designed by replacing the convolutional layer in an ordinary residual block with the MDSConv layer. The MRB is used as the major unit of the proposed MSRN. Furthermore, to enhance further the feature representation ability, the proposed network adds a high-level shortcut connection (HSC) on the cascaded two MRBs to aggregate lower level features and higher level features. Experimental results on three benchmark HSIs demonstrate the superiority of the proposed MSRN method over several state-of-the-art methods.

In recent years, convolutional neural networks (CNNs) have been increasingly leveraged for the classification of hyperspectral imagery, displaying notable advancements. To address the issues of insufficient spectral and spatial information extraction and high computational complexity in hyperspectral image classification, we introduce the MDRDNet, an integrated neural network model. This novel architecture is comprised of two main components: a Multiscale 3D Depthwise Separable Convolutional Network and a CBAM-augmented Residual Dilated Convolutional Network. The first component employs depthwise separable convolutions in a 3D setting to efficiently capture spatial–spectral characteristics, thus substantially reducing the computational burden associated with 3D convolutions. Meanwhile, the second component enhances the network by integrating the Convolutional Block Attention Module (CBAM) with dilated convolutions via residual connections, effectively counteracting the issue of model degradation. We have empirically evaluated the MDRDNet’s performance by running comprehensive experiments on three publicly available datasets: Indian Pines, Pavia University, and Salinas. Our findings indicate that the overall accuracy of the MDRDNet on the three datasets reached 98.83%, 99.81%, and 99.99%, respectively, which is higher than the accuracy of existing models. Therefore, the MDRDNet proposed in this study can fully extract spatial–spectral joint information, providing a new idea for solving the problem of large model calculations in 3D convolutions.

Gene Regulatory Networks (GRNs) reveal complex interactions between genes in organisms, crucial for understanding the life system's operation. The rapid development of biotechnology, especially single-cell RNA sequencing (scRNA-seq), has generated a large amount of scRNA-seq data, which can be analyzed to explore the regulatory relationships between genes at the single-cell level. Previous models used to construct GRNs mainly aim at constructing associative relationships between genes, but usually fail to accurately reveal the causality between genes. Therefore, we present a hybrid deep learning model called EfficientNet-resDDSC (the EfficientNet with Residual Blocks and Depthwise Separable Dilated Convolutions) to infer causality between genes. The model inherits the basic structure of EfficientNet-B0 and incorporates residual blocks as well as dilated convolutions. The model's ability to extract low-level features at the primary stage is enhanced by introducing residual blocks. The model combines Depthwise Separable Convolution (DSC) in the inverted linear bottleneck layers with the dilated convolutions to expand the model's receptive fields without increasing the computational effort. This design enables the model to comprehensively reveal potential relationships among different genes in high-dimensional and high-noise single-cell data. In comparison with the five existing deep learning network models, EfficientNet-resDDSC's overall performance is significantly better than others on four datasets. In this study, EfficientNet-resDDSC was further applied to construct GRNs for breast cancer patients, focusing on the related regulatory genes of the key gene BRCA1, which contributes to the advancement of breast cancer research and treatment strategies.

In this paper, an efficient architecture of depthwise separable convolutional neural network is presented. By simplifying the network architecture and adopting 16-bit fixed point form, the computation cost is substantially reduced with slightly decrease in image classification precision based on CIFAR10 dataset. In order to improve energy efficiency and reduce memory access, the custom processing element (PE) is proposed, which supports zero skipping, local accumulation and memory, as well as function multiplexing (for different convolution operations). Besides, the specific hardware architecture based on our custom PE is proposed and the hardware architecture can also support two dataflow modes in order to customize traditional convolution and depthwise separable convolution (DWC) dataflow. The hardware architecture is implemented on a Xilinx Zynq 7Z020 field-programmable gate array (FPGA) platform and the experimental results are implemented. By exploiting parallelism and data reuse, post synthesis simulation with a clock frequency of 50 MHz shows that the network achieves a peak performance of 9.6 GOPS and an energy efficiency of over 90.56 GOPS/W for single-frame runtime inference, achieving a 59× higher energy efficiency compared with the CPU Intel i5-8400. The results show that the proposed accelerator can classify each picture from Cifar10 in 10 ms, which is about 100 frames per second.The FPGA design achieves 41x speedup if compared to CPU, achieving real-time image classification at 100 fps.

Intelligent fault diagnosis methods based on Transformer variants have shown excellent performance. However, these methods generally model features globally, which means a poor anti-noise ability and greater computational complexity. Therefore, a lightweight framework for bearing fault diagnosis based on depthwise separable convolution (DSC) and sparse attention mechanism named DSC-SparseFormer is proposed. First, a guided-and-sparse self-attention (GSSA) module, which can adaptively adjust the region of interest for features, is proposed to extract features, thereby reducing sensitivity to redundant information and downgrading computational complexity. This is potentially the first time such a methodology has been proposed and applied. Second, discrete cosine transform (DCT) is applied to map time-domain signals into frequency domain to enrich feature space, and a time-frequency coupling attention (TFCA) module is developed to dynamically integrate time-domain and frequency-domain features. Finally, extensive experiments on the HUSTBearing dataset, DHU dataset, and HIT dataset demonstrate that DSC-SparseFormer achieves high-accuracy (HUSTBearing: 99.67%, DHU: 99.94%, and HIT: 99.93%) and lightweight (Params: 0.003 M, FLOPs: 6.07 M) fault diagnosis. The effectiveness of the GSSA and TFCA is validated by ablation studies, and feature visualisation is employed to improve the interpretability of the DSC-SparseFormer.

Diseases such as diabetes and hypertension can lead to change the shape of the retinal blood vessels. Segmentation of fundus images is a key step in the process of quantitative analysis of the disease, which is instructive in the analysis and diagnosis of clinical diseases. In this paper, a method for the segmentation of retinal image vessels based on fully convolutional network (FCN) with depthwise separable convolution and channel weighting is presented. Firstly, CLAHE and Gamma correction of the green channel of the fundus image are used to enhance the contrast. Then, in order to adapt to network training, the enhanced image is divided into patches to expand the data. Finally, the depthwise separable convolution instead of the standard convolution method is used to increase the network width. Meanwhile, the channel weighting module is introduced to explicitly model the relationship between the characteristic channels in order to improve the distinguishability of the features. The combination of them is applied to the FCN and the results of expert manual identification are used to supervise the experiment on the DRIVE database. The results show that the segmentation accuracy of the proposed method in DRIVE database reached 0.963 0 and AUC reached 0.983 1. The segmentation accuracy in STARE database reached 0.962 0 and AUC achieved 0.983 0. To some extent, the proposed method has better feature resolution and better segmentation performance.

With the rapid growth of artificial intelligence applications on embedded devices and mobile platforms, traditional convolutional neural network models are often difficult to meet the real-time and power consumption requirements of embedded devices due to their large number of parameters. This paper presents a lightweight improvement to the GoogLeNet model by replacing the standard convolution in the Inception module with Depthwise Separable Convolutions and simplifying the network structure. This approach effectively reduces the model’s parameter count and computational complexity. Experimental results show that the improved model reduces the parameter count by approximately $54 \%$ and the computational complexity by about $42 \%$. The training speed is significantly improved, disturbance resistance is enhanced, and the model is suitable for deployment on low-computational-power devices.

One difficult problem of keyword spotting is how to miniaturize its memory footprint while maintain a high precision. Although convolutional neural networks have shown to be effective to the small-footprint keyword spotting problem, they still need hundreds of thousands of parameters to achieve good performance. In this paper, we propose an efficient model based on depthwise separable convolution layers and squeeze-and-excitation blocks. Specifically, we replace the standard convolution by the depthwise separable convolution, which reduces the number of the parameters of the standard convolution without significant performance degradation. We further improve the performance of the depthwise separable convolution by reweighting the output feature maps of the first convolution layer with a so-called squeeze-and-excitation block. We compared the proposed method with five representative models on two experimental settings of the Google Speech Commands dataset. Experimental results show that the proposed method achieves the state-of-the-art performance. For example, it achieves a classification error rate of 3.29% with a number of parameters of 72K in the first experiment, which significantly outperforms the comparison methods given a similar model size. It achieves an error rate of 3.97% with a number of parameters of 10K, which is also slightly better than the state-of-the-art comparison method given a similar model size.

In this paper, a new bolt fault diagnosis method is developed to solve the fault diagnosis problem of wind turbine flange bolts using one-dimensional depthwise separable convolutions. The main idea is to use a one-dimensional convolutional neural network model to classify and identify the acoustic vibration signals of bolts, which represent different bolt damage states. Through the method of knock test and modal simulation, it is concluded that the damage state of wind turbine flange bolt is related to the natural frequency distribution of acoustic vibration signal. It is found that the bolt damage state affects the modal shape of the structure, and then affects the natural frequency distribution of the bolt vibration signal. Therefore, the damage state can be identified by identifying the natural frequency distribution of the bolt acoustic vibration signal. In the present one-dimensional depth-detachable convolutional neural network model, the one-dimensional vector is first convolved into multiple channels, and then each channel is separately learned by depth-detachable convolution, which can effectively improve the feature quality and the effect of data classification. From the perspective of the realization mechanism of convolution operation, the depthwise separable convolution operation has fewer parameters and faster computing speed, making it easier to build lightweight models and deploy them to mobile devices.

Addressing the urgent need for high real-time performance in military target image classification within the context of modern battlefield informatization and intelligent big data processing, this paper proposes a lightweight convolutional neural network model based on depth-wise separable convolution (DSC) and knowledge distillation. This approach effectively solves the long-standing challenge of balancing accuracy and speed in military target recognition in big data environments. Using a teacher-student framework, the method employs a pre-trained ResNet34 as the teacher model, distilling its knowledge into a lightweight student model constructed with depth-wise separable convolutions (based on ResNet18). The training incorporates a combined loss function comprising cross-entropy loss, KL divergence loss, and intermediate feature map mean squared error loss. Experimental results on the MSTAR dataset demonstrate that the proposed model maintains classification accuracy comparable to the teacher model while achieving a substantial parameter reduction (only 6.85% of ResNet34’s parameters) and significant inference speed improvement. This breakthrough enables high-real-time, high-precision military target image classification in resource-constrained battlefield environments.

Hybrid deep learning architecture for skin disease classification

Transmission lines in complex outdoor environments often suffer external damage in construction areas, severely affecting the stability of power systems. Traditional manual detection methods have problems of low efficiency and poor real-time performance. In deep learning-based detection methods, standard convolution has a large parameter count and computational complexity, making it difficult to deploy on edge devices; while lightweight depthwise separable convolution offers low computational cost, it suffers from insufficient feature extraction capability. This limitation stems from its independent processing of each channel’s information, making it unable to simultaneously meet the practical requirements for both lightweight design and high detection accuracy in transmission line monitoring applications. To address the above problems, this study proposes LFRE-YOLO, a lightweight external damage detection algorithm for transmission lines based on YOLOv10n. This study proposes LFRE-YOLO, a lightweight external damage detection algorithm based on YOLOv10n. First, we design a lightweight feature reuse and enhancement convolution (LFREConv) that overcomes the limitations of traditional depthwise separable convolution through cascaded dual depthwise convolution structure and residual connection mechanisms, significantly expanding the effective receptive field with minimal parameter increment and compensating for information loss caused by independent channel processing in depthwise convolution through feature reuse strategies. Second, based on LFREConv, we propose an efficient lightweight feature extraction module (LFREBlock) that achieves cross-channel information interaction enhancement and channel importance modeling. Additionally, we propose a lightweight feature reuse and enhancement detection head (LFRE-Head) that applies LFREConv to the regression branch, achieving comprehensive lightweight design of the detection head while maintaining spatial localization accuracy. Finally, we employ layer-adaptive magnitude-based pruning (LAMP) to prune the trained model, further optimizing the network structure through layer-wise adaptive pruning. Experimental results demonstrate significant improvements over YOLOv10n baseline: mAP50 increased from 92.0% to 94.1%, mAP50-95 improved from 66.2% to 70.2%, while reducing parameters from 2.27 M to 0.99 M, computational complexity from 6.5 G to 3.1 G, and achieving 86.9 FPS inference speed, making it suitable for resource-constrained edge computing environments.

Improvement of response speed and precision of distributed Brillouin optical fiber sensors using neural networks

In this paper, we present a fast and lightweight speech synthesis model that is suitable for on-device applications. By leveraging the techniques of long-short range attention, depth-wise separable convolution, and linear attention, we significantly reduce the model size and complexity of the baseline FastSpeech2-based Transformer framework. Unlike the baseline model that requires O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to compute attention and convolution operations because of nested-loop computations, our proposed model only requires O(N) computations due to the modification of a nested-loop into two cascaded single loops. Experimental results show that our proposed model is able to generate speech with a real-time factor of 0.26 and requires only 10.4 million parameters. Despite the reduction in model size and complexity, still, the generated speech quality of our model is nearly close to the baseline.

&amp;lt;h2&amp;gt;Abstract&amp;lt;/h2&amp;gt;\n&amp;lt;p&amp;gt;&amp;lt;strong&amp;gt;Objectives:&amp;lt;/strong&amp;gt;&amp;nbsp;The main objective of the work is to classify the psoriasis affected skin into one of different psoriasis types viz. erythrodermic, gutatte, inverse, nail, plaque, and pustular using depth wise separable convolution neural networks.&amp;lt;strong&amp;gt;&amp;nbsp;Methods:&amp;lt;/strong&amp;gt;&amp;nbsp;To identify the type of psoriasis disorder, the experiment uses MobileNet machine learning architecture which is based on depthwise separable convolutions. In the preprocessing step, the input images are segmented into 224 pixels X 224 pixels with the help of the keras KerasImageDataGenerator function and in next step, these segmented images are fed as input to 28 layers of reconstructed MobileNet architecture. A series of convolutions and depth-wise separable convolutions layers are applied on the input images. The rectified linear activation function is applied to introduce non-linearity in the network. Adam optimizer algorithm is used for training the network. Categorical cross-entropy is used for the comparison of the accuracy of the experimental results with the existing work of classification of psoriasis disorder.&amp;nbsp;&amp;lt;strong&amp;gt;Findings:&amp;lt;/strong&amp;gt;&amp;nbsp;Using MobileNet machine learning architecture, the experiments attained around 86% of classification efficiency, and an average F1􀀀 score of .94 as compared with VGG19 and ResNet-34, compared to the work done in(1).&amp;lt;strong&amp;gt;&amp;nbsp;Novelty:&amp;lt;/strong&amp;gt;&amp;nbsp;The novelty of the work lies in the prediction of the types of the psoriasis disorder accurately with less turnaround time with an accuracy of 86% consuming low processing capability which can be implemented on low powered hand-held devices.&amp;lt;/p&amp;gt;\n&amp;lt;p&amp;gt;&amp;lt;strong&amp;gt;Keywords:&amp;lt;/strong&amp;gt; Machine Learning; Depth Wise Separable Convolution; Psoriasis Disorder; ReLU Activation; Pointwise convolutions&amp;lt;/p&amp;gt;