MCCP: Multi-Collaboration Channel Pruning for Model Compression

Acceleration and compression on deep Convolutional Neural Networks (CNNs) have become a critical problem to develop intelligence on resource-constrained devices. Previous channel pruning can be easily deployed and accelerated without specialized hardware and software. However, weight-level redundancy is not well explored in channel pruning, which results in a relatively low compression ratio. In this work, we propose a Low-rank Approximated channel Pruning (LAP) framework to tackle this problem with two targeted steps. First, we utilize low-rank approximation to eliminate the redundancy within filter. This step achieves acceleration, especially in shallow layers, and also converts filters into smaller compact ones. Then, we apply channel pruning on the approximated network in a global way and obtain further benefits, especially in deep layers. In addition, we propose a spectral norm based indicator to coordinate these two steps better. Moreover, inspired by the integral idea adopted in video coding, we propose an evaluator based on Integral of Decay Curve (IDC) to judge the efficiency of various acceleration and compression algorithms. Ablation experiments and IDC evaluator prove that LAP can significantly improve channel pruning. To further demonstrate the hardware compatibility, the network produced by LAP obtains impressive speedup efficiency on the FPGA.

Channel pruning is one of the mainly used meth-ods in current network model compression. However, existing channel pruning methods lack effective hardware runtime latency guidance, making the reduction in model size not fully converted into inference latency reduction, which results in poor inference performance of the pruned network models. This paper proposes a CNN channel pruning framework, CNNBooster, that incorporates hardware runtime latency in-formation, whose two major contributions are as follows: First, CNNBooster automatically analyses the latency behavior of various CNNs with channel reduction on different GPU hard-ware platforms, which achieves efficient localization of pruna-ble coordinates. Second, CNNBooster uses a flexible grained latency-aware and param-aware pruning algorithm based on prunable coordinates to achieve a significant reduction in model inference latency and parameter size. We evaluate CNNBooster with three benchmark models (VGG16, ResNet18 and MobilenetV1) on three NVIDIA-series platforms (V100, RTX 2080 Ti and Jetson Nano). The experimental results show that CNNBooster can achieve a maximum 67.51 % latency re-duction and 1.4x performance improvement compare with current state-of-the-art works.

Channel pruning guided by spatial and channel attention for DNNs in intelligent edge computing

In view of the problems that the application of target detection model in edge equipment occupied too much memory and the operation was slow, a lightweight model based on YoloV5 is proposed to detect the wearing condition of safety helmet and anti-slip shoes. The channel prunning is carried out on the basis of the original model, and the channel with low weight is deleted to reduce the parameters of the model and improve the detection speed. The experimental results show that the recognition accuracy of the network after channel pruning is 93.2%, which is basically the same as that before channel pruning. Meanwhile, the number of parameters is reduced by 63.9%, and the detection speed is 123.45fps. It provides technical support for the development of embedded detection equipment.

991012980216903412 HKUST Electronic Theses Neural network compression via quantization, channel pruning and beyond by Zechun Liu thesis 2021 1 online resource (xii, 131 pages) : illustrations (some color) Not presently available for public access at author’s request *CONFIDENTIAL* Ph.D. Electronic…Read more ›

This article proposes a novel module called middle spectrum grouped convolution (MSGC) for efficient deep convolutional neural networks (DCNNs) with the mechanism of grouped convolution. It explores the broad "middle spectrum" area between channel pruning and conventional grouped convolution. Compared with channel pruning, MSGC can retain most of the information from the input feature maps due to the group mechanism; compared with grouped convolution, MSGC benefits from the learnability, the core of channel pruning, for constructing its group topology, leading to better channel division. The middle spectrum area is unfolded along four dimensions: groupwise, layerwise, samplewise, and attentionwise, making it possible to reveal more powerful and interpretable structures. As a result, the proposed module acts as a booster that can reduce the computational cost of the host backbones for general image recognition with even improved predictive accuracy. For example, in the experiments on the ImageNet dataset for image classification, MSGC can reduce the multiply-accumulates (MACs) of ResNet-18 and ResNet-50 by half but still increase the Top-1 accuracy by more than 1%. With a 35% reduction of MACs, MSGC can also increase the Top-1 accuracy of the MobileNetV2 backbone. Results on the MS COCO dataset for object detection show similar observations. Our code and trained models are available at https://github.com/hellozhuo/msgc.

Model compression methods have become popular in recent years, which aim to alleviate the heavy load of deep neural networks (DNNs) in real-world applications. However, most of the existing compression methods have two limitations: 1) they usually adopt a cumbersome process, including pretraining, training with a sparsity constraint, pruning/decomposition, and fine-tuning. Moreover, the last three stages are usually iterated multiple times. 2) The models are pretrained under explicit sparsity or low-rank assumptions, which are difficult to guarantee wide appropriateness. In this article, we propose an efficient decomposition and pruning (EDP) scheme via constructing a compressed-aware block that can automatically minimize the rank of the weight matrix and identify the redundant channels. Specifically, we embed the compressed-aware block by decomposing one network layer into two layers: a new weight matrix layer and a coefficient matrix layer. By imposing regularizers on the coefficient matrix, the new weight matrix learns to become a low-rank basis weight, and its corresponding channels become sparse. In this way, the proposed compressed-aware block simultaneously achieves low-rank decomposition and channel pruning by only one single data-driven training stage. Moreover, the network of architecture is further compressed and optimized by a novel Pruning & Merging (PM) module which prunes redundant channels and merges redundant decomposed layers. Experimental results (17 competitors) on different data sets and networks demonstrate that the proposed EDP achieves a high compression ratio with acceptable accuracy degradation and outperforms state-of-the-arts on compression rate, accuracy, inference time, and run-time memory.

This paper designs a lightweight high-precision transmission line component detection model, named grouped dense, monotonic self-regularized, and partial faster convolution, pruning, and distillation optimized—you only look once (GMPPD-YOLO), in transmission line inspection. It addresses the issue of low detection accuracy of target detection algorithms due to the complex background, large differences in target shape, location, texture, etc, as well as diversified and smaller defects in insulator and vibration hammer images taken by unmanned aerial vehicles from multiple angles. To enhance the model’s feature extraction capabilities in complex backgrounds and across different scales, the grouped dense C3 dense feature extraction module was designed, enabling the model to more effectively handle diverse defect forms. Simultaneously, the monotonic self-regularized pyramid pooling–fast (MSPPF) module is proposed to enhance the model’s capability to process multi-scale information. Additionally, the partial-faster C3 feature awareness module is designed to improve feature fusion performance, enhancing the model’s ability to perceive features at different scales. Finally, channel pruning was used to reduce redundant parameters, and knowledge distillation was employed to compensate for the accuracy loss caused by pruning. This approach further compressed the model size while ensuring its detection performance. The experimental results demonstrate that compared to the original YOLOv5s algorithm, the proposed GMPPD-YOLO algorithm achieves a reduction in parameters by 68.4%, a decrease in Giga floating-point operations per second by 58.2%, and a reduction in the model size by 66.4%, while achieving an increase in precision by 1%, mAP50 by 1.1%, and mAP95 by 0.4%. This confirms the significant potential of the GMPPD-YOLO algorithm for deployment in real-time drone-based power transmission line inspections.

Image inpainting techniques have recently been developed leveraging deep neural networks and have seen many real-world applications. However, image inpainting networks, which are typically based on generative adversarial network (GAN), suffer from high parameter complexities and long inference time. While there are some efforts to compress image-to-image translation GAN, compressing image inpainting networks has rarely been explored. In this paper, we aim to create a small and efficient GAN-based inpainting model by compressing the generator of the inpainting model without sacrificing the quality of reconstructed images. We propose novel channel pruning and knowledge distillation techniques that are specialized for image inpainting models with mask information. Experimental results demonstrate that our compressed inpainting model with only one-tenth of the model size achieves similar performance to the full model.KeywordsImage inpaintingNetwork pruningKnowledge distillation

For practical deep neural network design on mobile devices, it is essential to consider the constraints incurred by the computational resources and the inference latency in various applications. Among deep network acceleration approaches, pruning is a widely adopted practice to balance the computational resource consumption and the accuracy, where unimportant connections can be removed either channel-wisely or randomly with a minimal impact on model accuracy. The coarse-grained channel pruning instantly results in a significant latency reduction, while the fine-grained weight pruning is more flexible to retain accuracy. In this paper, we present a unified framework for the Joint Channel pruning and Weight pruning, named JCW, which achieves a better pruning proportion between channel and weight pruning. To fully optimize the trade-off between latency and accuracy, we further develop a tailored multi-objective evolutionary algorithm in the JCW framework, which enables one single round search to obtain the accurate candidate architectures for various deployment requirements. Extensive experiments demonstrate that the JCW achieves a better trade-off between the latency and accuracy against previous state-of-the-art pruning methods on the ImageNet classification dataset.

A lightweight real-time unified detection model for rice and wheat ears in complex agricultural environments

Chemical exchange saturation transfer (CEST) MRI is a powerful molecular imaging technique for detecting metabolites through proton exchange. While CEST MRI provides high sensitivity, its clinical application is hindered by prolonged scan time due to the need for imaging across numerous frequency offsets for parameter estimation. Since scan time is directly proportional to the number of frequency offsets, identifying and selecting the most informative frequency can significantly reduce acquisition time. We propose a novel deep learning-based framework that integrates frequency selection and parameter estimation to accelerate CEST MRI. Our method leverages channel pruning via batch normalization to identify the most informative frequency offsets while simultaneously training the network for accurate parametric map prediction. Using data from six healthy volunteers, channel pruning selects 13 informative frequency offsets out of 53 without compromising map quality. Images from selected frequency offsets were reconstructed using the MR Multitasking method, which employs a low-rank tensor model to enable under-sampling of k-space lines for each frequency offset, further reducing scan time. Predicted parametric maps of amide proton transfer (APT), nuclear overhauser effect (NOE), and magnetization transfer (MT) based on these selected frequencies were comparable in quality to maps generated using all frequency offsets, achieving superior performance compared to Fisher information-based selection methods from our previous work. This integrated approach has the potential to reduce the whole-brain CEST MRI scan time from the original 5:30 min to under 1:30 min without compromising map quality. By leveraging deep learning for frequency selection and parametric map prediction, the proposed framework demonstrates its potential for efficient and practical clinical implementation. Future studies will focus on extending this method to patient populations and addressing challenges such as B0 inhomogeneity and abnormal signal variation in diseased tissues.

Real-time defects detection for apple sorting using NIR cameras with pruning-based YOLOV4 network

Sugarcane stem node detection is critical for monitoring sugarcane growth, enabling precision cutting, reducing spuriousness, and improving breeding for resistance to downfall. However, in complex field environments, sugarcane stem nodes often suffer from reduced detection accuracy due to background interference and shadowing effects. For this reason, this paper proposes an improved sugarcane stem node detection model based on YOLO11. This study incorporates the ASF-YOLO (Attentional Scale Sequence Fusion based You Only Look Once) mechanism to enhance the feature fusion layer of YOLO11. Additionally, a high-resolution detection layer, P2, is integrated into the fusion module to improve the model’s ability to detect small objects—particularly sugarcane stem nodes—and to better handle multi-scale feature representations. Secondly, to better align with the P2 small-object detection layer, this paper adopts a shared convolutional detection head named LSDECD (Lightweight Shared Detail-Enhanced Convolutional Detection Head), which can better deal with small target detection while reducing the number of model parameters through parameter sharing and detail-enhanced convolution. Using soft-NMS (non-maximum suppression) to replace the original NMS and combining with Shape-IoU, a bounding box regression method that focuses on the shape and scale of the bounding box itself, makes the bounding box regression more accurate, and solves the problem of the impact of detection caused by occlusion and illumination. Finally, to address the increased complexity introduced by the addition of the P2 detection layer and the replacement of the detection head, channel pruning is applied to the model, effectively reducing its overall complexity and parameter count. The experimental results show that the model before pruning has 96.1% and 53.2% mean average precision mAP50 and mAP50:95, respectively, which are 11.9% and 11.1% higher than the original YOLO11n, and the model after pruning also has 10.8% and 9.3% higher than the original YOLO11n, respectively, and the number of parameters is reduced to 279,778, and model size is reduced to 1.3MB. The computational cost decreased from 11.6 GFlops to 6.6 GFlops.

Aiming at the problems of leakage, misdetection and low detection efficiency of transmission line foreign object detection in railroad environment, we propose an improved railroad transmission line foreign object detection algorithm MRS-YOLO based on YOLOv11n. Firstly, a multi-scale Adaptive Kernel Depth Feature Fusion (MAKDF) module is proposed and fused with the C3k2 module to form C3k2_MAKDF, which enhances the model's feature extraction capability for foreign objects of different sizes and shapes. Secondly, a novel Re-calibration Feature Fusion Pyramid Network(RCFPN) is designed as a Neck structure to improve the feature fusion capability of the model. Then, Spatial and Channel Reconstruction Detect Head (SC_Detect) based on spatial and channel preprocessing is designed to improve the overall detection accuracy of the model. Finally, the channel pruning technique is used to reduce the redundancy of the improved model, drastically reduce Parameters and GFLOPs, and improve the detection efficiency. The experimental results show that the mAP50 and mAP50:95 of the MRS-YOLO algorithm proposed in this paper are improved to 94.8% and 86.4%, respectively, which are 0.7 and 2.3 percentage points higher compared to the baseline, while Parameters and GFLOPs are reduced by 44.2% and 17.5%, respectively. It is demonstrated that the improved algorithm can be better applied to the task of foreign object detection in railroad transmission lines.