Joint Sparsity with Mixed Granularity for Efficient GPU Implementation

Pruning is recently prevalent in deep neural network compression to save memory footprint and accelerate network inference. Unstructured pruning, i.e., fine-grained pruning, helps preserve model accuracy, while structural pruning, i.e., coarse-grained pruning, is preferred for general-purpose platforms such as GPUs. This paper proposes a regularization-free structural pruning scheme to take advantage of both unstructured and structural pruning by heuristically mixing vector-wise fine-grained and block-wise coarse-grained pruning masks with an AND operation. Experimental results demonstrate that the proposal can achieve higher model accuracy and higher sparsity ratio of VGG-16 on CIFAR-10 and CIFAR-100 compared with commonly applied block and balanced sparsity.

Various sparse regularizers have been applied to machine learning problems, among which structured sparsity has been proposed for a better adaption to structured data. In this paper, motivated by effectively classifying linked data (e.g. Web pages, tweets, articles with references, and biological network data) where a group structure exists over the whole dataset and links exist between specific samples, we propose a joint sparse representation model that combines group sparsity and graph sparsity, to select a small number of connected components from the graph of linked samples, meanwhile promoting the sparsity of edges that link samples from different groups in each connected component. Consequently, linked samples are selected from a few sparsely-connected groups. Both theoretical analysis and experimental results on four benchmark datasets show that the joint sparsity model outperforms traditional group sparsity model and graph sparsity model, as well as the latest group-graph sparsity model.

The structure of block sparsity in multi-band signals is prevalent. Among the block-sparse signal problems for compressive sensing, the most presenting recovery algorithms require block sparsity as prior information, whereas it may not be available in many practical applications. In this paper, a block sparsity adaptive regularized orthogonal matching pursuit algorithm (BAROMP) for compressive sensing is presented. The proposed algorithm could guarantee the accuracy of recovery by both the adaptive process which chooses the candidate set automatically and the regularization process which decides the atoms in the final support set. The simulation results show that the recovery probability of BAROMP which does not require sparsity as prior information is near to BROMP.

Electrical impedance tomography (EIT) is a promising agile imaging modality that allows estimation of the electrical conductivity distribution at the interior of an object from boundary measurements, which is attractive in a broad spectrum of biomedical, geophysical, and industrial continuous monitoring applications. As the inverse problem of EIT image reconstruction generally suffers from severe ill-posedness, various regularization techniques have been adopted to impose moderate constraints so as to obtain desired, usually sparse solutions. Extensive investigation showed that an enhanced recover performance can be achieved by exploiting a priori knowledge on the sparse structures. For regularization incorporating such knowledge, Bayesian approach is considered a natural mechanism. In this paper, we introduce the concept of sparse Bayesian learning to EIT imaging, which yields improved accuracy and robustness in the image reconstruction performance by exploiting signal structures in terms of their block sparsity and intra-block correlation. The effectiveness of the proposed algorithm is confirmed by the phantom test results.

It is difficult to employ transformer models for computer vision in mobile devices due to their memory- and computation-intensive properties. Accordingly, there is ongoing research on various methods for compressing transformer models, such as pruning. However, general computing platforms such as central processing units (CPUs) and graphics processing units (GPUs) are not energy-efficient to accelerate the pruned model due to their structured sparsity. This paper proposes a low-power accelerator for transformers with various sizes of structured sparsity induced by pruning with different granularity. In this study, we can accelerate a transformer that has been pruned in a head-wise, line-wise, or block-wise manner. We developed a head scheduling algorithm to support head-wise skip operations and resolve the processing engine (PE) load imbalance problem caused by different number of operations in one head. Moreover, we implemented a sparse general matrix-to-matrix multiplication (sparse GEMM) module that supports line-wise and block-wise skipping. As a result, when compared with a mobile GPU and mobile CPU respectively, our proposed accelerator achieved <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$6.1\times$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$13.6\times$</tex> improvements in energy efficiency for the detection transformer (DETR) model and achieved approximately <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.6\times$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$7.9\times$</tex> improvements in the energy efficiency on average for the vision transformer (ViT) models.

In this paper, we investigate the problem of estimating a complex-valued Laplacian matrix with a focus on its application in the estimation of admittance matrices in power systems. The proposed approach is based on a constrained maximum likelihood estimator (CMLE) of the complex-valued Laplacian, which is formulated as an optimization problem with Laplacian and sparsity constraints. The complex-valued Laplacian is a symmetric, non-Hermitian matrix that exhibits a joint sparsity pattern between its real and imaginary parts. Thus, we present a group-sparse-based penalized log-likelihood approach for the Laplacian estimation. Leveraging the mixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell_{2,1}$</tex-math></inline-formula> norm relaxation of the joint sparsity constraint, we develop a new alternating direction method of multipliers (ADMM) estimation algorithm for the implementation of the CMLE of the Laplacian matrix under a linear Gaussian model. Next, we apply the proposed ADMM algorithms for the problem of estimating the admittance matrix under three commonly-used measurement models that stem from Kirchhoff's and Ohm's laws, each with different assumptions and simplifications: 1) the nonlinear alternating current (AC) model; 2) the decoupled linear power flow (DLPF) model; and 3) the direct current (DC) model. The performance of the ADMM algorithm is evaluated using data from the IEEE <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$33$</tex-math></inline-formula>-bus power system data under different settings. The numerical experiments demonstrate that the proposed algorithm outperforms existing methods in terms of mean-squared-error (MSE) and F-score, thus providing a more accurate recovery of the admittance matrix.

The success of recent deep convolutional neural networks (CNNs) depends on learning hidden representations that can summarize the important factors of variation behind the data. In this work, we describe Network Dissection, a method that interprets networks by providing meaningful labels to their individual units. The proposed method quantifies the interpretability of CNN representations by evaluating the alignment between individual hidden units and visual semantic concepts. By identifying the best alignments, units are given interpretable labels ranging from colors, materials, textures, parts, objects and scenes. The method reveals that deep representations are more transparent and interpretable than they would be under a random equivalently powerful basis. We apply our approach to interpret and compare the latent representations of several network architectures trained to solve a wide range of supervised and self-supervised tasks. We then examine factors affecting the network interpretability such as the number of the training iterations, regularizations, different initialization parameters, as well as networks depth and width. Finally we show that the interpreted units can be used to provide explicit explanations of a given CNN prediction for an image. Our results highlight that interpretability is an important property of deep neural networks that provides new insights into what hierarchical structures can learn.

Efficient subspace clustering and feature extraction via [formula omitted]-norm and [formula omitted]-norm minimization

This article introduces a probabilistic Bayesian model for addressing the problem of compressive multipolarization through-wall radar imaging (TWRI). The proposed approach formulates the task of wall-clutter mitigation and multipolarization image reconstruction as a Bayesian inference problem for a joint distribution between observed radar measurements and latent wall-clutter matrix and indoor target images. The joint probability distribution incorporates three prior beliefs: low-dimensional structure of the wall reflections, group sparsity structure of the target images, and joint sparsity among the polarization images. These signal attributes are modeled through hierarchical priors, whose parameters and hyperparameters are treated with a full Bayesian formulation. Furthermore, this article presents a variational Bayesian inference algorithm that estimates wall-clutter and multipolarization images as posterior distributions and optimizes the model parameters and hyperparameters simultaneously. Experimental results on simulated and real radar data show that the proposed model is very effective at removing wall clutter and enhancing target localization even when the radar measurements are significantly reduced.

In this paper, a downlink cooperative channel estimation scheme is proposed for the three-dimensional massive multiple inputs multiple outputs (3D-mMIMO) system operating in the frequency division duplexing (FDD) mode. In the proposed cooperative scheme, users have to cooperate with each other via device-to-device (D2D) communication protocol to jointly exploit the sparsity structure property of the channel. Motivated by the sparsity property of the mMIMO channel in the angle-time domain, a parametric feedback scheme is proposed, where the feedback overhead is decreased by sending a limited version of the estimated coefficients rather than all the coefficients back to the BS. Then, a compressive sensing (CS) algorithm is proposed, which we named weighted fast iterative shrinkage thresholding (WFISTA). In the WFISTA, we first introduce new weights and threshold function to the original FISTA to enhance its sparsity-undersampling trade-off in the single measurement vector (SMV) case, then we extend the proposed WFISTA to the case of multiple measurement vector (MMV) problem by adopting ReMBo (reduce MMV and boost) strategy. The proposed WFISTA has the ability to estimate the mMIMO system's channel coefficients by exploiting the joint sparsity structure through cooperation among users' equipment (UEs). Complexity analysis and the probability of decreasing the feedback overhead are provided for the proposed cooperative estimation scheme. The simulation results verify the efficiency of the proposed cooperative algorithm scheme compared to several joint channel estimations.

Sparse signal processing has shown a significant promise for the reconstruction of biomedical signals, which possess well-defined sparsity structures in an appropriate transform domain. In this paper, a reconstruction algorithm is proposed based on the multiple-measurement-vector model using the hidden sparsity and correlation structure of biomedical signals. This incorporates subspace filtering over the partial supports, initialized by the sparse Bayesian learning framework along with a maximum projection-based support-estimation technique. The proposed method outperforms the conventional algorithms with respect to correct support recovery rate and needs fewer measurements. The robustness is studied by employing real-world biomedical signals. The experimental results using multichannel fetal-electrocardiogram signals and respiratory signals, collected from the Physiobank database, show that the proposed technique is an improved method in terms of reconstruction quality and compression rate.

Sparse Bayesian learning using correlated hyperparameters for recovery of block sparse signals

This study assesses the accuracy of Elo-based rating models in sparse, fragmented networks of pairwise comparisons. Three different modifications to the standard Elo rating are introduced and combined to eight model variants which were tested on both real-world and artificially generated data from the domain of football. The first modification implements a noise-free input by replacing the match outcome with the matches betting odds. The second modification adapts the k-parameter to adhere for competition differences. The last introduces a league-consistent Elo adaptation. Forecasting accuracy, ranking quality, and comparison accuracy were assessed to compare model performance based on both a rich and a limited version of the dataset. Results show that Elo ratings are suitable for comparisons of teams that never meet in matches. Additionally, noise-free Elo continuously shows the best accuracy, while league-consistent Elo enhances accuracy when data availability is limited. The artificial dataset showed strong similarities to the real data, supporting its suitability for controlled experiments. Overall, the findings highlight the importance of the interplay of model choice and data availability, while demonstrating the general ability of Elo ratings to evaluate and compare teams, even in sparse network structures.

A novel SRC fusion method using hierarchical multi-scale LBP and greedy search strategy

Dictionary learning algorithms have been successfully used in both reconstructive and discriminative tasks, where the input signal is represented by a linear combination of a few dictionary atoms. While these methods are usually developed under l 1 sparsity constrain (prior) in the input domain, recent studies have demonstrated the advantages of sparse representation using structured sparsity priors in the kernel domain. In this paper, we propose a supervised dictionary learning algorithm in the kernel domain for hyperspectral image classification. In the proposed formulation, the dictionary and classifier are obtained jointly for optimal classification performance. The supervised formulation is task-driven and provides learned features from the hyperspectral data that are well suited for the classification task. Moreover, the proposed algorithm uses a joint (l 12 ) sparsity prior to enforce collaboration among the neighboring pixels. The simulation results illustrate the efficiency of the proposed dictionary learning algorithm.