Projecting on to the Multi-Layer Convolutional Sparse Coding Model

In this paper, we propose Multilayer network-based Visual Interpreter (MuVI), a framework for visual interpretability of Convolutional Neural Networks (CNNs) based on their mapping into multilayer networks. The peculiarity of MuVI is that it constructs a pixel-level heatmap of the salient parts of an image processed by a CNN, where the importance of each pixel depends on all layers of the CNN and not only on the final ones, as in the existing approaches in the literature. MuVI first maps the CNN into a multilayer network. It then uses this representation to identify the parts of the CNN that most influence the prediction results by extracting those paths within the multilayer network whose nodes correspond to the most active areas of the feature maps. The weight of the paths is given by the sum of the weights of the arcs corresponding to the activations across all feature maps of the CNN; this characteristic allows MuVI to consider all layers of the CNN, not just the last ones. Finally, MuVI constructs the visual interpretability heatmap by selecting the paths with the highest weights. The experimental tests performed show that MuVI is able to achieve very satisfactory results in terms of AUC insertion (0.25), AUC deletion (0.11), % Increase in Confidence (12.32), Average Drop % (51.22), Pointing Game Accuracy (0.28) and Computation time ( 26 . 226 s ). These results, taking all these measures together, are better than those obtained by the classical approaches already proposed in the literature, such as SmoothGrad, Grad-CAM, Grad-CAM++, and RISE. They are also comparable to state-of-the-art approaches in the literature, such as Score-CAM and HSIC. • A framework for visual interpretability of CNNs through a multilayer network. • Approaches to compute heatmaps from multilayer networks. • Visual interpretability for VGG16 and Imagenet. • Computation of heatmaps with pixel-level rather than area-level precision. • Competitive performance of the proposed framework compared to existing ones.

Early detection of plant nutrient deficiency is crucial for agricultural productivity. This study investigated the performance and interpretability of Convolutional Neural Networks (CNNs) for this task. Using the rice and banana datasets, we compared three CNN architectures (CNN, VGG-16, Inception-V3). Inception-V3 achieved the highest accuracy (93% for rice and banana), but simpler models such as VGG-16 might be easier to understand. To address this trade-off, we employed Explainable AI (XAI) techniques (SHAP and Grad-CAM) to gain insights into model decision-making. This study emphasises the importance of both accuracy and interpretability in agricultural AI and demonstrates the value of XAI for building trust in these models.

Convolutional Neural Networks (CNNs) have demonstrated an outstanding performance on a range of image classification problems in various domains. However, their major drawback is that they are “black box” and opaque classifiers. Taking into consideration the increasing demand for interpretable classification models, this paper introduces a novel meta-feature extraction scheme. This scheme is based on fuzzy sets, and it can be applied on the feature maps of a CNN. Initially, representative image prototypes are selected based on their deep feature map representation. Then, it constructs information granules from the feature maps, describing the content of each image class. It uses fuzzy sets to linguistically characterize the similarity between the deep feature maps of an image and the deep feature maps of the image prototypes. Thus, a classification outcome can be interpreted based on the features characterizing the different image classes involved in a classification problem. The experimental evaluation of the proposed scheme is performed on five publicly available image datasets. The results indicate that the proposed scheme outperforms other state-of-the-art classifiers, while providing an understandable interpretation of the classification result.

Sparse prior to image denoising is a classical research field with a long history in computer vision. We propose an end-to-end supervised neural network, named DnMLCSC-net, which is inspired via multi-layer convolutional sparse coding model embedded with symbiotic analysis–synthesis priors for natural image denoising. Unfolding a multi-layer, learned iterative soft thresholding algorithm (ML-LISTA) and developing into a convolutional recurrent neural network, all parameters in the model are updated adaptively to minimize mixed loss via gradient descent using backpropagation. In addition, a combined ReLU function is taken as the activation function. Inconsistent dilated convolution and batch normalization were empirically introduced into the encoding layers corresponding to the first iteration of ML-LISTA. Experimental results show that our network achieves a competitive denoising effect in comparison with several state-of-the-art denoising methods.

Hyperspectral images (HSIs) play a crucial role in remote sensing but are often degraded by complex noise patterns. Ensuring the physical property of the denoised HSIs is vital for robust HSI denoising, giving the rise of deep unfolding-based methods. However, these methods map the optimization of a physical model to a learnable network with a predefined depth, which lacks convergence guarantees. In contrast, Deep Equilibrium (DEQ) models treat the hidden layers of deep networks as the solution to a fixed-point problem and models them as infinite-depth networks, naturally consistent with the optimization. Under the framework of DEQ, we propose a Deep Equilibrium Convolutional Sparse Coding (DECSC) framework that unifies local spatial-spectral correlations, nonlocal spatial self-similarities, and global spatial consistency for robust HSI denoising. Within the convolutional sparse coding (CSC) framework, we enforce shared 2D convolutional sparse representation to ensure global spatial consistency across bands, while unshared 3D convolutional sparse representation captures local spatial-spectral details. To further exploit nonlocal self-similarities, a transformer block is embedded after the 2D CSC. Additionally, a detail enhancement module is integrated with the 3D CSC to promote image detail preservation. We formulate the proximal gradient descent of the CSC model as a fixed-point problem and transform the iterative updates into a learnable network architecture within the framework of DEQ. Experimental results demonstrate that our DECSC method achieves superior denoising performance compared to state-of-the-art methods.

In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.

This work investigates the problem of efficiently learning discriminative low-dimensional (LD) representations of multiclass image objects. We propose a generic end-to-end approach that jointly optimizes sparse dictionary and convolutions for learning LOW-dimensional discriminative image representations, named SparConvLow, taking advantage of convolutional neural networks (CNNs), dictionary learning, and orthogonal projections. The whole learning process can be summarized as follows. First, a CNN module is employed to extract high-dimensional (HD) preliminary convolutional features. Second, to avoid the high computational cost of direct sparse coding on HD CNN features, we learn sparse representation (SR) over a task-driven dictionary in the space with the feature being orthogonally projected. We then exploit the discriminative projection on SR. The whole learning process is consistently treated as an end-to-end joint optimization problem of trace quotient maximization. The cost function is well-defined on the product of the CNN parameters space, the Stiefel manifold, the Oblique manifold, and the Grassmann manifold. By using the explicit gradient delivery, the cost function is optimized via a geometrical stochastic gradient descent (SGD) algorithm along with the chain rule and the backpropagation. The experimental results show that the proposed method can achieve a highly competitive performance with the state-of-the-art (SOTA) image classification, object categorization, and face recognition methods, under both supervised and semi-supervised settings. The code is available at https://github.com/MVPR-Group/SparConvLow.

In the field of 3D Human Pose Estimation and Reconstruction based on body joints extracted from a 2D image; Exists challenges like self-occlusion and depth perception. These problems hinder approximate estimations. This article proposes a hybrid method that consists of semantic segmentation, sparse representation and 2D pose estimation for 3D estimation. Specifically, we train two fully Convolution neural networks (FCNs) to estimate 2D pose and semantic segmentation. Then, we narrow the estimation down using basic human body structure and results from the FCNs. Subsequently, utilizing the estimated 2D pose of the earlier step and a sparse representation model the 3D pose will estimate. Using a Convolution Neural Networks learning method for 2D human pose estimation and then by sparse representation will estimate. Results of the proposed method, show improvement relative to the previous works on 3D Human pose estimation. 3D Human Pose Estimation by this method demonstrates a significant mean error reduction compared to earlier studies.

Low contrast is a frequent challenge in image analysis, especially within medical imaging and highly saturated scenes. To address this issue, we present a nonlinear transformation for local contrast enhancement in digital images. Our method adapts the hyperbolic tangent function using two parameters: one to select the intensity range for modification and another to control the degree of enhancement. This approach outperforms conventional histogram-based techniques such as histogram equalization and specification in local contrast enhancement, without increasing computational cost, and produces smooth, artifact-free results in user-defined regions of interest. In addition, the proposed method was compared with CLAHE in MRIs, showing that, unlike CLAHE, the proposed method does not enhance the noise present in the background of the image. Furthermore, in deep learning contexts where dataset size is often limited, our method could serve as an effective data augmentation tool—generating varied contrast images while preserving anatomical structures, which improves neural network training for brain tumor detection in magnetic resonance imaging. The ability to manipulate local contrast may offer a pathway toward better interpretability of convolutional neural networks, as targeted contrast adjustments allow researchers to probe model sensitivity and enhance the explainability of classification and detection mechanisms.

Generally, emotions in short videos are transmitted through characters behaviour and text content in the video. The mainstream recognition model is based on a convolutional neural network. However, its processing ability for video data is limited, its recognition accuracy is not high enough, and it is limited in processing complex text. To solve these problems, a new text recognition emotion classification model is designed by improving the text feature fusion module on the basis of the convolutional neural network model. Moreover, a new human behaviour recognition emotion classification model is designed by introducing multi-head attention mechanisms on the basis of a 3D convolutional neural network model. These results confirmed that the accuracy of the improved text recognition model was around 75%, while the original convolutional model's average accuracy was only about 67%. The 3D convolutional model under the multi-head attention mechanism had the highest recognition accuracy, with a recognition accuracy of 94.5% on the UCF101 database, which was 12 to 39 percentage points higher than the model under other attention mechanisms. The improved convolutional network model for short video text classification and behaviour recognition has more advantages than traditional models. These research results have certain value for classification models and can serve as technical references.

In this paper, we propose the convolutional sparse coding based model in the wavelet domain for the task of single image super-resolution (SISR). The conventional sparse coding based approaches work on overlapping image patches and use the dictionary atoms to sparse code an image patch. Further, at the final stage, an overlap-add mechanism is used to get the final high-resolution image estimate. However, these algorithms fail to take into account the consistency present in the overlapping patches which limits their performance. We propose the use of wavelet integrated convolutional sparse coding approach where instead of dictionary atoms we utilize the convolution summations between the learned filters and their mappings for sparse representation based SISR. The use of wavelets is proposed owing to their unique directional and compact features. A pair of filters are learned along with a mapping function for each wavelet sub-band to exploit the consistency among patches. The proposed wavelet integrated convolutional sparse coding model helps capture useful contextual information. The proposed model is evaluated on publicly available datasets for different scale-up parameters. To show the efficacy of the proposed model we compare it with recent state-of-the-art algorithms. The visual results along with the quantitative ones indicate that the proposed model performs well for the tasks of super-resolution.

International audience

With the rapid development of science and technology, high-dimensional data emerge one after another. High-order tensors can be used to describe high-dimensional data structure, which can retain the hidden structure of data, but cannot obtain the deep features. Therefore, it is very important to establish a deep high order tensor model. In this paper, we propose a deep high-order tensor convolutional sparse coding model, which can automatically learn the deep convolutional kernel. Based on the learned deep convolutional kernel, a two-layer deep dictionary learning model is established. Then, the sparse representation coefficients are respectively solved, and a new weighted matching cost method is constructed, which combines shallow and deep features. The experimental results on the Middlebury 2014 dataset show that the proposed deep high-order tensor convolutional sparse coding is effective for stereo matching.

While many neural coding models have been proposed for V1, it remains an open question as to which model best describes the diversity of observed response properties. For instance, the canonical linear-nonlinear model (LN) partially explains some fundamental mechanistic and phenomenological properties of V1, but is unable to explain many nonlinear response properties that are likely associated with the keys to efficient and robust human vision. For example, surround suppression is one such nonlinear response property in which visual stimuli extending beyond the classical receptive field (CRF) selectively diminish neural responses. This property has been studied through electrophysiology experiments with synthetic stimuli (e.g., gratings). Surprisingly, high level sparse coding models implemented in a biologically plausible dynamical system have been shown to produce surround suppression effects that match individual and population observed responses [3]. More recently, surround suppression has been investigated experimentally using natural stimuli, and these experiments have shown an increase in the sparsity of measured responses [1,2]. Despite these findings, it remains unclear whether a functional sparse coding model is sufficient to produce the types of surround suppression observed with natural stimuli. In this abstract, we demonstrate that the surround suppression effects recently observed with natural stimuli are also emergent properties of a sparse coding model (Figure 1).

Compressed sensing (CS) reconstruction of nonsparse scenes is one of the difficulties in synthetic aperture radar (SAR) imaging technology. Although the conventional CS method with sparse representation has proven applicable for nonsparse SAR reconstruction, its disadvantages are unsatisfactory imaging quality and high computational complexity under downsampling. In this paper, a novel deep learning approach for nonsparse SAR scene reconstruction is proposed based on sparse representation and the iterative shrinkage threshold algorithm (ISTA). Specifically, we first develop a sparse representation-based imaging model associated with the ℓ₁ sparse regularizer in nonlinear transform domains. Then, the advantages of the recurrent neural network (RNN) and convolutional neural network (CNN) are incorporated into an ISTA-inspired deep unfolded network (DUN) called SR-ISTA-Net, in which all the parameters are layer-varied rather than handcrafted. The experiments verify that the proposed SR-ISTA-Net can provide high-quality reconstruction results under nonsparse scenes while substantially reducing imaging time.