Quantifying Explainability: Metrics and Methodologies for Dense Neural Network Interpretability

In this paper, we use graphics processing units (GPU) to accelerate sparse and arbitrary structured neural networks. Sparse networks have nodes in the network that are not fully connected with nodes in preceding and following layers, and arbitrary structure neural networks have different number of nodes in each layers. Sparse Neural networks with arbitrary structures are generally created in the processes like neural network pruning and evolutionary machine learning strategies. We show that we can gain significant speedup for full activation of such neural networks using graphical processing units. We do a prepossessing step to determine dependency groups for all the nodes in a network, and use that information to guide the progression of activation in the neural network. Then we compute activation for each nodes in its own separate thread in the GPU, which allows for massive parallelization. We use CUDA framework to implement our approach and compare the results of sequential and GPU implementations. Our results show that the activation of sparse neural networks lends very well to GPU acceleration and can help speed up machine learning strategies which generate such networks or other processes that have similar structure.

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today’s neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100× improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.

This paper presents a method for constructing a size compressed neural network with better or similar accuracy than a given dense neural network, therefore the compressed network requires less memory and computational resources. The presented method relies basically on learning successive mappings between the given dense neural network (teacher) hidden features and the size-compressed neural network (student) hidden features, where the latter is learned also to solve the initial task of the teacher network. The presented method is particularly compared to baselines where we specifically show that the additional learned mappings significantly improve the performance (accuracy and computation) of the student network.

Improving the skill of medium-range (3–8-day) severe weather prediction is crucial for mitigating societal impacts. This study introduces a novel approach leveraging decoder-only transformer networks to postprocess artificial intelligence (AI)-based weather forecasts, specifically from the Pangu-Weather model, for improved severe weather guidance. Unlike traditional postprocessing methods that use a dense neural network to predict the probability of severe weather using discrete forecast samples, our method treats forecast lead times as sequential “tokens,” enabling the transformer to learn complex temporal relationships within the evolving atmospheric state. We compare this approach against postprocessing of the Global Forecast System (GFS) using both a traditional dense neural network and our transformer, as well as configurations that exclude convective parameters to fairly evaluate the impact of using the Pangu-Weather AI model. Results demonstrate that the transformer-based postprocessing significantly enhances forecast skill compared to dense neural networks. Furthermore, AI-driven forecasts, particularly Pangu-Weather initialized from high-resolution analysis, exhibit superior performance to GFS in the medium range, even without explicit convective parameters. Our approach offers improved accuracy, and reliability, which also provides interpretability through feature attribution analysis, advancing medium-range severe weather prediction capabilities. Significance Statement This research aims to improve the prediction of general severe weather events, such as tornadoes, hail, and damaging winds, for forecasts made 3–8 days in advance. Our approach uses an advanced artificial intelligence (AI) technique, inspired by large language models, to refine severe weather predictions based on weather forecasts generated from AI weather models. We found this method can substantially boost forecast accuracy, in some cases nearly doubling prediction skill compared to simpler deep learning techniques. More accurate and earlier warnings give communities crucial extra time to prepare, potentially reducing the devastating impacts of severe storms and saving lives.

Dense convolutional neural networks based multiclass plant disease detection and classification using leaf images

Inferring and understanding the underlying connectivity structure of a system solely from the observed activity of its constituent components is a challenge in many areas of science. In neuroscience, techniques for estimating connectivity are paramount when attempting to understand the network structure of neural systems from their recorded activity patterns. To date, no universally accepted method exists for the inference of effective connectivity, which describes how the activity of a neural node mechanistically affects the activity of other nodes. Here, focussing on purely excitatory networks of small to intermediate size and continuous node dynamics, we provide a systematic comparison of different approaches for estimating effective connectivity. Starting with the Hopf neuron model in conjunction with known ground truth structural connectivity, we reconstruct the system’s connectivity matrix using a variety of algorithms. We show that, in sparse non-linear networks with delays, combining a lagged-cross-correlation (LCC) approach with a recently published derivative-based covariance analysis method provides the most reliable estimation of the known ground truth connectivity matrix. We outline how the parameters of the Hopf model, including those controlling the bifurcation, noise, and delay distribution, affect this result. We also show that in linear networks, LCC has comparable performance to a method based on transfer entropy, at a drastically lower computational cost. We highlight that LCC works best for small sparse networks, and show how performance decreases in larger and less sparse networks. Applying the method to linear dynamics without time delays, we find that it does not outperform derivative-based methods. We comment on this finding in light of recent theoretical results for such systems. Employing the Hopf model, we then use the estimated structural connectivity matrix as the basis for a forward simulation of the system dynamics, in order to recreate the observed node activity patterns. We show that, under certain conditions, the best method, LCC, results in higher trace-to-trace correlations than derivative-based methods for sparse noise-driven systems. Finally, we apply the LCC method to empirical biological data. Choosing a suitable threshold for binarization, we reconstruct the structural connectivity of a subset of the nervous system of the nematode C. elegans. We show that the computationally simple LCC method performs better than another recently published, computationally more expensive reservoir computing-based method. We apply different methods to this dataset and find that they all lead to similar performances. Our results show that a comparatively simple method can be used to reliably estimate directed effective connectivity in sparse neural systems in the presence of spatio-temporal delays and noise. We provide concrete suggestions for the estimation of effective connectivity in a scenario common in biological research, where only neuronal activity of a small set of neurons, but not connectivity or single-neuron and synapse dynamics, are known.

<i>Objective:</i> This paper assesses the diagnostic performance of deep learning methods for tumour detection in breast microwave sensing (BMS). <i>Methods:</i> A convolutional neural network (CNN) was used to predict the presence of a cancerous lesion in data from experimental scans of MRI-derived phantoms. An experimental dataset containing data from 1257 scans was used. The CNN was compared to a similarly sized dense neural network (DNN) and logistic regression classifier. <i>Results:</i> The CNN was able to exploit the sinogram data structure to achieve diagnostic performance significantly better than random classification, while neither the DNN nor logistic regression classifiers could generalize to unseen test data. The area under the curve of the receiver operating characteristic curve of the CNN classifier was estimated to be between (78 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3)&#x0025; and (90 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3)&#x0025;, where the upper estimate was obtained when the testing set was constrained to consist of phantoms with breast volumes that are within the volume bounds of the training set and when the tumour was located at the same vertical position as the system antennas. <i>Conclusion:</i> The results obtained in this investigation demonstrate the potential of combining deep learning and BMS systems for breast cancer detection. <i>Impact:</i> This paper provides an estimate of the diagnostic performance of an air-based BMS system using deep learning methods for automatic tumour detection on a sizable experimental dataset. The performance was found to be comparable to that of AI-assisted mammography and marks a first step toward larger-scale investigations in BMS.

Dense connected convolutional neural network (Dense Net) is a new architecture of deep convolutional neural network, which ensures the maximum information transmission between network layers by establishing the connection relationship between different layers. In the task of text remote supervised relation extraction, aiming at the limitation of existing neural network methods using shallow network to extract features, a deep convolution neural network model with dense connection based on attention mechanism is designed. In this model, the dense connection module and maximum pooling layer composed of 5 layer convolutional neural network are used as the sentence encoder. By combining the lexical, syntactic and semantic features of different levels, it helps the network to learn the features, so as to obtain more abundant semantic information of the input sentence. At the same time, it reduces the gradient vanishing phenomenon of deep neural network, and makes the network more capable of representing natural language Strong. Secondly, the word level attention is introduced to calculate the relevance between two entities and context words, so as to fully capture the semantic information of the entity context in the sentence; then the sentence level attention is constructed on multiple instances to reduce the problem of label error labeling; finally, the dependency inclusion relationship between different relationships is automatically learned through the attention of the relationship level. The average accuracy of the model on the NYT freebase dataset is 83.5%. Experimental results show that the model can effectively use features and improve the accuracy of remote supervised relationship extraction.

&lt;div class="section abstract"&gt;&lt;div class="htmlview paragraph"&gt;Predictive maintenance is crucial for Industry 4.0, and deep neural networks are a promising approach for predicting the capacity of electric batteries. However, few applications effectively utilize neural networks for this purpose with lithium-ion batteries. In this work, different deep learning models are developed, starting with simple neural networks, dense neural networks, convolutional networks, and recurrent networks. Using a public domain dataset, training, testing, and validation datasets were generated to predict battery capacity as a function of the number of cycles. Despite the limited number of samples in the dataset, deep learning techniques are employed to ensure robust prediction performance. The work presents the loss functions for each iteration of the algorithms and the average absolute error. The models made good generalizations over the test dataset within a short prediction time window. Finally, the work presents an average absolute error below 0.3, ensuring good convergence, avoiding data overfitting, and ensuring good generalization of the model to the test set. This work is a first estimate of the life cycle of electric vehicle batteries using small samples as inputs for estimating the remaining useful life of this electric vehicle component.&lt;/div&gt;&lt;/div&gt;

Retrieving the layout of different cluttered indoor scenes from monocular images is a challenging task, using a smaller deep neural network than the existing proposals. Previous geometric solutions are prone to failure in the presence of cluttered scenes because they depend strongly on hand-engineering features and the expectation of the possibility of the vanishing points’ calculation. With the growth of neural networks, the geometric methods were either replaced or fused within the emerging area of deep learning. The more recent solutions rely on dense neural networks with additional adjustments, either by the calculation of the vanishing points, position based, or by layout ranking. All these methods presented valid solutions to this challenge with the flaw of being computationally demanding. Here, we present a more lightweight solution, running the segmentation on a smaller neural network and introducing a discriminative classifier for the posterior layout ranking and optimization. Our proposed method is evaluated by two standard dataset benchmarks, achieving near state of the art results even with a fraction of the required parameters than the available state of the art methods.

This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices. The presented distillation methods are respectively based on Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The two methods basically rely on the successive dimension reduction of a given dense neural network (teacher) hidden features, and the learning of a smaller neural network (student) which solves the initial learning problem along with a mapping problem to the reduced successive features spaces. The presented methods are compared to baselines –learning the student networks from scratch–, and we show that the additional mapping problem significantly improves the performance (accuracy, memory and computing resources) of the student networks.

Structural MRI is the standard-of-care imaging modality for screening and diagnosis of most neurological conditions. However, the ability to reliably evaluate brain MRIs for disease characterization (whether by machines or experts) is often hampered by the presence of artifacts such as magnetic field inhomogeneity, aliasing, or patient motion; some of which may not be visually apparent. Reliable quality assessment of brain MRI scans would allow for excluding noisy acquisitions and reducing errors in subsequent downstream analyses. Since visual inspection is impractical for large volumes of data and subject to inter-observer variability, there is a need for accurate, automated Quality Assessment (QA) of MR images. Previous studies have investigated image quality metrics (IQMs) in order to quantify the effect of specific artifacts and quality degradation in an MR image. There has also been some recent work in developing machine learning models which use IQMs to enable robust QA across multiple sites. We build on this approach by leveraging a total of 64 IQMs (quantifying noise, artifacts, information, and general brain measurements) within a Dense Neural Network (DNN) model for QA of brain MRI scans using the publicly available multi-site ABIDE-I cohort (17 sites, 1102 subjects). In attempting to predict the "mean opinion score" of MR image quality (as assessed by expert radiologists), the DNN model yielded an accuracy of 87.3±2.1% in training (15 sites, 885 subjects) which generalized to a 79.5±1.9% accuracy in validation (2 sites, 246 subjects). This initial DNN model suggests the promise of deep learning to improve automated QA of MRI scans in large multi-site, multi-scanner cohorts.

Accelerated prediction of molecular properties for per- and polyfluoroalkyl substances using graph neural networks with adjacency-free message passing.

Basic image processing operations like median filtering and Gaussian blurring in general do not change the semantic content of an image, although they are commonly used to cover fingerprints of falsification that does alter image content such as copy-move and splicing. Therefore image forensics researchers are interested in detecting these basic operations. Some existing detectors track local inconsistencies in statistics of the image. However these statistics are very sensitive to image development process. Thus pre-processing operations can be damaging for performances of such detectors. In this paper, we focus on a very common pre-processing operation, i.e., re-sizing, and study how it affects performance when trying to detect several image processing operations on small patches, with Gaussian Mixture Model (GMM) as feature extractor and a Dense Neural Network (DNN) as classifier. We first show performance drops. We then introduce an adaptation method which relies on better fit to testing data for the feature extraction and fine-tuning for the neural network classifier. Experimental results show that our method is able to improve results with very few labeled testing samples. We also present comparisons with an improved version of a recent CNN(Convolutional Neural Network)-based method.

The rapid evolution of the Internet of Everything (IoE) has significantly enhanced global connectivity and multimedia content sharing, simultaneously escalating the unauthorized distribution of multimedia content, posing risks to intellectual property rights. In 2022 alone, about 130 billion accesses to potentially non-compliant websites were recorded, underscoring the challenges for industries reliant on copyright-protected assets. Amidst prevailing uncertainties and the need for technical and AI-integrated solutions, this study introduces two pivotal contributions. First, it establishes a novel taxonomy aimed at safeguarding and identifying IoE-based content infringements. Second, it proposes an innovative architecture combining IoE components with automated sensors to compile a dataset reflective of potential copyright breaches. This dataset is analyzed using a Bidirectional Encoder Representations from Transformers-based advanced Natural Language Processing (NLP) algorithm, further fine-tuned by a dense neural network (DNN), achieving a remarkable 98.71% accuracy in pinpointing websites that violate copyright.