A Systematic Literature Review on Hardware Reliability Assessment Methods for Deep Neural Networks

With the advancement of machine learning methods, audio sentiment analysis has become an active research area in recent years. For example, business organizations are interested in persuasion tactics from vocal cues and acoustic measures in speech. A typical approach is to find a set of acoustic features from audio data that can indicate or predict a customer's attitude, opinion, or emotion state. For audio signals, acoustic features have been widely used in many machine learning applications, such as music classification, language recognition, emotion recognition, and so on. For emotion recognition, previous work shows that pitch and speech rate features are important features. This thesis work focuses on determining sentiment from call center audio records, each containing a conversation between a sales representative and a customer. The sentiment of an audio record is considered positive if the conversation ended with an appointment being made, and is negative otherwise. In this project, a data processing and machine learning pipeline for this problem has been developed. It consists of three major steps: 1) an audio record is split into segments by speaker turns; 2) acoustic features are extracted from each segment; and 3) classification models are trained on the acoustic features to predict sentiment. Different set of features have been used and different machine learning methods, including classical machine learning algorithms and deep neural networks, have been implemented in the pipeline. In our deep neural network method, the feature vectors of audio segments are stacked in temporal order into a feature matrix, which is fed into deep convolution neural networks as input. Experimental results based on real data shows that acoustic features, such as Mel frequency cepstral coefficients, timbre and Chroma features, are good indicators for sentiment. Temporal information in an audio record can be captured by deep convolutional neural networks for improved prediction accuracy.

We present a physics‐informed deep neural network (DNN) method for estimating hydraulic conductivity in saturated and unsaturated flows governed by Darcy's law. For saturated flow, we approximate hydraulic conductivity and head with two DNNs and use Darcy's law in addition to measurements of hydraulic conductivity and head to train these DNNs. For unsaturated flow, we approximate unsaturated conductivity function and capillary pressure with DNNs and train these DNNs using measurements of capillary pressure and the Richards equation. Because it is difficult to measure unsaturated conductivity in the field, we assume that no measurements of unsaturated conductivity are available. The proposed approach enforces the partial differential equation (PDE) (Darcy or Richards equation) constraints by minimizing the PDE residual at select points in the simulation domain. We demonstrate that physics constraints increase the accuracy of DNN approximations of sparsely observed functions and allow for training DNNs when no direct measurements of the functions of interest are available. For the saturated conductivity estimation problem, we show that the physics‐informed DNN method is more accurate than the state‐of‐the‐art maximum a posteriori probability method. For the unsaturated flow in homogeneous porous media, we find that the proposed method can accurately estimate the pressure‐conductivity relationship based on the capillary pressure measurements only, even in the presence of measurement noise.

SUMMARY Traditional seafloor mapping relies on shipborne soundings which have limited spatial coverage. The Surface Water and Ocean Topography (SWOT) wide-swath altimetry satellite holds the potential for predicting more detailed seafloor topography. In this study, we integrate SWOT gravity data with single-beam shipborne depths to construct seafloor topography models in the Northwestern Pacific using the deep neural network (DNN) method. Compared to shipborne depth checkpoints, the root mean square (RMS) error of the differences between topography model predicted by DNN method and shipborne depths is approximately 97.5 m, improving by 19.5 per cent and 9.9 per cent compared to the gravity-geologic (GGM) method and the Smith and Sandwell (SAS) method respectively. Compared to traditional data, the integration of SWOT gravity data universally enhances prediction accuracy. Furthermore, the DNN method effectively demonstrates superior capability in balancing the characterization of overall structures with the retention of authentic topography features, which we demonstrated in the Mariana region of the NW Pacific Ocean. However, limited by spatial heterogeneity and physical mechanisms, accurate prediction of such complex, fine-scale topography using gravity data remains a significant challenge.

Multi-label learning based deep transfer neural network for facial attribute classification

Recently, smart farming research based on artificial intelligence (AI) has been widely applied in the field of agriculture to improve crop cultivation and management. Predicting the harvest time (time-to-harvest) of crops is important in smart farming to solve problems such as planning the production schedule of crops and optimizing the yield and quality. This helps farmers plan their labor and resources more efficiently. In this paper, our concern is to predict the time-to-harvest (i.e., survival time) of tomatoes on a smart farm. For this, it is first necessary to develop a deep learning modeling approach that takes into account the farm effect on the tomato plants, as each farm has multiple tomato plant subjects and outcomes on the same farm can be correlated. In this paper, we propose deep neural network (DNN) survival models to account for the farm effect as a fixed effect using one-hot encoding. The tomato data used in our study were collected on a weekly basis using the Internet of Things (IoT). We compare the predictive performance of our proposed method with that of existing DNN and statistical survival modeling methods. The results show that our proposed DNN method outperforms the existing methods in terms of the root mean squared error (RMSE), concordance index (C-index), and Brier score.

Deep Neural Networks (DNNs) have permeated multiple applications, including cutting-edge safety-critical domains, which require relevant computational power, often provided by Graphic Processing Units (GPUs). GPUs are manufactured with advanced semiconductor technologies that can be affected by faults during the operational phase (e.g., due to wear-out, aging, or environmental harshness), whose effects possibly reach the DNN outputs, in some cases leading to catastrophic consequences. Hence, hardware-aware reliability assessments of DNNs are crucial to be considered in the context of safety-critical systems (following regulations/standards of specific application domains). Application-level fault injection (FI) techniques (i.e., DNN parameter corruption) are often adopted for the reliability evaluation of DNNs; unfortunately, these approaches hardly represent fault effects from GPU hardware. This work proposes an FI strategy based on Hardware-Injection-Through-Program-Transformation (HITPT) to mimic the effect of permanent faults (PFs) at the GPU instruction level, enabling effective assessment of PFs on DNN’s reliability. Our approach provides a good trade-off between the fault effect evaluation’s accuracy and the required computational time. Using the proposed approach, for the first time, we systematically assessed the effects of PF in GPUs executing some DNN sample cases. The results indicate that the faults injected closer to the hardware, using our evaluation strategy, can produce a higher accuracy degradation than the evaluations performed by the typical application-level FI that modify only the DNN parameters. Furthermore, the proposed FI methodology provides insightful results to identify the most suitable fault-tolerance solutions (e.g., selective hardening or design diversity) for their application at thread levels inside GPU’s kernels.

This paper analyzes the memory access amount in different data reuse schemes and internal memory sizes in order to design hardware deep neural network (DNN) accelerator with better power efficiency. After comparing the trade-off between memory energy consumption and SRAM sizes, we decide to design the accelerator using mixed input/output reuse schemes. Experimental results show that the numbers of DRAM access and SRAM access are reduced compared with stateoi-the-art designs of similar designs for computation of the convolutional layers in the VGG-16 model.

Summary A meticulous interpretation of steady-state or unsteady-state relative permeability (Kr) experimental data is required to determine a complete set of Kr curves. In this work, different machine learning (ML) models were developed to assist in a faster estimation of these curves from steady-state drainage coreflooding experimental runs. These ML algorithms include gradient boosting (GB), random forest (RF), extreme gradient boosting (XGB), and deep neural network (DNN) with a main focus on and comparison of the two latter algorithms (XGB and DNN). Based on existing mathematical models, a leading-edge framework was developed where a large database of Kr and capillary pressure (Pc) curves were generated. This database was used to perform thousands of coreflood simulation runs representing oil-water drainage steady-state experiments. The results obtained from these simulation runs, mainly pressure drop along with other conventional core analysis data, were used to estimate analytical Kr curves based on Darcy’s law. These analytically estimated Kr curves along with the previously generated Pc curves were fed as features into the ML model. The entire data set was split into 80% for training and 20% for testing. The k-fold cross-validation technique was applied to increase the model’s accuracy by splitting 80% of the training data into 10 folds. In this manner, for each of the 10 experiments, nine folds were used for training and the remaining fold was used for model validation. Once the model was trained and validated, it was subjected to blind testing on the remaining 20% of the data set. The ML model learns to capture fluid flow behavior inside the core from the training data set. In terms of applicability of these ML models, two sets of experimental data were needed as input; the first was the analytically estimated Kr curves from the steady-state drainage coreflooding experiments, while the other was the Pc curves estimated from centrifuge or mercury injection capillary pressure (MICP) measurements. The trained/tested model was then able to estimate Kr curves based on the experimental results fed as input. Furthermore, to test the performance of the ML model when only one set of experimental data is available to an end user, a recurrent neural network (RNN) algorithm was trained/tested to predict Kr curves in the absence of Pc curves as an input. The performance of the three developed models (XGB, DNN, and RNN) was assessed using the values of the coefficient of determination (R2) along with the loss calculated during training/validation of the model. The respective crossplots along with comparisons of ground truth vs. artificial intelligence (AI)-predicted curves indicated that the model is capable of making accurate predictions with an error percentage between 0.2% and 0.6% on history-matching experimental data for all three tested ML techniques. This implies that the AI-based model exhibits better efficiency and reliability in determining Kr curves when compared to conventional methods. The developed ML models by no means replace the need to conduct drainage coreflooding or centrifuge experiments but act as an alternative to existing commercial platforms that are used to interpret experimental data to predict Kr curves. The two main advantages of the developed ML models are their capability of predicting Kr curves within a matter of a few minutes as well as with limited intervention from the end user. The results also include a comparison between classical ML approaches, shallow neural networks, and DNNs in terms of accuracy in predicting the final Kr curves. The research presented here is an extension of the state-of-the-art framework proposed by Mathew et al. (2021). However, the two main aspects of the current study are the application of deep learning for the prediction of Kr curves and the application of feature engineering. The latter not only reduces the training/testing time for the ML models but also enables the end user to obtain the final predictions with the least set of experimental data. The various models discussed in this research work currently focus on the prediction of Kr curves for drainage steady-state experiments; however, the work can be extended to capture the imbibition cycle as well.

Raindrop size distributions (DSD) and rain rate have been estimated from polarimetric radar data using different approaches with the accuracy depending on the errors both in the radar measurements and the estimation methods. Herein, a deep neural network (DNN) technique was utilized to improve the estimation of the DSD and rain rate by mitigating these errors. The performance of this approach was evaluated using measurements from a two-dimensional video disdrometer (2DVD) at the Kessler Atmospheric and Ecological Field Station in Oklahoma as ground truth with the results compared against conventional estimation methods for the period 2006–17. Physical parameters (mass-/volume-weighted diameter and liquid water content), rain rate, and polarimetric radar variables (including radar reflectivity and differential reflectivity) were obtained from the DSD data. Three methods—physics-based inversion, empirical formula, and DNN—were applied to two different temporal domains (instantaneous and rain-event average) with three diverse error assumptions (fitting, measurement, and model errors). The DSD retrievals and rain estimates from 18 cases were evaluated by calculating the bias and root-mean-squared error (RMSE). DNN produced the best performance for most cases, with up to a 5% reduction in RMSE when model errors existed. DSD and rain estimated from a nearby polarimetric radar using the empirical and DNN methods were well correlated with the disdrometer observations; the rain-rate estimate bias of the DNN was significantly reduced (3.3% in DNN vs 50.1% in empirical). These results suggest that DNN has advantages over the physics-based and empirical methods in retrieving rain microphysics from radar observations.

Despite the great success of deep neural networks in many fields, the lack of interpretability has severely limited their wide application in security-sensitive tasks. Although current interpretable methods of deep neural networks such as visualization, class activation mapping, and sensitivity analysis can help users intuitively understand the inner working mechanism of the neural networks to some extent, they are either too coarse in explanation or too complex in explanation form to read easily. In order to use semantic information which is more understandable and close to human thought to interpret the deep neural network and increase the readability of the interpretation,we propose a semantic interpretation method of deep neural network based on knowledge graph. The method takes the VGG16 network as an example, and by mining the key neuronsof the neural network, construct semantic dictionaries and knowledge maps of key neurons, and automatically generate human-understandable semantic explanatory statements based on the knowledge maps. The method provides a new idea to improve the transparency of the operation process of deep neural networks, and also provides a clearer reference basis for pruning and tuning of deep neural networks.

Sentence-level sentiment analysis is a natural language processing model growing rapidly and strongly due to its role in artificial intelligence systems. There are many approaches to developing and improving the performance of sentence-level sentiment analysis with satisfactory performance, among which deep neural network methods are notable. However, sentence-level sentiment analysis methods based on deep neural networks often have two limitations: (i) The system architecture is not deep enough; (ii) Sentences containing unclear sentiments cannot be processed. To solve the above two challenges, in this paper, we propose a new sentence-level sentiment analysis method called Fuzzy-enhanced Deep Neural Networks (FeDN2) by deepening deep neural networks by adding fuzzy and defuzzy classes. FeDN2 includes the following main layers: (i) BERT layer to convert sentences into vectors. (ii) Fuzzification layer to blur sentence vectors. (iii) Deep convolutional layers to extract high-level features. (iv) Defuzzification layer to convert high-level feature vectors into clear values. (v) Fully connected layer to learn non-linear combinations of these high-level features. (vi) Classification layer to identify the sentiment polarity of sentences. FeDN2 was tested on two benchmark datasets. The results demonstrated that it can improve the performance of previous sentence-level sentiment analysis methods based on deep neural networks.

In recent years, a lot of power allocation algorithms have been proposed to maximize spectral efficiency (SE) and energy efficiency (EE) for the distributed antenna systems (DAS). However, the traditional iterative power allocation algorithms are difficult to be implemented in reality because of their high computational complexity. With the development of machine learning algorithms, it has been proved that the machine learning method has excellent learning ability and low computational complexity, which can approximate the traditional iterative power allocation well and be easily to be implemented in reality. In this paper, we propose a new deep neural network (DNN) model for DAS. From the perspective of machine learning, traditional iterative algorithms can be regarded as a nonlinear mapping between user channel realizations and optimal power allocation schemes. Therefore, we train the DNN to learn the nonlinear mapping between the user channel realizations and the corresponding power allocation schemes based on the traditional iterative algorithm. Then, a power allocation schemes based on DNN method is developed to maximize SE and EE for DAS. The simulation results show that the proposed scheme can not only obtain the almost similar performance as the traditional iterative algorithm, but also reduce much online computational time.

Numerical solution of Generalized Burger–Huxley & Huxley’s equation using Deep Galerkin neural network method

Deep neural networks (DNNs) have achieved unprecedented success across many scientific and engineering fields in the last decades. Despite its empirical success, unfortunately, recent studies have shown that there are various failure modes and blindspots in DNN models which may result in unexpected serious failures and potential harms, e.g. the existence of adversarial examples and small perturbations. This is not acceptable especially for safety critical and high stakes applications in the real-world, including healthcare, self-driving cars, aircraft control systems, hiring and malware detection protocols. Moreover, it has been challenging to understand why and when DNNs will fail due to their complicated structures and black-box behaviors. Lacking interpretability is one critical issue that may seriously hinder the deployment of DNNs in high-stake applications, which need interpretability to trust the prediction, to understand potential failures, and to be able to mitigate harms and eliminate biases in the model. To make DNNs trustworthy and reliable for deployment, it is necessary and urgent to develop methods and tools that can (i) quantify and improve their robustness against adversarial and natural perturbations, and (ii) understand their underlying behaviors and further correct errors to prevent injuries and damages. These are the important first steps to enable Trustworthy AI and Trustworthy Machine Learning. In this talk, I will survey a series of research efforts in my lab contributed to tackling the grand challenges in (i) and (ii). In the first part of my talk, I will overview our research effort in Robust Machine Learning since 2017, where we have proposed the first attack-agnostic robustness evaluation metric, the first efficient robustness certification algorithms for various types of perturbations, and efficient robust learning algorithms across supervised learning to deep reinforcement learning. In the second part of my talk, I will survey a series of exciting results in my lab on accelerating interpretable machine learning and explainable AI. Specifically, I will show how we could bring interpretability into deep learning by leveraging recent advances in multi-modal models. I'll present recent works in our group on automatically dissecting neural networks with open vocabulary concepts, designing interpretable neural networks without concept labels, and briefly overview our recent efforts on demystifying black-box DNN training process, automated neuron explanations for Large Language Models and the first robustness evaluation of a family of neuron-level interpretation techniques.

Optimisation and interpretation of machine and deep learning models for improved water quality management in Lake Loktak