Deep learning acceleration in 14nm CMOS compatible ReRAM array: device, material and algorithm co-optimization

Biased outcomes in machine learning models can arise due to various factors, including limited training data, imbalanced class distribution, suboptimal training methodologies, and overfitting. In training neural networks with Stochastic Gradient Descent (SGD) and backpropagation, the choice of hyperparameters like learning rate and momentum is crucial to influencing the model’s performance. A comprehensive grid search study was conducted using static hyperparameters with standard SGD and dynamic hyperparameters with the Adam optimizer. The investigation focused on a multifaceted analysis across different tasks —- classification, segmentation, and detection — and was applied to four image-based applications: digit classification using the MNIST dataset, tuberculosis detection from chest X-ray images, lung segmentation in chest X-ray images, and the detection of malaria parasites in blood smear images. In the first comparative study on the MNIST dataset, the SGD algorithm consistently outperformed the Adam algorithm as noise levels increased. SGD held a slight advantage in accuracy over Adam in a noise-free environment. This advantage became more apparent as noise was introduced and increased to moderate levels. At high noise levels, both algorithms experienced a significant decline in performance, yet SGD maintained a relatively better accuracy compared to Adam. This trend underscored SGD’s superior ability to generalize across varying noise conditions. For TB detection in the second application, the DenseNet121 architecture was used, and it was found that Adam showed better performance on the larger TBX11K dataset. However, SGD outperformed Adam on a smaller subset of TBX11K. In the third comparative study for lung segmentation using COVID-19 and NLM datasets, SGD slightly outperformed Adam based on the mean and Hausdorff distances. In the case of malaria parasite detection using the YOLOv8 architecture, SGD and Adam optimizers showed varying performances across different conditions. Initially, both achieved high accuracies on the full dataset, with Adam slightly outperforming SGD. However, with decreasing dataset size, SGD maintained more consistent performance, while Adam’s accuracy fluctuated significantly. In noise tests, both showed equal accuracy on clean data, but under noise conditions, SGD maintained higher accuracies, suggesting better generalization capabilities. Further experiments assessed how well SGD and Adam could cope with domain shifts, mainly when using data from different countries. SGD’s generalization performance was superior to Adam’s performance in these experiments. In summary, although Adam is arguably the more popular of the two optimization techniques, SGD held its own in the experiments and showed superior performance when it came to generalization or classifying in noisy conditions.

Explainable optimization of deep learning model for COVID-19 detection using chest images

In this paper, we consider constrained optimization problems subject to a convex set C. Stochastic gradient descent (SGD) is a simple and popular stochastic optimization algorithm that has been the workhorse of machine learning for many years. We show a new and surprising fact about SGD, in that depending on the constraint set C, one can operate SGD with much lower-dimensional stochastic gradients without affecting its performance. In particular, we design an optimization algorithm that operates with the lower-dimensional (compressed) stochastic gradients, and establish that with the right set of parameters it has the exact same dimension-free convergence guarantees as that of regular SGD for popular convex and nonconvex optimization settings. We also present two applications of these bounds, one for improving the empirical risk minimization bounds in differentially private nonconvex optimization, and other for reducing communication costs with distributed SGD. Additionally, we also show that this connection between constraint set structure and gradient compression also extends beyond SGD to the conditional gradient (Frank-Wolfe) method. The geometry of the constraint set, captured by its Gaussian width, plays an important role in all our results.

SGD (Stochastic gradient descent) is widely used in deep learning, however SGD cannot get linear convergence and is not effective in large amounts of data. This paper use SSAG to improve the efficiency. SSAG contains two optimization strategies, one is stratified sampling strategy and the other is historical gradient averaging strategy. It has the advantages of fast convergence of variance, flexible application to big data, and easy work in deep network. This paper studies the efficiency of SSAG gradient optimization algorithm based on RNN framework. The proposed RNN framework comprises a feature extraction layer, a stacked RNN layer, and a transcription layer. The experimental results confirm that the accuracy of SSAG is better than the SGD and the Momentum. Both stratified sampling and historical averaging strategies have the effect of improving task accuracy. Experimental results verified that SSAG has better effect in image classification task.

Information Theory (IT) has been used in Machine Learning (ML) from early days of this field. In the last decade, advances in Deep Neural Networks (DNNs) have led to surprising improvements in many applications of ML. The result has been a paradigm shift in the community toward revisiting previous ideas and applications in this new framework. Ideas from IT are no exception. One of the ideas which is being revisited by many researchers in this new era, is Information Bottleneck (IB); a formulation of information extraction based on IT. The IB is promising in both analyzing and improving DNNs. The goal of this survey is to review the IB concept and demonstrate its applications in deep learning. The information theoretic nature of IB, makes it also a good candidate in showing the more general concept of how IT can be used in ML. Two important concepts are highlighted in this narrative on the subject, i) the concise and universal view that IT provides on seemingly unrelated methods of ML, demonstrated by explaining how IB relates to minimal sufficient statistics, stochastic gradient descent, and variational auto-encoders, and ii) the common technical mistakes and problems caused by applying ideas from IT, which is discussed by a careful study of some recent methods suffering from them.

Federated learning over wireless networks requires aggregating locally computed gradients at a server where the mobile devices send statistically distinct gradient information over heterogenous communication links. This paper proposes a Bayesian approach for wireless federated learning referred to as soft-sign stochastic gradient descent (soft-signSGD). The idea of soft-signSGD is to aggregate the one-bit quantized local gradients at the server by jointly exploiting i) the prior distributions of the local gradients, ii) the gradient quantizer function, and iii) channel distributions. This aggregation method is optimal in the sense of minimizing the mean-squared error (MSE) under a simplified Gaussian prior assumption on the local gradient. From simulations, we demonstrate that soft-signSGD considerably outperforms the conventional sign stochastic gradient descent algorithm when training and testing neural networks using the MNIST dataset and the CIFAR-10 dataset over heterogeneous wireless networks.

Deep learning algorithms can be used to design systems that report data on patients and deliver warnings to medical applications or electronic health information if there are changes in the patient's health. These systems could be created using deep Learning. This helps verify that patients get the proper effective care at the proper time for each specific patient. A healthcare decision support system was presented using the Internet of Things and deep learning methods. In the proposed system, we examined the capability of integrating deep learning technology into automatic diagnosis and IoT capabilities for faster message exchange over the Internet. We have selected the suitable Neural Network structure (number of best-hidden layers and activation function classes) to construct the e-health system. In addition, the e-health system relied on data from doctors to understand the Neural Network. In the validation method, the total evaluation of the proposed healthcare system for diagnostics provides dependability under various patient conditions. Based on evaluation and simulation findings, a dual hidden layer of feed-forward NN and its neurons store the tanh function more effectively than other NN. To overcome challenges, this study will integrate artificial intelligence with IoT. This study aims to determine the NN's optimal layer counts and activation function variations.

Hepatocellular carcinoma (HCC), the primary cancer of the liver, is life-threatening, with few or no symptoms, and detection in the early stage will help for successful treatment with surgery, and transplant for a better life quality. Here, we proposed two stacking classification models based on deep learning with differential hybrid feature selection for the early detection of HCC using novel non-invasive biomarker PIVKA-II. We showed how the variations in hybrid feature selection affect the performance of stacking classification and different supervised machine-learning algorithms on a metaclassifier. The base layers were support vector machine (SVM), gradient boosting (GB), and linear discriminant analysis (LDA). The meta classifier was a multilayer perceptron (MLP) with three different optimizers, stochastic gradient descent (SGD), adaptive moment estimation (ADAM), and Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS). Our first model outperformed the second with their hybrid features by improving accuracy by 1.5% and F1_score by 1% in both SGD and ADAM optimization, while MLP-LBFGS had a 1.4% increase in accuracy. The precision had hiked by 1.9%, 3.5%, and 1.7% in SGD, ADAM, and LBFGS, respectively, in Model-1. Matthew’s correlation coefficient (MCC) for MLP-SGD increased from 0.82 to 0.85, MLP-ADAM from 0.81 to 0.85, and MLP-LBFGS from 0.75 to 78 for the first model.

Disturbance rejection is a fundamental problem in control engineering, and there are many methods to achieve a good disturbance rejection. One of the standard methods for the disturbance rejection is to employ an integral compensator. This compensator integrates the error between the reference signal and the output, and use the integrated value with a specific coefficient as a compensating signal. In this work, we discuss the relationship between the machine learning theory and the integral compensation. We focus on single-input-single-output discrete-time time-invariant systems, and show that the integral compensation can be understood in the context of the machine learning in this case. In particular, the integral compensation is identical to the online optimization with the standard stochastic gradient descent [1]. The above idea gives two suggestions. First, the integral compensation may become faster by employing other types of stochastic gradient descent. Many algorithms of stochastic gradient descent have been proposed in the machine learning literature, and such algorithms may improve the classical integral compensation. Second, it may become possible to reject the time-invariant state-dependent disturbance. The modeling error is a typical example of such disturbance. By learning such a disturbance, the control performance for repetitive motions will be improved compared to the integral compensation. This presentation discusses the pros and cons of regarding the integral compensation as the stochastic gradient descent optimization.

Nowadays, data in power system have become big data and have attracted much attention due to the huge treasure buried in them. There is a great demand for forecasting electrical load and avoiding unexpected losses in power system. How to deal with the tremendous amount of data and capture the hidden information has become an important question. Deep learning can effectively extract the intrinsic features from data, achieving state-of-the-art performance in electrical load forecasting. However, it is often stuck in local minimum because of random initialization of parameters and structure selection in the stochastic gradient descent (SGD) so that its robust performance which is attached importance to in electrical practice is not guaranteed. To address that problem, in this paper, we proposed a robust method for accurate electrical load forecasting on the basis of the evolutionary algorithm which can avoid local minimum trap in deep learning trained by SGD. Specially, the deep learning model's structure and parameters will be optimized by not only gradient but also survival of the fittest through selection, crossover, mutation, and replacement on different models. Experimental results on the real electrical load data demonstrate the effectiveness of the evolutionary deep learning model in terms of accuracy of electrical load forecasting.

Deep learning is a sub-branch of artificial intelligence that acquires knowledge by training a neural network. It has many applications in the field of banking, automobile industry, agriculture, and healthcare industry. Deep learning has played a significant role in solving complex tasks related to computer vision, such as image classification, natural language processing, and object detection. On the other hand, optimizers also play an intrinsic role in training the deep learning model. Recent studies have proposed many deep learning models, such as VGG, ResNet, DenseNet, and ImageNet. In addition, there are many optimizers such as stochastic gradient descent (SGD), Adam, AdaDelta, Adabelief, and AdaMax. In this study, we have selected those models that require lower hardware requirements and shorter training times, which facilitates the overall training process. We have modified the Adam based optimizers and minimized the cyclic path. We have removed an additional hyper-parameter from RMSProp and observed that the optimizer works with various models. The learning rate is set to minimum and constant. The initial weights are updated after each epoch, which helps to improve the accuracy of the model. We also changed the position of the epsilon in the default Adam optimizer. By changing the position of the epsilon, it accumulates the updating process. We used various models with SGD, Adam, RMSProp, and the proposed optimization technique. The results indicate that the proposed method is effective in achieving the accuracy and works well with the state-of-the-art architectures.

Robust Prediction of Isodose Distribution with a Fully Convolutional Networks (FCN)-Based Deep Learning Model

The staggering innovations and emergence of numerous deep learning (DL) applications have forced researchers to reconsider hardware architecture to accommodate fast and efficient application-specific computations. Applications, such as object detection, image recognition, speech translation, as well as music synthesis and image generation, can be performed with high accuracy at the expense of substantial computational resources using DL. Furthermore, the desire to adopt Industry 4.0 and smart technologies within the Internet of Things infrastructure has initiated several studies to enable on-chip DL capabilities for resource-constrained devices. Specialized DL processors reduce dependence on cloud servers, improve privacy, lessen latency, and mitigate bandwidth congestion. As we reach the limits of shrinking transistors, researchers are exploring various application-specific hardware architectures to meet the performance and efficiency requirements for DL tasks. Over the past few years, several software optimizations and hardware innovations have been proposed to efficiently perform these computations. In this article, we review several DL accelerators, as well as technologies with emerging devices, to highlight their architectural features in application-specific integrated circuit (IC) and field-programmable gate array (FPGA) platforms. Finally, the design considerations for DL hardware in portable applications have been discussed, along with some deductions about the future trends and potential research directions to innovate DL accelerator architectures further. By compiling this review, we expect to help aspiring researchers widen their knowledge in custom hardware architectures for DL.

Power system fault type classification and location prediction is critical in assessing the reliability of the power system, and later restoring it to a stable operating point followed by a fault. State of the art methods include sequence component, impedance measurement from the origin of the fault, and traveling wave based methods for fault type detection and classification problem. It is important to identify and classify the fault as quickly as possible for restoring power system stability to normal operation. Machine Learning and Deep Learning methods allow the analysis of large data of fault voltages and currents by using fast and efficient algorithms. These methods require large amount of data, however, with the recent advances in the field of power system, data acquisition using smart meters and Phasor Measurement Units (PMU), huge amount of system-wide data can be made available to analyze the problem of fault type classification and location prediction. This paper presents a comparative study of Stochastic Gradient Descent (SGD) based Deep Neural Network (DNN) and Machine Learning (ML) applied to power system fault type and location prediction problem. DNN architecture uses 10 hidden layers, each layer having 60 units with hyperbolic tangent as activation function, and a combination of Support Vector Machine (SVM) and Principal Component Analysis (PCA) method are considered. Comparative results in terms of time taken to run the algorithms, and accuracy of the results obtained are presented for a 3 machine 9 bus system. Results indicate that SVM method is an optimal choice for fault classification with high accuracy for location prediction and low computational requirements compared with DNN.

Deep learning has emerged as a transformative technology in data science, revolutionizing various domains through its powerful capabilities. This paper explores the theoretical foundations, practical applications, and comparative analysis of deep learning models. The theoretical foundations section discusses key neural network architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers, highlighting their unique capabilities in processing different types of data. Optimization algorithms crucial for effective training, including Stochastic Gradient Descent (SGD) and Adam, are examined. Regularization techniques for preventing overfitting and enhancing generalization are also addressed. Practical applications in healthcare, finance, and retail showcase the real-world impact of deep learning. A comparative analysis of performance metrics demonstrates the superiority of deep learning models over traditional methods. Despite their advantages, deep learning models face limitations and challenges, including data dependency and interpretability issues. The paper concludes by emphasizing the ongoing research efforts to mitigate these challenges and ensure the continued advancement of deep learning in data science.