An Efficient Optimization Technique for Training Deep Neural Networks

Deep learning algorithms can be used to design systems that report data on patients and deliver warnings to medical applications or electronic health information if there are changes in the patient's health. These systems could be created using deep Learning. This helps verify that patients get the proper effective care at the proper time for each specific patient. A healthcare decision support system was presented using the Internet of Things and deep learning methods. In the proposed system, we examined the capability of integrating deep learning technology into automatic diagnosis and IoT capabilities for faster message exchange over the Internet. We have selected the suitable Neural Network structure (number of best-hidden layers and activation function classes) to construct the e-health system. In addition, the e-health system relied on data from doctors to understand the Neural Network. In the validation method, the total evaluation of the proposed healthcare system for diagnostics provides dependability under various patient conditions. Based on evaluation and simulation findings, a dual hidden layer of feed-forward NN and its neurons store the tanh function more effectively than other NN. To overcome challenges, this study will integrate artificial intelligence with IoT. This study aims to determine the NN's optimal layer counts and activation function variations.

Artificial Intelligence in the American Healthcare Industry: Looking Forward to 2030

The past few years have seen a surge of applying Deep Learning (DL) models for a wide array of tasks such as image classification, object detection, machine translation, etc. While DL models provide an opportunity to solve otherwise intractable tasks, their adoption relies on them being optimized to meet latency and resource requirements. Benchmarking is a key step in this process but has been hampered in part due to the lack of representative and up-to-date benchmarking suites. This is exacerbated by the fast-evolving pace of DL models. This paper proposes DLBricks, a composable benchmark generation design that reduces the effort of developing, maintaining, and running DL benchmarks on CPUs. DLBricks decomposes DL models into a set of unique runnable networks and constructs the original model's performance using the performance of the generated benchmarks. DLBricks leverages two key observations: DL layers are the performance building blocks of DL models and layers are extensively repeated within and across DL models. Since benchmarks are generated automatically and the benchmarking time is minimized, DLBricks can keep up-to-date with the latest proposed models, relieving the pressure of selecting representative DL models. Moreover, DLBricks allows users to represent proprietary models within benchmark suites. We evaluate DLBricks using $50$ MXNet models spanning $5$ DL tasks on $4$ representative CPU systems. We show that DLBricks provides an accurate performance estimate for the DL models and reduces the benchmarking time across systems (e.g. within $95\%$ accuracy and up to $4.4\times$ benchmarking time speedup on Amazon EC2 c5.xlarge).

In order to increase prediction accuracy, deep learning models must be trained by adjusting parameters to minimize a loss function. In supervised learning, the mapping between inputs and their right outputs is learned by training models using labeled input examples. In order to minimize mistakes, predictions are compared to actual outcomes, and optimization methods are used to adjust parameters. Until convergence is reached, these algorithms go through several cycles of iteration. Stochastic Gradient Descent (SGD), Momentum SGD, RMSProp, AMSGrad, Adam, Yogi, and Lion are the seven optimization techniques that are evaluated in this study based on training accuracy, test accuracy, training loss, and sensitivity to learning rate. MNIST and CIFAR-10 were the two benchmark datasets used in the experiments. SGD with a learning rate of 0.5 had the best test accuracy of 99.14% and the highest training accuracy of 99.89% on MNIST. With test accuracy of 99.15% and 98%, respectively, at a learning rate of 1e-2, Momentum SGD and Adam likewise demonstrated strong performance. Optimizers like Yogi and Lion, on the other hand, performed competitively at lower learning rates but suffered at higher ones; at 1e-5, Lion's test accuracy was 98.69%. All optimizers displayed comparatively decreased accuracies for CIFAR-10, which was indicative of the dataset's increased complexity. Momentum SGD outperformed other optimizers including Adam, Yogi, and Lion, achieving the highest training accuracy of 98.90% and the best test accuracy of 72.94% at a learning rate of 1e-2. Lion showed better performance and stability on both datasets at a low learning rate of 1e-5. These results highlight how crucial it is to choose learning rates and optimization techniques that are specific to the features of each dataset.

In the age of pervasive connectivity, cryptography is a vital defensive measure for information security, and the security of cryptographic protection is of critical importance. Deep learning technology has recently made significant strides in areas like image classification and natural language processing, garnering considerable interest. Compared with classic cryptographic algorithms, modern block ciphers are more intricate, and the mappings between plaintext and ciphertext are less distinct, rendering the extraction of plaintext features from ciphertexts by neural networks as almost infeasible. However, the symbiosis of deep learning and traditional differential cryptanalysis holds promise for enhancing crypto-attack performance. Thus, the integration of deep learning theory and methods into the field of cryptography is becoming a significant trend in technological advancement. In this context, cryptanalysis is progressively developing in the direction of intelligence and automation, with an increasing number of researchers employing deep learning to assist in cryptanalytic tasks. This review aims to delve into the current research trends surrounding deep learning-supported differential cryptanalysis. It commences with a thorough recapitulation of differential analysis in cryptography and introduces common models in deep learning, along with their characteristics. Moreover, it encapsulates the design of differential classifiers powered by deep learning, inclusive of various optimization techniques utilized within these algorithms. The paper also posits directions for future research focus. Despite challenges, deep learning possesses vast potential in reinforcing conventional differential cryptanalysis, providing deeper insights for security analysis and response strategies, and serving as a valuable tool and perspective for the design and appraisal of future cryptographic solutions.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

Abstract: Neural networks have become a cornerstone of modern artificial intelligence applications, achieving remarkable success in areas such as image recognition, natural language processing, and predictive analytics. However, achieving high accuracy while avoiding overfitting remains a major challenge in deep learning systems. This paper explores various optimization techniques for improving neural network performance, including regularization methods, hyperparameter tuning, data augmentation, and advanced architectures. The study reviews approaches such as dropout, batch normalization, early stopping, and ensemble learning to enhance generalization. Additionally, the role of optimization algorithms like stochastic gradient descent (SGD), Adam, and RMSProp is discussed. Evaluation metrics such as accuracy, precision, recall, and loss functions are used to measure performance. Challenges such as model complexity, computational cost, and data imbalance are also addressed. The paper concludes with future directions focusing on automated machine learning (AutoML) and explainable AI. Overall, optimizing neural networks plays a crucial role in developing robust and reliable AI systems. Keywords: Neural Networks, Overfitting, Regularization, Deep Learning, Optimization Techniques, Dropout, Batch Normalization, Hyperparameter Tuning

Biased outcomes in machine learning models can arise due to various factors, including limited training data, imbalanced class distribution, suboptimal training methodologies, and overfitting. In training neural networks with Stochastic Gradient Descent (SGD) and backpropagation, the choice of hyperparameters like learning rate and momentum is crucial to influencing the model’s performance. A comprehensive grid search study was conducted using static hyperparameters with standard SGD and dynamic hyperparameters with the Adam optimizer. The investigation focused on a multifaceted analysis across different tasks —- classification, segmentation, and detection — and was applied to four image-based applications: digit classification using the MNIST dataset, tuberculosis detection from chest X-ray images, lung segmentation in chest X-ray images, and the detection of malaria parasites in blood smear images. In the first comparative study on the MNIST dataset, the SGD algorithm consistently outperformed the Adam algorithm as noise levels increased. SGD held a slight advantage in accuracy over Adam in a noise-free environment. This advantage became more apparent as noise was introduced and increased to moderate levels. At high noise levels, both algorithms experienced a significant decline in performance, yet SGD maintained a relatively better accuracy compared to Adam. This trend underscored SGD’s superior ability to generalize across varying noise conditions. For TB detection in the second application, the DenseNet121 architecture was used, and it was found that Adam showed better performance on the larger TBX11K dataset. However, SGD outperformed Adam on a smaller subset of TBX11K. In the third comparative study for lung segmentation using COVID-19 and NLM datasets, SGD slightly outperformed Adam based on the mean and Hausdorff distances. In the case of malaria parasite detection using the YOLOv8 architecture, SGD and Adam optimizers showed varying performances across different conditions. Initially, both achieved high accuracies on the full dataset, with Adam slightly outperforming SGD. However, with decreasing dataset size, SGD maintained more consistent performance, while Adam’s accuracy fluctuated significantly. In noise tests, both showed equal accuracy on clean data, but under noise conditions, SGD maintained higher accuracies, suggesting better generalization capabilities. Further experiments assessed how well SGD and Adam could cope with domain shifts, mainly when using data from different countries. SGD’s generalization performance was superior to Adam’s performance in these experiments. In summary, although Adam is arguably the more popular of the two optimization techniques, SGD held its own in the experiments and showed superior performance when it came to generalization or classifying in noisy conditions.

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)

Abstract. To Electronic health records (EHRs) are a treasure trove of patient information, including clinical notes. However, extracting meaningful insights from these unstructured narratives remains a challenge. This paper explores the potential of natural language processing (NLP) and deep learning to unlock the rich clinical narrative data and revolutionize healthcare. Clinical notes capture a wealth of information beyond structured data points. They document patient history, symptoms, treatment progress, and physician observations. Analyzing this narrative unlocks a deeper understanding of a patient's health journey, aiding in NLP can identify subtle patterns in language usage that might be indicative of specific diseases. This can assist clinicians in arriving at more accurate diagnoses, particularly for complex cases. Deep learning models can analyze narratives to identify patients at higher risk of developing certain conditions. This allows for proactive interventions and preventative measures. Extracting details about a patient's lifestyle, social context, and emotional well-being from narratives can facilitate the development of personalized treatment plans that cater to individual needs. NLP techniques like named entity recognition, sentiment analysis, and topic modeling enable the extraction of key clinical entities, emotions, and themes from narratives. Deep learning, with its ability to learn complex relationships in text data, empowers NLP tasks. Deep learning models can be trained on massive datasets of clinical notes, achieving superior performance in tasks like information extraction compared to traditional NLP methods. Deep learning models can handle the vast amount of text data generated in healthcare settings, enabling large- scale analysis of clinical narratives. By learning from diverse patient populations, deep learning models can improve their generalizability and adapt to variations in language usage within clinical documentation. Integrating NLP and deep learning into clinical workflows promises a future where the patient narrative is not just documented but actively analyzed to inform better healthcare decisions. This can lead to earlier diagnoses, more effective treatments, and ultimately, improved patient outcomes. By addressing the challenges and fostering responsible development, we can unlock the true potential of clinical narratives and empower a new era of data-driven healthcare.

1 - Congruence of deep learning in biomedical engineering: future prospects and challenges

Although the existing traditional image classification methods have been widely applied in practical problems, there are some problems in the application process, such as unsatisfactory effects, low classification accuracy, and weak adaptive ability. This method separates image feature extraction and classification into two steps for classification operation. The deep learning model has a powerful learning ability, which integrates the feature extraction and classification process into a whole to complete the image classification test, which can effectively improve the image classification accuracy. However, this method has the following problems in the application process: first, it is impossible to effectively approximate the complex functions in the deep learning model. Second, the deep learning model comes with a low classifier with low accuracy. So, this paper introduces the idea of sparse representation into the architecture of the deep learning network and comprehensively utilizes the sparse representation of well multidimensional data linear decomposition ability and the deep structural advantages of multilayer nonlinear mapping to complete the complex function approximation in the deep learning model. And a sparse representation classification method based on the optimized kernel function is proposed to replace the classifier in the deep learning model, thereby improving the image classification effect. Therefore, this paper proposes an image classification algorithm based on the stacked sparse coding depth learning model-optimized kernel function nonnegative sparse representation. The experimental results show that the proposed method not only has a higher average accuracy than other mainstream methods but also can be good adapted to various image databases. Compared with other deep learning methods, it can better solve the problems of complex function approximation and poor classifier effect, thus further improving image classification accuracy.

Deep learning is employed in many applications, such as computer vision, natural language processing, robotics, and recommender systems. Large and complex neural networks lead to high accuracy; however, they adversely affect many aspects of deep learning performance, such as training time, latency, throughput, energy consumption, and memory usage in the training and inference stages. To solve these challenges, various optimization techniques and frameworks have been developed for the efficient performance of deep learning models in the training and inference stages. Although optimization techniques such as quantization have been studied thoroughly in the past, less work has been done to study the performance of frameworks that provide quantization techniques. In this paper, we have used different performance metrics to study the performance of various quantization frameworks, including TensorFlow automatic mixed precision and TensorRT. These performance metrics include training time and memory utilization in the training stage along with latency and throughput for graphics processing units (GPUs) in the inference stage. We have applied the automatic mixed precision (AMP) technique during the training stage using the TensorFlow framework, while for inference we have utilized the TensorRT framework for the post-training quantization technique using the TensorFlow TensorRT (TF-TRT) application programming interface (API).We performed model profiling for different deep learning models, datasets, image sizes, and batch sizes for both the training and inference stages, the results of which can help developers and researchers to devise and deploy efficient deep learning models for GPUs.

Deep learning has emerged as a transformative technology in data science, revolutionizing various domains through its powerful capabilities. This paper explores the theoretical foundations, practical applications, and comparative analysis of deep learning models. The theoretical foundations section discusses key neural network architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers, highlighting their unique capabilities in processing different types of data. Optimization algorithms crucial for effective training, including Stochastic Gradient Descent (SGD) and Adam, are examined. Regularization techniques for preventing overfitting and enhancing generalization are also addressed. Practical applications in healthcare, finance, and retail showcase the real-world impact of deep learning. A comparative analysis of performance metrics demonstrates the superiority of deep learning models over traditional methods. Despite their advantages, deep learning models face limitations and challenges, including data dependency and interpretability issues. The paper concludes by emphasizing the ongoing research efforts to mitigate these challenges and ensure the continued advancement of deep learning in data science.

Document classification is a prevalent task in Natural Language Processing (NLP), which has an extensive range of applications in the biomedical domains such as biomedical literature indexing, automatic diagnosis codes assignment, tweets classification for public health topics, and patient safety reports classification. Nevertheless, manual classification of biomedical articles published every year into specific predefined categories becomes a cumbersome task. Hence, building an automatic document classification for biomedical databases emerges as a significant task among the scientific community. In recent years, Deep Learning (DL) models like Deep Neural Networks (DNN), Convolution Neural Networks (CNN), Recurrent Neural Networks (RNN), and Ensemble Deep Learning models are widely used in the area of text document classification for better classification performance compared to Machine Learning (ML) algorithms. The major advantage of using DL models in document classification is that it provides rich semantic and grammatical information for document representation through pre-trained word embedding. Hence, this paper investigates the deployment of the various state-of-the-art DL based classification models in automatic classification of benchmark biomedical datasets. Finally, the performance of all the aforementioned constitutional classifiers is compared and evaluated through the well-defined performance evaluation metrics such as accuracy, precision, recall, and f1-measure.