Accelerating Tensor Swapping in GPUs With Self-Tuning Compression

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Distant speech recognition (DSR) remains to be an open challenge, even for the state-of-the-art deep neural network (DNN) models. Previous work has attempted to improve DNNs under constantly distant speech. However, in real applications, the speaker-microphone distance (SMD) can be quite dynamic, varying even within a single utterance. This paper investigates how to alleviate the impact of dynamic SMD on DNN models. Our solution is to incorporate the frame-level SMD information into DNN training. Generation of the SMD information relies on a universal extractor that is learned on a meeting corpus. We study the utility of different architectures in instantiating the SMD extractor. On our target acoustic modeling task, two approaches are proposed to build distance-aware DNN models using the SMD information: simple concatenation and distance adaptive training (DAT). Our experiments show that in the simplest case, incorporating the SMD descriptors improves word error rates of DNNs by 5.6% relative. Further optimizing SMD extraction and integration results in more gains.

Optimizing makespan and resource utilization for multi-DNN training in GPU cluster

COVID-19 is most commonly diagnosed using a testing kit but chest X-rays and computed tomography (CT) scan images have a potential role in COVID-19 diagnosis. Currently, CT diagnosis systems based on Artificial intelligence (AI) models have been used in some countries. Previous research studies used complex neural networks, which led to difficulty in network training and high computation rates. Hence, in this study, we developed the 6-layer Deep Neural Network (DNN) model for COVID-19 diagnosis based on CT scan images. The proposed DNN model is generated to improve accurate diagnostics for classifying sick and healthy persons. Also, other classification models, such as decision trees, random forests and standard neural networks, have been investigated. One of the main contributions of this study is the use of the global feature extractor operator for feature extraction from the images. Furthermore, the 10-fold cross-validation technique is utilized for partitioning the data into training, testing and validation. During the DNN training, the model is generated without dropping out of neurons in the layers. The experimental results of the lightweight DNN model demonstrated that this model has the best accuracy of 96.71% compared to the previous classification models for COVID-19 diagnosis.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Deep neural network modeling of unknown partial differential equations in nodal space

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

Background and Objectives: This study investigated the usefulness of deep neural network (DNN) models based on 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) and blood inflammatory markers to assess the therapeutic response in pyogenic vertebral osteomyelitis (PVO). Materials and Methods: This was a retrospective study with prospectively collected data. Seventy-four patients diagnosed with PVO underwent clinical assessment for therapeutic responses based on clinical features during antibiotic therapy. The decisions of the clinical assessment were confirmed as ‘Cured’ or ‘Non-cured’. FDG-PETs were conducted concomitantly regardless of the decision at each clinical assessment. We developed DNN models depending on the use of attributes, including C-reactive protein (CRP), erythrocyte sedimentation ratio (ESR), and maximum standardized FDG uptake values of PVO lesions (SUVmax), and we compared their performances to predict PVO remission. Results: The 126 decisions (80 ‘Cured’ and 46 ‘Non-cured’ patients) were randomly assigned with training and test sets (7:3). We trained DNN models using a training set and evaluated their performances for a test set. DNN model 1 had an accuracy of 76.3% and an area under the receiver operating characteristic curve (AUC) of 0.768 [95% confidence interval, 0.625–0.910] using CRP and ESR, and these values were 79% and 0.804 [0.674–0.933] for DNN model 2 using ESR and SUVmax, 86.8% and 0.851 [0.726–0.976] for DNN model 3 using CRP and SUVmax, and 89.5% and 0.902 [0.804–0.999] for DNN model 4 using ESR, CRP, and SUVmax, respectively. Conclusions: The DNN models using SUVmax showed better performances when predicting the remission of PVO compared to CRP and ESR. The best performance was obtained in the DNN model using all attributes, including CRP, ESR, and SUVmax, which may be helpful for predicting the accurate remission of PVO.

Recently deep-learning (DL) techniques have been widely adopted in side-channel power analysis. A DL-assisted SCA generally consists of two phases: a deep neural network (DNN) training phase and a follow-on attack phase using the trained DNN. However, currently the two phases are not well aligned, as there is no conclusion on what metric used in the training can result in the most effective attack in the second phase. When traditional loss functions such as negative log-likelihood (NLL) are used in training a DNN, the trained model does not yield optimal follow-on attack. Recently some information theoretical SCA leakage metrics are proposed, either as the validation metric to stop the DNN training with traditional loss functions, or as both the validation metric and the training loss function. None of those proposed metrics, however, directly measures the SCA effectiveness. We propose to conduct DNN training directly with a common SCA effectiveness metric, Guessing Entropy (GE). We overcome the prior practical difficulty of using GE in DNN training by utilizing the GEEA estimation algorithm introduced in CHES 2020. We show that using GEEA as either the validation metric or the loss function produces DNN models that lead to much more effective follow-on attacks. Our work consolidates the DL-assisted SCA framework with a consistent metric, which shows great potential to be adopted as the universal SCA-oriented DNN training framework.

Deep Neural Network (DNN) models have achieved excellent performance in many inference tasks and have been widely used in many intelligent applications. However, DNN models often require a lot of computational resources to complete the inference tasks, which hinders the deployment of such models to resource‐constrained edge devices. In order to extend the application scenarios of DNN models, the edge‐cloud collaborative inference methods, represented by model partition, have attracted much research attention in recent years. In scenarios that have multiple edge devices deployed, the edge‐cloud collaborative inference method requires partial migration of tasks, but traditional scheduling methods only migrate tasks at the task level. In this paper, we propose two task scheduling methods, which can solve the problem of partial migration of tasks in multiedge scenarios. The first scheduling method is based on the optimal cutting of a single DNN. The cutting positions of all the models are the same, regardless of the influence of external factors. This method is suitable for chain and directed acyclic graph‐ (DAG‐) type DNNs. The second scheduling method takes external factors such as congestion and queuing delay at the cloud side into consideration, which dynamically selects the cutting position of each DNN to optimize the overall delay and thus is applicable to chain DNN models. The experimental results show that, compared with the baseline method, our proposed scheduling method can reduce the delay by up to 6.48x.

Decision of the model complexity is a significant challenge in contemporary data-driven modeling applications. Designing neural architecture involves the process of determining the optimal model complexity for deep neural networks (DNNs) models in order to uncover relationships in real-world data patterns. Consequently, optimizing DNN architectures is crucial for enhancing the practical approximation performance of DNN models to real-world data. This study implements a data-driven neuroevolution scheme for the optimal neural architecture search (NAS) and demonstrates a data-driven engineering application for cooling load estimation. The proposed neuroevolution scheme aims at evolving to the best generalizing DNN model that well suits the modeling complexity requirements of the dataset. To this end, the objective function for the neural architecture optimization process is simplified to the mean square error of the test dataset, which enables to reduce the risk of insufficient generalization during NAS. By employing this objective function for evolution field optimization (EFO), the proposed neuroevolution process can automatically achieve the optimal model complexity, preventing overfitting and underfitting cases, and thereby attaining almost the best generalization for the dataset. For this purpose, this approach combines parametric learning with the backpropagation algorithm and structural learning with EFO-based neural architecture search to address data-driven, optimal complexity DNN model generation problems. Effectiveness of the method is demonstrated in the cooling load estimation problem of residential buildings, and performances of the optimal DNN models with four objective functions are analyzed. The design of objective function for the best generalizing model is also elaborated.

In this article, for the first time, we propose channel characteristic-based deep neural network (DNN) models for accurate eye-height (EH) and eye-width (EW) estimation of high bandwidth memory (HBM) silicon interposer channels. The proposed models preprocess the design parameters that are highly relevant to the characteristics of the HBM channels. By taking account of the contribution of each design parameter to the eye diagram, the proposed models can accurately estimate the EH and EW of the channels even with a limited number of datasets. For verification, the proposed DNN models were applied to the microstrip and stripline channels of an HBM silicon interposer. Only redistributed layer (RDL) was used to clearly see the effect of reflecting the channel characteristics of the proposed method. We compared the proposed DNN models with various regression methods and a conventional fully connected multilayer DNN model. As a result, the proposed DNN models reduced the EH and EW error rates by 22.7 and 43.9% compared to the other regression methods. In addition, the proposed DNN models not only reduced the error rates by 22.0–28.4% but also reduced the computing cost by 8.0–9.4%, compared to the conventional DNN model. Moreover, we compared the proposed models with various DNN models having other preprocessing structures. By showing 26.7 and 28.8% lower EH and EW error rates than the other DNN models, we validated that the proposed models properly consider the most dominant design factors in preprocessing.

Distributed processing using high-performance computing resources is essential for developers to train large-scale deep neural networks (DNNs). The major impediment to distributed DNN training is the communication bottleneck during the parameter exchange among the distributed DNN training workers. The communication bottleneck increases training time and decreases the utilization of the computational resources. Our previous study, SoftMemoryBox (SMB1) presented considerably superior performance compared to message passing interface (MPI) in the parameter communication of distributed DNN training. However, SMB1 had disadvantages such as the limited scalability of the distributed DNN training due to the restricted communication bandwidth from a single memory server, inability to provide a synchronization function for the shared memory buffer, and low portability/usability as a consequence of the kernel-level implementation. This paper proposes a scalable, shared memory buffer framework, called SoftMemoryBox II (SMB2), which overcomes the shortcomings of SMB1. With SMB2, distributed training processes can easily share virtually unified shared memory buffers composed of memory segments provided from remote memory servers and can exchange DNN parameters at high speed through the shared memory buffer. The scalable communication bandwidth of the SMB2 framework facilitates the reduction of DNN distributed training times compared to SMB1. According to intensive evaluation results, the communication bandwidth of the proposed SMB2 is 6.3 times greater than that of SMB1 when the SMB2 framework is scaled out to use eight memory servers. Moreover, the training time of SMB2-based asynchronous distributed training of five DNN models is up to 2.4 times faster than SMB1-based training.

Industrial software technology: R Mitchell (ed) Peter Peregrinus Ltd (on behalf of the Institution of Electrical Engineers), London, UK (1987) 289pp £37.00

Artificial Intelligence (AI) in healthcare enables predicting diabetes using data-driven methods instead of the traditional ways of screening the disease, which include hemoglobin A1c (HbA1c), oral glucose tolerance test (OGTT), and fasting plasma glucose (FPG) screening techniques, which are invasive and limited in scale. Machine learning (ML) and deep neural network (DNN) models that use large datasets to learn the complex, nonlinear feature interactions, but the conventional ML algorithms are data sensitive and often show unstable predictive accuracy. Conversely, DNN models are more robust, though the ability to reach a high accuracy rate consistently on heterogeneous datasets is still an open challenge. For predicting diabetes, this work proposed a hybrid DNN approach by integrating a bidirectional long short-term memory (BiLSTM) network with a bidirectional gated recurrent unit (BiGRU). A robust DL model, developed by combining various datasets with weighted coefficients, dense operations in the connection of deep layers, and the output aggregation using batch normalization and dropout functions to avoid overfitting. The goal of this hybrid model is better generalization and consistency among various datasets, which facilitates the effective management and early intervention. The proposed DNN model exhibits an excellent predictive performance as compared to the state-of-the-art and baseline ML and DNN models for diabetes prediction tasks. The robust performance indicates the possible usefulness of DL-based models in the development of disease prediction in healthcare and other areas that demand high-quality analytics.