GPU Domain Specialization via Composable On-Package Architecture

As the security issue of cloud-offloaded Deep Learning (DL) inference is drawing increasing attention, running DL inference in Trusted Execution Environments (TEEs) has become a common practice. Latency prediction of TEE-hosted DL model inference is essential for many scenarios, such as DNN model architecture searching with a latency constraint or layer scheduling in model-parallelism inference. However, existing solutions fail to address the memory over-commitment issue in resource-constrained environments inside TEEs.This paper presents Sectum, an accurate latency predictor for DL inference inside TEE enclaves. We first perform a synthetic empirical study to analyze the relationship between inference latency and memory occupation. Sectum predicts inference latency following a two-stage design based on some critical observations. First, Sectum uses a Graph Neural Network (GNN)-based model to detect whether a given model would trigger memory over-commitment in TEEs. Then, combining operator-level latency modeling with linear regression, Sectum could predict the latency of a model. To evaluate Sectum, we design a large dataset that contains the latency information of over 6k CNN models. Our experiments demonstrate that Sectum could achieve over 85% ±10% accuracy of latency prediction. To our knowledge, Sectum is the first method to predict TEE-hosted DL inference latency accurately.

In this paper, we propose a simulated annealing method for timeliness and energy aware deep learning (DL) job assignment. In the proposed method, we make a decision of server position for DL training jobs in order to consider both the timeliness of deep neural network (DNN) model updating and the associated energy consumption. For timeliness management, we design three penalty functions; step, linear, and exponential functions, so as to practically penalize the service quality degradation of DL inference due to delay of DL training. For energy management, we formulate the operation cost optimization problem considering energy consumption for DL training. By using these management schemes, our method is able to find the trade-off of the timeliness and energy consumption of DL training. Especially, in order to overcome the non-smoothness of the defined problem, we design a simulated annealing (SA) based metaheuristic which finds approximated optimal solution for DL training job assignment. For the performance evaluation, we show the preliminary experimental results of DL training jobs with AlexNet, Inception-V3, and ResNet on NVIDIA GPU devices.

In recent years, deep reinforcement learning has combined the advantages of reinforcement learning and deep learning, and has made great progress in decision-making tasks. However, the training of deep reinforcement learning requires frequent interactions between the agent and the environment and repeated experiments. Adversaries have chances to poison the sample data collected by the agent by attacking the experimental environment in the training process, thereby bringing security risks and serious consequences to the training process of reinforcement learning. This work is committed to addressing the security risks in the field of deep reinforcement learning. However, this work improves the algorithm from the perspective of sample data filtering, and improves the security performance of deep reinforcement learning algorithm. There are two contributions in this work: one is to defend against adversarial attacks against deep reinforcement learning through cluster analysis and sample value evaluation; the other is to propose a deep reinforcement learning algorithm based on sample value evaluation on the basis of deterministic strategy gradient algorithm. The algorithm uses the clustering method to classify the sample pool, and measures the contribution value and security risk of the sample to the model training through the sample value evaluation. The classic game experiments show that the proposed algorithm is safe and effective. It reduces the threat of the agent falling into the adversarial sample attack and improves the training performance of deep reinforcement learning.

This study aims to evaluate the implementation of the Deep Learning (Pembelajaran Mendalam/PM) training program for teachers at SMPN 2 Praya Tengah using the Stake evaluation model, which consists of antecedents, transactions, and outcomes. The study employed a qualitative approach, with data collected through interviews, observations, and documentation. The research subjects included five teachers participating in the training, the school principal, and the training instructor. The findings indicate that in the antecedents aspect, the Deep Learning training was conducted based on teachers’ real needs to improve the quality of more meaningful and student-centered learning. Institutional support was generally adequate, although limitations were found in facilities and funding. In the transactions aspect, the training process was participatory and collaborative, utilizing various methods such as discussions, simulations, and hands-on practice that encouraged active teacher engagement, despite challenges related to time management and depth of material coverage. In the outcomes aspect, the training had a positive impact on improving teachers’ understanding, self-confidence, and skills in designing and implementing deep learning in the classroom; however, implementation still faced constraints related to limited time and students’ readiness. Based on these results, it can be concluded that the Deep Learning (PM) training for teachers at SMPN 2 Praya Tengah is effective and relevant to school needs, yet it requires continuous support and follow-up mentoring to ensure its impact contributes optimally to improving the quality of learning.

There is an increasing emphasis on securing deep learning (DL) inference pipelines for mobile and IoT applications with privacy-sensitive data. Prior works have shown that privacy-sensitive data can be secured throughout deep learning inferences on cloud-offloaded models through trusted execution environments such as Intel SGX. However, prior solutions do not address the fundamental challenges of securing the resource-intensive inference tasks on low-power, low-memory devices (e.g., mobile and IoT devices), while achieving high performance. To tackle these challenges, we propose SecDeep, a low-power DL inference framework demonstrating that both security and performance of deep learning inference on edge devices are well within our reach. Leveraging TEEs with limited resources, SecDeep guarantees full confidentiality for input and intermediate data, as well as the integrity of the deep learning model and framework. By enabling and securing neural accelerators, SecDeep is the first of its kind to provide trusted and performant DL model inferencing on IoT and mobile devices. We implement and validate SecDeep by interfacing the ARM NN DL framework with ARM TrustZone. Our evaluation shows that we can securely run inference tasks with 16× to 172× faster performance than no acceleration approaches by leveraging edge-available accelerators.

Recently, deep learning has been exploited in the field of medical image analysis. However, the training of deep learning models with medical images is time-consuming since most medical image data are three-dimensional volumes or high-resolution two-dimensional images. Moreover, the optimization of numerous hyperparameters strongly affects the performance of deep learning. If a framework for training deep learning with hyperparameter optimization on a supercomputer system can be realized, it is expected to accelerate the training of deep learning with medical images. In this study, we described our novel environment for training deep learning with medical images on the supercomputer system in our institute (Reedbush-H supercomputer system) based on asynchronous parallel Bayesian optimization. We trained two types of automated lesion detection application in a constructed environment. The constructed environment enabled us to train deep learning with hyperparameter tuning in a short time.

Deep Learning (DL) techniques have been widely deployed in many application domains. The growth of DL models’ size and complexity demands distributed training of DL models. Since DL training is complex, software implementing distributed DL training is error-prone. Thus, it is crucial to test distributed deep learning software to improve its reliability and quality. To address this issue, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">differential</i> testing technique—D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup>, which leverages a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distributed</i> equivalence rule that we create to test distributed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deep</i> learning software. The rationale is that the same model trained with the same model input under different distributed settings should produce equivalent prediction output within certain thresholds. The different output indicates potential bugs in the distributed deep learning software. D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> automatically generates a diverse set of distributed settings, DL models, and model input to test distributed deep learning software. Our evaluation on two of the most popular DL libraries, i.e., PyTorch and TensorFlow, shows that D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> detects 21 bugs, including 12 previously unknown bugs.

Deep Learning (DL) has the potential to optimize machine learning in both the scientific and clinical communities. However, greater expertise is required to develop DL algorithms, and the variability of implementations hinders their reproducibility, translation, and deployment. Here we present the community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the goal of lowering these barriers. GaNDLF makes the mechanism of DL development, training, and inference more stable, reproducible, interpretable, and scalable, without requiring an extensive technical background. GaNDLF aims to provide an end-to-end solution for all DL-related tasks in computational precision medicine. We demonstrate the ability of GaNDLF to analyze both radiology and histology images, with built-in support for k-fold cross-validation, data augmentation, multiple modalities and output classes. Our quantitative performance evaluation on numerous use cases, anatomies, and computational tasks supports GaNDLF as a robust application framework for deployment in clinical workflows.

Efforts to leverage the benefits of Deep Learning(DL) models for performing inference in resource constrained embedded devices is very popular nowadays. Researchers worldwide are trying to come up with software and hardware accelerators that make pre-trained DL models suitable for running the inference phase in these devices. Apart from software and hardware accelerators, DL model partitioning and offloading to Cloud or Edge network servers is becoming more and more practicable with increasing importance of Edge Computing. DL inference workflow partitioning and offloading can augment software / hardware acceleration for improved latency and energy efficiency in resource constrained embedded systems. Efficacy of a computation offloading to system is dependent on proper profiling of timing and energy required for processing DL algorithms. In this work we implement a DL inference offloading system using Raspberry Pi 3 based robot vehicle with a hardware accelerator from Intel (Neural Compute Stick). We report the workload partitioning approach, detailed experimental results and performance improvements achieved. We demonstrate that the current approach of DL execution profiling without considering dynamic system load of the edge device results in sub-optimal partitioning of DL algorithm and provide a solution approach to that.

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet the actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification. However, deep learning has the following problems in the application of medical image classification. First, it is impossible to construct a deep learning model with excellent performance according to the characteristics of medical images. Second, the current deep learning network structure and training strategies are less adaptable to medical images. Therefore, this paper first introduces the visual attention mechanism into the deep learning model so that the information can be extracted more effectively according to the problem of medical images, and the reasoning is realized at a finer granularity. It can increase the interpretability of the model. Additionally, to solve the problem of matching the deep learning network structure and training strategy to medical images, this paper will construct a novel multiscale convolutional neural network model that can automatically extract high-level discriminative appearance features from the original image, and the loss function uses the Mahalanobis distance optimization model to obtain a better training strategy, which can improve the robust performance of the network model. The medical image classification task is completed by the above method. Based on the above ideas, this paper proposes a medical classification algorithm based on a visual attention mechanism-multiscale convolutional neural network. The lung nodules and breast cancer images were classified by the method in this paper. The experimental results show that the accuracy of medical image classification in this paper is not only higher than that of traditional machine learning methods but also improved compared with other deep learning methods, and the method has good stability and robustness.

Implementing deep learning technology with FPGA as an accelerator has become a popular application due to its efficiency and performance. However, given the tremendous data generated on medical diagnosis, normal inference speed is not sufficient. Hence, the FPGA technology is implemented for fast inference. In this context, the FPGA accelerates the deep learning inference process for fast breast cancer classification with minimal latency on real-time deployment. This paper summarizes the findings of model deployment across various computing devices in deep learning technology with FPGA. The study includes model performance evaluation, throughput, and latency comparison with different batch sizes to the extent of expected delay for real-world deployment. The result concludes that FPGA is the most suitable to act as a deep learning inference accelerator with a high throughput-to-latency ratio and fast parallel inference.

The paradigm of Intelligent DataPlane (IDP) embeds deep learning (DL) models on the network dataplane to enable intelligent traffic analysis at line-speed. However, the current use of the match-action table (MAT) abstraction on the dataplane is misaligned with DL inference, leading to several key limitations, including accuracy degradation, limited scale, and lack of generality. This paper proposes Pegasus to address these limitations. Pegasus translates DL operations into three dataplane-oriented primitives to achieve generality: Partition, Map, and SumReduce. Specifically, Partition "divides" high-dimensional features into multiple low-dimensional vectors, making them more suitable for the dataplane; Map "conquers" computations on the low-dimensional vectors in parallel with the technique of Fuzzy Matching, while SumReduce "combines" the computation results. Additionally, Pegasus employs Primitive Fusion to merge computations, improving scalability. Finally, Pegasus adopts full-precision weights with fixed-point activations to improve accuracy. Our implementation on a P4 switch demonstrates that Pegasus can effectively support various types of DL models, including Multi-Layer Perceptron (MLP), Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and AutoEncoder models on the dataplane. Meanwhile, Pegasus outperforms state-of-the-art approaches with an average accuracy improvement of up to 22.8%, along with up to 248× larger model size and 212× larger input scale.

Homomorphic encryption (HE) is the ultimate tool for performing secure computations even in untrusted environments. Application of HE for deep learning (DL) inference is an active area of research, given the fact that DL models are often deployed in untrusted environments (e.g., third-party servers) yet inferring on private data. However, existing HE libraries [somewhat (SWHE), leveled (LHE) or fully homomorphic (FHE)] suffer from extensive computational and memory overhead. Few performance optimized high-speed homomorphic libraries are either suffering from certain approximation issues leading to decryption errors or proven to be insecure according to recent published attacks. In this article, we propose architectural tricks to achieve performance speedup for encrypted DL inference developed with exact HE schemes without any approximation or decryption error in homomorphic computations. The main idea is to apply quantization and suitable data packing in the form of bitslicing to reduce the costly noise handling operation, Bootstrapping while achieving a functionally correct and highly parallel DL pipeline with a moderate memory footprint. Experimental evaluation on the MNIST dataset shows a significant ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$37\times$ </tex-math></inline-formula> ) speedup over the nonbitsliced versions of the same architecture. Low memory bandwidths (700 MB) of our design pipelines further highlight their promise toward scaling over larger gamut of Edge-AI analytics use cases.

currently artificial intelligence (AI) witnesses a revolution motivated by the power of machine learning and deep neural networks. Many fields have made a breakthrough via applying deep learning algorithms to their domain of applications. This paper introduces an insight into efforts recently presented to exploit the deep learning algorithms in electronic warfare (EW) research. Some of EW problems are reformulated in accordance with the deep learning basic framework, as a guide to facilitate an effective transition toward ingesting deep learning algorithms into EW domain of applications. Along with problems reformulation, we point out the current and previous work related to the different problems, if any. The last section in the paper is devoted to addressing the benchmarking requirements and the challenges that face the transition toward deep learning techniques. This paper isn't intended only for a comprehensive survey besides it introduces an aggregated vision for what is achieved and what is possible to achieve by applying deep learning in EW applications.

Tasks across diverse application domains can be posed as large-scale optimization problems, these include graphics, vision, machine learning, imaging, health, scheduling, planning, and energy system forecasting. Independently of the application domain, proximal algorithms have emerged as a formal optimization method that successfully solves a wide array of existing problems, often exploiting problem-specific structures in the optimization. Although model-based formal optimization provides a principled approach to problem modeling with convergence guarantees, at first glance, this seems to be at odds with black-box deep learning methods. A recent line of work shows that, when combined with learning-based ingredients, model-based optimization methods are effective, interpretable, and allow for generalization to a wide spectrum of applications with little or no extra training data. However, experimenting with such hybrid approaches for different tasks by hand requires domain expertise in both proximal optimization and deep learning, which is often error-prone and time-consuming. Moreover, naively unrolling these iterative methods produces lengthy compute graphs, which when differentiated via autograd techniques results in exploding memory consumption, making batch-based training challenging. In this work, we introduce ∇-Prox, a domain-specific modeling language and compiler for large-scale optimization problems using differentiable proximal algorithms. ∇-Prox allows users to specify optimization objective functions of unknowns concisely at a high level, and intelligently compiles the problem into compute and memory-efficient differentiable solvers. One of the core features of ∇-Prox is its full differentiability, which supports hybrid model- and learning-based solvers integrating proximal optimization with neural network pipelines. Example applications of this methodology include learning-based priors and/or sample-dependent inner-loop optimization schedulers, learned with deep equilibrium learning or deep reinforcement learning. With a few lines of code, we show ∇-Prox can generate performant solvers for a range of image optimization problems, including end-to-end computational optics, image deraining, and compressive magnetic resonance imaging. We also demonstrate ∇-Prox can be used in a completely orthogonal application domain of energy system planning, an essential task in the energy crisis and the clean energy transition, where it outperforms state-of-the-art CVXPY and commercial Gurobi solvers.