XLA-NDP: Efficient Scheduling and Code Generation for Deep Learning Model Training on Near-Data Processing Memory

This paper presents a Data Science-oriented application for image classification tasks that is able to automatically: a) gather images needed for training Deep Learning (DL) models with a built-in search engine crawler; b) remove duplicate images; c) sort images using built-in pre-trained DL models or user’s own trained DL model; d) apply data augmentation; e) train a DL classification model; f) evaluate the performance of a DL model and system by using an accuracy calculator as well as the Accuracy Per Consumption (APC), Accuracy Per Energy Cost (APEC), Time to closest APC (TTCAPC) and Time to closest APEC (TTCAPEC) metrics calculators. Experimental results show that the proposed Computer Vision application has several unique features and advantages, proving to be efficient regarding execution time and much easier to use when compared to similar applications.

Recently, deep learning has been exploited in the field of medical image analysis. However, the training of deep learning models with medical images is time-consuming since most medical image data are three-dimensional volumes or high-resolution two-dimensional images. Moreover, the optimization of numerous hyperparameters strongly affects the performance of deep learning. If a framework for training deep learning with hyperparameter optimization on a supercomputer system can be realized, it is expected to accelerate the training of deep learning with medical images. In this study, we described our novel environment for training deep learning with medical images on the supercomputer system in our institute (Reedbush-H supercomputer system) based on asynchronous parallel Bayesian optimization. We trained two types of automated lesion detection application in a constructed environment. The constructed environment enabled us to train deep learning with hyperparameter tuning in a short time.

Artificial intelligence (AI) has the ability to predict rheological properties and constituent composition of 3D-printed materials with appropriately trained models. However, these models are not currently available for use. In this work, we trained deep learning (DL) models to (1) predict the rheological properties, such as the storage (G’) and loss (G”) moduli, of 3D-printed polyacrylamide (PAA) substrates, and (2) predict the composition of materials and associated 3D printing parameters for a desired pair of G’ and G”. We employed a multilayer perceptron (MLP) and successfully predicted G’ and G” from seven gel constituent parameters in a multivariate regression process. We used a grid-search algorithm along with 10-fold cross validation to tune the hyperparameters of the MLP, and found the R2 value to be 0.89. Next, we adopted two generative DL models named variational autoencoder (VAE) and conditional variational autoencoder (CVAE) to learn data patterns and generate constituent compositions. With these generative models, we produced synthetic data with the same statistical distribution as the real data of actual hydrogel fabrication, which was then validated using Student’s t-test and an autoencoder (AE) anomaly detector. We found that none of the seven generated gel constituents were significantly different from the real data. Our trained DL models were successful in mapping the input–output relationship for the 3D-printed hydrogel substrates, which can predict multiple variables from a handful of input variables and vice versa.

Robust perception systems allow farm robots to recognize weeds and vegetation, enabling the selective application of fertilizers and herbicides to mitigate the environmental impact of traditional agricultural practices. Today's perception systems typically rely on deep learning to interpret sensor data for tasks such as distinguishing soil, crops, and weeds. These approaches usually require substantial amounts of manually labeled training data, which is often time-consuming and requires domain expertise. This paper aims to reduce this limitation and propose an automated labeling pipeline for crop-weed semantic image segmentation in managed agricultural fields. It allows the training of deep learning models without or with only limited manual labeling of images. Our system uses RGB images recorded with unmanned aerial or ground robots operating in the field to produce semantic labels exploiting the field row structure for spatially consistent labeling. We use the rows previously detected to identify multiple crop rows, reducing labeling errors and improving consistency. We further reduce labeling errors by assigning an "unknown" class to challenging-to-segment vegetation. We use evidential deep learning because it provides predictions uncertainty estimates that we use to refine and improve our predictions. In this way, the evidential deep learning assigns high uncertainty to the weed class, as it is often less represented in the training data, allowing us to use the uncertainty to correct the semantic predictions. Experimental results suggest that our approach outperforms general-purpose labeling methods applied to crop fields by a large margin and domain-specific approaches on multiple fields and crop species. Using our generated labels to train deep learning models boosts our prediction performance on previously unseen fields with respect to unseen crop species, growth stages, or different lighting conditions. We obtain an IoU of 88.6% on crops, and 22.7% on weeds for a managed field of sugarbeets, where fully supervised methods have 83.4% on crops and 33.5% on weeds and other unsupervised domain-specific methods get 54.6% on crops and 11.2% on weeds. Finally, our method allows fine-tuning models trained in a fully supervised fashion to improve their performance in unseen field conditions up to +17.6% in mean IoU without additional manual labeling.

Deep learning has rapidly advanced in both performance and development, but the training of deep learning models is often time-consuming and resource intensive, and needing adjustment on a large number of parameters. This research looks at whether or not Bio inspired optimization (GA and PSO), can help deepen deep learning itself, so to speak. By providing these nature inspired methods, the magic of automated model setting, these methods can help you to automatically find the best model setting for you to speed up training, increase accuracy, and reduce overfitting. Through employing GA and PSO on tasks such as hyperparameter tuning and optimization of models are deep learning models made more adaptive, efficient and robust. The techniques are then compared with traditional optimization methods and their use in building better deep learning models with less manual effort is demonstrated.

Transient stability assessment (TSA) based on deep learning has recently attracted broad attention. The offline training of deep learning model typically requires a large number of labeled samples, and the training process is very time-consuming. Moreover, it is difficult to ensure that the trained model is suitable for all the operation conditions of power systems. In order to dramatically reduce the computation cost, and further enhance the adaptability of the model, an innovative method is proposed based on the active transfer learning (TL). First, the deep belief network (DBN) is designed as the base model and trained with a large number of labeled samples to obtain better evaluation performance for TSA. Second, when the topologies or operation conditions change substantially, active learning with information entropy is exploited to select the minimum number of the most informative samples, which effectively reduces the time of samples generation. Third, Maximum Mean Discrepancy (MMD) is calculated to select different transfer schemes. Only the last BP layer needs to be fine-tuned for the scenarios with a low MMD, but for those with high MMD, the entire network needs to be fine-tuned. The transfer time is greatly shortened on the premise of ensuring the transfer effect. The comprehensive experimental results of three power systems illustrate the effectiveness of the proposed method. Note to Practitioners—This research is motivated by automatic adaptive assessment of power system transient stability. The main objective is to realize online fast and continuous prediction with limited samples. This kind of demand is very normal in the industrial systems where it is expensive, time-consuming and difficult to obtain labeled samples. This paper proposes an active transfer learning method to select critical samples and update the deep learning model, which greatly reduces the time of samples generation and improves the updating efficiency of model. The proposed method is also applicable to make classification prediction or to address regression problems of industrial scenarios based on deep learning models.

With the development of high-throughput technologies, genomics datasets rapidly grow in size, including functional genomics data. This has allowed the training of large Deep Learning (DL) models to predict epigenetic readouts, such as protein binding or histone modifications, from genome sequences. However, large dataset sizes come at a price of data consistency, often aggregating results from a large number of studies, conducted under varying experimental conditions. While data from large-scale consortia are useful as they allow studying the effects of different biological conditions, they can also contain unwanted biases from confounding experimental factors. Here, we introduce Metadata-guided Feature Disentanglement (MFD)-an approach that allows disentangling biologically relevant features from potential technical biases. MFD incorporates target metadata into model training, by conditioning weights of the model output layer on different experimental factors. It then separates the factors into disjoint groups and enforces independence of the corresponding feature subspaces with an adversarially learned penalty. We show that the metadata-driven disentanglement approach allows for better model introspection, by connecting latent features to experimental factors, without compromising, or even improving performance in downstream tasks, such as enhancer prediction, or genetic variant discovery. The code will be made available at https://github.com/HealthML/MFD.

Curious Containers: A framework for computational reproducibility in life sciences with support for Deep Learning applications

In the case of neonates, especially low birth weight preterm and high-risk infants, portable X-rays are frequently used. However, the image quality of portable X-rays is significantly lower compared to standard adult or pediatric X-rays, leading to considerable challenges in identifying abnormalities. Although attempts have been made to introduce deep learning to address these image quality issues, the poor quality of the images themselves hinders the training of deep learning models, further emphasizing the need for image enhancement. Additionally, since neonates have a high cell division rate and are highly sensitive to radiation, increasing radiation exposure to improve image quality is not a viable solution. Therefore, it is crucial to enhance image quality through preprocessing before training deep learning models. While various image enhancement methods have been proposed, Contrast Limited Adaptive Histogram Equalization (CLAHE) has been recognized as an effective technique for contrast-based image improvement. However, despite extensive research, the process of setting CLAHE’s hyperparameters still relies on a brute force, manual approach, making it inefficient. To address this issue, we propose a method called Bayesian Optimization CLAHE(BO-CLAHE), which leverages Bayesian optimization to automatically select the optimal hyperparameters for X-ray images used in diagnosing lung diseases in preterm and high-risk neonates. The images enhanced by BO-CLAHE demonstrated superior performance across several classification models, with particularly notable improvements in diagnosing Transient Tachypnea of the Newborn (TTN). This approach not only reduces radiation exposure but also contributes to the development of AI-based diagnostic tools, playing a crucial role in the early diagnosis and treatment of preterm and high-risk neonates.

In this work we propose a workflow to deal with overlaid images—images with superimposed text and company logos—, which is very common in underwater monitoring videos and surveillance camera footage. It is demonstrated that it is possible to use Explaining Artificial Intelligence to improve deep learning models performance for image classification tasks in general. A deep learning model trained to classify metal surface defect, which previously had a low performance, is then evaluated with Layer-wise relevance propagation—an Explaining Artificial Intelligence technique—to identify problems in a dataset that hinder the training of deep learning models in a wide range of applications. Thereafter, it is possible to remove this unwanted information from the dataset—using different approaches: from cutting part of the images to training a Generative Inpainting neural network model—and retrain the model with the new preprocessed images. This proposed methodology improved F1 score in 20% when compared to the original trained dataset, validating the proposed workflow.

A novel deep metric learning model for imbalanced fault diagnosis and toward open-set classification

Deep learning has proven to be a powerful approach to solving challenging problems, ranging from natural language processing, speech recognition, to computer vision. The proposed model has capable of matching the expanding amount of facts as well as complexity of the algorithms is demands of deep researching methodologies. These demands cannot be met with traditional computing environments. This is why cloud computing technologies have developed, offering a scalable and affordable alternative for executing deep learning algorithms. Cloud computing platforms provide the resources necessary to run deep learning workloads including compute, storage, and networking. This means you no longer have to spend lots of money on expensive hardware and makes it easier for teams and researchers to deploy and train deep learning models faster. Cloud computing has ushered in a new era of deep learning developments by mobilizing the power of specialized hardware, notably GPUs, to accelerate the training and performance of deep learning models.

The coverage of the dataset is one of the important factors for training deep learning models. This coverage measures how well the dataset covers the cases, which can be defined by metrics, where deep learning models should learn. Hence, the lower coverage has negative impacts on the training of deep learning models. In this paper, we introduce a concept to achieve high coverage in terms of multiple metrics and achieve a better coverage of the dataset. In particular, we propose data assets inspired by software assets of software product line engineering. A data asset indicates a subset of a dataset extracted by various metrics. Optimizing the proportion of such data assets in the dataset improves the coverage in terms of the corresponding metrics. In other words, we achieve the better coverage of the dataset based on data assets. To propose this approach, we describe our future research plan.

Deep Learning (DL) has seen rapid adoption in all domains. Since training DL models is expensive, both in terms of time and resources, application workflows that make use of DL increasingly need to operate with a large number of derived learning models, which are obtained through transfer learning and fine-tuning. At scale, thousands of such derived DL models are accessed concurrently by a large number of processes. In this context, an important question is how to design and develop specialized DL model repositories that remain scalable under concurrent access, while addressing key challenges: how to query the DL model architectures for specific patterns? How to load/store a subset of layers/tensors from a DL model? How to efficiently share unmodified layers/tensors between DL models derived from each other through transfer learning? How to maintain provenance and answer ancestry queries? State of art leaves a gap regarding these challenges. To fill this gap, we introduce EvoStore, a distributed DL model repository with scalable data and metadata support to store and access derived DL models efficiently. Large-scale experiments on hundreds of GPUs show significant benefits over state-of-art with respect to I/O and metadata performance, as well as storage space utilization.

GPUs have been favored for training deep learning models due to their highly parallelized architecture. As a result, most studies on training optimization focus on GPUs. There is often a trade-off, however, between cost and efficiency when deciding on how to choose the proper hardware for training. In particular, CPU servers can be beneficial if training on CPUs was more efficient, as they incur fewer hardware update costs and better utilizing existing infrastructure. This paper makes several contributions to research on training deep learning models using CPUs. First, it presents a method for optimizing the training of deep learning models on Intel CPUs and a toolkit called ProfileDNN, which we developed to improve performance profiling. Second, we describe a generic training optimization method that guides our workflow and explores several case studies where we identified performance issues and then optimized the Intel Extension for PyTorch, resulting in an overall 2x training performance increase for the RetinaNet-ResNext50 model. Third, we show how to leverage the visualization capabilities of ProfileDNN, which enabled us to pinpoint bottlenecks and create a custom focal loss kernel that was two times faster than the official reference PyTorch implementation.