Finding exact hitting set solutions for systems biology applications using heterogeneous GPU clusters

As the deep learning model grows larger, training model with a single computational resource becomes impractical. To solve this, hybrid parallelism, which combines data and pipeline parallelism emerges to train large models with multiple GPUs. In practice, using heterogeneous GPU clusters to train large models is a common need due to the upgrade of a part of hardware. However, existing hybrid parallelism approaches in the heterogeneous environment do not work well in communication efficacy, workload balance among GPUs and utilizing the memory constrained GPU. To address these problems, we present a parallel DNN training approach, Hybrid Parallelism on Heterogeneous clusters (HPH). In HPH, we propose a topology designer that minimizes the communication time cost. Furthermore, HPH uses a partition algorithm that automatically partitions DNN layers among workers to maximize throughput. Besides, HPH adopts recomputation-aware scheduling to reduce memory consumption and further reschedule the pipeline to eliminate the extra time overhead of recomputation. Our experimental results on a 32-GPU heterogeneous cluster show that HPH achieves up to 1.42x training speed-ups compared with the state-of-the-art approach.

Clusters of GPUs are becoming commonly used to execute computationally demanding applications. Due to the frequent changes in GPU architecture, many clusters contain heterogeneous types of GPUs, leading to the problem of load distribution among the machines. In this work, we propose a load distribution algorithm for scientific applications executed in heterogeneous GPU clusters. The algorithm finds a distribution of data that minimizes the execution time of the application, by guaranteeing that all GPUs spend the same amount of time processing its assigned kernels and data. We use the algorithm to execute the simulation of large scale neuronal networks. We show that the algorithm effectively balances the load among the GPUs and reduces the execution time of the application.

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents HAP, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. HAP jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A*-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, HAP explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that HAP achieves up to 2.41x speed-up on heterogeneous clusters. • Computing methodologies → Distributed computing methodologies.

HASE: Hardware-Aware Scheduling for Inference Tasks in Heterogeneous GPU Clusters

Heterogeneous transfer clustering can translate knowledge from some related heterogeneous source domains to the target domain without any supervision. Existing works usually use a large amount of complete co-occurrence data to learn the projection functions mapping heterogeneous data to a common latent feature subspace. However, in many real applications, it is not practical to collect abundant co-occurrence data, while the available co-occurrence data are always incomplete. Another commonly encountered problem is that the complex structure of real heterogeneous data may result in substantial degeneration in clustering performance. To address these issues, we propose a heterogeneous transfer clustering method specifically designed for partial co-occurrence data (HTCPC). It is superior to the existing methods in three facets. First, HTCPC fully uses the partial co-occurrence data in both source and target domains to learn a latent space, maximally extracting useful knowledge for clustering from limited information. Second, it incorporates multi-layer hidden representations, accurately preserving the complex hierarchical structure of data. Third, it enforces approximately orthogonal constraint in representations, effectively characterizing the latent subspace with minimal redundancy. An efficient algorithm has been derived and implemented to realize the proposed HTCPC. A series of experiments on the real datasets have illustrated the advantage of the proposed approach compared with state-of-the-art methods.

BackgroundSuccessfully modeling high-dimensional data involving thousands of variables is challenging. This is especially true for gene expression profiling experiments, given the large number of genes involved and the small number of samples available. Random Forests (RF) is a popular and widely used approach to feature selection for such "small n, large p problems." However, Random Forests suffers from instability, especially in the presence of noisy and/or unbalanced inputs.ResultsWe present RKNN-FS, an innovative feature selection procedure for "small n, large p problems." RKNN-FS is based on Random KNN (RKNN), a novel generalization of traditional nearest-neighbor modeling. RKNN consists of an ensemble of base k-nearest neighbor models, each constructed from a random subset of the input variables. To rank the importance of the variables, we define a criterion on the RKNN framework, using the notion of support. A two-stage backward model selection method is then developed based on this criterion. Empirical results on microarray data sets with thousands of variables and relatively few samples show that RKNN-FS is an effective feature selection approach for high-dimensional data. RKNN is similar to Random Forests in terms of classification accuracy without feature selection. However, RKNN provides much better classification accuracy than RF when each method incorporates a feature-selection step. Our results show that RKNN is significantly more stable and more robust than Random Forests for feature selection when the input data are noisy and/or unbalanced. Further, RKNN-FS is much faster than the Random Forests feature selection method (RF-FS), especially for large scale problems, involving thousands of variables and multiple classes.ConclusionsGiven the superiority of Random KNN in classification performance when compared with Random Forests, RKNN-FS's simplicity and ease of implementation, and its superiority in speed and stability, we propose RKNN-FS as a faster and more stable alternative to Random Forests in classification problems involving feature selection for high-dimensional datasets.

Physical systems with very large number of variables (say with tens of thousands of variables) may be solved with already available digital computers by tearing the system apart into a large number of small subdivisions, solving each subdivision separately, afterward interconnecting the partial solutions by a set of transformations to obtain outright the exact solution of the original system, Among the many advantages of the tearing (or tensorial) method is the red 3 ction of the amount of original calculations to a small fraction of about 2/n2where n is the number of subdivisions. Another advantage is the reduction of the number of elements in inverse matrices to a fraction smaller than 1/ \sqrt{n} . The same saving of labor appears also in smaller systems employing slide-rule calculations.

While genome-wide gene expression data are generated at an increasing rate, the repertoire of approaches for pattern discovery in these data is still limited. Identifying subtle patterns of interest in large amounts of data (tens of thousands of profiles) associated with a certain level of noise remains a challenge. A microarray time series was recently generated to study the transcriptional program of the mouse segmentation clock, a biological oscillator associated with the periodic formation of the segments of the body axis. A method related to Fourier analysis, the Lomb-Scargle periodogram, was used to detect periodic profiles in the dataset, leading to the identification of a novel set of cyclic genes associated with the segmentation clock. Here, we applied to the same microarray time series dataset four distinct mathematical methods to identify significant patterns in gene expression profiles. These methods are called: Phase consistency, Address reduction, Cyclohedron test and Stable persistence, and are based on different conceptual frameworks that are either hypothesis- or data-driven. Some of the methods, unlike Fourier transforms, are not dependent on the assumption of periodicity of the pattern of interest. Remarkably, these methods identified blindly the expression profiles of known cyclic genes as the most significant patterns in the dataset. Many candidate genes predicted by more than one approach appeared to be true positive cyclic genes and will be of particular interest for future research. In addition, these methods predicted novel candidate cyclic genes that were consistent with previous biological knowledge and experimental validation in mouse embryos. Our results demonstrate the utility of these novel pattern detection strategies, notably for detection of periodic profiles, and suggest that combining several distinct mathematical approaches to analyze microarray datasets is a valuable strategy for identifying genes that exhibit novel, interesting transcriptional patterns.

Gene expression profiles can be extracted from DNA in order to obtain relevant information related to kidney transplant. Successful kidney transplant from donor to patient depends on the fitness of both kidneys, so more and more study should be conducted particularly in kidney transplant classification. The common problem of kidney transplant classification is large amount of genes data from various samples. In this research, we demonstrate L1 Feature Selection Ensemble Classifier based on Data Clustering to select informative genes in order to classify gene expression profiles. After classification on data clustering, ensemble classifier produces 97% overall accuracy with precision, recall, F-Test and Kappa Coefficient reaches 95.7%, 91.3%, 93.5%, 90.3% respectively.

The immune system is highly complex, and its malfunctioning can result in many complex disorders. Understanding its inner workings is crucial to designing optimal immunotherapies, developing new vaccines, or understanding autoimmune diseases, just to name a few. Immune-related diseases present unique challenges due to our limited understanding of the complex molecular and cellular interactions involved, as well as the scarcity of available therapeutic options. Recent years have witnessed the progressive development of high-throughput experimental technologies to probe the immune system. This large amount of data has facilitated the emergence of statistical and machine-learning models focused on unravelling the intricate complexities of the immune system. With this vision in mind, a workshop titled "Computational modelling of immunological mechanisms: From statistical approaches to interpretable machine learning" was organized on Sunday, September 18th, 2022 at the 21st European Conference on Computational Biology (ECCB) in Sitges, Spain. The workshop, led by María Rodríguez Martínez, Anna Niarakis, and Matteo Barberis, explored recent statistical models, high-throughput data analyses, and machine learning models to understand immunological mechanisms. More than 60 participants attended the workshop, comprising students, early-career and senior researchers, as well as professionals from diverse domains including Immunology, Systems Biology, Computational Biology, Computer Science, and Bioinformatics. To conclude the workshop, a round table was organized to foster discussions on the existing challenges and chart a roadmap for the development of the next generation of computational models dedicated to investigating the cellular and molecular functions that underlie the immune system.

Large amounts of data are currently produced via the “tooled up” foundation of “omic” technologies. Proteomics and metabolomics which are typical for post-genomic profiling techniques have been utilized in crop development. The current genomics projects of Musa spp. including banana and plantain are complemented with high-throughput metabolomics and proteomics approaches. While metabolic analyses globally focus on small molecules of plants under a specific condition, proteomics identify and quantify differentially expressed proteins. Data from metabolite and protein profiling are analyzed by bioinformatics and complement the more commonly used transcriptomics approaches. This chapter reviews the current developments in different areas of Musa metabolome and proteome studies. Experimental and analytical tools such as NMR, MS, 2D-gel electrophoresis, GC-MS, MALDI-TOF have been described as important requisites for the generation of Musa metabolome and proteome data. In general, systems biology then forms the expected integration of results from transcriptomics, metabolomics, proteomics and bioinformatics approaches in Musa genomics research.

Today, accelerator cards like GPUs are an important constituent of HPC clusters. For certain GPU-intense applications, the trend is shifting toward multi-GPU systems with four or more GPUs per compute node. This can increase the performance per dollar and the performance per watt. The Linpack benchmark is the standard tool for measuring the compute performance of supercomputers. Its standard implementation, HPL, cannot make use of GPUs on its own and other GPU-enabled versions of HPL show reduced efficiency on multi-GPU systems with four or more GPUs per node. In previous efforts we have developed a GPU-accelerated Linpack implementation in particular for AMD GPUs, which we employed on several clusters of a couple of hundred nodes each. Gustafson's law predicts perfect weak scaling with the number of processor cores, but it is not directly applicable to the number of GPUs in a system due to shared resources. In this paper we develop a model for general relations between the number of GPUs and the minimum problem size required to use them efficiently taking into account the limited PCIe bandwidth, memory bandwidth, and CPU resources. Based on this, we present an approach to scale our Linpack implementation to eight GPUs and achieve good GPU utilization in Linpack on future systems. Finally, we examine how energy efficiency, which has improved significantly with dual-and quad-GPU servers, scales to many-GPU systems.

Systems biology came about as growing numbers of scientists and engineers from different fields created experimental methods and algorithms that supported the analysis of very large quantities of biological data. Systems Biology: A Very Short Introduction outlines the exciting processes and possibilities in this new field. It describes how modern biology enabled us to learn how intricately the expression of every gene is controlled, how signalling systems keep organisms running smoothly, how complicated even the simplest cells are, and how computational methods may help us understand these complex systems of life. It explores what this field is about, why it is needed, and how it will affect our understanding of life, particularly in the areas of personalized medicine, drug development, food and energy production, and sustainable stewardship of our environments.

We present Gandivafair, a distributed, fair share scheduler that balances conflicting goals of efficiency and fairness in GPU clusters for deep learning training (DLT). Gandivafair provides performance isolation between users, enabling multiple users to share a single cluster, thus, maximizing cluster efficiency. Gandivafair is the first scheduler that allocates cluster-wide GPU time fairly among active users. Gandivafair achieves efficiency and fairness despite cluster heterogeneity. Data centers host a mix of GPU generations because of the rapid pace at which newer and faster GPUs are released. As the newer generations face higher demand from users, older GPU generations suffer poor utilization, thus reducing cluster efficiency. Gandivafair profiles the variable marginal utility across various jobs from newer GPUs, and transparently incentivizes users to older GPUs by a novel resource trading mechanism that maximizes cluster efficiency without affecting fairness guarantees of any user. With a prototype implementation and evaluation in a heterogeneous 200-GPU cluster, we show that Gandivafair achieves both fairness and efficiency under realistic multi-user workloads.

The genome is now accessible to the endoscopist