Multi-GPU System Design with Memory Networks

We present MGPU, a C++ programming library targeted at single-node multi-GPU systems. Such systems combine disproportionate floating point performance with high data locality and are thus well suited to implement real-time algorithms. We describe the library design, programming interface and implementation details in light of this specific problem domain. The core concepts of this work are a novel kind of container abstraction and MPI-like communication methods for intra-system communication. We further demonstrate how MGPU is used as a framework for porting existing GPU libraries to multi-device architectures. Putting our library to the test, we accelerate an iterative non-linear image reconstruction algorithm for real-time magnetic resonance imaging using multiple GPUs. We achieve a speed-up of about 1.7 using 2 GPUs and reach a final speed-up of 2.1 with 4 GPUs. These promising results lead us to conclude that multi-GPU systems are a viable solution for real-time MRI reconstruction as well as signal-processing applications in general.

This paper analyzes the optimization features of machine learning (ML) model training procedures using multi-GPU systems to enhance cyber security in telecommunication networks. A key aspect of the study is the use of data parallelism, which allows the distribution of the training load across multiple GPUs, significantly reducing training time and improving model accuracy-critical factors for rapid threat detection in cyberspace. A novel approach for optimizing data batch size using Mutual Information (MI) is proposed, which harmonizes the utilization of computational resources with the information content of the training data. MI helps to determine the optimal data batch size that minimizes training errors and improves model accuracy without a significant increase in training time. Experimental results demonstrate the substantial advantages of multi-GPU configurations compared to single-GPU setups, providing faster training and improved model accuracy. It was particularly emphasized that MI-guided batch size tuning significantly outperforms traditional manual tuning methods, ensuring higher validation accuracy and reducing training time. The study showed that the MI-based approach is an effective tool for addressing the problem of optimizing ML model training processes in real-world scenarios where cyber security is critical. The proposed methods allow ML models to train faster and more accurately identify potential threats, making them particularly relevant for telecommunication networks where a rapid response to new threats in real time is required. The implementation of modern computational technologies such as multi-GPU systems and MI-optimized training enhances the efficiency and accuracy of machine learning models. This, in turn, improves cyber security measures and ensures a more reliable defence of telecommunication networks against malicious attacks. It is noted that the proposed approaches can be adapted not only for cyber security but also for other areas where high model accuracy and fast training are important. Future research prospects include the development of new machine learning methods, particularly deep neural networks, the exploration of alternative computational architectures such as quantum computing or distributed systems, and their integration into real-time systems. Special attention should be paid to the ethical aspects of implementing automated cyber security systems, particularly in preventing bias in algorithms and ensuring fairness in their application.

Efficient breadth first search on multi-GPU systems

Graphics processing units leverage on a large array of parallel processing cores to boost the performance of a specific streaming computation pattern frequently found in graphics applications. Unfortunately, while many other general purpose applications do exhibit the required streaming behavior, they also possess unfavorable data layout and poor computation-to-communication ratios that penalize any straight-forward execution on the GPU. In this paper we describe an efficient and scalable code generation framework that can map general purpose streaming applications onto a multi-GPU system. This framework spans the entire core and memory hierarchy exposed by the multi-GPU system. Several key features in our framework ensure the scalability required by complex streaming applications. First, we propose an efficient stream graph partitioning algorithm that partitions the complex application to achieve the best performance under a given shared memory constraint. Next, the resulting partitions are mapped to multiple GPUs using an efficient architecture-driven strategy. The mapping balances the workload while considering the communication overhead. Finally, a highly effective pipeline execution is employed for the execution of the partitions on the multi-GPU system. The framework has been implemented as a back-end of the StreamIt programming language compiler. Our comprehensive experiments show its scalability and significant performance speedup compared with a previous state-of-the-art solution.

Spread spectrum (SS) signal acquisition in satellite communication is a very computation intensive technique, which hinders the development of real-time spread spectrum signal acquisition. In this paper, in order to achieve real-time acquisition, we propose a multi-GPU based SS signal acquisition algorithm. First, sliding correlation, the computation kernel, is formulated and efficiently parallelized by CUDA. Second, a CUDA-enabled SS signal acquisition algorithm is implemented by adopting the CUDA-enabled sliding correlation. Third, a multi-GPU based algorithm is implemented by using multi-GPU programming. The performance is evaluated in a real SS telemetry system. Real-time acquisition is achieved in all cases by using a single K40 GPU. Furthermore, an average of 374.7 $\times$ speedup (max 473 $\times$ ) in 6 datasets is achieved when using four K40 GPUs. Good scalability is observed when varying the parameters.

GPUs have been accelerating a wide range of algorithms and applications with their massively parallel computing capabilities. Today, as GPU programs work under the close supervision of a CPU, computing platforms are usually equipped with both CPUs and GPUs. Most existing GPU programs underutilize the computing capabilities of the CPU since the CPU is frequently idle while GPUs are executing. Additionally, existing work primarily focuses on using a single GPU, thus failing to utilize the computing power of multi-GPU systems. In this dissertation, we propose a new heterogeneous computing paradigm--Collaborative Heterogeneous Computing. Collaborative Heterogeneous Computing leverages fine-grained CPU-GPU communication mechanisms in computation with the use of CPUs and multiple GPUs simultaneously. Collaborative Heterogeneous Computing can be divided into CPU-GPU collaborative computing and Multi-GPU collaborative computing. For CPU-GPU collaborative computing, we first categorize 7 CPU-GPU collaborative execution patterns. With the guidance of the summarized CPU-GPU execution patterns, we implement Hetero-Mark, a benchmark suite that enables users to explore CPU-GPU collaborative execution patterns. We also design and develop Multi2Sim-HSA, an HSA system emulator that emulates behaviors of CPUs and GPUs in CPU-GPU collaborative execution applications. In addition, we observe that CPU-GPU communication over the PCIe network causes congestion and can become a major performance bottleneck. To address this issue, we propose a priority-based PCIe scheduling algorithm that can improve device utilization and satisfy Quality of Service (QoS) requirements. For multi-GPU collaborative computing, we develop MGPUMark, a benchmark suite that explores multi-GPU collaborative execution and fine-grained inter-GPU communication. Benchmarks in MGPUMark enable exploration of inter-GPU communication patterns. We also design and develop MGPUSim, a high-performance, high-flexibility, and high-accuracy GPU simulator. MGPUSim can faithfully model behaviors of GPUs in multi-GPU collaborative computing. We identify that the main performance bottlenecks occur due to long-latency, low-bandwidth, inter-GPU interconnects. Thus, reducing inter-GPU communication traffic can improve multi-GPU system performance. We develop the Locality APIs, a set of GPU programming APIs that enables programmers to place the computing threads and associated data on a multi-GPU system to reduce inter-GPU traffic. We also introduce Progressive Page-Splitting Migration (PASI), a hardware-based solution that allows GPUs to adjust data placement to avoid inter-GPU traffic.--Author's abstract

High-level domain-specific languages for array processing on the GPU are increasingly common, but they typically only run on a single GPU. As computational power is distributed across more devices, languages must target multiple devices simultaneously. To this end, we present a compositional translation that fissions data-parallel programs in the Accelerate language, allowing subsequent compiler and runtime stages to map computations onto multiple devices for improved performance---even programs that begin as a single data-parallel kernel.

Large-scale brain networks undergo widespread changes with older age and in neurodegenerative diseases such as Alzheimer's disease (AD). Research in young adults (YA) suggest that the underlying functional architecture of brain networks remains relatively consistent between rest and task states. However, it remains unclear whether the same is true in aging and to what extent any changes may be related to accumulation of AD pathology such as β-amyloid (Aβ) and tau. Here, we examined age-related differences in functional connectivity (FC) between rest and an object-scene mnemonic discrimination task using fMRI in young and older adults (OA; both females and males). We used an a priori episodic memory network (EMN) parcellation scheme associated with object and scene processing, that included anterior-temporal regions and posterior-medial regions. We also used positron emission topography to measure Aβ and tau in older adults. The correlation between rest and task FC (i.e., FC similarity) was reduced in older compared with younger adults. Older adults with lower FC similarity in EMN had higher levels of tau in the same EMN regions and performed worse during object, but not scene, trials during the fMRI task. These findings link AD pathology, particularly tau, to a less stable functional architecture in memory networks. They also suggest that smaller changes in FC organization between rest and task states may facilitate better performance in older age. Interpretations are limited by methodological factors related to different acquisition directions and durations between rest and task scans.SIGNIFICANCE STATEMENT The brain's large-scale network organization is relatively consistent between rest and task states in young adults (YA). We found that memory networks in older adults (OA) were less correlated between rest and (memory) task states compared with young adults. Older adults with less correlated brain networks also had higher levels of Alzheimer's disease (AD) pathology in the same regions, suggesting that a less stable network architecture may reflect the early evolution of AD. Older adults with less correlated brain networks also performed worse during the memory task suggesting that more similar network organization between rest and task states may facilitate better performance in older age.

The adoption of unified memory and demand paging has simplified programming and eased memory management in discrete GPUs. However, long-latency page faults cause significant performance overhead. While several software-based mechanisms have been proposed to address this issue, they suffer from inefficiency when page prefetching and pre-eviction are combined. For example, a state-of-the-art page replacement policy, hierarchical page eviction (HPE), is inefficient when prefetching is enabled. Furthermore, the prefetcher semantics-aware pre-evicting policy, which pre-evicts continuous pages in bulk the way they were brought in by the prefetcher, may cause thrashing for some irregular applications.In this paper, coordinated page prefetch and eviction (CPPE) is proposed to manage memory oversubscription in GPUs with unified memory. CPPE incorporates a modified page eviction policy, MHPE, and an access pattern-aware prefetcher in a fine-grained manner: MHPE is aware of prefetch semantics and the prefetcher prefetches pages according to access patterns in eviction candidates selected by MHPE. Simulation results show that, when the GPU memory is 75% and 50% oversubscribed, CPPE achieves an average speedup of 1.56x and 1.64x (up to 10.97x) over the state-of-the-art baseline, which combines a sequential-local prefetcher and LRU pre-eviction policy. CPPE also outperforms other approaches, including Random/reserved LRU with the sequential-local prefetcher, and simply disabling prefetching under memory oversubscription.

Memory consolidation involves reorganization at both the synaptic and system levels. The latter involves gradual reorganization of the brain regions that support memory and has been mostly highlighted using hippocampal-dependent tasks. The standard memory consolidation model posits that the hippocampus becomes gradually less important over time in favor of neocortical regions. In contrast, this reorganization of circuits in amygdala-dependent tasks has been less investigated. Moreover, this question has been addressed using primarily lesion or cellular imaging approaches thus precluding the comparison of recent and remote memory networks in the same animals. To overcome this limitation, we used microPET imaging to characterize, in the same animals, the networks activated during the recall of a recent versus remote memory in an olfactory cued fear conditioning paradigm. The data highlighted the drastic difference between the extents of the two networks. Indeed, although the recall of a recent odor fear memory activates a large network of structures spanning from the prefrontal cortex to the cerebellum, significant activations during remote memory retrieval are limited to the piriform cortex. These results strongly support the view that amygdala-dependent memories also undergo system-level reorganization, and that sensory cortical areas might participate in the long-term storage of emotional memories.

High Performance Computing is steadily embracing heterogeneous systems for supporting a wide variety of workloads. Currently there are two main sources of heterogeneity in compute nodes: (a) different compute elements such as multicore CPUs, GPUs, FPGAs, etc. and (b) different types of memory including DDR, HBM, SSDs. Multiple compute elements and memory types present many opportunities for accelerating applications featuring stages characterized by different compute intensity, sequential or parallel execution, cache sensitivity, etc. At the same time programmers are facing multiple challenges in making necessary adaptations in their codes. In this study we employ IBM’s OpenMP 4.5 implementation to program hybrid nodes with multiple CPUs and GPUs and manage on-node memories and application data. Through code samples we provide application developers with numerous options for memory management and data management. We consider simple functions using arrays and also complex and nested data structures.

The QR factorisation is a cornerstone of numerical linear algebra, essential for solving overdetermined linear systems, eigenvalue problems, and various scientific computing tasks. However, computing it for ill-conditioned tall-and-skinny (TS) matrices on large-scale distributed-memory systems, particularly those with multiple GPUs, presents significant challenges in balancing numerical stability, high performance, and efficient communication. Traditional Householder-based QR methods provide numerical stability but perform poorly on TS matrices due to their reliance on memory-bound kernels. This paper introduces a novel algorithm for computing the QR factorisation of ill-conditioned TS matrices based on CholeskyQR methods. Although CholeskyQR is fast, it typically fails due to severe loss of orthogonality for ill-conditioned inputs. To solve this, our new algorithm, mCQRGSI+, combines the speed of CholeskyQR with stabilising techniques from the Gram–Schmidt process. It is specifically optimised for distributed multi-GPU systems, using adaptive strategies to balance computation and communication. Our analysis shows the method achieves accuracy comparable to Householder QR, even for extremely ill-conditioned matrices (condition numbers up to 1016). Scaling experiments demonstrate speedups of up to 12× over ScaLAPACK and 16× over SLATE’s CholeskyQR2. This work delivers a method that is both robust and highly parallel, advancing the state-of-the-art for this challenging class of problems.

General Purpose Graphics Processing Units (GPGPU) have been used in high performance computing platforms to accelerate the performance of scientific applications such as simulations. With the increased computing resources required for large-scale network simulation, one GPU device may not have enough memory and computation capacities. It is therefore necessary to enhance the system scalability by introducing multiple GPU devices. It is also attractive to investigate the performance scalability of Multi-GPU simulations. This paper describes the simulation of information propagation on multiple GPU devices, including the optimized network simulation algorithms, the network partitioning and replication strategy, and the data synchronization scheme. The experimental results for scalable random networks show that the number of simulation steps, computation time, synchronization time, and data transfer time all affect the overall simulation performance. In order to compare with random networks, we also conduct simulations of scale-free networks. We can observe that the node replication ratio in scale-free networks is smaller than that in random networks and therefore the cost of data transfer and synchronization is significantly reduced. This indicates that the network structure is also an important factor that influences the simulation performance in a Multi-GPU system.

We present a new two-stage fine-grained air Particulate Matter (PM) prediction system using a variety of deep memory networks. Our model is significantly simpler than traditional weather report systems, which rely heavily on the atmospheric reaction equations and pollutant emissions inventories. Pollutant inventories are notoriously difficult to obtain and are often packed with declination for multiple reasons, while reaction equations are impossible to exhaust. These traditional models also tend to perform poorly when affected by strong convective weather. In contrast, our model does not need precise hand collected inventories of pollution sources. It can utilize the potential of Deep-Neural-Networks (DNN) to reveal the relationship among different locations and even find relations between known social events and air quality. Both potentially provide a valid path to air pollution control. The key to our approach is a well-tuned sophisticated attention based network that uses multiple GPUs, allowing us to transform a traditional sparse prediction problem into a sequence-to-sequence learning problem and train it end-to-end. By evaluating the model on over 1,625 instances of data of Northern China collected by our team, we show that it is not only computationally efficient but also accurately feasible compared with other optional models.

Spiking neural networks are being investigated both as biologically plausible models of neural computation and also as a potentially more efficient type of neural network. Recurrent neural networks in the form of networks of gating memory cells have been central in state-of-the-art solutions in problem domains that involve sequence recognition or generation. Here, we design an analog Long Short-Term Memory (LSTM) cell where its neurons can be substituted with efficient spiking neurons, where we use subtractive gating (following the subLSTM in [1]) instead of multiplicative gating. Subtractive gating allows for a less sensitive gating mechanism, critical when using spiking neurons. By using fast adapting spiking neurons with a smoothed Rectified Linear Unit (ReLU)-like effective activation function, we show that then an accurate conversion from an analog subLSTM to a continuous-time spiking subLSTM is possible. This architecture results in memory networks that compute very efficiently, with low average firing rates comparable to those in biological neurons, while operating in continuous time.