The Renoir Dataflow Platform: Efficient Data Processing without Complexity

Next generation science will increasingly come to rely on the ability to perform efficient, on-the-fly analytics of data generated by high-performance computing (HPC) simulations, modeling complex physical phenomena. Scientific computing workflows are stymied by the traditional chaining of simulation and data analysis, creating multiple rounds of redundant reads and writes to the storage system, which grows in cost with the ever-increasing gap between compute and storage speeds in HPC clusters. Recent HPC acquisitions have introduced compute node-local flash storage as a means to alleviate this I/O bottleneck. We propose a novel approach, Active Flash, to expedite data analysis pipelines by migrating to the location of the data, the flash device itself. We argue that Active Flash has the potential to enable true out-of-core data analytics by freeing up both the compute core and the associated main memory. By performing analysis locally, dependence on limited bandwidth to a central storage system is reduced, while allowing this analysis to proceed in parallel with the main application. In addition, offloading work from the host to the more power-efficient controller reduces peak system power usage, which is already in the megawatt range and poses a major barrier to HPC system scalability. We propose an architecture for Active Flash, explore energy and performance trade-offs in moving computation from host to storage, demonstrate the ability of appropriate embedded controllers to perform data analysis and reduction tasks at speeds sufficient for this application, and present a simulation study of Active Flash scheduling policies. These results show the viability of the Active Flash model, and its capability to potentially have a transformative impact on scientific data analysis.

Energy Management Systems (EMSs) are used to monitor energy consumption in buildings with the purpose of improving energy efficiency, by identifying savings opportunities and misuse situations. To achieve that, an EMS collects energy metering data streams from a network of energy meters in the building. Sensor data must be processed in (near) real-time, to support a timely decision making process. Currently, EMSs are using traditional DBMSs to process such data, introducing a persistence step that translates to an unacceptable latency on data evaluation. Moreover, sensor data monitoring queries are not elegantly supported by the SQL query language, thus hampering the ability of an EMS to process energy metering data in realtime. Data Stream Management Systems (DSMSs) are used to process data streams efficiently in several domains. Many sensor network monitoring applications have been implemented upon DSMSs resulting in significant improvements on performance and overall resource usage. This work validates the hypothesis proposed in [1] that, to process energy metering data streams in realtime, EMSs should be supported by DSMSs, instead of DBMSs. We introduce an EMS’s Data Processing Architecture supported by a DSMS that supports the implementation of an EMS capable of performing real-time data processing. We validate our solution through a benchmark evaluation against a DBMS based architecture. The results clearly show that the DSMSbased EMS outperformed the state of the art approach, both in data evaluation latency and query language expressibility— demonstrating its adequacy to process energy metering data streams in real-time.

Mutual information is a well-established and broadly used concept in information theory. It allows to quantify the mutual dependence between two variables -- an essential task in data analysis. For static data, a broad range of techniques addresses the problem of estimating mutual information. However, the assumption of static data is not applicable for today's dynamic data sources such as data streams: In contrast to static approaches, an online estimator must be able to deal with the evolving, changing, and infinite nature of the stream. Furthermore, some tasks require the estimation to be available online while processing the raw data stream. Our proposed solution Mise (Mutual Information Stream Estimation) allows a user to issue mutual information queries in arbitrary time windows. As a key feature, we introduce a novel sampling scheme, which ensures an equal treatment of queries over multiple time scales, e.g., ranging from milliseconds up to decades. We thoroughly analyze the requirements of such a multiscale sampling scheme, and evaluate the resulting quality of Mise in a broad range of experiments.

Modern High Performance Computing (HPC) applications process very large amounts of data. A critical research challenge lies in transporting input data to the HPC center from a number of distributed sources, e.g., scientific experiments and web repositories, etc., and offloading the result data to geographically distributed, intermittently available end-users, often over under-provisioned connections. Such end-user data services are typically performed using point-to-point transfers that are designed for well-endowed sites and are unable to reconcile the center's resource usage and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. To overcome these inefficiencies, decentralized HPC data services are emerging as viable alternatives. In this paper, we develop and enhance such distributed data services by designing CATCH, a Cloud-based Adaptive data Transfer service for HPC. CATCH leverages a bevy of cloud storage resources to orchestrate a decentralized data transport with fail-over capabilities. Our results demonstrate that CATCH is a feasible approach, and can help improve the data transfer times at the HPC center by as much as 81.1\% for typical HPC workloads.

The research area of Multimedia Content Analysis (MMCA) considers all aspects of the automated extraction of knowledge from multimedia archives and data streams. To satisfy the increasing computational demands of emerging MMCA problems, there is an urgent need to apply High Performance Computing (HPC) techniques. As most MMCA researchers are not also experts in the field of HPC, there is a demand for programming models and tools that can help MMCA researchers in applying these techniques. Ideally, such models and tools should be efficient and easy to use.At present there are several user transparent library-based tools available that aim to satisfy both these conditions. All such tools use a data parallel approach in which data structures (e.g. video frames) are scattered among the available compute nodes. However, for certain MMCA applications a data parallel approach induces intensive communication, which significantly decreases performance. In these situations, we can benefit from applying alternative parallelization approaches.This paper presents an innovative user transparent programming model for MMCA applications that employs task parallelism. We show our programming model to be a viable alternative that is capable of outperforming existing user transparent data parallel approaches. As a result, the model is an important next step towards our goal of integrating data and task parallelism under a familiar sequential programming interface.KeywordsSource NodeApplication Programming InterfaceHigh Performance ComputingTask GraphRuntime SystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This scientific article addresses two critical tasks in data analysis—time series classification and clustering, particularly focusing on heart sound recordings. One of the main challenges in analyzing time series lies in the difficulty of comparing different series due to their variability in length, shape, and amplitude. Various algorithms were employed to tackle these tasks, including the Long Short-Term Memory (LSTM), KNN, recurrent neural network for classification and the K-means and DBSCAN methods for clustering. The study emphasizes the effectiveness of these methods in solving classification and clustering problems involving time series data containing heart sound recordings. The results indicate that LSTM is a powerful tool for time series classification due to its ability to retain contextual information over time. In contrast, KNN demonstrated high accuracy and speed in classification, though its limitations became apparent with larger datasets. For clustering tasks, the K-means method proved to be more effective than DBSCAN, showing higher clustering quality based on metrics such as silhouette score, Rand score, and others. The data used in this research were obtained from the UCR Time Series Archive, which includes heart sound recordings from various categories: normal sounds, murmurs, additional heart sounds, artifacts, and extra systolic rhythms. The analysis of results demonstrated that the chosen classification and clustering methods could be effectively used for diagnosing heart diseases. Furthermore, this research opens up new opportunities for further improvement in data processing and analysis methods, particularly in developing new medical diagnostic tools. Thus, this work illustrates the effectiveness of machine learning algorithms for time series analysis and their significance in improving cardiovascular disease diagnosis.

Cloud computing and big data: Technologies and applications

Understanding data can be tricky and sometimes overwhelming. Using automation helps a data analysis team improve their methods more consistently. Specifically, automating software tasks is the best way to manage the many repetitions and repetitive jobs that come with thorough data analysis. When starting a project that involves software, automation, and working together, DevOps is the best way to go. DevOps improves all parts of the data analysis process, allowing teams to automatically handle software-related tasks in data analysis and work well with everyone involved in the project. The combination of DevOps, Data Analytics, and Data Engineering introduced the new DataOps which is used for data processing and analytics. DataOps improves communication, collaboration, and faster data pipeline

The lightweight nature, application portability, and deployment flexibility of containers is driving their widespread adoption in cloud solutions. Data analysis and deep learning (DL)/machine learning (ML) applications have especially benefited from containerization. As such data analysis is adopted in high performance computing (HPC), the need for container support in HPC has become paramount. However, containers face crucial performance and I/O challenges in HPC. One obstacle is that while there have been HPC containers, such solutions have not been thoroughly investigated, especially from the aspect of their impact on the crucial HPC I/O throughput. To this end, this paper provides a first-of-its-kind empirical analysis of state-of-the-art representative container solutions (Docker, Podman, Singularity, and Charliecloud) in HPC environments. We also explore how containers interact with an HPC parallel file system like Lustre. We present the design of an analysis framework that is deployed on all nodes in an HPC environment, and captures CPU, memory, network, and file I/O statistics from the nodes and the storage system. We are able to garner key insights from our analysis, e.g., Charliecloud outperforms other container solutions in terms of container start-up time, while Singularity and Charliecloud are equivalent in I/O throughput. But this comes at a cost, as Charliecloud invokes the most metadata and I/O operations on the underlying Lustre file system. By identifying such trade-offs and optimization opportunities, we can enhance HPC containers performance and the ML/DL applications that increasingly rely on them.

The pig farming sector in Europe is a very large and highly intensified livestock sector. Among other critical challenges such as environmental impact, economic return, resource usage, food safety, and fitting farming in more and more urbanized areas, there also is a fundamental question on how to fulfill the concerns of society about animal welfare and provide the best care for every individual. Pigs are very intelligent, kind, and curious animals. About all of this, I did not have a clue when I applied for a PhD position at ILVO and the KU Leuven as “Researcher Innovative Technologies in Pig Husbandry.” The job description form indicated that the applicant needed a degree in Bioscience Engineering or Veterinary Science, neither of which I had. Now, 8 years later, I’m part of flagship EU projects in precision livestock farming, and I have been combining many different aspects like sensors, Radio Frequency Identification (RFID), data analysis, decision support tools, Internet of Things (IoT), and High-Performance Computing (HPC). My personal key topic is to use these innovative technologies to work toward more individualized tailor-made care for each pig. This can bring the sector from a mainly group-based management to individual monitoring in order to detect if something is wrong with an individual pig, even if it’s hard to spot with the human eye.

The pig farming sector in Europe is a very large and highly intensified livestock sector. Among other critical challenges such as environmental impact, economic return, resource usage, food safety, and fitting farming in more and more urbanized areas, there also is a fundamental question on how to fulfill the concerns of society about animal welfare and provide the best care for every individual. Pigs are very intelligent, kind, and curious animals. About all of this, I did not have a clue when I applied for a PhD position at ILVO and the KU Leuven as “Researcher Innovative Technologies in Pig Husbandry.” The job description form indicated that the applicant needed a degree in Bioscience Engineering or Veterinary Science, neither of which I had. Now, 8 years later, I’m part of flagship EU projects in precision livestock farming, and I have been combining many different aspects like sensors, Radio Frequency Identification (RFID), data analysis, decision support tools, Internet of Things (IoT), and High-Performance Computing (HPC). My personal key topic is to use these innovative technologies to work toward more individualized tailor-made care for each pig. This can bring the sector from a mainly group-based management to individual monitoring in order to detect if something is wrong with an individual pig, even if it’s hard to spot with the human eye.

During the last decade, Heterogeneous systems are emerging for high performance computing [1]. In order to achieve high performance computing (HPC), existing technologies and programming models aims to see rapid growth toward intra-node parallelism [2]. The current high computational system and applications demand for a massive level of computation power. In last few years, Graphical processing unit (GPU) has been introduced an alternative of conventional CPU for highly parallel computing applications both for general purpose and graphic processing. Rather than using the traditional way of coding algorithms in serial by single CPU, many multithreading programming models has been introduced such as CUDA, OpenMP, and MPI to make parallel processing by using multicores. These parallel programming models are supportive to data driven multithreading (DDM) principle [3]. In this paper, we have presented performance based preliminary evaluation of these programming models and compared with the conventional single CPU serial processing system. We have implemented a massive computational operation for performance evaluation such as complex matrix multiplication operation. We used data driven multithreaded HPC system for performance evaluation and presented the results with a comprehensive analysis of these parallel programming models for HPC parallelism.

High Performance Computing (HPC) is a strategical resource that allows research communities and developers to fulfill the processing demand (1 ExaFlops/Sec) for future Exascale Computing system which is expected in the end of current decade. In order to provide an extensive level of performance, many powerful and energy efficient devices (MIC, GPU) and parallel programming models have been proposed. One solution deal HPC applications is optimized utilization of these parallel computing based devices and programming models. In this paper we have proposed a novel approach hybrid Tri-level parallel computing model. The objective of proposed model was to provide a platform that can render both fine-grain and course-grain parallelism for massive computation. For hybrid proposed model, three promising parallel computing models were considered including CUDA (parallelize GPU thread), OpenMP (parallelize CPU threads) and MPI (communicate among homogeneous / heterogeneous cores). Further to validate proposed model, we considered Gauss Jacobi iterative solver for 2-D Laplace equation and implemented in proposed hybrid model. Experiments performed at various mesh size on conventional CPU and proposed hybrid model. Consequently, hybrid model performed much faster as compared to serial and single GPU computation. Nevertheless, based on significant results, hybrid parallel model could be considered as initiative model to solve other HPC applications in linear / non-linear system.

The Geo-INQUIRE (Geosphere INfrastructure for QUestions into Integrated REsearch) project, supported by the Horizon Europe Programme, is aimed at enhancing services to make data and high-level products accessible to the broad Geoscience scientific community. Geo-INQUIRE’s goal is to encourage curiosity-driven studies into understanding the geosphere dynamics at the interface between the solid Earth, the oceans and the atmosphere using long data streams, high-performance computing and cutting-edge facilities. The Geo-INQUIRE Transnational Access (TA) covers both virtual and on-site access to a variety of state of the art laboratories, facilities, experimental sites (testbeds) and computational resources with the aim of enabling the development of excellent ground-breaking science. Six research infrastructures located across Europe, referred to as “testbeds”, will provide locations for users to perform experiments in a variety of environments from the Earth’s surface (both on land and at sea) to the subsurface; over different spatial scales: from small-scale experiments in laboratories to kilometric submarine fibre cables. These sites are: the Bedretto Laboratory (Switzerland); the Ella-Link Geolab (Portugal); the Liguria-Nice-Monaco submarine infrastructure (Italy/France); the Irpinia Near-Fault Observatory (Italy); the Eastern Sicily facility (Italy); and the Corinth Rift Laboratory (Greece). In addition, ECCSEL-ERIC is providing access to 5 of its research facilities focussing on CO2 Capture, Utilisation, Transport and Storage. The facilities providing access are: Svelvik CO2 Field Lab (Norway), PITOP Borehole Geophysical Test Site (Italy), Sotacarbo Fault Laboratory (Italy), Catenoy experimental site and gas-water-rock interaction Laboratory in Oise (France) and the Mobile Seismic Array (the Netherlands) which is fully mobile and can be deployed anywhere in the world. TA will be also offered for software and workflows belonging to the EPOS-ERIC and the ChEESE Centre of Excellence for Exascale in Solid Earth. These are grounded on simulation of seismic waves and rupture dynamics in complex media, tsunamis, subaerial and submarine landslides. HPC-based Probabilistic Tsunami, Seismic and Volcanic Hazard workflows are offered to assess hazard at high-resolution with extensive uncertainty exploration. Support and collaboration will be offered to the awardees to facilitate the access and usage of HPC resources for tackling geoscience problems. Geo-INQUIRE will grant TA to researchers to develop their own lab-based or numerical experiments with the aim of advancing scientific knowledge of Earth processes while fostering cross-disciplinary research across Europe. The data and products generated during the TAs will be made available to the scientific community via the project’s strict adherence to FAIR principles. To be granted, researchers submit a proposal to the TA calls that will be issued three times during the project life. The first call was launched on the 9th January. Calls will be advertised on the Geo-INQUIRE website https://www.geo-inquire.eu/ and through the existing community channels.

Chapter 4 - Frameworks for Big Data Integration, Warehousing, and Analytics