The eXtreme-DataCloud project: data management services for the next generation distributed e-infrastructures

The development of data management services capable to cope with very large data resources is a key challenge to allow the future einfrastructures to address the needs of the next generation extreme scale scientific experiments. To face this challenge, in November 2017 the H2020 eXtreme DataCloud - XDC project has been launched. Lasting for 27 months and combining the expertise of eight large European research organisations, the project aims at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. The targeted platforms are the current and next generation e-Infrastructures deployed in Europe, such as the European Open Science Cloud (EOSC), the European Grid Infrastructure (EGI), and the Worldwide LHC Computing Grid (WLCG). The project is use-case driven with a multidisciplinary approach, addressing requirements from research communities belonging to a wide range of scientific domains: High Energy Physics, Astronomy, Photon and Life Science, Medical research. XDC is aimed at implementing data management scalable services, combining already established data management and orchestration tools, to address the following high level topics: policy driven data management based on Quality-of-Service, Data Life-cycle management, smart placement of data with caching mechanisms to reduce access latency, meta-data with no predefined schema handling, execution of pre-processing applications during ingestion, data management and protection of sensitive data in distributed e-infrastructures, intelligent data placement based on access patterns. This contribution introduces the project, presents the foreseen overall architecture and the developments that are being carried on to implement the requested functionalities.

INFN-CNAF is one of the Worldwide Large Hadron Collider Computing Grid (WLCG) Tier-1 data centers, providing support in terms of computing, networking, storage resources and services also to a wide variety of scientific collaborations, ranging from physics to bioinformatics and industrial engineering [1]. Recently, several collaborations working with our data center have developed computing and data management workflows that require access to S3 storage services and/or the integration with POSIX capabilities. Nevertheless, the access to the data must be regulated by federated authentication and authorization mechanisms, such as OpenID Connect (OIDC), which is largely adopted by communities like WLCG [2] and within the European Open Science Cloud (EOSC) [3]. In the present work, the possibility to regulate POSIX access by integrating JSON Web Token (JWT) [4] authentication, provided by INDIGO-IAM as Identity Provider [5], with solutions based on S3 (for object storage) and WebDAV (for hierarchical storage) protocols has been evaluated an related results have been reported. In such respect, a comparison between the performance yielded by S3 and WebDAV protocols has been carried out within the same distributed environment with the aim to better identify the solution most suitable for the different use cases.

The modern security landscape for distributed computing in High Energy Physics (HEP) includes a wide range of threats employing different attack vectors. The nature of these threats is such that the most effective method for dealing with them is to work collaboratively, both within the HEP community and with partners further afield - these can, and should, include institutional and campus security teams. In parallel with this work, an appropriate technology stack is essential, incorporating current work on Big Data analytics. The work of the Worldwide LHC Computing Grid (WLCG) Security Operations Center (SOC) Working Group (WG) [1] is to pursue these goals to form a reference design (or guidelines) for WLCG sites of different types. The strategy of the group is to identify necessary components - starting with threat intelligence (MISP [2]) and network intrusion detection (Bro [3]), building a working model over time. We present on the progress of the working group thus far, in particular on the programme of workshops now underway. These workshops give an opportunity to engage with sites to allow the development of advice and procedures for deployment, as well as facilitating wider discussions on how to best work with trust groups at different levels. These trust groups vary in scope but can include institutes, National Grid Infrastructures and the WLCG as a whole.

In the near future, High Energy Physics experiments’ storage and computing needs will go far above what can be achieved by only scaling current computing models or current infrastructures. Considering the LHC case, for 10 years a federated infrastructure (Worldwide LHC Computing Grid, WLCG) has been successfully developed. Nevertheless, the High Luminosity (HL-LHC) scenario is forcing the WLCG community to dig for innovative solutions. In this landscape, one of the initiatives is the exploitation of Data Lakes as a solution to improve the Data and Storage management. The current Data Lake model foresees data caching to play a central role as a technical solution to reduce the impact of latency and network load. Moreover, even higher efficiency can be achieved through a smart caching algorithm: this motivates the development of an AI-based approach to the caching problem. In this work, a Reinforcement Learning-based cache model (named QCACHE) is applied in the CMS experiment context. More specifically, we focused our attention on the optimization of both cache performances and cache management costs. The QCACHE system is based on two distinct Q-Learning (or Deep Q-Learning) agents seeking to find the best action to take given the current state. More explicitly, they try to learn a policy that maximizes the total reward (i.e. hit or miss occurring in a given time span). While the addition Agent is taking care of all the cache writing requests, clearly the eviction agent deals with the decision to keep or to delete files in the cache. We will present an overview of the QCACHE framework an the results in terms of cache performances, obtained using using “Real-world” data, will be compared respect to standard replacement policies (i.e. we used historical data requests aggregation used to predict dataset popularity filtered for Italian region). Moreover, we will show the planned subsequent evolution of the framework.

This research program in high energy physics is carried out under the general supervision of a committee which is composed of G.W. Brandenburg, G.J. Feldman, M.E. Franklin, R.J. Glauber, K. Kinoshita, F.M. Pipkin, K. Strauch, R. Wilson, and H. Yamamoto. Professor G.J. Feldman currently serves as chair of this committee. Dr. Brandenburg is the Director of the High Energy Physics Laboratory and administers the DOE high energy physics contract. In the fall of 1991 S. Mishra will join this committee. Harvard is planning to make one or two additional senior faculty appointments in experimental high energy physics over the next two years. The principal goals of the work described here are to carry out forefront programs in high energy physics research and to provide first rate educational opportunities for students. The experimental program supported through HEPL is carried out at the major accelerator centers in the world and addresses some of the most important questions in high energy physics. Harvard`s educational efforts are concentrated in graduate education, where they are currently supporting thirteen research students. In addition, undergraduate students work in projects at HEPL during the academic year and over summers. These budget projections cover all of the Harvard based high energy physics experimental activities. The {open_quotes}umbrella{close_quotes} nature of this contract greatly simplifies support of essential central technical and computer services and helps the group to take advantage of new physics opportunities and to respond to unexpected needs. The funding for the operation of the HEPL facility is shared proportionally by the experimental groups. Harvard financially supports this high energy physics research program in many ways.

The core activity within the newly created H2020 project eXtreme DataCloud project will be the policy-driven orchestration of federated data management for data intensive sciences like High Energy Physics, Astronomy, Photon and Life Science. Well-known experts in this field will work on combining already established data management and orchestration tools to provide a highly scalable solution supporting the entire European Scientific Landscape. The work will cover Data Life Cycle Management as well as smart data placement on meta data, including storage availability, network bandwidth and data access patterns. Mechanisms will be put in place to trigger computational resources based on data ingestion and data movements. This paper presents the first architecture of this endeavor.

Knowledge infrastructure for integrated data management and analysis supporting new approach methods in predictive toxicology and risk assessment

Open Science is directly linked to the present trend of Research Data Management. To have fully open research data, systems that allow free access to the data need to be present. As a result, research institutions have tried to adapt to the changes required to be deemed as Open Science supporters. Several institutions implement a Current Research Information System (CRIS) and an Open Science Policy for all their published research data.Interoperable data between different systems implemented is a key feature of Open Science. In Europe, the European Open Science Cloud provides a set of standards and guidelines that must be followed so the implemented systems offer an interoperable service to their researchers.Previous research revealed that Albania is ready to take its first steps toward Open Science, and future ideas for how to contribute to the field were proposed. There were two significant findings: first, there is no CRIS implementation in Albanian institutions, and second, there is no strategy or policy to support Open Science. The research also proposed an Albanian Open Science Digital Repository.In this paper, we show how the implementation of the system is done and how it relates to the newly proposed Open Science Policy. The goal is to demonstrate how Albanian institutions are implementing advances towards Open Science.In conclusion, this paper demonstrates Albanian researchers' support for Open Science and the European Open Science Cloud.

Experimental observations and advanced computer simulations in High Energy Physics (HEP) paved the way for the recent discoveries at the Large Hadron Collider (LHC) at CERN. Currently, Monte Carlo simulations account for a very significant amount of computational resources of the Worldwide LHC Computing Grid (WLCG). The current growth in available computing performance will not be enough to fulfill the expected demand for the forthcoming High Luminosity run (HL-LHC). More efficient simulation codes are therefore required. This study focuses on evaluating the impact of different build methods on the simulation execution time. The Geant4 toolkit, the standard simulation code for the LHC experiments, consists of a set of libraries which can be either dynamically or statically linked to the simulation executable. Dynamic libraries are currently the preferred build method. In this work, three versions of the GCC compiler, namely 4.8.5, 6.2.0 and 8.2.0 have been used. In addition, a comparison between four optimization levels (Os, O1, O2 and O3) has also been performed. Static builds for all the GCC versions considered, exhibit a reduction in execution times of about 10%. Switching to newer GCC version results in an average of 30% improvement in the execution time regardless of the build type. In particular, a static build with GCC 8.2.0 leads to an improvement of about 34% with respect to the default configuration (GCC 4.8.5, dynamic, O2). The different GCC optimization flags do not affect the execution times.

EOSC-Life brings together the 13 Life Science ‘ESFRI’ research infrastructures (LS RIs) to create an open, digital and collaborative space for biological and medical research. The project is starting to publish ‘FAIR’ data and a catalogue of services provided by participating RIs for the management, storage and reuse of data in the European Open Science Cloud (EOSC). This space will be accessible to European research communities.

International mass digitization efforts through infrastructures like the European Distributed System of Scientific Collections (DiSSCo), the US resource for Digitization of Biodiversity Collections (iDigBio), the National Specimen Information Infrastructure (NSII) of China, and Australia’s digitization of National Research Collections (NRCA Digital) make geo- and biodiversity specimen data freely, fully and directly accessible. Complementary, overarching infrastructure initiatives like the European Open Science Cloud (EOSC) were established to enable mutual integration, interoperability and reusability of multidisciplinary data streams including biodiversity, Earth system and life sciences (De Smedt et al. 2020). Natural Science Collections (NSC) are of particular importance for such multidisciplinary and internationally linked infrastructures, since they provide hard scientific evidence by allowing direct traceability of derived data (e.g., images, sequences, measurements) to physical specimens and material samples in NSC. To open up the large amounts of trait and habitat data and to link these data to digital resources like sequence databases (e.g., ENA), taxonomic infrastructures (e.g., GBIF) or environmental repositories (e.g., PANGAEA), proper annotation of specimen data with rich (meta)data early in the digitization process is required, next to bridging technologies to facilitate the reuse of these data. This was addressed in recent studies (Younis et al. 2018, Younis et al. 2020), where we employed computational image processing and artificial intelligence technologies (Deep Learning) for the classification and extraction of features like organs and morphological traits from digitized collection data (with a focus on herbarium sheets). However, such applications of artificial intelligence are rarely—this applies both for (sub-symbolic) machine learning and (symbolic) ontology-based annotations—integrated in the workflows of NSC’s management systems, which are the essential repositories for the aforementioned integration of data streams. This was the motivation for the development of a Deep Learning-based trait extraction and coherent Digital Specimen (DS) annotation service providing “Machine learning as a Service” (MLaaS) with a special focus on interoperability with the core services of DiSSCo, notably the DS Repository (nsidr.org) and the Specimen Data Refinery (Walton et al. 2020), as well as reusability within the data fabric of EOSC. Taking up the use case to detect and classify regions of interest (ROI) on herbarium scans, we demonstrate a MLaaS prototype for DiSSCo involving the digital object framework, Cordra, for the management of DS as well as instant annotation of digital objects with extracted trait features (and ROIs) based on the DS specification openDS (Islam et al. 2020). Source code available at: https://github.com/jgrieb/plant-detection-service

SeaDataNet is a major pan-European infrastructure for managing and providing access to marine data sets, acquired by European organisations from research cruises and other observational activities in European coastal marine waters, regional seas and the global ocean. Founding partners are National Oceanographic Data Centres (NODCs), and major marine research institutes. The SeaDataNet network gradually expanded its network of data centres and infrastructure, during a series of dedicated EU RTD projects, and by engaging as core data management infrastructure and network in leading European Commission initiatives such as the European Marine Observation and Data network (EMODnet), Copernicus Marine Service (CMS), and the European Open Science Cloud (EOSC).SeaDataNet develops, governs and promotes common standards, vocabularies, software tools, and services for marine data management, which are widely adopted. A core service is the CDI data discovery and access service which provides online unified discovery and access to vast resources of data sets, managed by 115+ connected SeaDataNet data centres from 34 countries around European seas, both from research and monitoring organisations. Currently, it gives access to more than 3 Million data sets, originating from 1000+ organisations in Europe, covering physical, geological, chemical, biological and geophysical data, acquired in European waters and global oceans. Standard metadata and data formats are used, supported by an ever-increasing set of controlled vocabularies, resulting in rich and highly FAIR metadata and data sets. SeaDataNet provides core services in EMODnet Chemistry, Bathymetry, and Physics for bringing together and harmonizing large amounts of marine data sets, which are used by EMODnet groups for generating thematic data products.EMODnet Bathymetry is active since 2008 and maintains a Digital Terrain Model (DTM) for the European seas. This is published every 2 years, each time extending coverage, and improving quality and precision. The DTMs are produced from surveys and aggregated data sets that are referenced with metadata via the SeaDataNet Catalogue services. Bathymetric survey data sets are gathered and populated by national hydrographic services, marine research institutes, and companies in the SeaDataNet CDI Data Discovery & Access service. Currently, this amounts to more than 45.000 datasets from 78 data providers. A major selection of these datasets has been used for preparing the 2024 release of the EMODnet DTM for all European waters and Caribbean, which has been published on the EMODnet portal. Currently, work is ongoing for a new 2026 version. The EMODnet DTM has a grid resolution of 1/16 * 1/16 arc minutes (circa 115 * 115 m), covering all European seas. It is based upon circa 22.000+ in situ datasets. It can be downloaded in tiles and viewed as map layers in the EMODnet portal. The maps are derived from EMODnet Bathymetry OGC WMS, WMTS, and WFS services. The EMODnet Bathymetry products are very popular and in 2024 – 2025 more than 100.000 EMODnet DTM files were downloaded, and more than 60 million OGC service requests were registered over the 2 years. EMODnet Bathymetry is also managing the European contribution to the international Seabed 2030 project.

SeaDataNet is a pan-European infrastructure that manages and provides access to marine datasets collected by European organisations through research cruises and observational activities in coastal waters, regional seas, and the global ocean. It was founded by National Oceanographic Data Centres (NODCs) and major marine research institutes. The network has expanded through successive EU-funded RTD projects and by contributing to major European initiatives such as EMODnet, Copernicus Marine Service, ENVRI, and the European Open Science Cloud (EOSC).SeaDataNet develops and promotes widely adopted standards, vocabularies, software tools, and services that support FAIR marine data management. Its core service, the CDI (Common Data Index), provides unified online discovery and access to in situ marine observation data managed by more than 115 data centres in 34 countries. The service currently offers access to over 3 million datasets from more than 1,000 European organisations, covering physical, chemical, biological, geological, and geophysical data from European waters and the global ocean. The use of standard metadata, formats, and controlled vocabularies ensures rich, highly FAIR datasets.SeaDataNet also delivers core data services for EMODnet Chemistry, Bathymetry, and Physics, harmonising large volumes of marine data that support the production of thematic data products, including Essential Climate Variable (ECV) and Essential Ocean Variable (EOV) datasets.Environmental science increasingly relies on large, heterogeneous, and rapidly growing data collections that must be efficiently accessed, subsetted, and harmonised for use in models, digital twins, AI workflows, and Virtual Research Environments (VREs). The fully open-source Beacon software, developed by MARIS https://beacon.maris.nl/, addresses these challenges by enabling cloud-native, high-performance data lakes that are fast to deploy and access. Beacon supports parameter harmonisation using metadata annotations based on NERC vocabularies, ECV vocabularies, and the I-ADOPT methodology adopted in ENVRI-HUB Next.To improve the ease of access to subsets of the SeaDataNet CDI data collection, a Beacon instance containing all the open SeaDataNet data was set-up. This now allows users to obtain real-time access to data subsets in multiple data formats (NetCDF, Parquet, Zarr) and flexible querying from Jupyter Notebooks or a newly developed Beacon studio (user interface) for non-technical users. Within ENVRI-HUB Next, this SeaDataNet instance enables on-the-fly access to ECV-related subsets from millions of files via Jupyter Notebooks, ready for use in the Analytical Framework.The presentation focuses on this use case, the technical solution, and its potential applicability for other Research Infrastructures supporting EOSC use cases.

Thesis/Objective – The profession of data steward is not fully defined. In particular, in Polish scientific institutions, where even the name of this position is still hard to find, this is a new phenomenon that requires a deeper analysis. It is difficult to specify not only the definition of this profession, or rather the role, but also the scope of tasks and required competencies. All this depends on the role of the data steward in the unit, at the university or in the research team. The aim of the article is to present the origins of the creation of a new specialisation in Poland, which is the role of a data steward. In addition, the forms of education for data stewards in Poland are discussed, as well as initiatives and projects aimed at defining the professional profile, qualifications and career paths of data stewards. Research methods: The critical literature review was used to analyse the content of foreign and Polish LIS literature published in the years 2018-2022. Also, official documents and websites, project descriptions, working group’s, and Task Force’s working papers were analysed. In addition, a survey was carried out and analysed among the members of the Polish data stewards working group (DSCCIN PL) on organisational structures and positions dedicated to research data management existing at Polish universities. Results and conclusions – The first data stewards appeared in libraries. This resulted directly from the policy of the National Science Centre (NCN) but above all also from the fact that the first unit providing support for researchers in the field of data management – the Open Science Competence Center – was established in the Library of the Gdańsk University of Technology (Gdańsk Tech). The Gdańsk Tech Library has established numerous contacts with international networks and organizations, which resulted in active cooperation and placing it in international structures. In particular as a national chapter of the GO-FAIR network (DSCC-IN PL: Data Stewardship Competence Centers Implementation Network – Poland). This initiated the creation of the Polish working group of data stewards, which is now the foundation of the nascent profession and provides a field for the exchange of knowledge and experience. The lack of educational programs at university or in any other form of vocational education has been filled in a sense through the training activities of the Competence Center and the only Data Steward School in Poland. It was only in 2022 that the increased activity of the National Science Center and the Ministry of Education and Science began. In conjunction with the work on the national policy for open data as well as activities within the European Open Science Cloud, a coherent environment began to emerge for the development of a new specialization at the level of Polish universities and scientific institutions.

Motivations. In the last decade many projects related to grids have been carried out in Europe at both national and international levels with an important economic contribution by the European Commission.The grid middleware developed within these projects has been deployed into the European grid infrastructure (EGI) made by more than 350 sites all over Europe. The Italian Grid Infrastructure (IGI) is part of EGI and is one of the most important and widest national grid infrastructures in Europe since it provides about 33000 CPU cores, 17 PB of disk space and 9 PB of tape capacity spread over more than 50 sites. Although the grid infrastructure has been initially built according to the needs of a few scientific communities (high energy physics, earth observation and Bioinformatics among the others), it has been gradually evolving in order to provide grid services to a wider and wider user base. Several scientific communities are observing that as the instruments become more and more powerful the need for storage and computing is increasing day by day. This will also increase the number of users that could benefit from a geographical distributed computing grid infrastructure. Methods. In order to support new communities (users and resource providers), various activities have been started. Training has been considered very important, so tutorial for users and grid administrators are regularly organized. In addition great effort has been devoted to understanding the user needs, by defining appropriate use cases, and to supporting the user communities to port their applications on the grid environment (the so-called "gridification"). An important effort is also spent in developing new tools that could make the interaction between the final users and the grid as easy as possible. In particular web tools to submit jobs to the grid infrastructure have been deployed and used by some bioinformatics communities. In order to address the needs of users relying on high-level tools like Workflow managers (e.g. Taverna and other similar tools), a front-end web service has been developed. This web interface could be used as a bridge towards the EGI/IGI grid infrastructure. The IGI community is also providing a service that allows the exploitation of Relational Databases over the grid infrastructure, assuring a high level of security and privacy. Results. The usage of the standard EGI/IGI resources and services, together with the high level services that IGI is providing on top of the grid, has provided the end users with the capability of carrying out their high demanding computing activities in an easy and reliable way. In the past years, indeed, IGI has supported several bioinformatics communities to "gridify" many different applications such as: ASPic, PAML, MrBayes, CSTGrid, DNAfan, BLAST, BayeSSC, FT-Comar, Muscle, Gene Ontology DB analysis, ABCtoolbox, EMBOSS, Bowtie, SAMtools, Illumina Solexa data processing, AmpliconNoise, BioPython, HMMER. As a result the CPU consumed by various Italian bioinformatics groups on the IGI grid infrastructure has exceeded the 10 years in a few days of activity, thus hugely reducing the overall time needed for the execution of the jobs. Thanks to IGI the bioinformatics users have carried out their analysis in an easy and transparent way, both through simple web interfaces and through complex WorkFlow managers, reducing the time needed to get the results of about 2-3 orders of magnitude. The IGI infrastructure has also been exploited by the Computational Biology group of the Bologna University, in the frame of DUCK (Distributed Unified Computing for Knowledge, a collaboration between multidisciplinary Academic and Research Institutions located in the Emilia Romagna region), to run protein annotation application based on the Bologna Annotation Resource (BAR) method, and to perform massively parallel genome sequencing including about 18 millions of protein sequences. More than 150 computing nodes in the IGI grid infrastructure have been used, successfully dropping the computational times if compared to the computing resources of the local cluster available to the group. The protein annotation application reached a dropping factor of 120, i. e. the computation was performed in few weeks instead of years. Data management facilities offered by the Grid were also exploited to easily handle input and output files. To provide an easy-to-use service to the user communities IGI is developing a web portal that will hide the complexity of the authentication/authorization mechanisms and will also integrate the computing frameworks needed by the different user communities. A prototype of this portal can be easily set up for the bioinformatics community. Availability http://www.italiangrid.it/