NIST Big Data Interoperability Framework: Volume 3, Use Cases and General Requirements

NoSQL databases are an effective solution for storing and processing large data, but these databases are heterogeneous. They offer different data storage models, implementations and languages to developers and users. This wide variety of platforms makes it difficult data interoperability, data integration and even data migration from one system to another. This paper proposes a literature review of papers related to big data interoperability. This paper also compares different big data interoperability approaches using a multi-criteria analysis by applying the Complex Proportional Assessment Method (COPRAS).

The application of MapReduce cloud computing simulators for research and development is becoming popular, due to their efficiency and ease of utilization. This ignited the development of several cloud simulators for algorithm testing and performance analysis of dynamic MapReduce environments. The selection of appropriate simulator for a specific research remains a challenge. We have designed a MapReduce classification framework to guide cloud and big data researchers towards suitable tools. We have reviewed eleven MapReduce specific simulators. Our evaluation first revealed thirty general functional requirements for more widely applicable cloud simulators. Then, we focused on specific concerns of MapReduce related simulations and filtered the general requirements down to the most relevant thirteen. Our evaluation highlighted the strengths and weakness of several MapReduce simulators. IoT-based applications, stream processing and replaying of production cluster workloads are key criteria absent from many simulators. Therefore, we identified these as gaps, that simulator developers could focus on when extending their works towards MapReduce oriented simulations. Finally, researchers simulating dynamic behaviors of Hadoop clusters should select simulators efficient in parameter tuning.

The enormous volumes of data created and maintained by industries, research institutions are on the verge of outgrowing its infrastructure. The advancements in the organization's work flow include data storage, data management, data maintenance, data integration, and data interoperability. Among these levels, data integration and data interoperability can be the two major focus areas for the organizations which tend to implement advancements in their workflow. Overall, data integration and data interoperability influence the organization's performance. The data integration and data interoperability are complex challenges for the organizations deploying big data architectures due to the heterogeneous nature of data used by them. Therefore, it requires a comprehensive approach to negotiate the challenges in integration and interoperability. This paper focuses on the challenges of data integration and data interoperability in big data.

As one of the most popular queries in big data era, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> nearest neighbors ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> NN) query plays a significant role in various applications, such as medical diagnosis, signal processing, and recommendation systems. Meanwhile, driven by the advancement of the cloud service, an emerging trend among applications is to outsource the dataset and the corresponding <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> NN query services to the cloud. However, as the cloud is not fully trusted, those applications will face vital privacy concerns, and thus they usually encrypt data before outsourcing them to the cloud. Because encrypted data are outsourced to cloud, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> NN query over encrypted data has become increasingly attractive, and many solutions have been put forth in recent years. However, existing solutions cannot fully satisfy the objects of returning exact query results, protecting database privacy and query privacy, achieving high query efficiency, and imposing low computational costs at the user side. To address these issues, in this paper, we propose a new practical and privacy-preserving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> NN query scheme. Specifically, we first refine the general security requirements for the matrix encryption by systematically analyzing existing algorithms. Then, we design a novel asymmetric matrix encryption (AME) to securely achieve Euclidean distance computation and two distances comparison in a single-party and non-interactive way. Then, based on the AME scheme, we propose a privacy-preserving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> NN query scheme, in which a max-heap of size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> is used to accelerate query efficiency. Detailed security analysis shows that our proposed scheme is really privacy-preserving. In addition, extensive performance evaluations are conducted, and the results demonstrate that our proposed scheme is also highly efficient.

UNSTRUCTURED The successful implementation of the European Health Data Space (EHDS) for the secondary use of data (known as EHDS2) hinges on overcoming significant challenges, including the proper implementation of interoperability standards, harmonization of diverse national approaches to data governance, and the integration of rapidly evolving AI technologies. This work addresses these challenges by developing an interactive toolkit that leverages insights from 7 leading cancer research projects (Integration of Heterogeneous Data and Evidence towards Regulatory and HTA Acceptance [IDERHA], European Federation for Cancer Images [EUCAIM], Artificial intelligence Supporting Cancer Patients across Europe [ASCAPE], Personalised Health Monitoring and Decision Support Based On Artificial Intelligence and Holistic Health Records [iHelp], Central repository for digital pathology [Bigpicture], Piloting an infrastructure for the secondary use of health data [HealthData@EU] pilot, and improving cancer diagnosis and prediction with AI and big data [INCISIVE]) to guide in shaping the EHDS2 interoperability framework. Building upon the foundations laid by the Towards the European Health Data Space (TEHDAS) joint action (JA) and the new European Interoperability Framework (EIF), the toolkit incorporates several key innovative features. First, it provides interactive and user-friendly entry modules to support European projects in creating their own interoperability frameworks aligned with the evolving EHDS2 requirements technical and governance requirements. Second, it guides projects in navigating the complex landscape of health data standards, emphasizing the need for a balanced approach to implementing the EHDS2 recommended standards for data discoverability and sharing. Third, the toolkit fosters collaboration and knowledge sharing among projects by enabling them to share their experiences and best practices in implementing standards and addressing interoperability challenges. Finally, the toolkit recognizes the dynamic nature of the EHDS2 and the evolving regulatory landscape, including the impact of AI regulations and related standards. This allows for continuous adaptation and improvement, ensuring the toolkit remains relevant and useful for future projects. In collaboration with HSbooster.eu, the toolkit will be disseminated to a wider audience of projects and experts, facilitating broader feedback and continuous improvement. This collaborative approach will foster harmonized standards implementation across projects that ultimately contribute to the development of a common EHDS2 interoperability framework.

Over the last few years, the European Union (EU) has placed significant emphasis on the interoperability of critical infrastructures (CIs). One of the main CI transportation infrastructures are ports. The control systems managing such infrastructures are constantly evolving and handle diverse sets of people, data, and processes. Additionally, interdependencies among different infrastructures can lead to discrepancies in data models that propagate and intensify across interconnected systems. This article introduces “BigDaM”, a Big Data Management framework for critical infrastructures. It is a cutting-edge data model that adheres to the latest technological standards and aims to consolidate APIs and services within highly complex CI infrastructures. Our approach takes a bottom-up perspective, treating each service interconnection as an autonomous entity that must align with the proposed common vocabulary and data model. By injecting strict guidelines into the service/component development’s lifecycle, we explicitly promote interoperability among the services within critical infrastructure ecosystems. This approach facilitates the exchange and reuse of data from a shared repository among developers, small and medium-sized enterprises (SMEs), and large vendors. Business challenges have also been taken into account, in order to link the generated data assets of CIs with the business world. The complete framework has been tested in the main EU ports, part of the transportation sector of CIs. Performance evaluation and the aforementioned testing is also being analyzed, highlighting the capabilities of the proposed approach.

measurement, and testing-related efforts. These external perspectives can come from industry, academia, government, and others.

Recent advances in large-scale experimental facilities ushered in an era of data-driven science. These large-scale data increase the opportunity to answer many fundamental questions in basic science. However, these data pose new challenges to the scientific community in terms of their optimal processing and transfer. Consequently, scientists are in dire need of robust high performance computing (HPC) solutions that can scale with terabytes of data. In this thesis, I address the challenges in three major aspects of scientific big data processing as follows: 1) Developing scalable software and algorithms for data- and compute-intensive scientific applications. 2) Proposing new cluster architectures that these software tools need for good performance. 3) Transferring the big scientific dataset among clusters situated at geographically disparate locations. In the first part, I develop scalable algorithms to process huge amounts of scientific big data using the power of recent analytic tools such as, Hadoop, Giraph, NoSQL, etc. At a broader level, these algorithms take the advantage of locality-based computing that can scale with increasing amount of data. The thesis mainly addresses the challenges involved in large-scale genome analysis applications such as, genomic error correction and genome assembly which made their way to the forefront of big data challenges recently. In the second part of the thesis, I perform a systematic benchmark study using the above-mentioned algorithms on different distributed cyberinfrastructures to pinpoint the limitations in a traditional HPC cluster to process big data. Then I propose the solution to those limitations by balancing the I/O bandwidth of the solid state drive (SSD) with the computational speed of high-performance CPUs. A theoretical model has been also proposed to help the HPC system designers who are striving for system balance. In the third part of the thesis, I develop a high throughput architecture for transferring these big scientific datasets among geographically disparate clusters. The architecture leverages the power of Ethereum's Blockchain technology and Swarm's peer-to-peer (P2P) storage technology to transfer the data in secure, tamper-proof fashion. Instead of optimizing the computation in a single cluster, in this part, my major motivation is to foster translational research and data interoperability in collaboration with multiple institutions.

Since its first release in 1998, the Digital 21 Strategy has been serving the blueprint for Hong Kong to develop the information and communications technology infrastructure. With government leadership as an important component, the Strategy has put forward the development of an e-government that can realize one-stop delivery of public electronic services. In 2003, Hong Kong Government has launched the Interoperability Framework (IF) as an e-government initiative to facilitate implementation of cross-department joined-up government services. IF comprises two parts: (1) definition of a set of recommended technical specifications as a single point of reference for departments and contractors to implement joined-up projects, (2) a framework for formulating and managing XML message standards for G2G and G2B data exchange. This paper discusses about the second part of the IF, mainly on the XML Schema and Design Guide, and its reallife applications.

One of the main obstacles towards the promotion of IoT adoption and innovation is data interoperability. Facilitating cross-domain interoperability is expected to be the core element for the realisation of the next generation of the IoT computing paradigm that is already taking shape under the name of Internet of Everything (IoE). In this article, an analysis of the current status on IoT semantic interoperability is presented that leads to the identification of a set of generic requirements that act as fundamental design principles for the specification of interoperability enabling solutions. In addition, an extension of NGSIv2 data model and API (de-facto) standards is proposed aiming to bridge the gap among IoT and social media and hence to integrate user communities with cyber-physical systems. These specifications have been utilised for the implementation of the IoT2Edge interoperability enabling mechanism which is evaluated within the context of a catastrophic wildfire incident that took place in Greece on July 2018. Weather data, social media activity, video recordings from the fire, sensor measurements and satellite data, linked to the location and the time of this fire incident have been collected, modeled in a uniform manner and fed to an early fire detection decision support system. The findings of the experiment certify that achieving minimum data interoperability with light-weight, plug-n-play mechanisms can be realised with significant benefits for our society.

Digital governance reforms are frequently evaluated through visible artefacts, such as portals, apps, and platforms, yet sustained public value depends on institutional mechanisms that secure lawful data use, interoperability, accountability, and inclusion. The goal of this study is to structure international learning in digital governance through three interlinked lenses, namely lessons, policy transfer, and adaptation to national context. Methodologically, the study applies qualitative policy synthesis and comparative institutional reasoning to trace how international practices are translated into domestic reform design. Within this approach, lessons are formulated as conditional propositions that connect outcomes to causal mechanisms and to enabling conditions, and these propositions are assessed through triangulated evidence rather than single-source success narratives. Policy transfer is treated as a process of institutional translation shaped by actors, channels, and incentives, which explains why reform elements often travel as modular components rather than as complete models. Adaptation is examined through dependency and fit assessment, focusing on whether the receiving context can operationalize the prerequisites that make transferred mechanisms work in practice. The analysis indicates that lessons become transferable only when outcomes, mechanisms, enabling conditions, and evidence standards are made explicit, because this reduces artefact imitation and improves the precision of reform design. It also shows that policy transfer most commonly moves components such as interoperability frameworks, service standards, and digital identity, but implementation success depends on institutionalizing hidden dependencies, including clear legal bases, enforceable authority, data stewardship, procurement discipline, and the sequencing of safeguards and capacity building before scaling. Overall, international practice is most useful when knowledge travels as enforceable routines and institutional capabilities and when performance management prioritizes completion, administrative burden reduction, equity, trust, and resilience over digitization counts. Future research should refine comparable indicators for inclusion and trust, test causal links between interoperability investments and burden reduction, and examine how AI-supported administrative decisions reshape accountability and legitimacy.

In recent years, the application of Big Data analytics in public safety and emergency management has become increasingly important. With the growing volume of data generated by various sources such as social media, IoT devices, sensors, and emergency calls, Big Data technologies offer the potential to enhance decision-making processes, improve response times, and mitigate risks during emergencies. This paper explores the role of Big Data analytics in public safety, highlighting its use in disaster prediction, resource allocation, real-time decision-making, and crisis management. The paper also discusses the challenges faced by agencies in implementing Big Data solutions, including data privacy, security concerns, and interoperability issues, and provides recommendations for overcoming these obstacles to maximize the benefits of Big Data in emergency management.

Earth observation (EO) data are remote sensing image data obtained by air-borne or space-borne sensors, which has the characters of distribution and heterogeneity. Because these data are owned by different space agencies. With Petabyte-scale data volumes of Earth Observation (EO) obtained, the EO data has been one of the most important part of big data. The characters of distribution and heterogeneity of big EO data storage and processing systems owned by different space agencies brings challenges to integrate and use these data. To meet the big EO data challenges, the new generation Spatial data infrastructure is needed to provide the solution and implementation technique to big EO data storage, access, process and analysis. The concept and architecture of Next-generation national EO Spatial Data Infrastructure (EO-SDI) is proposed in the paper. Key techniques are introduced to solve the complexity of distributed and heterogeneous EO data integration and on-demand processing. An interoperable national EO data system of systems is developed to integrate distributed EO archives of eight EO data centers.

Complex global food chains and rising demand for high-quality halal product requires better and more efficient management. Traditionally, the Halal Assurance System (HAS) and Quality Management System (QMS) have been implemented separately, resulting in overlapping processes and redundant documentation. In the era of digital transformation and Industry 4.0, integrating these systems digitally improves traceability, compliance, and overall organization’s performance. This review aimed to examine current trends, technologies, and research gaps related to the digitalization of integrated Halal and Quality Management Systems (HAS–QMS). A comprehensive literature analysis highlights the role of emerging technologies—such as blockchain, the Internet of Things (IoT), artificial intelligence (AI), cloud computing, and big data analytics—in supporting halal integrity and quality assurance across the food value chain. The findings indicated that digital integration facilitates real-time monitoring, automated documentation, and improved audit readiness. However, challenges persist in areas such as data interoperability, standard harmonization, regulatory acceptance, and the digital readiness of small and medium enterprises. The paper proposed a conceptual framework for the digitalized integration of HAS and QMS, aligning technological capabilities with halal and quality compliance requirements. To align with industry 5.0, future research should standardize models, test performance, and integrate sustainability and ethical standards.

Data interoperability is a major issue in data management for data science and big data analytics. Probabilistic data integration (PDI) is a specific kind of data integration where extraction and integration problems such as inconsistency and uncertainty are handled by means of a probabilistic data representation. This allows a data integration process with two phases: (1) a quick partial integration where data quality problems are represented as uncertainty in the resulting integrated data, and (2) using the uncertain data and continuously improving its quality as more evidence is gathered. The main contribution of this paper is an iterative approach for incorporating evidence of users in the probabilistically integrated data. Evidence can be specified as hard or soft rules (i.e., rules that are uncertain themselves).