SmartLake: A smart datalake for short and long tail data types

The Breagh tight gas field is located in the Southern North Sea and currently delivers gas from 8 wells via a normally unmanned installation (NUI) to a processing facility in Teesside. In the coming years there is scope to install a second NUI and drill an additional nine wells. Platform and onshore sensors deliver high-frequency pressure, temperature, and flow rate data for each well, in addition to key facility data. The high volume and variable quality of this high frequency and real time data directly affects the way in which daily engineering activities are performed. Originally, the engineering team used conventional methods of data handling, storage and management, including manual manipulation, and spreadsheet computations. This was a time consuming exercise which led to excessive replication of data and a lack of audit trail behind the data used for any analysis. Allocation of field production to individual wells is a key production engineering workflow. The flow rate data is the prime input for every piece of analysis conducted by the reservoir engineering team; decline curve analysis, rate and pressure transient analysis, material balance and history matching the numerical simulation model. The results of this analysis directly impacts upon routine production strategy & optimisation, workovers, well planning, future development strategy and reserves. The platform Venturi flow meters have proven unreliable throughout the production life of Breagh, leading to the evaluation of several alternative allocation methods, all of which are manual and time intensive. This paper presents workflows for production data management and production allocation. Various production allocation methods are used and this paper demonstrates the impact of data accuracy on long-term development decisions, and short term interventions due to flow assurance issues. Thereby, the asset team can shift from a reactive, to a proactive position and generate value for the business.

Workflow engines used to script scientific experiments involving numerical simulation, data analysis, instruments, edge sensors, and artificial intelligence have to deal with the complexities of hardware, software, resource availability, and the collaborative nature of science. In this paper, we survey workflow engines used in data-intensive and compute-intensive discovery pipelines from scientific disciplines such as astronomy, high energy physics, earth system science, bio-medicine, and material science and present a qualitative analysis of their respective capabilities. We compare 5 popular workflow engines and their differentiated approach to job orchestration, job launching, data management and provenance, security authentication, ease-ofuse, workflow description, and scripting semantics. The comparisons presented in this paper allow practitioners to choose the appropriate engine for their scientific experiment and lead to recommendations for future work.

The healthcare industry has been using data and technology driven innovation for a long time now with Evidence Based Medicine (EBM) being at the helm. Effectively using technology is vital towards successful data management in healthcare. A convenient approach that can collect, store, evaluate, and analyze health data will be beneficial to all the stakeholders in the healthcare delivery chain. With the advancement, clubbed with the availability and affordability of technology, it has become an effective tool for providing the right- care backed with accurate, consolidated evidence. Such approach not only helps in improving the quality of care, it also helps in early detection of diseases and effective treatment, Big Data is providing the necessary data management tools for the same. A simple Extract-Transform-Load (ETL) procedure using Hadoop can be used. As Big data analyses large amounts of data to uncover hidden patterns, correlations and other insights, its role in making the right, predictive decisions for quality care of the patients and better health planning is possible. Big data technologies such as Hadoop bring significant cost advantages when it comes to storing large amounts of data and combined with the speed of Hadoop and in-memory analytics, with the ability to analyze new sources of data, we are able to analyze information instantly – and make decisions which is very essential in the healthcare provider segment. Hadoop does not only store, but also process any and many file data: big or small, can be plain text files or binary files like images and even multiple different version of some particular data format across different time periods. It consists of three core components: a distributed file system, a parallel programming framework, and a resource/job management system. In healthcare, patient records, health plans, insurance information and other types of information is extremely difficult to manage – but, this very data is full of key insights, the need is for an effective data management and analytics that can be pragmatic. By analyzing large amounts of information – both structured and unstructured – rapidly, health care providers can provide lifesaving diagnoses. Keywords: Big Data, EBM, ETL, Hadoop

Most of the domains of science are facing an explosion in the data that they have to collect and process in order to conduct research. This is true for both: scientific domains dealing with experimental data, e.g. biology, sociology, astronomy etc, but also scientific domains dealing with simulation data, e.g. seismology, physics etc. To maximize the potential outcome of scientific data analysis, respective data management applications need to fulfil the following coarse tasks: fast on-demand data processing, and effective storage and consolidation of diverse data collections. These two tasks are in general hard to realize because of: (a) the big data size, (b) the diversity of data formats, (c) their conceptual dependencies, (d) disperse data locations, and (e) intensive and systematic nature of scientific queries. We present the characteristics of big scientific data collections and their necessities in terms of data management. Based on this discussion, we discuss the structure of a framework for the processing and consolidation of heterogeneous scientific data collections. Such a framework aims to mediate between the user and a set of available data management technologies, such as relational DBMSs, key-value stores and column stores, in order to efficiently direct data management operations (insertions, updates) and especially requests (queries) to the appropriate data management application. The framework aims to distribute, dissect, and schedule data management actions, as well as integrate results, in a way that reduces response time. This entails the accommodation of methods for the selective parallelism and serialization depending on partial results and response times. Also, this entails the accommodation of methods for the gradual alteration of data formats and storage, e.g. storage of semi-structured data or raw data in files into relational databases. Furthermore, we discuss the processing of scientific query bulks or workflows with the possibility to retrieve early partial results and calibrate query parameters.

This report contains the groundwater and surface water monitoring data that were obtained during calendar year (CY) 2006 at the U.S. Department of Energy (DOE) Y-12 National Security Complex (hereafter referenced as Y-12) on the DOE Oak Ridge Reservation (ORR) in Oak Ridge, Tennessee. The CY 2006 monitoring data were obtained from wells, springs, and surface water sampling locations in three hydrogeologic regimes at Y-12 (Figure A.1). The Bear Creek Hydrogeologic Regime (Bear Creek Regime) encompasses a section of Bear Creek Valley (BCV) between the west end of Y-12 and the west end of the Bear Creek Watershed (directions are in reference to the Y-12 grid system). The Upper East Fork Poplar Creek Hydrogeologic Regime (East Fork Regime) encompasses the Y-12 industrial facilities and support structures in BCV. The Chestnut Ridge Hydrogeologic Regime (Chestnut Ridge Regime) encompasses a section of Chestnut Ridge directly south of Y-12. Section 2 of this report provides background information pertinent to groundwater and surface water quality monitoring in each hydrogeologic regime, including the topography and bedrock geology, surface water drainage, groundwater system, and extent of groundwater contamination. The CY 2006 groundwater and surface water monitoring data in this report were obtained from sampling and analysis activities implemented under the Y-12 Groundwater Protection Program (GWPP) managed by BWXT Y-12, L.L.C. (BWXT), and from sampling and analysis activities implemented under several monitoring programs managed by Bechtel Jacobs Company LLC (BJC). Cooperative implementation of the monitoring programs directed by the Y-12 GWPP and BJC (i.e., preparing SAPs, coordinating sample collection, and sharing data) ensures that the CY 2006 monitoring results fulfill requirements of all the applicable monitoring drivers with no duplication of sampling and analysis efforts. Section 3 of this report contains a summary of information regarding the groundwater and surface water sampling and analysis activities implemented under the Y-12 GWPP including sampling locations and frequency; quality assurance (QA)/quality control (QC) sampling; sample collection and handling; field measurements and laboratory analytes; data management and data quality objective (DQO) evaluation; and groundwater elevation monitoring. However, this report does not include equivalent information regarding the groundwater and surface water sampling and analysis activities associated with the monitoring programs implemented by BJC. Such details are deferred to the respective programmatic plans and reports issued by BJC (see Section 3.0). Collectively, the groundwater and surface water monitoring data obtained during CY 2006 by the Y-12 GWPP and BJC address DOE Order 450.1 (Environmental Protection Program) requirements for monitoring groundwater and surface water quality in areas: (1) which are, or could be, affected by operations at Y-12 (surveillance monitoring); and (2) where contaminants from Y-12 are most likely to migrate beyond the boundaries of the ORR (exit pathway/perimeter monitoring). Section 4 of this report presents a summary evaluation of the monitoring data with regard to the respective objectives of surveillance monitoring and exit pathway/perimeter monitoring, based on the analytical results for the principal groundwater and surface water contaminants at Y-12: nitrate, uranium, volatile organic compounds (VOCs), gross alpha activity, and gross beta activity. Section 5 of this report summarizes the most pertinent findings regarding the principal contaminants, along with recommendations proposed for ongoing groundwater and surface water quality monitoring performed under the Y-12 GWPP. Narrative sections of this report reference several appendices. Figures (maps and diagrams) and tables (excluding data summary tables presented in the narrative sections) are in Appendix A and Appendix B, respectively. Appendix C contains construction details for the wells in each regime that were sampled during CY 2006 by either the Y-12 GWPP or BJC. Field measurements recorded during collection of the groundwater and surface water samples and results of laboratory analyses of the samples are in Appendix D (Bear Creek Regime), Appendix E (East Fork Regime and surrounding areas), and Appendix F (Chestnut Ridge Regime). Appendix G contains data for the QA/QC samples associated with monitoring performed in each regime by the Y-12 GWPP.

Between 1953 and 1970, studies on the long-term effects of external x-ray and {gamma} irradiation on inbred and hybrid mouse stocks were carried out at the Biological and Medical Research Division, Argonne National Laboratory. The results of these studies, plus the mating, litter, and pre-experimental stock records, were routinely coded on IBM cards for statistical analysis and record maintenance. Also retained were the survival data from studies performed in the period 1943-1953 at the National Cancer Institute, National Institutes of Health, Bethesda, Maryland. The card-image data files have been corrected where necessary and refiled on hard disks for long-term storage and ease of accessibility. In this report, the individual studies and data files are described, and pertinent factors regarding caging, husbandry, radiation procedures, choice of animals, and other logistical details are summarized. Some of the findings are also presented. Descriptions of the different mouse stocks and hybrids are included in an appendix; more than three dozen stocks were involved in these studies. Two other appendices detail the data files in their original card-image format and the numerical codes used to describe the animal`s exit from an experiment and, for some studies, any associated pathologic findings. Tabular summaries of sample sizes, dose levels, and other variables are also given to assist investigators in their selection of data for analysis. The archive is open to any investigator with legitimate interests and a willingness to collaborate and acknowledge the source of the data and to recognize appropriate conditions or caveats.

Geospatial datasets are currently constructed, managed, and utilized individually according to the spatial scale of the real world, such as the ground/surface/underground or indoor/outdoor, as well the particular purpose of the geospatial data used for location-based services. In addition, LBS applications use an optimal data model and data format according to their particular purpose, and thus, various datasets exist to represent the same spatial features. Such duplicated geospatial datasets and geographical feature-based GIS data cause serious problems in the financial area, compatibility issues among LBS systems, and data integration problems among the various geospatial datasets generated independently for different systems. We propose a geospatial data fusion model called the topological relation-based data fusion model (TRDFM) using topological relations among spatial objects in order to integrate different geospatial datasets and different data formats. The proposed model is a geospatial data fusion model implemented in a spatial information application and is used to directly provide spatial information-based services without data conversion or exchange of geometric data generated by different data models. The proposed method was developed based on an extension of the AnchorNode concept of IndoorGML. The topological relationships among spatial objects are defined and described based upon the basic concept of IndoorGML. This paper describes the concept of the proposed TRDFM and shows an experimental implementation of the proposed data fusion model using commercial 3D GIS software. Finally, the limitations of this study and areas of future research are summarized.

Chapter 19 and Chapter 20 are on the emerging approaches to data management for a new geospatial science research. This chapter gives the essences and the methodologies of data matter perspective, and it has two sections; Section A highlights the subject theme; the essences of geospatial science data matter; while Section B expands that into the geospatial science data methodologies. Chapter 20 is the about the data management optimization perspective. It has only one section; Section C, that develops further the essence and methodologies of geospatial data cultivated in these two previous sections. The whole analytical discussion is in the emerging fields and how they had optimized and totally changed the geospatial science data management panorama.

Geospatial information has been indispensable for many application fields, including traffic planning, urban planning, and energy management. Geospatial data are mainly stored in relational databases that have been developed over several decades, and most geographic information applications are desktop applications. With the arrival of big data, geospatial information applications are also being modified into, e.g., mobile platforms and Geospatial Web Services, which require changeable data schemas, faster query response times, and more flexible scalability than traditional spatial relational databases currently have. To respond to these new requirements, NoSQL (Not only SQL) databases are now being adopted for geospatial data storage, management, and queries. This paper reviews state-of-the-art geospatial data processing in the 10 most popular NoSQL databases. We summarize the supported geometry objects, main geometry functions, spatial indexes, query languages, and data formats of these 10 NoSQL databases. Moreover, the pros and cons of these NoSQL databases are analyzed in terms of geospatial data processing. A literature review and analysis showed that current document databases may be more suitable for massive geospatial data processing than are other NoSQL databases due to their comprehensive support for geometry objects and data formats and their performance, geospatial functions, index methods, and academic development. However, depending on the application scenarios, graph databases, key-value, and wide column databases have their own advantages.

Geoinformatics researchers and practitioners have been facing challenges of geospatial data processing ever since the debut of digital mapping, the predecessor term for today’s geospatial information systems. Over time, due to the advancement in data collection technologies (such as satellite imagery and GPS), challenges of geospatial data processing have become widespread. Today, every individual or organization that either routinely or sporadically must handle geospatial data has a better understanding and much appreciation for the complex nature of geospatial data processing. Of several characteristics of geospatial phenomena that make geospatial data to be complex for processing, data-intensive computing, one of the main characteristics of geospatial big data that can be addressed by employing special computing hardware and advanced computing techniques, is discussed in this chapter. To that end, since geospatial big data is defined and different approaches for addressing challenges of this emergent paradigm are discussed in all other chapters of this book, in this chapter high-performance computing (HPC), commonly considered as one main approach for handling data-intensive problems, is focused.

Freshwater aquaculture in Dharmasraya Regency plays a vital role in local food security and economic development, yet data management practices remain limited to static tabular records. This lack of spatial representation hampers effective planning, monitoring, and decision-making by the Fisheries and Food Agency. To address this issue, the present study develops a web-based Geographic Information System (GIS) tailored for freshwater aquaculture that integrates spatial visualization with comprehensive attribute data, including fish species, cultivation methods, pond size, and production outputs. The novelty of this research lies in three aspects: the integration of attribute and geospatial data into a unified platform, district-level filtering and route mapping to aquaculture sites, and the use of open-source technologies such as Leaflet.js and Chart.js to ensure interactivity, accessibility, and scalability. The system was developed using the Waterfall model, covering requirement analysis, design, implementation, and black-box testing for validation. Results show that the GIS successfully displays aquaculture distribution maps, interactive production graphs, and user-friendly data management features that align with stakeholder needs. Compared to previous GIS applications, which often emphasize coastal aquaculture or rely on satellite imagery, this system directly supports inland aquaculture management at the district level. Beyond technical contributions, the platform delivers practical benefits by enabling local authorities to monitor aquaculture spatially, improving transparency for the public, and supporting sustainable fisheries development. This study thus demonstrates how a district-level WebGIS can bridge the gap between static data and actionable decision support in aquaculture management.

Input/output (I/O) of geospatial raster data often becomes the bottleneck of parallel geospatial processing due to the large data size and diverse formats of raster data. The open‐source Geospatial Data Abstraction Library (GDAL), which has been widely used to access diverse formats of geospatial raster data, has been applied recently to parallel geospatial raster processing. This article first explores the efficiency and feasibility of parallel raster I/O using GDAL under three common ways of domain decomposition: row‐wise, column‐wise, and block‐wise. Experimental results show that parallel raster I/O using GDAL under column‐wise or block‐wise domain decomposition is highly inefficient and cannot achieve correct output, although GDAL performs well under row‐wise domain decomposition. The reasons for this problem with GDAL are then analyzed and a two‐phase I/O strategy is proposed, designed to overcome this problem. A data redistribution module based on the proposed I/O strategy is implemented for GDAL using a message‐passing‐interface (MPI) programming model. Experimental results show that the data redistribution module is effective.

Australia invests significant resources in environmental data acquisition, management and publication. Although data are abundant, users are typically hampered by an inability to discover, access and re-use the data. The National Environmental Information Infrastructure (NEII) activity will improve the effectiveness and efficiency of discovering, accessing and re-using environmental data beyond its primary purpose. The NEII is envisioned as a federation of environmental data nodes adopting common standards to create a data infrastructure with an initial focus on nationally significant environmental data. Its primary focus is on the discovery and re-use of national environmental data that is already well-managed, but that currently has limited application beyond its original business purpose. In common with spatial data infrastructures, the NEII encompasses common data models, exchange formats and standard network protocols along with centralised catalogues of uniform metadata descriptions. It also includes standardised models for describing environmental measurements, monitoring sites and methods used to observe the environment to address the unique requirements of environmental information. These architectural elements are more fully described in the NEII Reference Architecture (Bureau of Meteorology, 2014b, www.neii.gov.au). The Bureau of Meteorology is leading the development of the NEII including provision of core coordination and integration infrastructure, as well as the governance and collaboration framework for its development and operation. However its ongoing success is dependent on establishing enduring partnerships with major national environmental information organisations. The NEII Programme is framed around five focus areas including (a) Communication, (b) Engagement, (c) Policies and frameworks, (d) Data management, and (e) ICT build. This paper provides a more detailed overview of one component of the Data Management area regarding the development of the NEII Conformance Framework. Conformance means following established guidelines, specifications and standards or working towards them. A conformance framework is a structured set of guidelines that detail the levels at which a participant provides services that comply with the NEII architecture. The NEII Conformance Framework has been developed to (a) enable partners to set priorities against specific expectations that will jointly lead to developing a sustainable federated environmental information system; (b) enable users of the NEII to rapidly assess the suitability of data for their business needs; and (c) provide a common measurement system to monitor progress in the development of the NEII. The framework adopts a capability maturity model approach to describe the steady improvement required for NEII nodes to progress from a basic data release (experimental data service with limited operational support) to an enduring federated system providing environmental information. The design of the model is informed by other maturity-based approaches for open data such as the Open Data Institute's certification approach (Open Data Institute 2015). The paper also presents a worked example of the conformance framework against the Bureau of Meteorology's Australian Hydrologic Geospatial Fabric NEII data services. The paper concludes with an overview of major learnings from the programme to-date. These relate primarily to reducing the cost of participation in NEII by data custodians given the challenge of delivering data using NEII standards; and developing approaches to better support users to use the portfolio of NEII data given some of the methods of data delivery may be new to non-technical users. The NEII programme has prioritised both these and developed work packages to achieve improvements.

This chapter comprehensively explores the evolving landscape of geospatial data storage and management, elucidating its critical role in unlocking the transformative potential of geospatial information across diverse applications. Focusing on recent advancements and future directions, this chapter delves into the intricacies of geospatial data management, encompassing vector data, raster data, and point-cloud data storage. Challenges such as data volume, variety, velocity, veracity, and standardization are meticulously addressed, highlighting the complexities of managing geospatial data effectively. Additionally, the chapter examines various storage options, including file formats, database systems, and cloud storage services, offering insights into their advantages and disadvantages. Moreover, it underscores the pivotal aspects of geospatial data management, such as data quality, metadata, and data integration.

Effective integration of petroleum engineering workflows with robust data management practices is essential for maximizing the value of seismic data in both exploration and production phases. This review synthesizes current best practices for data governance, storage architectures, and collaborative platforms that enable seamless access to high?fidelity seismic volumes. We examine advanced techniques for optimizing seismic acquisition parameters, processing algorithms, and attribute extraction to enhance reservoir characterization and drill planning. Emphasis is placed on scalable data pipelines, cloud?native infrastructures, and machine?learning–driven analytics that accelerate decision?making while ensuring data integrity and reproducibility. Through a series of industry case studies, we highlight how integrated engineering–data ecosystems have improved exploration success rates, reduced nonproductive time, and boosted hydrocarbon recovery. Finally, we discuss emerging trends—such as digital twins, real?time data streaming, and AI?augmented interpretation—and propose a roadmap for future research that bridges the gap between unconventional reservoir challenges and next?generation data management solutions.