Provenance as Essential Infrastructure for Data Lakes

The selection of the best data storage and management system stands essential because machine learning (ML) continues expanding its industry-driven innovation. Today's two most prevalent large-scale data processing systems are data lakes and data warehouses, which provide unique strengths and barriers when applied to ML workloads. This paper thoroughly compares data lakes and data warehouses by Analyzing their operational speed, abilities, and price efficiency alongside data, management controls, and ML integration capabilities. Data lakes showcase their superiority in processing unstructured together with semi-structured information because they serve deep learning and big data analytics requirements. A data warehouse offers optimized querying and structured storage, which is suitable for traditional business intelligence applications and ML platforms. The execution speed of data warehouses is faster compared to data lakes, but data lakes enable enhanced real-time abilities and flexibility for large-scale ML work. The research approach consists of conducting a feature-based examination of both architectures, combined with real-world examples, performance scaling data, and cost measurements. This research finds that AI analytics operate most successfully through data lakes; however, structured ML jobs need data warehouses for efficient operation. The combination of data lakehouse technology presents a new possibility for joining both paradigms to create more efficient environments for machine learning applications.

Modeling metadata in data lakes—A generic model

The deployment and optimization of data lakes in financial data management is investigated in this research article. Concerning an ever-growing volume and diversity of data, conventional data management technologies are showing insufficient capability for financial organizations. Providing a scalable and flexible infrastructure for storing and evaluating enormous volumes of organized and unstructured data, data lakes provide a good answer. With an eye on data governance, security, and analytics, this paper investigates many approaches for building, running, and managing data lakes in the financial sector. By means of an extensive literature study and case studies, we pinpoint areas of best practices and difficulties in implementing data lake solutions for financial institutions. The work ends with suggestions for further studies and useful field applications.

In medical domain, huge amounts of data are generated at all times. These data are usually difficult to access, with poor data quality and many data islands. Besides, with a wide range of sources and complex structure, these data contain essential information and are difficult to manage. However, few existing data management frameworks based on Data Lake excel in solving the persistence and the analysis efficiency for medical multi-source heterogeneous data. In this paper, we propose an efficient Multi-source Heterogeneous Data Lake Platform (MHDP) to realize the efficient medical data management. Firstly, we propose an efficient and unified method based on Data Lake to store data of different types and different sources persistently. Secondly, based on the unified data store, an efficient multi-source heterogeneous data fusion is implemented to effectively manage data. Finally, an efficient data query strategy is carried out to assist doctors in medical decision-making. In-depth analysis on applications shows that MHDP delivers better performance for data management in medical domain.

Nowadays, the IoT ecosystem is evolving rapidly, with multiple heterogeneous sources producing high volumes of data and processes transforming this data into meaningful or “smart” information. These volumes of data, including IoT data, need to be stored in repositories that can host raw, unprocessed, relational and non-relational types of data, such as Data Lakes. Due to the weakness of metadata management, security & access control is one of the main challenges of Big Data storage architectures as Data Lakes can be replaced without oversight of the contents. Recently, the Blockchain technology has been introduced as an effective solution to build trust between different entities, where trust is either nonexistent or unproven, and to address security and privacy concerns. In this paper we introduce DLMetaChain, an extended Data Lake metadata mechanism that consists of data from heterogeneous data sources which interact with IoT data. The extended mechanism mainly focuses on developing an architecture to ensure that the data in the Data Lake is not modified or altered by taking into advantage the capabilities of the Blockchain.

Modern physics experiments are often led by large collaborations including scientists and institutions from different parts of the world. To cope with the ever increasing computing and storage demands, computing resources are nowadays offered as part of a distributed infrastructure. A critical challenge for present and future experiments is an efficient and reliable data distribution and access system. Rucio is a framework for data management, access and distribution, originally developed by the ATLAS experiment and later adopted by several scientific collaborations. It is currently used by the second generation gravitational wave (GW) detectors LIGO and Virgo, and is being evaluated for the Einstein Telescope (ET), the future third generation interferometer in preparation in Europe. In this contribution, the on-going R&D for integration of Rucio within the ET computing infrastructure will be outlined. The activities include the setup of a Data Lake based on Rucio for future ET Mock Data Challenges and the customization of Rucio features for the GW community. The evaluation of RucioFS, a FUSE mount filesystem to provide the user with the well known POSIX-like view of the Rucio catalogue, and the possible implementation of additional features in it will be described. Moreover, the need for the integration of the ET Data Lake with mock Data Lakes belonging to other experiments within the astrophysics and GW communities will be addressed. This is a critical feature as data analysts in this field often require access to open data from other experiments for sky localisation and multi-messenger analysis.

The impact of next-generation sequencing on genomics

Data quality is paramount in data-driven decision-making processes, especially when dealing with large volumes of data in environments like data warehouses and data lakes. These systems store vast amounts of raw and processed data from multiple sources, making data management and quality assurance complex yet critical. With the growing adoption of Artificial Intelligence (AI), new techniques and tools have emerged that can significantly enhance data quality. This paper discusses how AI can improve the quality of data within both data warehouses and data lakes by automating data cleansing, validation, anomaly detection, and ensuring consistency. It explores the benefits, challenges, and methodologies for integrating AI tools into these systems. Keywords: Data quality, AI in data quality, data warehouse, data lakes, big data, data processing, data cleansing, data profiling.

The energy data scope is very broad including oil and gas, coal, minerals, new energy, renewable and conversion energy, electricity, and others. The different volume, variety, veracity, and velocity of data have challenge to address with the energy data model. The works focus on the development of data model for big data storage implementation schemes in the energy data. The data model has produces by analyzed the national energy big data architecture, develop the data mapping and correlation, big data master data management with energy industry standardization, big data management portal for upstream energy and downstream energy. The research has goal to establishing a foundation for building technology and big data management in the energy sector which includes petroleum, coal, geothermal and renewable energy, and in the future can be the basis for predictive analysis and national energy production.

Background: With the development of information technology and the popularization of the Internet, a large amount of data has been generated and collected. These data typically have the characteristics of massive, diverse, and high-speed growth, and traditional data storage devices and technologies are no longer able to meet the storage and analysis needs of data. And people need more and more storage space to store personal files, photos, videos, and other data. Traditional personal computer hard drives have limited capacity and cannot meet the growing data demand. The emergence of cloud storage provides more flexible and efficient solutions for big data storage and analysis, as well as larger capacity and more convenient access methods, solving the challenges of personal storage and sharing. Method: Based on comparative analysis, explore the advantages and disadvantages of cloud storage in the fields of personal data storage and sharing, as well as big data storage and analysis. Result: Cloud storage has certain advantages and limitations in the fields of personal storage and sharing, as well as big data storage and analysis. Users need to consider these factors comprehensively based on their own needs and actual situation.

The phenomenon of big data is currently agrowing topic in the world of information technology. From someof the literature mentioned that manage big data can createsignificant value for the world economy, improving productivityand competitiveness of enterprises and the public sector as wellas creating a large economic surplus for consumers. However,from some of the information obtained, big data is still not widelyapplied in the company or organization. This study aimed toexplore more information about the big data and proceed withmaking an application prototype big data management. Toexperiment with big data storage that is database, this researchuse NoSQL database technology that can map the needs of bothstructured and unstructured. And this research will be carriedout migration of Relational Database (RDBMS) into the databaseMongoDB. Prototype will be create with the object of study isstructured and unstructured data. The expected result of thisresearch is a model or prototype of big data management thatcan help organizations and companies (especially education) tomake decisions based on various types of data.

In the new environment of intelligent manufacturing, enterprise quality data has increased exponentially. How to manage, utilize, mine and analyze quality data has become a key issue in modern quality management. This article expands the definition of the product quality data package in the intelligent manufacturing environment, and proposes a big data-based product quality data package construction and management solution, gives a quality data fusion method based on business decision, outlines the application of quality data package. Finally, a chip manufacturing company was used to verify the feasibility of the product quality data package construction and management plan.

Spatial big data architecture: From Data Warehouses and Data Lakes to the LakeHouse

In recent years, many fields that experience a sudden proliferation of data, which increases the volume of data that must be processed and the variety of formats the data is stored in have been identified. This causes pressure on existing compute infrastructures and data analysis methods, as more and more data are considered as a useful source of information for making critical decisions in particular fields. Among these fields exist several areas related to human life, e.g., various branches of medicine, where the uncertainty of data complicates the data analysis, and where the inclusion of fuzzy expert knowledge in data processing brings many advantages. In this paper, we show how fuzzy techniques can be incorporated in big data analytics carried out with the declarative U-SQL language over a big data lake located on the cloud. We define the concept of big data lake together with the Extract, Process, and Store process performed while schematizing and processing data from the Data Lake, and while storing results of the processing. Our solution, developed as a Fuzzy Search Library for Data Lake, introduces the possibility of massively parallel, declarative querying of big data lake with simple and complex fuzzy search criteria, using fuzzy linguistic terms in various data transformations, and fuzzy grouping. Presented ideas are exemplified by a distributed analysis of large volumes of biomedical data on Microsoft Azure cloud. Results of performed tests confirm that the presented solution is highly scalable on the Cloud and is a successful step toward soft and declarative processing of data on a large scale. The solution presented in this paper directly addresses three characteristics of big data, i.e., volume , variety , and velocity , and indirectly addresses, veracity and value .

Earth and Space Science Open Archive PosterOpen AccessYou are viewing the latest version by default [v2]Hypnos Board: A Low-Cost All-In-One Solution for Environment Sensor Power Management, Data Storage, and Task SchedulingAuthorsBrysonGotoiDBaoNguyeniDJohnSelkeriDChetUdelliDSee all authors Bryson GotoiDCorresponding Author• Submitting AuthorOregon State UniversityiDhttps://orcid.org/0000-0002-9224-3290view email addressThe email was not providedcopy email addressBao NguyeniDOregon State UniversityiDhttps://orcid.org/0000-0002-5692-3176view email addressThe email was not providedcopy email addressJohn SelkeriDOregon State UniversityiDhttps://orcid.org/0000-0001-9751-6094view email addressThe email was not providedcopy email addressChet UdelliDOregon State UniversityiDhttps://orcid.org/0000-0001-5815-8319view email addressThe email was not providedcopy email address