Big Data Pipeline with ML-Based and Crowd Sourced Dynamically Created and Maintained Columnar Data Warehouse for Structured and Unstructured Big Data

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

The proliferation of digital data sources and formats has led to the apparition of data lakes, systems where numerous data sources coexist, with less (or no) control and coordination among the sources, than previously practised in enterprise databases and data warehouses. While most data lakes are designed for very large number of tables, Connec-tionLens [2,3] is a data lake system for structured, semi-structured, and unstructured data, which it integrates into a single graph; the graph can be explored via graph queries with keyword search [4] and entity path enumeration [5] . In this paper, we describe ConnectionStudio, a userfriendly platform leveraging ConnectionLens, and integrating feedback from non-expert users, in particular, journalists. Our main insights are: (i) improve and entice exploration by giving a first global view; (ii) facilitate tabular exports from the integrated graph; (iii) provide interactive means to improve the graph constructions. The insights can be used to further advance the exploration and usage of data lakes for non-IT users.

Big data is a valuable asset for organisation as it analyses and help to understand the customers, changes within their business environment, market analysis and future trends. The big data is multifaceted (different data types and versatile), and mostly exists in unstructured formats. The extraction of value from this data is challenging. The usability and productivity of this multifaceted unstructured data is greatly compromised. A number of factors and associated reasons affect the usability of unstructured big data. The present research work investigates these factors and associated reasons behind the usability issues of multifaceted unstructured big data. The identification of these factors contribute to develop solutions to reduce the lack of usability of highly unstructured big data. A detailed study of existing literature followed by survey questionnaire has been conducted to identify the factors and their reasons. Descriptive statistics has been used to analyse and interpret the data and results.

Data quality is paramount in data-driven decision-making processes, especially when dealing with large volumes of data in environments like data warehouses and data lakes. These systems store vast amounts of raw and processed data from multiple sources, making data management and quality assurance complex yet critical. With the growing adoption of Artificial Intelligence (AI), new techniques and tools have emerged that can significantly enhance data quality. This paper discusses how AI can improve the quality of data within both data warehouses and data lakes by automating data cleansing, validation, anomaly detection, and ensuring consistency. It explores the benefits, challenges, and methodologies for integrating AI tools into these systems. Keywords: Data quality, AI in data quality, data warehouse, data lakes, big data, data processing, data cleansing, data profiling.

The selection of the best data storage and management system stands essential because machine learning (ML) continues expanding its industry-driven innovation. Today's two most prevalent large-scale data processing systems are data lakes and data warehouses, which provide unique strengths and barriers when applied to ML workloads. This paper thoroughly compares data lakes and data warehouses by Analyzing their operational speed, abilities, and price efficiency alongside data, management controls, and ML integration capabilities. Data lakes showcase their superiority in processing unstructured together with semi-structured information because they serve deep learning and big data analytics requirements. A data warehouse offers optimized querying and structured storage, which is suitable for traditional business intelligence applications and ML platforms. The execution speed of data warehouses is faster compared to data lakes, but data lakes enable enhanced real-time abilities and flexibility for large-scale ML work. The research approach consists of conducting a feature-based examination of both architectures, combined with real-world examples, performance scaling data, and cost measurements. This research finds that AI analytics operate most successfully through data lakes; however, structured ML jobs need data warehouses for efficient operation. The combination of data lakehouse technology presents a new possibility for joining both paradigms to create more efficient environments for machine learning applications.

Today big data pipelines are increasingly adopted by service applications representing a key enabler for enterprises to compete in the global market. However, the management of non-functional aspects of the big data pipeline (e.g., security, privacy) is still in its infancy. As a consequence, while functionally appealing, the big data pipeline does not provide a transparent environment, impairing the users’ ability to evaluate its behavior. In this paper, we propose a security assurance methodology for big data pipelines grounded on the DevSecOps development paradigm to increase trustworthiness allowing reliable security and privacy by design. Our methodology models and annotates big data pipelines with non-functional requirements verified by assurance checks ensuring requirements to hold along with the pipeline lifecycle. The performance and quality of our methodology are evaluated in a real walkthrough analytics scenario.

The personalized health care service utilizes the relational patient data and big data analytics to tailor the medication recommendations. However, most of the health care data are in unstructured form and it consumes a lot of time and effort to pull them into relational form. This study proposes a novel data lake architecture to reduce the data ingestion time and improve the precision of healthcare analytics. It also removes the data silos and enhances the analytics by allowing the connectivity to the third-party data providers (such as clinical lab results, chemist, insurance company, etc.). The data lake architecture uses the Hadoop Distributed File System (HDFS) to provide the storage for both structured and unstructured data. This study uses K-means clustering algorithm to find the patient clusters with similar health conditions. Subsequently, it employs a support vector machine to find the most successful healthcare recommendations for the each cluster. Our experiment results demonstrate the ability of data lake to reduce the time for ingesting data from various data vendors regardless of its format. Moreover, it is evident that the data lake poses the potential to generate clusters of patients more precisely than the existing approaches. It is obvious that the data lake provides an unified storage location for the data in its native format. It can also improve the personalized healthcare medication recommendations by removing the data silos.

Spatial big data architecture: From Data Warehouses and Data Lakes to the LakeHouse

Microsoft Fabric is an end-to-end data analytical platform designed for achieve a Unified solutions in a Business Intelligence and Enterprises. It incorporates data movements, processing, ingestion, transformation, real-time event routing, and report building. It provide services like Data Engineering, Data Factory, Data Science, Real-Time Analytics, Data Warehouse, and Databases. This paper examines Microsoft Fabric and its features especially about a One Lake. Microsoft Fabric applied in a real-time analytics, data storage, advanced machine learning capabilities, and robust security protocols. Collection of files, folders and tables which represent the database is a data house . A Data lake is a cloud-based big data analytics service. Which allows storing, processing, and analyzing large amounts of structured and unstructured data. It integrates with other Azure services to provide a full data analysis solution[1].

Financial institutions in emerging economies face significant cybersecurity challenges due to the increasing sophistication of cyber threats and limited resources for detection and prevention. This paper introduces Big Data-Driven Cybersecurity Models, designed to address gaps in threat detection and prevention by leveraging advanced analytics, machine learning, and real-time data processing. These models aim to strengthen the cybersecurity frameworks of financial institutions, particularly in emerging economies, by improving threat intelligence, detection accuracy, and response efficiency. The proposed models focus on the integration of big data technologies, such as distributed computing frameworks, data lakes, and advanced visualization tools, to handle large volumes of structured and unstructured data. By analyzing data from diverse sources, including transaction logs, network traffic, and customer interactions, the models enable the identification of anomalous patterns and potential security threats. Machine learning algorithms enhance predictive capabilities, allowing institutions to detect previously unknown threats and mitigate risks proactively. A key component of the models is their scalability and adaptability to resource-constrained environments. The framework includes cloud-based solutions to reduce infrastructure costs and ensure accessibility for smaller financial institutions. Additionally, the models emphasize the importance of fostering cybersecurity awareness and regulatory compliance to address human and organizational vulnerabilities. Validation through case studies and simulations demonstrates the effectiveness of these models in mitigating cybersecurity risks. Results show significant improvements in threat detection rates, response times, and cost-efficiency. These findings underscore the potential of big data-driven approaches to enhance the resilience of financial institutions against evolving cyber threats. This research provides actionable insights for policymakers, financial institutions, and technology developers seeking to strengthen cybersecurity in emerging economies. By adopting the proposed models, financial institutions can bridge critical gaps in threat detection and prevention, ensuring the security and trust necessary for sustainable economic growth.

The term “Big Data” refers to a large volume of information usually in terabytes and petabytes. It includes both structured and unstructured data. Unstructured data is conventionally text-heavy, but may also contain data such as facts, dates and numbers. To use this unstructured information effectively, it needs to be processed and organized. Taxonomy is considered as a powerful way of organizing the information. For automatic taxonomy generation, various techniques have been proposed in the past. However, the substantial nature of big data presently crosses the processing abilities of traditional techniques. Thus, to meet this challenge an extensible and scalable technique is required to potentially accelerate the process of taxonomy generation and its evolution upon arrival of new data, hence catering large amount of unstructured big data. This paper proposes a technique for both the taxonomy generation and evolution on Apache Spark infrastructure. The proposed technique is evaluated on a text dataset of a computing domain. The evaluation results show that the technique presented in this paper outperformed the existing techniques in terms of time and quality metrics. The time and quality-based evaluation showed that the use of MapReduce environment has resolved the scalability issues of current taxonomy generation and evolution process.

Geospatial data in databases have shifted to conform to the characteristics of big-data in tandem with the development of the Internet, mobile Internet, cloud computing, and especially, spatial data acquisition technologies. Faced with tackling spatial big data, traditional spatial database management techniques based on Relational Database Management Systems have encountered problems including the unstructured characteristics of the spatial object, the high scalability of storage capacity, and the high concurrency in big data application environment. This paper focuses on the mainstream of NoSQL databases that successfully deal with unstructured big data and are widely used in Internet applications, but lack of spatial characteristics. The data operational and query modes cannot meet the requirments of GIS applications. To resolve this problem, this paper proposes a strategy that takes a NoSQL database as a warehouse for spatial big data and a traditional spatial database as the application server. The storage system architecture and the key technology and solutions are discussed. A prototype system was developed based on MongoDB, PostgreSQL and SQLite to verify the feasibility and effectiveness of the strategy.


 
 
 The data warehouse is a very famous solution for analyzing business data from heterogeneous sources. Unfortunately, a data warehouse only can analyze structured data. Whereas, nowadays, thanks to the popularity of social media and the ease of creating data on the web, we are experiencing a flood of unstructured data. Therefore, we need an approach that can "structure" the unstructured data into structured data that can be processed by the data warehouse. To do this, we propose a schema extraction approach using Google Cloud Platform that will create a schema from unstructured data. Based on our experiment, our approach successfully produces a schema from unstructured data. To the best of our knowledge, we are the first in using Google Cloud Platform for extracting a schema. We also prove that our approach helps the database developer to understand the unstructured data better.
 
 

Data Lakes are the methods for storing and managing large quantities of unstructured data. Modern enterprises and small businesses, regardless of their size, can use this data to derive valuable insights about their business, such as process improvements or product usage. Although this approach to extracting insights is powerful, only some studies describe the actual implementation architectures of data lakes and warehouses. The article provides a broad overview of setting up a data lake on AWS (Amazon Web Services). It covers setting up an Application Programming Interface (API) to consume data, store data, visualize data, and the ability to create data lakes across multiple AWS accounts quickly with a single Command-line Interface (CLI) command. This is useful for creating a scalable data lake or data warehouse setup that doesn’t require much manual work. We describe how such design can be done using infrastructure as a code approach to achieve this and propose AWS architecture for solving the task of compelling data storage. The article provides a diagram of the proposed architecture accompanied by a high-level description and theoretical background.

Development and deployment of a big data pipeline for field-based high-throughput cotton phenotyping data