Integrated Data Architecture for Business

Big Data Analytics is a major branch of data science where the huge amount raw data is processed to get insight for relevant business processes. Integration of big data, its analytics along with Service Oriented Architecture (SOA) is need of the hour, such integration shall render reusability and scalability to various business processes. This chapter explains the concept of Big Data and Big Data Analytics at its implementation level. The Chapter further describes Hadoop and its technologies which are one of the popular frameworks for Big Data Analytics and envisage integrating SOA with relevant case studies. The chapter demonstrates the SOA integration with Big Data through, two case studies of two different scenarios are incorporated that integrates real world implementation with theory and enables better understanding of the industrial level processes and practices.

Intelligent big data analytics is an emerging paradigm for integrating big data, analytics, and artificial intelligence. The objective of this paper is to provide a managerial framework of intelligent big data analytics. More specifically, this paper proposes a managerial framework of intelligent big data analytics, which consists of intelligent big data analytics as a science, technology, system, service and management for improving business decision making. Then it elaborates intelligent big data analytics for management taking into account main managerial functions: planning, organising, leading and controlling. The proposed approach in this paper might facilitate the research and development of business analytics, big data analytics, business intelligence, artificial intelligence and data science.

Application of Big Data analytics to exploit for insight and assist in decision making have seen phenomenal growth among organizations in recent years. Although many organizations have invested heavily in Big Data technologies, there still seems to be a lack of common reference framework for organizations to derive business values and answer critical business questions from Big Data Analytics (BDA). In this paper, a vendor-neutral and technology-agnostic Big Data Analytics Activity Reference Framework is presented to close the gap. The framework describes the high level Big Data Analytics activities and serves as a reference to practitioners to decide on the selection of the infrastructure, platform and computing layers. This reference framework is aimed at serving as a launching pad for researchers and implementers to customize their respective Big Data Analytics projects before implementing with the relevant technology and tools.

Chapter 4 - Frameworks for Big Data Integration, Warehousing, and Analytics

A huge volume of digitized clinical data is generated and accumulated rapidly since the widespread adoption of Electronic Medical Records (EMRs). These big data in healthcare hold the promise of propelling healthcare evolving from a proficiency-based art to a data-driven science, from a reactive mode to a proactive mode, from one-size-fits-all medicine to personalized medicine. This paper first discusses the research background - big data analytics in healthcare, the research framework of big data analytics in healthcare, analysis of medical process, and the literature summary of treatment pattern mining. Then the challenges for data-driven typical treatment pattern mining are highlighted, including similarity measure between treatment records, typical treatment pattern extraction, evaluation and recommendation, when considering the rich temporal and heterogeneous medical information in EMRs. Furthermore, three categories of typical treatment patterns are mined from doctor order content, duration, and sequence view respectively, which can provide a data-driven guideline to achieve the “5R” goal for rational drug use and clinical pathways.

Among the most revolutionary technologies are big data analytics, artificial intelligence (AI) and robotics, machine learning (ML), cybersecurity, blockchain technology, and cloud computing. The project was focused on how to create a Big Data Analytics Architecture Framework to increase the production capability and global trade for Sub-Saharan Africa’s oilseeds and textile industries (SSA). The infrastructure, e-commerce, and disruptive technologies in the oilseeds and textile industries, as well as global e-commerce, all demand large investments. The pragmatic paradigm served as the foundation for the research approach. This study employed a review of the literature, document analysis, and focus groups. For the oilseeds and textile sectors in SSA, a Big Data analytics architectural framework was created. The Hadoop platform was created as a framework for big data analytics. The open-source Hadoop platform offers the analytical tools and computing capacity needed to handle such massive data volumes. It supports E-commerce and is based on the Hadoop platform, which offers the analytical tools and computing power needed to handle such massive data volumes. The low rate of return on investments made in breeding, seed production, processing, and marketing limits the competitiveness of the oil crop or legume seed markets.

Big data analytics is a fast-evolving phenomenon shaped by interactions among individuals, organizations, and society. However, its ethical implications for these stakeholders remain empirically underexplored and not well understood. We present empirical findings from a Delphi study that identified, defined, and examined the key concepts that underlie ethical issues in big data analytics. We then analyze those concepts using stakeholder theory and discourse ethics and suggest ways to balance interactions between individuals, organizations, and society in order to promote the ethical use of big data analytics. Our findings inform practitioners and policymakers concerned with ethically using big data analytics and provide a basis for future research.

Climate change is accelerating, necessitating sophisticated forecasting techniques to alleviate its impacts efficiently. The huge amount and diversity of climate data frequently exceed the endurance of traditional computing techniques, creating problems with accuracy and processing speed. This research presents a quantum-enhanced framework for big data analytics in climate science by combining quantum machine learning, optimisation, and simulation methods. The recommended strategy empowers specific analysis of high-dimensional climate datasets, contributing noteworthy enhancements in predicting critical phenomena such as temperature irregularities, rainfall trends, and catastrophic weather patterns. By including quantum algorithms, the framework attains scalability and real-time competencies, minimising computational processing times while improving predictive accuracy. These results demonstrate the advantage of quantum-enhanced models over classical approaches by giving innovative tools for policymakers and industry leaders to help them make data-driven, well-informed decisions. The study highlights the revolutionary potential of quantum computing in addressing global climate challenges, making new paths for environmentally friendly remedies. As quantum technologies develop, their use in climate science will motivate innovative solutions for the endurance and preservation of the environment. Major Findings: This study highlights the necessity for cross-disciplinary collaboration to completely integrate quantum computing advancements into climate modelling, guaranteeing actionable and accurate perceptions. The results show how important quantum-enhanced analytics can be in helping to develop sustainable decision-making to deal with the escalating climate crisis.

A big size of bridge data is being constantly generated—from a variety of distributed sources and in heterogeneous formats—in the form of national bridge inventory (NBI) data, national bridge elements (NBE) data, bridge inspection and maintenance reports, traffic data, etc. Big bridge data analytics open unprecedented opportunities to leverage this large amount of data for improved bridge deterioration prediction and understanding, and enhanced maintenance decision making. In this paper, the authors propose a big bridge data analytics framework that consists of: (1) semantic data/information extraction and integration, (2) semantic data/information analysis and mining, and (3) semantic data/information retrieval and visualization. The big size of the data, in addition to it being scattered and heterogeneity of meaning and format, adds to the challenge of data integration and analysis. As such, at the cornerstone of this framework is an ontology of bridge deterioration knowledge, which aims to facilitate the semantic integration and analysis of the data based on content and domain-specific meaning, and to bridge the terminology gap across different sources. This paper focuses on providing an overview of the framework for big bridge data analytics and presenting the proposed bridge deterioration ontology as the first step towards implementing this framework.

The proliferation of multimedia devices over the Internet of Things (IoT) generates an unprecedented amount of data. Consequently, the world has stepped into the era of big data. Recently, on the rise of distributed computing technologies, video big data analytics in the cloud has attracted the attention of researchers and practitioners. The current technology and market trends demand an efficient framework for video big data analytics. However, the current work is too limited to provide a complete survey of recent research work on video big data analytics in the cloud, including the management and analysis of a large amount of video data, the challenges, opportunities, and promising research directions. To serve this purpose, we present this study, which conducts a broad overview of the state-of-the-art literature on video big data analytics in the cloud. It also aims to bridge the gap among large-scale video analytics challenges, big data solutions, and cloud computing. In this study, we clarify the basic nomenclatures that govern the video analytics domain and the characteristics of video big data while establishing its relationship with cloud computing. We propose a service-oriented layered reference architecture for intelligent video big data analytics in the cloud. Then, a comprehensive and keen review has been conducted to examine cutting-edge research trends in video big data analytics. Finally, we identify and articulate several open research issues and challenges, which have been raised by the deployment of big data technologies in the cloud for video big data analytics. To the best of our knowledge, this is the first study that presents the generalized view of the video big data analytics in the cloud. This paper provides the research studies and technologies advancing video analyses in the era of big data and cloud computing.

A framework to simplify pre-processing location-based social media big data for sustainable urban planning and management

Examining how influential Big Data Analytic Capabilities (BDAC) are in improving Firm Performance (FPER) in the Indonesian setting is the objectives of this study, which has seen a significant increase in Big Data expenditure in Asia but lacks research in this area. Additionally, the study proposes a theoretical model as a framework for developing BDAC. The IT Capability Framework and the mediation functions of both Process-oriented Dynamic Capability (PODC) and Business Process Agility (BPA serve as the foundation for this study model. A quantitative research method using purposive and snowball sampling techniques is applied to a sample of 53 Indonesian companies. The result is BPA is significant while the PODC is identified to be insignificant in mediating the BDAC and FPER relationship. This study provides a framework for understanding Big Data Analytics and empirical evidence of BDAC’s impact on the success of Indonesian firms.

The use of advanced analytics techniques has become pivotal for the Digital Transformation of the Oil and Gas Industry. Most of these models are used to predict and avoid the off-spec behaviors of both equipment and functional units of the plant and also for predicting overshooting events in advance allows plant’s operators to avoid production down-time. From a Machine Learning perspective, predicting off-specs situation and peaks in time signal is a complex task, due to the great rarity of events. For the very same reason, using standard data science measures – like Area Under the Curve (AUC), Recall and Precision – can lead to misleading performance indicators. In fact, a model that predicts no off-spec would have a high AUC just because of the unbalanced classes, leading to many false alarms. In this paper we present a business-oriented validation framework for big data analytics and machine learning models applied to a upstream production plant. This allow to evaluate both the effort required to operators and the expected benefit that could be achieved. The validation metrics defined take the classical Data Science measures to the business domain. This allow to adapt the model to the very specific use case and end user addressing the specific upstream plants constraints. This framework allows to define the optimal tradeoff between effort required and preventable events, providing statistics and KPIs to evaluate it. Normalized Recall (NR) takes into account both the percentage of events intercepted and the effort required, in terms of Attention Time (AT), when the operator should pay attention to the equipment involved. Plant operators can now have an idea of the results they can achieve with respect to the maximum effort required. Moreover, to prove the goodness of the model, we defined the lift in the NR as the ratio of the model NR and the NR that would be obtained by randomly distributing the same number of alarms. We applied this framework to specific use cases obtaining an expected recall of 40-50% with an expected effort of 5-10% of the time (considering more than 6 months). The effort is actually lower, since the operator is not requested to be fully committed to the alarm. The innovative framework developed is able to demonstrate the real operating capability of the analytics implemented on field, highlighting both the effort required to operators and the accuracy of machine learning tools.

Forensic analyst skills are at stake for processing of growing data from IoT based environment platforms. Tangible sources often have the size limits, but that’s not the case for communication traffic source. Hence, increasing the thirst for an efficient benchmarking for big data analysis. Available solutions to date have used an anomaly-based approach or have proposed approaches based on the deviation from a regular pattern. To tackle the seized bytes, authors have proposed an approach for big data forensics, with efficient sensitivity and precision. In the presented work, a generalized forensic framework has been proposed that use Google’s programming model, MapReduce as the backbone for traffic translation, extraction, and analysis of dynamic traffic features. For the proposed technique, authors have used open source tools like Hadoop, Hive, and Mahout and R. Apart from being open source, these tools support scalability and parallel processing. Also, comparative analysis of globally accepted machine learning models of P2P malware analysis in mocked real-time is presented. Dataset from CAIDA was taken and executed in parallel to validate the proposed model. Finally, the forensic performance metrics of the model shows the results with the sensitivity of 99%.

The complexity of the Internet and the volume of network traffic have dramatically increased in the last few years, making it more challenging to design scalable Network Traffic Monitoring and Analysis (NTMA) systems. Critical NTMA applications such as the detection of network attacks and anomalies require fast mechanisms for on-line analysis of thousands of events per second, as well as efficient techniques for off-line analysis of massive historical data. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks and anomalies, but this higher dimensionality comes with an extra data processing overhead. In this paper we present Big-DAMA, a big data analytics framework (BDAF) for NTMA applications. Big-DAMA is a flexible BDAF, capable to analyze and store big amounts of both structured and unstructured heterogeneous data sources, with both stream and batch processing capabilities. Big-DAMA uses off-the-shelf big data storage and processing engines to offer both stream data processing and batch processing capabilities, decomposing separate engines for stream, batch and query, following a Data Stream Warehouse (DSW) paradigm. Big-DAMA implements several algorithms for anomaly detection and network security using supervised and unsupervised machine learning (ML) models, using off-the-shelf ML libraries. We apply Big-DAMA to the detection of different types of network attacks and anomalies, benchmarking multiple supervised ML models. Evaluations are conducted on top of real network measurements collected at the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling. Big-DAMA can speed up computations by a factor of 10 when compared to a standard Apache Spark cluster, and can be easily deployed in cloud environments, using hardware virtualization technology.