SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

All applications in current trends need to use Machine Learning and Big Data in this huge data world.Machine learning is a type of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so.Machine learning use historical data as input to predict new output values.Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves.To harvest, process and analyze the data we need a data processing engine.Our project is to provide them with a machine learning engine which runs on.a specified port on their servers which collects the data sets in real time.This which can be queried using a simple Query Language and analyze the database on the query.The purpose of our project is to create a Data Processing Engine for Big Data Analytics. Machine learning and Big Data plays a major role in the current huge data world. Our project is to maintain a dedicated machine learning and data processing engine, so that even small developers without the knowledge of machine learning and big data analysis can produce expected results based on Machine Learning to make their application smooth. All applications in current trends need to use Machine Learning and Big Data in this huge data world. Machine learning is a subset of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so. Machine learning uses historical data as input to predict new output values. Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves. To harvest, process and analyze the data we need a data processing engine. This project is to provide them a data processing engine which runs on a specified port on their servers which collects the data sets in real time. This can be queried using a simple Query Language and analyze the data and produce respected results using ML based on the query

This paper is concerned with the study and implementation of effective Data Emplacement Algorithm in large set of databases called Big Data and proposes a model for improving the efficiency of data processing and storage utilization for dynamic load imbalance among nodes in a heterogeneous cloud environment. With the era of explosive information and data receiving, more and more fields need to deal with massive, large scale of data. A method has been proposed with an improved Data Placement algorithm called Effective Data Emplacement Algorithm with computing capacity of each node as a predominant factor that promotes and improves the efficiency in data processing in a short duration time from large set of data. The adaptability of the proposed model can be obtained by minimizing the time with processing efficiency through the computing capacity of each node in the cluster. The proposed solution improves the performance of the heterogeneous cluster environment by effectively distributing data based on the performance oriented sampling as the experimental results made with word count applications.

Application of Big Data analytics to exploit for insight and assist in decision making have seen phenomenal growth among organizations in recent years. Although many organizations have invested heavily in Big Data technologies, there still seems to be a lack of common reference framework for organizations to derive business values and answer critical business questions from Big Data Analytics (BDA). In this paper, a vendor-neutral and technology-agnostic Big Data Analytics Activity Reference Framework is presented to close the gap. The framework describes the high level Big Data Analytics activities and serves as a reference to practitioners to decide on the selection of the infrastructure, platform and computing layers. This reference framework is aimed at serving as a launching pad for researchers and implementers to customize their respective Big Data Analytics projects before implementing with the relevant technology and tools.

A huge volume of digitized clinical data is generated and accumulated rapidly since the widespread adoption of Electronic Medical Records (EMRs). These big data in healthcare hold the promise of propelling healthcare evolving from a proficiency-based art to a data-driven science, from a reactive mode to a proactive mode, from one-size-fits-all medicine to personalized medicine. This paper first discusses the research background - big data analytics in healthcare, the research framework of big data analytics in healthcare, analysis of medical process, and the literature summary of treatment pattern mining. Then the challenges for data-driven typical treatment pattern mining are highlighted, including similarity measure between treatment records, typical treatment pattern extraction, evaluation and recommendation, when considering the rich temporal and heterogeneous medical information in EMRs. Furthermore, three categories of typical treatment patterns are mined from doctor order content, duration, and sequence view respectively, which can provide a data-driven guideline to achieve the “5R” goal for rational drug use and clinical pathways.

Cloud computing is growing massively in the world of information technology. As a result of which the number of servers and virtual machines is increasing day by day. It leads to huge energy consumption and carbon emission. In this regard, task scheduling has drawn the attention of cloud service providers to reduce energy consumption and increase cloud utilization. This is a crucial issue in a heterogeneous cloud environment. In this paper, energy consumption issues of cloud datacenters are addressed and an Energy-Aware Cloud Task Scheduling (EACTS) algorithm is proposed. It is taking the concept from traditional min-min, max-min, and sufferage heuristics and merge with the energy model. These heuristics are implemented on the heterogeneous cloud environment. EACTS energy model focuses on the estimation of energy consumption in cloud datacenters. The proposed algorithms estimated the makespan, cloud utilization, and energy consumption for a benchmark dataset. The experimental results obtained from EACTS algorithm provides a valid arbitration between energy efficiency and makespan. It has shown the comparison between above mentioned algorithms for various scheduling parameters.

Intelligent big data analytics is an emerging paradigm for integrating big data, analytics, and artificial intelligence. The objective of this paper is to provide a managerial framework of intelligent big data analytics. More specifically, this paper proposes a managerial framework of intelligent big data analytics, which consists of intelligent big data analytics as a science, technology, system, service and management for improving business decision making. Then it elaborates intelligent big data analytics for management taking into account main managerial functions: planning, organising, leading and controlling. The proposed approach in this paper might facilitate the research and development of business analytics, big data analytics, business intelligence, artificial intelligence and data science.

In a heterogeneous cloud environment, the manual grading of computing assets is the first step in the process of configuring IT infrastructures to ensure optimal utilization of resources. Grading the efficiency of computing assets is however, a difficult, subjective and time consuming manual task. Thus, an automatic efficiency grading algorithm is highly desirable. In this paper, we compare the effectiveness of the different criteria used in the manual grading task for automatically determining the efficiency grading of a computing asset. We report results on a dataset of 1,200 assets from two different data centers in IBM Toronto. Our preliminary results show that electrical costs (associated with power and cooling) appear to be even more informative than hardware and age based criteria as a means of determining the efficiency grade of an asset. Our analysis also indicates that the effectiveness of the various efficiency criteria is dependent on the asset demographic of the data centre under consideration.

Big Data Analytics is a major branch of data science where the huge amount raw data is processed to get insight for relevant business processes. Integration of big data, its analytics along with Service Oriented Architecture (SOA) is need of the hour, such integration shall render reusability and scalability to various business processes. This chapter explains the concept of Big Data and Big Data Analytics at its implementation level. The Chapter further describes Hadoop and its technologies which are one of the popular frameworks for Big Data Analytics and envisage integrating SOA with relevant case studies. The chapter demonstrates the SOA integration with Big Data through, two case studies of two different scenarios are incorporated that integrates real world implementation with theory and enables better understanding of the industrial level processes and practices.

The insights that firms gain from big data analytics (BDA) in real time is used to direct, automate and optimize the decision making to successfully achieve their organizational goals. Data management (DM) and advance analytics (AA) tools and techniques are some of the key contributors to making BDA possible. This paper aims to investigate the characteristics of BD, processes of data management, AA techniques, applications across sectors and issues that are related to their effective implementation and management within broader context of BDA. A range of recently published literature on the characteristics of BD, DM processes, AA techniques are reviewed to explore their current state, applications, issues and challenges learned from their practice. The finding discusses different characteristics of BD, a framework for BDA using data management processes and AA techniques. It also discusses the opportunities/applications and challenges managers dealing with these technologies face for gaining competitive advantages in businesses. The study findings are intended to assist academicians and managers in effectively quantifying the data available in an organization into BD by understanding its properties, understanding the emerging technologies, applications and issues behind BDA implementation.

Among the most revolutionary technologies are big data analytics, artificial intelligence (AI) and robotics, machine learning (ML), cybersecurity, blockchain technology, and cloud computing. The project was focused on how to create a Big Data Analytics Architecture Framework to increase the production capability and global trade for Sub-Saharan Africa’s oilseeds and textile industries (SSA). The infrastructure, e-commerce, and disruptive technologies in the oilseeds and textile industries, as well as global e-commerce, all demand large investments. The pragmatic paradigm served as the foundation for the research approach. This study employed a review of the literature, document analysis, and focus groups. For the oilseeds and textile sectors in SSA, a Big Data analytics architectural framework was created. The Hadoop platform was created as a framework for big data analytics. The open-source Hadoop platform offers the analytical tools and computing capacity needed to handle such massive data volumes. It supports E-commerce and is based on the Hadoop platform, which offers the analytical tools and computing power needed to handle such massive data volumes. The low rate of return on investments made in breeding, seed production, processing, and marketing limits the competitiveness of the oil crop or legume seed markets.

To avoid the drawbacks of a pricing mechanism in heterogeneous cloud environments that considers only single resources, we propose a multi-resource combinatorial pricing mechanism in this paper. This approach jointly considers the principal resources (i.e., CPU, memory, storage, and bandwidth) with the goal of minimizing the total cost. Then, a cost-optimized resource provisioning policy (CORPP) based on game theory is applied to this mechanism that considers the Nash equilibrium between cloud users as well as the Stackelberg equilibrium between users and the cloud provider. The experimental results show that the proposed combinatorial pricing mechanism is more suitable in a heterogeneous cloud environment when running various types of cloudlets. Moreover, CORPP is effective in reducing users' total costs when using both random cloudlets and PlanetLab cloudlets.

Well organized datacentres with interconnected servers constitute the cloud computing infrastructure. User requests are submitted through an interface to these servers that provide service to them in an on-demand basis. The scientific applications that get executed at cloud by making use of the heterogeneous resources being allocated to them in a dynamic manner are grouped under NP hard problem category. Task scheduling in cloud poses numerous challenges impacting the cloud performance. If not handled properly, user satisfaction becomes questionable. More recently researchers had come up with meta-heuristic type of solutions for enriching the task scheduling activity in the cloud environment. The prime aim of task scheduling is to utilize the resources available in an optimal manner and reduce the time span of task execution. An improvised seagull optimization algorithm which combines the features of the Cuckoo search (CS) and seagull optimization algorithm (SOA) had been proposed in this work to enhance the performance of the scheduling activity inside the cloud computing environment. The proposed algorithm aims to minimize the cost and time parameters that are spent during task scheduling in the heterogeneous cloud environment. Performance evaluation of the proposed algorithm had been performed using the Cloudsim 3.0 toolkit by comparing it with Multi objective-Ant Colony Optimization (MO-ACO), ACO and Min-Min algorithms. The proposed SOA-CS technique had produced an improvement of 1.06%, 4.2%, and 2.4% for makespan and had reduced the overall cost to the extent of 1.74%, 3.93% and 2.77% when compared with PSO, ACO, IDEA algorithms respectively when 300 vms are considered. The comparative simulation results obtained had shown that the proposed improvised seagull optimization algorithm fares better than other contemporaries.

The complexity of the Internet and the volume of network traffic have dramatically increased in the last few years, making it more challenging to design scalable Network Traffic Monitoring and Analysis (NTMA) systems. Critical NTMA applications such as the detection of network attacks and anomalies require fast mechanisms for on-line analysis of thousands of events per second, as well as efficient techniques for off-line analysis of massive historical data. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks and anomalies, but this higher dimensionality comes with an extra data processing overhead. In this paper we present Big-DAMA, a big data analytics framework (BDAF) for NTMA applications. Big-DAMA is a flexible BDAF, capable to analyze and store big amounts of both structured and unstructured heterogeneous data sources, with both stream and batch processing capabilities. Big-DAMA uses off-the-shelf big data storage and processing engines to offer both stream data processing and batch processing capabilities, decomposing separate engines for stream, batch and query, following a Data Stream Warehouse (DSW) paradigm. Big-DAMA implements several algorithms for anomaly detection and network security using supervised and unsupervised machine learning (ML) models, using off-the-shelf ML libraries. We apply Big-DAMA to the detection of different types of network attacks and anomalies, benchmarking multiple supervised ML models. Evaluations are conducted on top of real network measurements collected at the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling. Big-DAMA can speed up computations by a factor of 10 when compared to a standard Apache Spark cluster, and can be easily deployed in cloud environments, using hardware virtualization technology.

This chapter describes data integration issues in big data analytics and proposes an integrated data integration framework for big data analytics. The main focus of this chapter is to address the issues of data integration from the architectural point of view. Addressing the issues of data integration from the architectural point of view will lead to a better understanding of the current situation and better construction of proposed solutions to those issues since architectural approach can give us a holistic and comprehensive view of the problems. The chapter also discusses future research directions of the proposed integrated data architecture framework.

Climate change is accelerating, necessitating sophisticated forecasting techniques to alleviate its impacts efficiently. The huge amount and diversity of climate data frequently exceed the endurance of traditional computing techniques, creating problems with accuracy and processing speed. This research presents a quantum-enhanced framework for big data analytics in climate science by combining quantum machine learning, optimisation, and simulation methods. The recommended strategy empowers specific analysis of high-dimensional climate datasets, contributing noteworthy enhancements in predicting critical phenomena such as temperature irregularities, rainfall trends, and catastrophic weather patterns. By including quantum algorithms, the framework attains scalability and real-time competencies, minimising computational processing times while improving predictive accuracy. These results demonstrate the advantage of quantum-enhanced models over classical approaches by giving innovative tools for policymakers and industry leaders to help them make data-driven, well-informed decisions. The study highlights the revolutionary potential of quantum computing in addressing global climate challenges, making new paths for environmentally friendly remedies. As quantum technologies develop, their use in climate science will motivate innovative solutions for the endurance and preservation of the environment. Major Findings: This study highlights the necessity for cross-disciplinary collaboration to completely integrate quantum computing advancements into climate modelling, guaranteeing actionable and accurate perceptions. The results show how important quantum-enhanced analytics can be in helping to develop sustainable decision-making to deal with the escalating climate crisis.