A Scalable Distributed Framework for Efficient Analytics on Ordered Datasets

African buffalo optimized multinomial softmax regression based convolutional deep neural network for software fault prediction

Event log files are the most common source of information for the characterization of events in large scale systems. However the large size of these files makes the task of manual analysing log messages to be difficult and error prone. This is the reason why recent research has been focusing on creating algorithms for automatically analysing these log files. In this paper we present a novel methodology for extracting templates that describe event formats from large datasets presenting an intuitive and user-friendly output to system administrators. Our algorithm is able to keep up with the rapidly changing environments by adapting the clusters to the incoming stream of events. For testing our tool, we have chosen 5 log files that have different formats and that challenge different aspects in the clustering task. The experiments show that our tool outperforms all other algorithms in all tested scenarios achieving an average precision and recall of 0.9, increasing the correct number of groups by a factor of 1.5 and decreasing the number of false positives and negatives by an average factor of 4.

Event logging and log files are playing an important role in system and network security. Log files record computer system activities, are used to provide requirements of reliability, security and accountability applications. Information stored in log files can be obtained from different devices, not necessarily clock synchronized, and they do not arrive in the same order they are generated. Nevertheless, log information has to be coherent in time to be useful. To support the events we propose to use Lamport's logic clocks, originated at different sources, in a causal relationship. As a result the administrator will count all the events involved general idea in a computer incident. A model implementation is also presented.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

Smart health is quickly boosted by technological advancements: smart sensors, body sensor network, internet of medical things and big data. Vast amounts of smart health big data from ubiquitous sensors pose unprecedented challenges to the security and privacy protection, which is extremely critical in healthcare applications. The vital signs, user daily behaviors, medicine recommendations, and so many other health data are vulnerable to different attacks, due to the fact that wearable/mobile monitors have very strict performance/power constraints, which limit the complexity of security protocols. In this paper, we study how to leverage a natural vital signal (Electrocardiogram - ECG) for user identification purpose, without introducing new hardware sensing devices. ECG is not only a gold standard cardiac signal, but also unique to each individual. We investigate a phase-domain deep patient-ECG image learning framework, to tackle key challenges in ECG biometric user identification: high diversities of ECG morphologies due to heart diseases, and time-consuming/ineffective heartbeat localization methods & manual feature engineering. The ultimate goal is to make the smart health security zero-effort: use `phase-domain transformation' to enable blind signal segmentation without localizing heartbeats; create a computer image processing-like task by `pixelating' phase-domain ECG trajectories to ECG images; and enable automatic (non-manual) `deep feature learning' using a deep convolutional neural network. Evaluated on two patient-ECG databases, this zero-effort framework achieves an accuracy as high as 97.2%, and greatly outperforms state-of-the-art studies in terms of the generalization ability and/or performance. This study is expected to enable highly challenging patient-ECG biometric user identification, by generalizable blind signal segmentation and deep feature learning strategies, in the era of smart health boosted by internet of medical things and big medical data.

Application of Big Data analytics to exploit for insight and assist in decision making have seen phenomenal growth among organizations in recent years. Although many organizations have invested heavily in Big Data technologies, there still seems to be a lack of common reference framework for organizations to derive business values and answer critical business questions from Big Data Analytics (BDA). In this paper, a vendor-neutral and technology-agnostic Big Data Analytics Activity Reference Framework is presented to close the gap. The framework describes the high level Big Data Analytics activities and serves as a reference to practitioners to decide on the selection of the infrastructure, platform and computing layers. This reference framework is aimed at serving as a launching pad for researchers and implementers to customize their respective Big Data Analytics projects before implementing with the relevant technology and tools.

Intelligent big data analytics is an emerging paradigm for integrating big data, analytics, and artificial intelligence. The objective of this paper is to provide a managerial framework of intelligent big data analytics. More specifically, this paper proposes a managerial framework of intelligent big data analytics, which consists of intelligent big data analytics as a science, technology, system, service and management for improving business decision making. Then it elaborates intelligent big data analytics for management taking into account main managerial functions: planning, organising, leading and controlling. The proposed approach in this paper might facilitate the research and development of business analytics, big data analytics, business intelligence, artificial intelligence and data science.

This study targets service-delivery improvements in South African public hospitals by pairing Big Data analytics with machine learning, implemented under a quantitative Cross-Industry Standard Process for Data Mining (CRISP-DM) workflow. We (i) identified determinants of effective Big Data analytics in public healthcare, (ii) compared analytical methods, (iii) built a predictive model of service outcomes, and (iv) assessed its performance and interpretability. Using a synthetic hospital dataset, we engineered features and trained a stacked ensemble (XGBoost, Random Forest, CatBoost, LightGBM) to classify outcomes as Improved, Unchanged, or Worsened. Baseline models (e.g., Support Vector Machine (SVM), logistic regression) were explored but not retained due to scalability and multiclass performance constraints. The final ensemble achieved 84% accuracy with class-wise Area Under the Curves (AUCs) of 0.93-0.95. Post-hoc SHapley Additive exPlanations (SHAP) analysis highlighted Satisfaction Rating, Age, and log (Wait Time) as the most influential features, and revealed interaction effects such as Wait-length interactions that align with operational intuition. To support practical use, we prototyped a Streamlit application that delivers near-real-time, explainable predictions for front-line decision support. The findings indicate that Machine Learning (ML)-driven Big Data analytics can provide scalable, interpretable insights to inform predictive decision-making, resource allocation, and patient-flow management in the public sector. While results are demonstrated on synthetic data, the approach establishes a deployable pathway for hospital analytics in South Africa and a foundation for validation on real-world datasets.

A huge volume of digitized clinical data is generated and accumulated rapidly since the widespread adoption of Electronic Medical Records (EMRs). These big data in healthcare hold the promise of propelling healthcare evolving from a proficiency-based art to a data-driven science, from a reactive mode to a proactive mode, from one-size-fits-all medicine to personalized medicine. This paper first discusses the research background - big data analytics in healthcare, the research framework of big data analytics in healthcare, analysis of medical process, and the literature summary of treatment pattern mining. Then the challenges for data-driven typical treatment pattern mining are highlighted, including similarity measure between treatment records, typical treatment pattern extraction, evaluation and recommendation, when considering the rich temporal and heterogeneous medical information in EMRs. Furthermore, three categories of typical treatment patterns are mined from doctor order content, duration, and sequence view respectively, which can provide a data-driven guideline to achieve the “5R” goal for rational drug use and clinical pathways.

In the past two decades, new developments in computing, sensing and crowdsourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called “big data” to inform research and the decision-making process are virtually endless. In general analyses have to be done across multiple data sets in order to bring out the most value of big data. A first important step is to identify temporal correlations between data sets. Given the characteristics of big data in term of volume and velocity, techniques that identify correlations not only need to be scalable, but also need to help users in ordering the correlation across temporal resolutions so that they can focus on important relationships. There is a large body of work in this area, however, most of them either only deal with small data sets, using a fixed temporal resolution, or does not provide a quantifiable measure of a correlation significance. In this paper, we present a method based on mutual information to identify correlations in large data sets. Discovered correlations are suggested to users in an order based on their significance. Our method supports an adaptive streaming technique that minimizes duplicated computation and is implemented on top of Apache Spark for scalability using big data platforms. We also provide a comprehensive evaluation using real-world data sets from NYC Open Data, and compare our findings against a recent study.

The proliferation of multimedia devices over the Internet of Things (IoT) generates an unprecedented amount of data. Consequently, the world has stepped into the era of big data. Recently, on the rise of distributed computing technologies, video big data analytics in the cloud has attracted the attention of researchers and practitioners. The current technology and market trends demand an efficient framework for video big data analytics. However, the current work is too limited to provide a complete survey of recent research work on video big data analytics in the cloud, including the management and analysis of a large amount of video data, the challenges, opportunities, and promising research directions. To serve this purpose, we present this study, which conducts a broad overview of the state-of-the-art literature on video big data analytics in the cloud. It also aims to bridge the gap among large-scale video analytics challenges, big data solutions, and cloud computing. In this study, we clarify the basic nomenclatures that govern the video analytics domain and the characteristics of video big data while establishing its relationship with cloud computing. We propose a service-oriented layered reference architecture for intelligent video big data analytics in the cloud. Then, a comprehensive and keen review has been conducted to examine cutting-edge research trends in video big data analytics. Finally, we identify and articulate several open research issues and challenges, which have been raised by the deployment of big data technologies in the cloud for video big data analytics. To the best of our knowledge, this is the first study that presents the generalized view of the video big data analytics in the cloud. This paper provides the research studies and technologies advancing video analyses in the era of big data and cloud computing.

This paper proposes an ontology of big data analytics and examines how to enhance business intelligence through big data analytics as a service by presenting a big data analytics services-oriented architecture (BASOA), and applying BASOA to business intelligence, where our surveyed data analysis showed that the proposed BASOA is viable for developing business intelligence and enterprise information systems. This paper also discusses the interrelationship between business intelligence and big data analytics. The proposed approach in this paper might facilitate the research and development of business analytics, big data analytics, and business intelligence as well as intelligent agents.KeywordsBig data analyticse-commerceBusiness intelligenceIntelligent agents

A big size of bridge data is being constantly generated—from a variety of distributed sources and in heterogeneous formats—in the form of national bridge inventory (NBI) data, national bridge elements (NBE) data, bridge inspection and maintenance reports, traffic data, etc. Big bridge data analytics open unprecedented opportunities to leverage this large amount of data for improved bridge deterioration prediction and understanding, and enhanced maintenance decision making. In this paper, the authors propose a big bridge data analytics framework that consists of: (1) semantic data/information extraction and integration, (2) semantic data/information analysis and mining, and (3) semantic data/information retrieval and visualization. The big size of the data, in addition to it being scattered and heterogeneity of meaning and format, adds to the challenge of data integration and analysis. As such, at the cornerstone of this framework is an ontology of bridge deterioration knowledge, which aims to facilitate the semantic integration and analysis of the data based on content and domain-specific meaning, and to bridge the terminology gap across different sources. This paper focuses on providing an overview of the framework for big bridge data analytics and presenting the proposed bridge deterioration ontology as the first step towards implementing this framework.

Healthcare industry is constantly undergoing changes owing to the advancements made in its medical and technological dimensions. Healthcare Informatics has evolved over decades by leveraging upon best of the computer technologies available. It has come a long way from being just a database to a comprehensive source of useful information for analytics and research. Emergence of technology paradigms like Big Data, Internet of Things (IoT), Complex Even Processing (CEP), Cloud Computing etc are changing the way things were. Healthcare informatics is now at juncture to find a complete solution to all its woes by adopting an integrated solution framework based on Big Data, IoT and CEP. This paper proposes one such framework which can act as reference for implementing a holistic healthcare informatics and analytics ecosystem.

Big Data Analytics is a major branch of data science where the huge amount raw data is processed to get insight for relevant business processes. Integration of big data, its analytics along with Service Oriented Architecture (SOA) is need of the hour, such integration shall render reusability and scalability to various business processes. This chapter explains the concept of Big Data and Big Data Analytics at its implementation level. The Chapter further describes Hadoop and its technologies which are one of the popular frameworks for Big Data Analytics and envisage integrating SOA with relevant case studies. The chapter demonstrates the SOA integration with Big Data through, two case studies of two different scenarios are incorporated that integrates real world implementation with theory and enables better understanding of the industrial level processes and practices.