Big data processing and analytics for process industries

This paper describes the architecture of a cross-sectorial Big Data platform for the process industry domain. The main objective was to design a scalable analytical platform that will support the collection, storage and processing of data from multiple industry domains. Such a platform should be able to connect to the existing environment in the plant and use the data gathered to build predictive functions to optimize the production processes. The analytical platform will contain a development environment with which to build these functions, and a simulation environment to evaluate the models. The platform will be shared among multiple sites from different industry sectors. Cross-sectorial sharing will enable the transfer of knowledge across different domains. During the development, we adopted a user-centered approach to gather requirements from different stakeholders which were used to design architectural models from different viewpoints, from contextual to deployment. The deployed architecture was tested in two process industry domains, one from the aluminium production and the other from the plastic molding industry.

The unprecedented proliferation of miniaturized sensors and intelligent communication, computing, and control technologies have paved the way for the development of the Industrial Internet of Things. The IIoT incorporates machine learning and massively parallel distributed systems such as clouds, clusters, and grids for big data storage, processing, and analytics. In IIoT, end devices continuously generate and transmit data streams, resulting in increased network traffic between device-cloud communication. Moreover, it increases in-network data transmissions. requiring additional efforts for big data processing, management, and analytics. To cope with these engendered issues, this article first introduces a novel concentric computing model (CCM) paradigm composed of sensing systems, outer and inner gateway processors, and central processors (outer and inner) for the deployment of big data analytics applications in IIoT. Second, we investigate, highlight, and report recent research efforts directed at the IIoT paradigm with respect to big data analytics. Third, we identify and discuss indispensable challenges that remain to be addressed for employing CCM in the IIoT paradigm. Lastly, we provide several future research directions (e.g., real-time data analytics, data integration, transmission of meaningful data, edge analytics, real-time fusion of streaming data, and security and privacy).

Data processing has become an important field in today's big data-dominated world. The data has been generating at a tremendous pace from different sources. There has been a change in the nature of data from batch-data to streaming-data, and consequently, data processing methodologies have also changed. Traditional SQL is no longer capable of dealing with this big data. This chapter describes the nature of data and various tools, techniques, and technologies to handle this big data. The chapter also describes the need of shifting big data on to cloud and the challenges in big data processing in the cloud, the migration from data processing to data analytics, tools used in data analytics, and the issues and challenges in data processing and analytics. Then the chapter touches an important application area of streaming data, sentiment analysis, and tries to explore it through some test case demonstrations and results.

Big data and data analytics are currently the buzzwords in both academia and industry to become data driven. Big data has been the trending topic in the accounting industry also. Big data and data analytics will have an important impact on accounting and accountants. Big data will improve the quality of accounting information and the accounting profession will continue to provide real-time and dynamic information to assist in decision-making. The purpose of this research is to investigate the impact of big data and big data analytics in accounting. Data analytics is one of the most recent developments in the accounting context. This study is qualitative in nature and adopted a literature review methodology to gain a better understanding of the study area. This literature review seeks to provide a description and evaluation of the impact of big data analytics on accounting. This research found that big data presents great opportunities for decision making in accounting and risks analysis, which indicated that companies could improve their performance, measure performance, manage risks and allow effective real-time decision-making with data analytics. This research revealed that accountants can create more value in a world of big data analytics and encourages accountants to get started with big data to find answers to risks in business operations as well as understand financial performance. It shows that relying on big data analytics will open new possibilities for accountants. This study contributes to the research literature in the area of big data analytics on accounting. The limitations of this study are that it utilizes few recent peer reviewed articles in the general accounting practice, therefore not exhaustive in describing how big data and big data analytics impacts accounting.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

While the potential benefits of Big Data adoption are significant, and some initial successes have already been realized, there remain many research and technical challenges that must be addressed to fully realize this potential. The Big Data processing, storage and analytics, of course, are major challenges that are most easily recognized. However, there are additional challenges related for instance to Big Data collection, integration, and quality enforcement. This paper proposes a hybrid approach to Big Data quality evaluation across the Big Data value chain. It consists of assessing first the quality of Big Data itself, which involve processes such as cleansing, filtering and approximation. Then, assessing the quality of process handling this Big Data, which involve for example processing and analytics process. We conduct a set of experiments to evaluate Quality of Data prior and after its pre-processing, and the Quality of the pre-processing and processing on a large dataset. Quality metrics have been measured to access three Big Data quality dimensions: accuracy, completeness, and consistency. The results proved that combination of data-driven and process-driven quality evaluation lead to improved quality enforcement across the Big Data value chain. Hence, we recorded high prediction accuracy and low processing time after we evaluate 6 well-known classification algorithms as part of processing and analytics phase of Big Data value chain.

Mesoeconomics represents a critical intermediate layer of economic analysis, bridging the gap between microeconomics and macroeconomics. With the growing complexity of global supply chains, traditional economic frameworks and data analysis techniques have proven insufficient. The increasing availability and use of big data technologies have highlighted the relevance of mesoeconomic approaches in analyzing networked economic interactions across international supply chains. The research aims to explore how mesoeconomic frameworks can be effectively utilized to analyze global supply chain networks through big data analytics. It seeks to identify the advantages of applying mesoeconomic analysis in understanding the structure, flow, and dynamics of supply chains, as well as the challenges associated with data integration and analytics. The research employs a qualitative and conceptual analysis of mesoeconomic theory in the context of supply chains, supplemented by a review of current big data analytics methods used by companies. The analysis includes an examination of various data sources (primary and secondary), network structures, and classification techniques that support the processing and interpretation of large-scale data related to goods and services flows. The findings suggest that mesoeconomics, when combined with big data analytics, offers a robust framework for capturing the complexities of global supply chains. This approach enhances the understanding of relationships among products, producers, and processes, enabling more efficient and informed decision-making. However, challenges such as data quality, interoperability, privacy, and security remain significant barriers to the full realization of big data’s potential in supply chain analysis.

This chapter provides a brief overview of the growing role played by big data, data analytics, and artificial intelligence (AI) in the accounting profession. The term 'Big Data' along with other trending topics such as 'Data Analytics' and 'Artificial Intelligence (AI)' have become buzzwords in the accounting profession in recent years. The skills of accounting professionals have evolved as technology has made rapid advances from using pencil and paper to typewriters and calculators, and eventually spreadsheets and accounting software. Data analytics in accounting is a relatively new skill set that is growing significantly in all areas of the accounting profession. Accounting professionals who can discern patterns and trends in big data and translate them into compelling strategic narratives will find themselves at the center of the twenty-first-century business world. Therefore, accounting professionals can capitalize on numerous opportunities in this rapidly evolving, disruptive, but ultimately advantageous environment by embracing big data, data analytics, and artificial intelligence (AI) to stay ahead of the competition.

This paper investigates how lean production levelling methods can be better applied in the process industry with support from key enabling technologies from Industry 4.0. To investigate such a topic, a literature study is conducted in three main areas, namely production planning in the process industry, Lean Production, and Industry 4.0. Based on the findings from the literature review, a conceptual framework is developed to illustrate the ability of Internet of Things, Big Data Analytics, and the further integration of IT systems to provide increased reliability for materials, processes, equipment, and forecasts that improves the utilization of Heijunka (production levelling) practices in the process industry.

Big data analytics in health is an emerging area due to the urgent need to derive actionable intelligence from the large volumes of healthcare data to efficiently manage the healthcare system and to improve health outcomes. In this paper we present a ‘big’ healthcare data analytics platform—termed as Healthcare Analytics for Intelligence and Learning (HAIL)—that is an end-to-end healthcare data analytics solution to derive data-driven actionable intelligence and situational awareness to inform and transform health decision-making, systems management and policy development. The innovative aspects of HAIL are: (a) the integration of data-driven and knowledge-driven analytics approaches, (b) a sand-box environment for healthcare analysts to develop and test health policy/process models by exploiting a range of data preparation, analytical and visualization methods, (c) the incorporation of specialized healthcare data standards, terminologies and concept maps to support data analytics, and (d) text analytics to analyze unstructured healthcare data. The architecture of HAIL comprises the following four main modules (fig 1): (A) Health Data Integration module that entails a semantics-based metadata manager to synthesize health data originating from a range of healthcare institutions to formulate a rich contextualized data resource. The data integration is achieved through ETL workflows designed by health analysts and researchers, (B) Health Analytics Module provides a range of healthcare analytics capabilities including, (i) Exploratory Analytics using data mining to perform data clustering, classification and association tasks, (ii) Predictive Analytics to predict future trends/outcomes derived from past observations of the healthcare processes, (iii) Text Analytics to analyze unstructured texts (such as clinical notes, discharge summaries, referral notes, clinical guidelines, etc.), (iv) Simulation-based Analytics to simulate what-if questions based on simulation models, (v) Workflow analytics to interact with modeled clinical workflows to understand the affects of various confounding factors, (vi) Semantic Analytics to infer contextualized relationships, anomalies and deviations through reasoning over a semantic health data model, and (vii) Informational Analytics to present summaries, aggregations, charts and reports, (C) Data Visualization Module offers a range of interactive data visualizations, such as geospatial visualizations, causal networks, 2D and 3D graphs, pattern clusters and interactive visualizations to explore high dimensional data, (D) Data Analytics Workbench is an interactive workspace to enable data health analysts to specify and set-up their analytics process in terms of data preparation, selection and set-up of analytical methods and the selection of visualization methods. Using the workbench analysts can design sophisticated data analytics workflows/models using a range of data integration, analytical and visualization methods. The HAIL platform is available via a web portal and the desktop application, and it is deployed on a cloud infrastructure. HAIL can be connected with existing health data sources to provide front-end data analytics. Fig 2 shows the technical architecture. The HAIL platform has been applied to analyze a real-life clinical healthcare situations using actual data from the provincial clinical data warehouse. We will present two case studies of the use of HAIL. In conclusion, the HAIL platform addresses a critical need for healthcare analytics to impact health decision-making, systems management and policy development.

All applications in current trends need to use Machine Learning and Big Data in this huge data world.Machine learning is a type of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so.Machine learning use historical data as input to predict new output values.Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves.To harvest, process and analyze the data we need a data processing engine.Our project is to provide them with a machine learning engine which runs on.a specified port on their servers which collects the data sets in real time.This which can be queried using a simple Query Language and analyze the database on the query.The purpose of our project is to create a Data Processing Engine for Big Data Analytics. Machine learning and Big Data plays a major role in the current huge data world. Our project is to maintain a dedicated machine learning and data processing engine, so that even small developers without the knowledge of machine learning and big data analysis can produce expected results based on Machine Learning to make their application smooth. All applications in current trends need to use Machine Learning and Big Data in this huge data world. Machine learning is a subset of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so. Machine learning uses historical data as input to predict new output values. Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves. To harvest, process and analyze the data we need a data processing engine. This project is to provide them a data processing engine which runs on a specified port on their servers which collects the data sets in real time. This can be queried using a simple Query Language and analyze the data and produce respected results using ML based on the query

Evaluating maturity level of big data management and analytics in industrial companies

Data Science is associated with new discoveries, the discovery of value from the data. It is a practice of deriving insights and developing business strategies through transformation of data in to useful information. It has been evaluated as a scientific field and research evolution in disciplines like statistics, computing science, intelligence science, and practical transformation in the domains like science, engineering, public sector, business and lifestyle. The field encompasses the larger areas of artificial intelligence, data analytics, machine learning, pattern recognition, natural language understanding, and big data manipulation. It also tackles related new scientific challenges, ranging from data capture, creation, storage, retrieval, sharing, analysis, optimization, and visualization, to integrative analysis across heterogeneous and interdependent complex resources for better decision-making, collaboration, and, ultimately, value creation. In this paper we entitled epicycles of analysis, formal modeling, from data analysis to data science, data analytics -A keystone of data science, The Big data is not a single technology but an amalgamation of old and new technologies that assistance companies gain actionable awareness. The big data is vital because it manages, store and manipulates large amount of data at the desirable speed and time. Big data addresses detached requirements, in other words the amalgamate of multiple un-associated datasets, processing of large amounts of amorphous data and harvesting of unseen information in a time-sensitive generation.
 As businesses struggle to stay up with changing market requirements, some companies are finding creative ways to use Big Data to their growing business needs and increasingly complex problems. As organizations evolve their processes and see the opportunities that Big Data can provide, they struggle to beyond traditional Business Intelligence activities, like using data to populate reports and dashboards, and move toward Data Science- driven projects that plan to answer more open-ended and sophisticated questions.
 Although some organizations are fortunate to have data scientists, most are not, because there is a growing talent gap that makes finding and hiring data scientists in a timely manner is difficult. This paper, aimed to demonstrate a close view about Data science, big data, including big data concepts like data storage, data processing, and data analysis of these technological developments, we also provide brief description about big data analytics and its characteristics , data structures, data analytics life cycle, emphasizes critical points on these issues.

The objective of this paper was to propose a multi-agents model for automating big data processing, to improve the process of decision-making and urban planning. The huge amounts of collected data from different domains, such as urban management and remote sensing, are characterized as big data with a spatial component. Smart data is the approach to deal with big data characteristics and challenges by focusing on the Value aspect. The focus on smart data on the relevant data and the mechanism of automation and collaboration of the agents, will contribute to the efficient automating for big data analytics and processing. The proposed approach is based on a collection of agents, and adopt the concept of smart data, this paradigm focus on the aspect value from the big data and help to retrieve the useful information from the large volumes of data by intelligent processing. The proposed model describes the functionalities of the agents. The agents receive data in real time, perform the operations of storing data, pre-processing, streaming processing and batch processing and finally transfer the results of analysis to the services and applications. Machine learning techniques can be used to enhance the aspect of cognition of the agents; through a case study, we used supervised learning methods to build a classification model to support the process of urban planning by predicting the type of habitat adequate for the population based on their demographic and socio-economic characteristics.KeywordsSmart dataBig dataMulti-agent systemAutomationMachine learningUrban planningBig data analyticsDecision-making

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.