A Survey on Distributed Frameworks for Machine Learning Based Big Data Analysis

With the advent of the cloud era, big data has attracted more and more attention. At present, due to the expansion of the scale of chain educational training institutions and the complexity of management, the information stored in the data of management platform is mostly unstructured data model. The large amount and complexity of unstructured data become the main difficulties in big data analysis and decision-making. This paper analyzes and discusses the application of chain educational training institutions based on the background of big data, and build a platform for big data analysis and decision-making of chain educational training institutions based on cloud computing. The platform greatly improves the ability to process massive data and discovers the internal relationship and rules between big data. It aims to make better use of big data to serve chain educational training institutions, promote the continuous development and progress of chain educational training institutions, and even lay the foundation for the global chain educational training institutions.

The emergence of the digital era has led to growth in various types of data in a cloud. In fact, there may be three fourth of the total data will be treated as big data. In many organisations, massive volume of both structured and unstructured data sit idle. Various categories of data are complex for pre-processing, analysing, storing and visualising. Cloud computing provides suitable platform for big data analytics for the storage and for predicting customer behaviour to sell products. Unstructured data like emails, notes, messages, documents, notifications and Twitter comments (including from IoT devices) remains untapped and is not stored in a relational database. Valuable information on pricing, customer behaviour and competitors may be inhumed within unstructured data. This makes cloud-based analytics as an effective research field to address several issues and risks need to be reduced. So we propose a method to extract and cluster sentiment information from various types of unstructured text data from social networks by using SVM classifiers combined with lexicons and machine learning for sentiment analysis of customer behaviour feedback. The method has performed efficient data collection, data loading and efficiently performs sentiment analysis on deep and hidden web.

The article substantiates the necessity of applying artificial intelligence (AI) technologies as a digital tool for the project-based approach in local governance. Local self-government plays a critical role in ensuring the functioning of communities, particularly during the current crisis caused by the Russian military invasion. Integrating modern AI-based management approaches enhances process automation, big data analysis, and resource optimization, improving efficiency. The study identifies key directions for AI application in local governance: project planning, control, and timeline management through analytics services; risk analysis and forecasting via machine learning algorithms; task automation for project execution; resource optimization using efficiency indicators; personalized communication with virtual assistants, chatbots, and digital democracy tools; intelligent HR services for team coordination; and informed decision-making based on big data analysis with reduced risks. An algorithm for AI adaptation in local selfgovernment has been proposed, encompassing stages such as task identification, data analysis for decision-making, process automation, resource monitoring, risk management, and personnel training. Examples of AI tools at the local level include «smart city» services for infrastructure monitoring, transportation optimization, and waste management; digital participation tools for public opinion analysis; and intelligent systems for strategic planning and tax procedure simplification. The study highlights the ethical implementation of AI with data protection and its benefits, including time savings, transparency, resource optimization, and enhanced communication. Prospective technologies include machine learning, natural language processing, computer vision, big data analysis, blockchain governance, and smart platforms for digital participation. These tools promote sustainable community development and improve public service delivery, ensuring higher quality of life for residents.

The generation of a huge volume of structured, semi-structured and unstructured real-time health monitoring data and its storage in the form of electronic health records (EHRs) need to be processed and analyzed intelligently to provide timely healthcare. A big data analytic platform is an alternative to the traditional warehouse paradigms for the processing, analysis and storage of the tremendous volume of healthcare data. However, the manual analysis of these voluminous, multi-variate patients data is tedious and error-prone. Therefore, an intelligent solution method is highly essential to perform multiple correlation analyses for disease diagnosis and prediction. In this paper, first, a structural framework is proposed to process the huge volume of cardiological big data generated from the hospital and patients. Then, an intelligent analytical model for the cardiological big data analysis is proposed by combining the concept of artificial neural network (ANN) and particle swarm optimization (PSO) to predict the abnormalities in the cardiac health of a person. In the proposed cardiac disease prediction model, an extensive electrocardiogram (ECG) data analysis method is developed to identify the probable normal and abnormal cardiac feature points. Simulation results show the effects of a number of attributes for improving the accuracy of the cardiac disease prediction and data processing time in the cloud with an increase in the number of the cardiac patients.

Big data analytics requires technologies to efficiently process large quantities of data. Moreover, especially in decision making, it not only requires individual intellectual capabilities in the analytical activities but also collective knowledge. Very often, people with diverse expert knowledge need to work together towards a meaningful interpretation of the associated results for new insight. Thus, a big data analysis infrastructure must both support technical innovation and effectively accommodate input from multiple human experts. In this chapter, we aim to advance our understanding on the synergy between human and machine intelligence in tackling big data analysis. Sensemaking models for big data analysis were explored and used to inform the development of a generic conceptual architecture as a means to frame the requirements of such an analysis and to position the role of both technology and human in this synergetic relationship. Two contrasting real-world use case studies were undertaken to test the applicability of the proposed architecture for the development of a supporting platform for big data analysis. Reflection on this outcome has further advanced our understanding on the complexity and the potential of individual and collaborative sensemaking models for big data analytics.

Big data technology is widely spread around the world, and is constantly developing and applying. In order to enhance the application value of big data analysis platform, it is necessary to constantly improve the data analysis and processing capacity of big data platform, so as to build a complete data analysis platform, realize resource sharing and real-time data collection. As a key point of contemporary information development, big data analysis platform is of great significance to promote social data exchange. Based on this, this paper focuses on two aspects: first, it describes the construction process and content of big data platform; second, it summarizes the relevant application and development of big data platform for reference.

1935P AI-driven big data analytics for effective response of cancer therapy (AIBERT drug discovery and development platform)

Data integrity verification of the outsourced big data in the cloud environment: A survey

The use of the big data analytics (BDA) platform is increasingly becoming prevalent in the data sciences. However, BDA processes consume resources and time excessively. Automating BDA processes is a cognitive approach to the BDA domain, which is most impaired by its heavy consumption of time and resources. However, the BDA workflow is highly dependent on diversified constraints because of the high variability, veracity, and volume of data processing to accomplish highly influential and sophisticated requirements. The workflow has to pass rigorous and diverse data mining steps, each step contains several tasks and these tasks made by many sub tasks, that it must be accomplished to progress to the next step in the workflow. This increases the available solution space for BDA processes. The intelligent heuristic approach is needed to address the domain-specific concerns which are large solution space, and awareness of constraints are caused the BDA planning. Therefore, we propose to use GraphPlan-based dynamic workflow generation for the BDA domain. Our empirical studies prove that the proposed sophisticated method satisfied planning requirements and outperformed a related planning technique.

Distributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.

In recent years, Data Mining techniques such as classification, clustering, association, regression etc. are widely used in healthcare field to help analyzing and predicting disease and improving the quality and efficiency of medical services. This paper presents a web-based platform for big data analysis of healthcare using Data Mining techniques. The platform consists of three main layers: Apache Spark Layer, Workflow Layer and Web Service Layer. Apache Spark Layer provides basic Apache Spark functionalities as regular Resilient Distributed Datasets (RDD) operations. Meanwhile, this layer provides a cache mechanism to maximize the use of the results as much as possible which were calculated before. Workflow Layer encapsulates a variety of nodes for Data Mining, which have different roles such as data source, algorithm model or evaluation tool. These nodes can be organized into a workflow which is a directed acyclic graph (DAG), and then it will be submitted to Apache Spark Layer to execute. And we have implemented many models including Naive Bayes model, Decision Tree model and Logistic Regression model etc. for healthcare big data. Web Service Layer implements rich restful API including data uploading, workflow composition and analysis task submission. We also provide a web graphical interface for the user. Through the interface users can achieve efficient Data Mining without any programming which can greatly help the medical staff who don’t understand programming to diagnose the patients’ condition more accurately and efficiently.

With the accelerated process of agricultural modernization, the accurate acquisition of agricultural environmental information has become a major trend. A field microclimate monitoring system based on wireless sensor network is constructed based on the latest concept model of the Internet of things and fuzzy control theory; it mainly composed of data acquisition system, data storage system and visualization platform for big data analysis. The data acquisition system is highly integrated with data acquisition and data transmission to obtain real-time data of farmland environment in different terrain areas, including meteorology, hydrology, soil, growth etc. Wireless transmission will transmit real-time data through the GPRS network to the big data analysis platform. The big data analysis platform presents the site data information and analyzes the rule of historical data through visualization technology, realizes the meteorological disaster early warning and forecast, and provides effective decision-making service information based on the agricultural fuzzy theory. Finally, we analyze all kinds of hardware interference problems and software defects encountered in the debugging process, and propose new solutions through the experimental data obtained from actual production applications. It has been proved that the field microclimate monitoring system runs steadily and meets the demand of agricultural monitoring.

A project titled “Establishing the advanced disaster reduction management system by fusion of real-time disaster simulation and big data assimilation,” was launched as Core Research for Evolutional Science and Technology (CREST) by the Japan Science and Technology Agency (JST). Intended to save as many lives as possible in future national crises involving earthquake and tsunami disasters, the project works on a disaster mitigation system of the big data era, based on cooperation of large-scale, high-resolution, real-time numerical simulations and assimilation of real-time observation data. The world’s most advanced specialists in disaster simulation, disaster management, mathematical science, and information science work together to create the world’s first analysis platform for real-time simulation and big data that effectively processes, analyzes, and assimilates data obtained through various observations. Based on quantitative data, the platform designs proactive measures and supports disaster operations immediately after disaster occurrence. The project was launched in 2014 and is working on the following issues at present.Sophistication and fusion of simulations and damage prediction models using observational big data: Development of a real-time simulation core system that predicts the time evolution of disaster effect by assimilating of location information, fire information, and building collapse information which are obtained from mobile terminals, satellite images, aerial images, and other new observation data in addition to sensing data obtained by the undersea high-density seismic observation network.Latent structure analysis and major disaster scenario creation based on a huge amount of simulation results: Development of an analysis and extraction method for the latent structure of a huge amount of disaster scenarios generated by simulation, and creation of severe scenarios with minimum “unexpectedness” by controlling disaster scenario explosion (an explosive increase in the number of predicted scenarios).Establishment of an earthquake and tsunami disaster mitigation big data analysis platform: Development of an earthquake and tsunami disaster mitigation big data analysis platform that realizes analyses of a huge number of disaster scenarios and increases in speed of data assimilation, and clarifies the requirements for operation of the platform as a disaster mitigation system.The project was launched in 2014 as a 5-year project. It consists of element technology development and system fusion, feasibility study as a next-generation disaster mitigation system (validation with/without introduction of the developed real-time simulation and big data analysis platform) in the affected areas of the Great East Japan Earthquake, and test operations in affected areas of the Tokyo metropolitan earthquake and the Nankai Trough earthquake.

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate (CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System (HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.