Comparison of the Statistical and Autoencoder Approach for Anomaly Detection in Big Data

Outlier detection (or anomaly detection) is a fundamental task in data mining. Outliers are data that deviate from the norm and outlier detection is often compared to “finding a needle in a haystack.” However, the outliers may generate high value if they are found, value in terms of cost savings, improved efficiency, compute time savings, fraud reduction and failure prevention. Detection can identify faults before they escalate with potentially catastrophic consequences. Big Data refers to large, dynamic collections of data. These vast and complex data appear problematic for traditional outlier detection methods to process but, Big Data provides considerable opportunity to uncover new outliers and data relationships. This chapter highlights some of the research issues for outlier detection in Big Data and covers the solutions used and research directions taken along with an analysis of some current outlier detection approaches for Big Data applications.

Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.

The definitions of anomaly detection and big data were presented. Due to the sampling and storage burden of anomaly detection in big data, compressive sensing theory was introduced and used in anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The anomaly detection method was used for through wall human detection. The experiments for detecting human behind Brick wall based on UWB radar signal was carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of human being through compressed signals.KeywordsAnomaly detectionBig dataThrough wall human detectionCompressive sensing

In this work, we present the first distributed solution for the Local Outlier Factor (LOF) method -- a popular outlier detection technique shown to be very effective for datasets with skewed distributions. As datasets increase radically in size, highly scalable LOF algorithms leveraging modern distributed infrastructures are required. This poses significant challenges due to the complexity of the LOF definition, and a lack of access to the entire dataset at any individual compute machine. Our solution features a distributed LOF pipeline framework, called DLOF. Each stage of the LOF computation is conducted in a fully distributed fashion by leveraging our invariant observation for intermediate value management. Furthermore, we propose a data assignment strategy which ensures that each machine is self-sufficient in all stages of the LOF pipeline, while minimizing the number of data replicas. Based on the convergence property derived from analyzing this strategy in the context of real world datasets, we introduce a number of data-driven optimization strategies. These strategies not only minimize the computation costs within each stage, but also eliminate unnecessary communication costs by aggressively pushing the LOF computation into the early stages of the DLOF pipeline. Our comprehensive experimental study using both real and synthetic datasets confirms the efficiency and scalability of our approach to terabyte level data.

Physical education and training are essential ways to improve the physical quality of the nation, and China has incorporated "building a healthy China" and "fitness for all" into its national development strategy, integrating a strong sports nation into the Chinese dream. The study of digital recording and automated training in sports is of profound value. Motion capture technology can digitally record the training process in a digital physical education training system. At the same time, accurate modeling and calculation can analyze the training effects and give appropriate guidance and feedback. This study develops a new and improved hierarchical K-means algorithm by combining the known classification algorithm K-means with a hierarchical algorithm. The performance of the old and new algorithms are compared and then applied to physical education training data to produce clustering results and analysis to reduce the model, which is used to reduce the number of parameters in the model and improve the recognition speed. The experimental results demonstrate that the relevant models proposed in this study achieve an average accuracy of 91.27% and 92.26%, respectively, which is better than a single network model and can effectively use big data for health event detection. The empirical results show that the improved model algorithm outperforms the single network model and can detect health events using big data.

Intrusion detection in big data using hybrid feature fusion and optimization enabled deep learning based on spark architecture

In the era of Big Data, where vast datasets fuel innovation across industries, the paramount concern remains safeguarding individual privacy. This article explores how statistical approaches ensure data confidentiality amidst the proliferation of digital information. Differential privacy techniques introduce calibrated noise to protect identities while preserving data utility, crucial for compliance and trust in data-driven decision-making. Secure Multiparty Computation (MPC) enables collaborative analysis without exposing raw data, supporting privacy in sectors like healthcare and finance. Privacy-preserving data mining techniques integrate encryption and anonymization to extract insights while shielding sensitive information. Anonymization and de-identification methods further bolster privacy by masking identifiable data, essential for adhering to stringent regulations like GDPR and HIPAA. As data generation escalates, advancing these statistical methods is essential for maintaining privacy integrity in the evolving landscape of Big Data applications. Keywords: Big Data, Privacy, Statistical approaches, Differential privacy, Secure multiparty computation.

SummaryIn recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.

For the quality of the wine big data identification technology, the introduction of data mining classification algorithm, effectively according to the content of several impact compounds in wine level identification;Are introduced including the Logistic regression and BP neural network and SVM classification algorithm, in view of the three algorithms identify the modeling analysis of wine quality. Data mining is closely related to big data, applying data mining to the wine in the quality detection of big data, can quickly to the quality of the wine.

The unprecedented development of the Internet of Things (IoT), artificial intelligence (AI), and Big Data has stimulated a boom of social networks such as Twitter, WeChat, Facebook, etc., generating a huge amount of social data that are worth further analysis. Social computing has an important focus on mining the deep relationships between social organizations, networks, and media. The increasing volumes and complexities make big social data mining more and more difficult. Hybrid Human–Artificial Intelligence (H-AI) is an approach combining both human intelligence and AI, so as to handle demanding problems in a harmonious way. By adopting H-AI in social computing, it would provide more possibilities for social data analysis, relationship discovery, outlier detection, and prediction, and is proving to be an emerging and promising direction for AI and big data research.

DNA sequencing, imaging procedures, and wireless healthcare monitoring devices are all examples of high-throughput, data-intensive precision medicine assays and technologies that have necessitated new methods for analysing, integrating, and interpreting the enormous volumes of data they produce. While several statistical approaches have been developed to deal with the "big data" generated by such tests, previous experience with artificial intelligence (AI) techniques suggests that they may be especially well-suited. Furthermore, data-intensive biomedical technologies applied to study have shown that people differ greatly at the genetic, biochemical, physiological, exposure, and behavioural levels, particularly with regards to disease processes and treatment receptivity. This indicates the need to 'personalise' medications so that they better suit the complex and often individual needs of each patient. AI can play a significant role in the clinical research and development of new personalised health products, from selecting relevant contribute to sustainable to testing their utility, because of the importance of data-intensive assays in revealing appropriate intervention objectives and approaches for personalising medicines. The work here presents a variety of ways in which AI can contribute to the progress of personalised medicine, and we argue that the success of this endeavour is critically dependent on the improvement of appropriate assays and methods for storing, aggregating, accessing, and ultimately combining the data they generate. In addition, the manuscript also discusses the potential future research directions and highlights the shortcomings of various AI methods.

The rapid growth of data production across diverse fields created big data as an essential area for both academic study and practical application during recent times. The spread of cloud computing provides businesses with an adaptable infrastructure that enables economical and scalable processing and storage of enormous data collections. Popular use of modern data mining methods requires dealing with major technical obstacles while analyzing massive and complicated information collections. Data mining as an intelligent methodology emerged because it unites artificial intelligence methods with machine learning techniques together with statistical approaches and database operations to resolve these difficulties. In this review paper demonstrates extensive details about Intelligent data mining solutions that operate within cloud infrastructure while handling big data. The paper presents essential details about the basic principles followed by a classification scheme which includes techniques like classification and clustering alongside association rule mining with deep learning methods and their applications for big data needs. This study analyzes data mining system integration models for clouds while it assesses their operational abilities and scalability and makes note of essential difficulties including heterogeneity and real-time processing requirements.

This chapter discusses the concept of natural language processing (NLP), specifically how the NLP is applied. The chapter further outlines the typical functions of the NLP. In addition, it explores the NLP in strategy and strategy implementation.The NLP is a branch of computer science, AI and linguistics that focuses on the interaction between computer programs and human language. There are three functions of the NLP, namely, machine translation (MT), text summarisation and sentiment analysis (SA). MT is a branch of computer linguistics that analyses the use of computerised tools to translate the context derived from this, from one language to the next. The language that is being translated is the human language.Automatic text summarisation refers to a process of bringing about a succinct and meaningful summary of the text. SA is a way of finding out the polarity or strength of the opinion that is expressed in written text. The opinion could be either positive or negative. MT has three major approaches, namely, rule-based MT, statistical MT and neural MT. A rule-based system would typically require the expert to know both the source and the target languages. This knowledge is then used to develop syntactic, semantic and morphological rules to achieve the translation. The statistical approach uses statistical models based on the analysis of bilingual text corpora. The neural MT approach uses neural networks to achieve machine translation. There are two broad categories of text summarisation approaches, namely, extractive summarisation and abstractive summarisation. Furthermore, there are four popular types of SA, namely, the fine-grained SA, emotion detection, aspect-based SA and multilingual SA. The firm that deploys the NLP and ML in Big Data Analytics would gain crucial and better insights, setting the firm apart from its competitors.

Feature engineering in big data analytics for IoT-enabled smart manufacturing – Comparison between deep learning and statistical learning

Climate change and global warming are among the most important environmental issues and require sufficient and urgent global action to protect the Earth for future generations. One of the main approaches used to reduce CO2 emissions and mitigate the worst impacts of climate change is carbon capture technology. Carbon capture technologies have the potential to capture carbon from the atmosphere and convert it into fuels that can be used in environmentally friendly energy production. Innovative technologies can increase the potential for carbon capture, which can play an important role in combating climate change. A better understanding of the mechanisms that extract, store and release carbon from the atmosphere will allow us to make more accurate predictions for assessing carbon capture potentials. There have been many efforts by scientists, industry sectors and policy makers in the search for new technologies to reduce greenhouse gas emissions and achieve net zero emissions targets. Research and development work to create new technology involves complex processes and requires a digital system to optimize big data forecasting as well as reduce production time. A mathematical and statistical approach such as machine learning plays an important role in solving research problems, providing fast results and cost-effective tools for predicting big data. Effective policies and international cooperation for carbon capture can increase the potential for carbon capture. New policies and cooperation models can incentivize investment in carbon capture projects, which can increase potential. These new approaches can be used to better understand the potential for carbon capture and develop effective solutions to tackle climate change. However, work in this area is still ongoing and more research and development is needed in the future.