Fault detection and explanation through big data analysis on sensor streams

Data in the maritime domain is growing at an unprecedented rate, e.g., terabytes of oceanographic data are collected every month, and petabytes of data are already publicly available. Big data from heterogeneous sources such as sensors, buoys, vessels, and satellites could potentially fuel a large number of interesting applications for environmental protection, security, fault prediction, shipping routes optimization, and energy production. However, because of several challenges related to big data and the high heterogeneity of the data sources, such applications are still underdeveloped and fragmented. In this paper, we analyze challenges and requirements related to big maritime data applications and propose a scalable data management solution. A big data architecture meeting these requirements is described, and examples of its implementation in concrete scenarios are provided. The related data value chain and use cases in the context of a European project, BigDataOcean, are also described.

Outlier detection (or anomaly detection) is a fundamental task in data mining. Outliers are data that deviate from the norm and outlier detection is often compared to “finding a needle in a haystack.” However, the outliers may generate high value if they are found, value in terms of cost savings, improved efficiency, compute time savings, fraud reduction and failure prevention. Detection can identify faults before they escalate with potentially catastrophic consequences. Big Data refers to large, dynamic collections of data. These vast and complex data appear problematic for traditional outlier detection methods to process but, Big Data provides considerable opportunity to uncover new outliers and data relationships. This chapter highlights some of the research issues for outlier detection in Big Data and covers the solutions used and research directions taken along with an analysis of some current outlier detection approaches for Big Data applications.

The detection of incipient faults of the current fault diagnosis systems in Nuclear Power Plants is inherently limited. Active research in machine learning algorithms like Adaptive Neuro-Fuzzy Inference System (ANFIS) is providing promising results in the prediction of faults. This paper explored four different configurations of Adaptive Neuro-Fuzzy Inference System (ANFIS) methodology in a bid to come up with a superior model that not only had a high sensitivity in the detection of incipient faults but also had superior prediction capabilities. The data-driven ANFIS schemes were used to predict a sensitive fault signature and to evaluate the models, Small Break Loss of Coolant Accident (SBLOCA) transient events were modeled in Qinshan I Nuclear Power Plant. Coefficient of determination, normal probability plot of residuals and mean absolute percent error were used to assess the competencies of the estimation of the models.

To enhance the accuracy of current anomaly detection and fault prediction in power systems, a CNN-LSTM hybrid model based on deep learning is proposed. By combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks, the model effectively extracts spatiotemporal features and captures long-term temporal dependencies. A temporal attention mechanism is introduced to improve the model’s ability to perceive critical temporal sequences, while an adaptive threshold setting algorithm is designed to accommodate the dynamic changes of the power grid. Experimental results demonstrate that the proposed method exhibits excellent performance in anomaly detection and fault prediction, significantly improving accuracy and robustness compared to traditional methods, and showing promising application prospects. Analysis indicates that the model effectively addresses the challenges of current anomaly detection and demonstrates high stability in multi-step fault prediction.

This paper proposes a fault detection method based on an extended state observer (ESO). This method of fault detection is achieved in two-stage process, residual generation and fault prediction. Residual generation uses an ESO to estimate the state variables of the three-tank system. Fault prediction uses an ellipsoid threshold method to predict the faults. Twelve kinds of stepwise faults are detected successfully in the three-tank system. The experimental results elaborate the effectiveness of the online fault detection based on ESO.

Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.

The amount of data in the maritime domain is rapidly increasing due to the increase in devices that can collect marine information, such as sensors, buoys, ships, and satellites. Maritime data is growing at an unprecedented rate, with terabytes of marine data being collected every month and petabytes of data already being made public. Heterogeneous marine data collected through various devices can be used in various fields such as environmental protection, defect prediction, transportation route optimization, and energy efficiency. However, it is difficult to manage vessel related data due to high heterogeneity of such marine big data. Additionally, due to the high heterogeneity of these data sources and some of the challenges associated with big data, such applications are still underdeveloped and fragmented. In this paper, we propose the Vessel Data Lakehouse architecture consisting of the Vessel Data Lake layer that can handle marine big data, the Vessel Data Warehouse layer that supports marine big data processing and AI, and the Vessel Application Services layer that supports marine application services. Our proposed a Vessel Data Lakehouse that can efficiently manage heterogeneous vessel related data. It can be integrated and managed at low cost by structuring various types of heterogeneous data using an open source-based big data framework. In addition, various types of vessel big data stored in the Data Lakehouse can be directly utilized in various types of vessel analysis services. In this paper, we present an actual use case of a vessel analysis service in a Vessel Data Lakehouse by using AIS data in Busan area.

Advancements in sensing technology and wireless networks allow digital communication without human interferences. It is referred to as Industry 4.0, aiming to automate and decentralize product manufacturing. Nowadays, it has emerged widely in various industries such as health care, stock market, enterprise resource planning, industrial manufacturing, and so on. It produces enormous amounts of data with different formats, which can’t be fit in existing databases (i.e., relational database management system). Here, big data comes with a vital role to transform and manage the produced data (sensory data) into actionable ideas. It’s able to discover the patterns, trends, preferences among data and establish meaningful relations. Big data boosts the efficiency of industry with respect to product development, cost and supply chain optimization, fault prediction, and the like. The authors present an idea of the closeness between big data and blockchain in the industrial sector. The review part of this chapter summarizes the auxiliary tools and methods of big data that have been proposed recently. This research work further highlights recent inclinations, prospects, and downsides of big data in Industry 4.0.

With the development of smart grid and the rapid expansion of power grid scale, it is very difficult to grasp the operational state of power equipment timely and accurately. Combined with the current status of the application of big data technology and data mining analysis methods in equipment status assessment, this paper expounds the connotation and purpose, data sources and characteristics, basic framework and key technical issues involved in big data analysis of power equipment status, and summarizes and discusses big data. Typical application scenarios and application effects of technology in power equipment status assessment, and propose challenges in research and application as well as future development trends.

Data is becoming the world's new natural resource and big data use grows quickly. The trend of computing technology is that everything is merged into the Internet and 'big data' are integrated to comprise complete information for collective intelligence. With the increasing size of big data, refining big data themselves to reduce data size while keeping critical data (or useful information) is a new approach direction. In this paper, we provide a novel data consumption model, which separates the consumption of data from the raw data, and thus enable cloud computing for big data applications. We define a new Data-as-a-Product (DaaP) concept, a data product is a small sized summary of the original data and can directly answer users' queries. Thus, we separate the mining of big data into two classes of processing modules: the refine modules to change raw big data into small sized data products, and application-oriented mining modules to discover desired knowledge further for applications from well-defined data products. Our practices of mining big stream data, including medical sensor stream data, streams of text data and trajectory data, demonstrated the efficiency and precision of our DaaP model for answering users' queries.

SummaryIn recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.

Software fault prediction plays a crucial role in enhancing software quality and reliability by enabling early detection of defect-prone modules. This study proposes a machine learning-based predictive framework to identify software faults using historical software metrics and defect data. The objective is to evaluate and compare the performance of various machine learning algorithms—Random Forest, XGBoost, Support Vector Machine (SVM), and Neural Networks—for their effectiveness in fault detection. Public benchmark datasets comprising object-oriented metrics such as CK, Halstead, and McCabe were used after undergoing pre-processing steps including normalization; class balancing using SMOTE, and feature selection through PCA and RFE. Models were trained and tested using stratified k-fold cross-validation, and evaluated using metrics such as Accuracy, Precision, Recall, F1-score, and ROC-AUC. Among the models, XGBoost outperformed others, achieving the highest F1-score and ROC-AUC, followed by Random Forest. Feature importance analysis and SHAP value interpretation provided insights into key predictors contributing to software faults, enhancing the model's transparency and practical usability. The study demonstrates that ensemble learning methods significantly improve the fault prediction process, making them suitable for deployment in real-world software quality assurance workflows. These findings contribute to the ongoing advancement of predictive software engineering and open new avenues for integrating intelligent fault detection systems in continuous integration and DevOps environments.

The definitions of anomaly detection and big data were presented. Due to the sampling and storage burden of anomaly detection in big data, compressive sensing theory was introduced and used in anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The anomaly detection method was used for through wall human detection. The experiments for detecting human behind Brick wall based on UWB radar signal was carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of human being through compressed signals.KeywordsAnomaly detectionBig dataThrough wall human detectionCompressive sensing

Vibration Analysis on motor parts has been done before fitment to detect minute surface and sub-surface cracks and other defects. While there are many other fault detection methods, vibration analysis has been used due to its low requirements in equipment, resources and fast data acquisitions. Vibration Condition Monitoring is the process through which the current working condition of a machine can be ascertained by using vibration analysis. This paper deals with the condition monitoring and prediction of faults in a two- motor. Since each part in the motor is distinct, during working, each part produces its own distinct vibration signature. Using accelerometers an ADXL 335 and an in- built Smartphone accelerometer, the vibrations produced were recorded. Faults were introduced into the motor by replacing the normal parts with faulty parts. These data were imported to MATLAB software. From the vibration data recorded before and after the introduction of the faults, time and frequency domain plots were created. This work present a method for fault detection of motor using vibration signals and with use of FFT, location of fault can be detected.

As explained in the literature, it is very hard to measure premise variables of a Takagi-Sugeno (TS) fuzzy system. Therefore, in this paper, a fault detection and prediction (FDP) scheme is designed for a class of TS fuzzy systems with immeasurable (unknown) premise variables and external disturbances. A fault detection (FD) observer is designed to approximate the system output and the premise variables. Subsequently, a FD residual is generated by comparing the observer output with respect to the system output. The FD residual is evaluated to detect any faults in the system. Further, time-to-failure (TTF) of the TS fuzzy system is obtained by using a mathematical equation. Note the parameter update law and TTF scheme utilize the approximated premise variables since they are not measurable. Stability of the fault detection and TTF prediction results are verified using Lyapunov theory. Finally, a simulation study using a truck-trailer system is presented to verify the theoretical claims.