Cleaning method for abnormal energy big data based on sparse self-coding

With the rapid development of wind farm worldwide, monitoring the status of numerous wind turbines becomes the essential work. Abnormal data in wind power curve (WPC) are quite important for wind farm operations and maintenances because they usually reveal wind turbine failures or some extreme conditions. This paper proposes a new algorithm of WPC abnormal data detection and cleaning by image thresholding based on minimization of dissimilarity-and-uncertainty-based energy (MDUE). The basic idea is to transform the scattered data into a digital image and the problem of data cleaning is turned into an image segmentation problem. For all data pixels, the confidences of being classified as normal class are computed and make up a grey level feature image. Then the optimum threshold is determined by searching through the energy space based on intensity-based class uncertainty and shape dissimilarity. Finally, the normal and three types of abnormal data are marked after applying image thresholding to the feature image. The algorithm is compared with several data-based algorithms and a recently published image-based algorithm. A large number of experiments conducted on real-world WPC data collected from 37 wind turbines in two wind farms verified the superior performance of the proposed method.

Various factors such as communication interference or failures, maintenance or protection shutdowns of wind turbines, and downrating under grid dispatch control contribute to the presence of significant abnormal data in the actual operating data collected from wind turbines. In this manuscript, a method is proposed to identify abnormal data in wind power using boundary modeling. The abnormal data are classified into three types based on distribution characteristics. A stepwise procedure is presented for identifying abnormal data. The bottom-stacked abnormal data are initially identified using rule-based filtering. The scattered abnormal data are preliminarily identified through region partition using clustering, combined with the quartile method. Datasets comprising wind speed range and mean wind speed at various wind power levels are acquired. A boundary model is developed by analyzing the correlation between wind power and respective wind speed ranges, as well as the correlation between power and corresponding mean wind speeds, to refine the identification of middle-stacked and scattered abnormal data. Finally, the effectiveness and universality of the proposed method are verified using both actual data and simulated data. Quantitative analysis is performed using wind power curve modeling to evaluate the accuracy of the abnormal data identification results.

Wind farm power data is an essential data source for characteristics analysis of wind farms, and its precision highly influences the operation and control of wind power. Existing methods for abnormal wind power data detection are mainly based on supervised or unsupervised algorithms, which may suffer from huge effort consumption on artificial label-setting or improvable detection precision respectively. In this paper, a semi-supervised anomaly detection method based on Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is proposed. Supervision from a limited number of labelled data is utilized to guide the anomaly detection process. Simulation based on realistic wind farm power data from China is performed to verify the method validity. The simulation results illustrate that the proposed method can obtain a satisfactory performance on avoiding false dismissals and false alarms among irregular-shaped wind power clusters, as well as an improved distinguishing ability around the confusing boundaries between normal and abnormal data.

Aiming at a large number of abnormal data in wind turbine operation data, considering the wind turbine operation characteristics and data clustering, this paper proposes a method for cleaning abnormal power data of wind turbines based on high-dimensional space clustering. First, based on the control strategy differences and operating mechanisms of wind turbines, the mechanism cleaning of abnormal wind power data is performed; Second, consider the wind speed, rotor speed, and power, and use Gaussian mixture model (GMM) to cluster the data in three-dimensional space to achieve the preliminary cleaning of abnormal power data of wind turbines; Finally, on the basis of the preliminary cleaned data, an optimized multi-directional quartile method is used in the Copula space for refined cleaning. The actual data of a certain type of wind turbine in North China is selected for example analysis. The simulation results show that the method proposed in this paper is reasonable and effective.

In order to improve the effect of network public opinion data abnormal behavior detection, an artificial intelligence-based network public opinion data abnormal behavior detection method is proposed. By constructing the network public opinion data model, recognizing the evolution rule of network public opinion data, locating the abnormal data area according to the behavior detection algorithm, and using the probability neural network under artificial intelligence to detect the abnormal data behavior. The experimental results show that the detection method proposed this time is 28.12% and 84.37% higher than the two traditional methods when detecting large-scale public opinion abnormal behavior data. It can be seen that the detection method based on artificial intelligence is not restricted by the volume of network data, and the detection effect is better.KeywordsArtificial intelligenceNetwork public opinionData abnormal behaviorDetection method

Aiming at the problem that the detection of abnormal data in multidimensional data sets is not processed by dimensionality reduction, which leads to low detection accuracy, high false positive rate and long detection time, a method for detecting abnormal data in multidimensional data sets based on spectral clustering is proposed. Firstly, the data in the multidimensional data set is clustered by Laplacian matrix to preliminarily classify the data; secondly, the local linear embedding (LLE) algorithm is used to reduce the dimensionality of the classified data, and the high-dimensional data set is expressed by feature vectors to remove redundant information in the multidimensional data set; finally, the processed multidimensional data set is input into the support vector machine model, and the abnormal data is detected according to the calculation of regression estimation value. Experimental results show that the proposed algorithm has higher accuracy, lower false positive rate and shorter detection time for detecting abnormal data in multidimensional data sets.

As a key component in the hydraulic brake system of automobiles, the brake joint directly affects the braking performance and driving safety of the vehicle. Therefore, improving the quality of brake joints is crucial. During the processing, due to the complexity of the welding material and welding process, the weld seam is prone to various defects such as cracks, pores, undercutting, and incomplete fusion, which can weaken the joint and even lead to product failure. Traditional weld seam detection methods include destructive testing and non-destructive testing; however, destructive testing has high costs and long cycles, and non-destructive testing, such as radiographic testing and ultrasonic testing, also have problems such as high consumable costs, slow detection speed, or high requirements for operator experience. In response to these challenges, this article proposes a defect detection and classification method for laser welding seams of automotive brake joints based on machine vision inspection technology. Laser-welded automotive brake joints are subjected to weld defect detection and classification, and image processing algorithms are optimized to improve the accuracy of detection and failure analysis by utilizing the high efficiency, low cost, flexibility, and automation advantages of machine vision technology. This article first analyzes the common types of weld defects in laser welding of automotive brake joints, including craters, holes, and nibbling, and explores the causes and characteristics of these defects. Then, an image processing algorithm suitable for laser welding of automotive brake joints was studied, including pre-processing steps such as image smoothing, image enhancement, threshold segmentation, and morphological processing, to extract feature parameters of weld defects. On this basis, a welding seam defect detection and classification system based on the cascade classifier and AdaBoost algorithm was designed, and efficient recognition and classification of welding seam defects were achieved by training the cascade classifier. The results show that the system can accurately identify and distinguish pits, holes, and undercutting defects in welds, with an average classification accuracy of over 90%. The detection and recognition rate of pit defects reaches 100%, and the detection accuracy of undercutting defects is 92.6%. And the overall missed detection rate is less than 3%, with both the missed detection rate and false detection rate for pit defects being 0%. The average detection time for each image is 0.24 s, meeting the real-time requirements of industrial automation. Compared with infrared and ultrasonic detection methods, the proposed machine-vision-based detection system has significant advantages in detection speed, surface defect recognition accuracy, and industrial adaptability. This provides an efficient and accurate solution for laser welding defect detection of automotive brake joints.

Abnormal signals are inevitable in big data acquired from harsh industrial environments. Abnormal data detection is a crucial component of condition monitoring for rotating parts and is also the premise of data cleaning, compensation, and mining. To detect abnormal data segments of rolling bearings, this paper proposes a dynamic adaptive local outlier factor (DALOF) anomaly detection method. First, a data dynamic segmentation method based on sliding windows is designed to determine samples with variable lengths. Then, a time-domain feature extraction and fusion method based on principal component analysis is exploited to reduce the feature space discrepancy. To improve the accuracy of abnormal data detection, a data quality evaluation model is established to assess each data segment using DALOF. The validity of the proposed method is also verified by analyzing signals including missing data, random interference data, and drift data. Several other methods are respectively applied to identify these abnormal data to further demonstrate the benefits of the developed methodology.

The high proportion abnormal data in photovoltaic power generation output has the characteristics of overlapping and multi Gaussian distribution. How to accurately remove the high proportion abnormal data in photovoltaic power generation output is a key issue in achieving accurate prediction of photovoltaic power. The article analyzes the characteristics of high proportion abnormal data in photovoltaic power generation output, and studies a detection method for high proportion abnormal data in photovoltaic power generation output based on Gaussian mixture model. We have established an algorithm model for the Gaussian mixture model, using the Expectation Maximization (EM) algorithm to estimate the parameters of the Gaussian mixture model, and using the algorithm model to detect and eliminate high proportion abnormal data in photovoltaic power generation output. Actual case analysis and comparative experiments show that the method proposed in the article can accurately detect high proportion abnormal data from multiple classifications, making it more suitable for detecting high proportion abnormal data in photovoltaic power generation output compared to traditional abnormal data detection methods.

An advanced entropy-based feature with a frame-level vocal effort likelihood space modeling for distant whisper-island detection

With the development of science and technology, China's highway construction and development is faster and faster, and the highway safety accidents are also increasing. The traditional highway mobile violation detection system has the shortcomings of high cost, difficult maintenance, and low precision, high rate of missing detection. Therefore, this paper proposes the research of highway mobile violation detection robot based on machine vision research. On the basis of machine vision, this paper constructs a mobile illegal detection robot system for expressway. In addition, we use the combination of background subtraction and frame subtraction to detect moving objects. In the research, we select the accident prone section of an expressway, and take retrograde, parking, lane changing and speeding as the indicators to test the robot system in this paper. The results show that the correct detection rate is 90.9%, the missing detection rate is 4.55%, and the false detection rate is 4.55%. It can be seen that the robot based on machine vision has the advantages of high correct detection rate, low missing detection rate and false detection rate. In addition, this paper based on machine vision of highway mobile illegal detection robot is relatively small, simple structure, low cost, has great advantages in practical applications.

During the operation of wind turbines, there are a lot of abnormal data collected from wind power data, which cannot be used to truly evaluate the performance and operation state of wind turbines. According to the distribution characteristics of these abnormal data, an outlier delete method using Bayesian change point-quartile combined algorithm is proposed. Firstly, to identify and delete the abnormal data below wind power curve, based on the Bayesian theorem, change point position is identified. Finally, the quartile method is utilized to delete the upper sparse outliers. The proposed method is applied to 10-min field monitoring wind power data, root mean square error (RMSE), program running time and the rejection rate of field wind turbine data are used to measure model accuracy, the results show that compared with the least squares (LS) change point method, the method proposed in this paper is feasible and effective.

The accuracy of the pitch angle deviation directly affects the guidance accuracy of the laser seeker. During the guidance process, the abnormal pitch angle deviation data will be produced when the seeker is affected by interference sources. In this paper, a new abnormal data detection method based on Smooth Multi-Kernel Polarization Support Vector Data Description (SMP-SVDD) is proposed. In the proposed method, the polarization value is used to determine the weight of the multi-kernel combination coefficient to obtain the multi-kernel polarization function, in which the particle swarm optimization is used to find the optimal kernels for higher detection accuracy. Besides, by using smoothing mechanism, the constrained quadratic programming problem is translated to be smooth and differentiable. Then, this problem can be solved by the conjugate gradient method, which could reduce the computational complexity. In experimental section, abundant simulation experiments were designed and the experimental results verify that the proposed SMP-SVDD method could achieve higher detection accuracy and low computational cost compared with different detection methods in different guidance stages.

Principal component analysis (PCA) is widely used for anomaly detection, specifically by computing the distances of each point in the dataset to the (sample) subspace spanned by the principal components of the sample covariance matrix. Points with distances above a threshold are labeled as anomalous. Although it is typically unknown in practice, the distribution from which the sample is drawn has its own (population) subspace, spanned by the principal components of the distribution's covariance matrix, and the thresholded distances of each point in the dataset from this subspace determine the true labels of the points. It follows that the sample covariance matrix produces false alarms (points labeled anomalous that are not) and missed detections (points labeled non-anomalous that are). In this paper we study how the false alarm rate (FAR) and missed detection rate (MDR) depend upon the number of samples, focusing on the case where the sample points are drawn from a normal distribution with a covariance matrix having a single spike.

One of the most basic characteristic features of every smart device in a network based on the Internet of Things (IoT) is to gather a larger set of data that has been created and then transfer the gathered data to the destination/receiver server through the internet. Thus, IoT-based networks are most vulnerable to simple or complex attacks that need to be identified in the early stage of data transmission for saving the network from these malicious attacks. The chief goal of the proposed work is to design and form the intelligent intrusion detection system (I-IDS) using the machine learning models such that the attacks can be identified in the IoT network. The model is built considering the normal and malicious attacks on the data that are generated in IoT smart environment. To simulate such a model, a testbed is built where a wireless router, a DHT11 sensor, and a node MCU are being used during the design phase. An attacker or adversarial system is built to perform poisoning and sniffing attacks using a laptop system. The node captures the sensor values and transmits the data to the ThinkSpeak platform, during the normal phase via the wireless gateway, and in the attack phase, the malicious attacker interprets the data, modifies it while transmitting from node to the ThinkSpeak server. Thus, the attack called Man-In-The-Middle (MITM) is performed and classified as abnormal data. Various machine learning algorithms are performed on the data, and finally, the results obtained using a probabilistic model called as Markov model have a high performance evaluated based on the I-IDS IoT network. The results obtained during the experimental analysis show that the Markov model has obtained a 100% detection rate and 19% of false alarm rate (FAR) with high precision and low error rate. The algorithms such as naïve Bayes classifier, support vector machine (SVM), decision tree, and Adaboost are considered in comparison with the Markov model. The optimal solution is obtained concerning other evaluation metrics like sensitivity, F1, and true-positive rate (TPR). Therefore, the integrated network of IoT-WSN with its performance metrics is tabulated to show the potentials of securing a network system. Additionally, the proposed work gives a high level of security for IoT smart environment as compared with the other machine learning algorithms using the novel technique of intelligent IDS.