The Application of Spark-Based Gaussian Mixture Model for Farm Environmental Data Analysis

PurposePower transformers are vital components of an electrical network. A defective transformer can cause instability and blackouts in parts of the network. An accurate classification of different transformer faults results in a relatively accurate fault diagnosis and timely corrective actions. It is possible to increase productivity and reduce costs by using fault detection of power transformers through the analysis of gases dissolved in oil. The proposed technique is a suitable tool to help the utilities and engineers in charge of preventive maintenance by reducing the costs of different fault diagnosis tests for power transformers.Design/methodology/approachIn this paper, the IEC 60599 standard along with clustering and classification methods are used to classify power transformer’s fault types. K-means and Fuzzy C-means clustering methods are used for clustering, and the support vector machine (SVM) method is used for classification of different types of faults in ‎power ‎transformers. The performance of K-means and SVM methods is improved by using the Grasshopper Optimization Algorithm (GOA). The efficiency of the proposed methods is evaluated using real field data of power transformers. The purpose of this study is to propose hybrid methods including K-means-GOA clustering and SVM-GOA classification for accurate fault diagnosis. These methods have been used for the first time in fault diagnosis determination of power transformers through gas analysis. The Silhouette criteria is used in this paper to compare the efficiency of different clustering methods.FindingsSimulation results of the paper are based on the gas chromatography data related to 266 different real power transformers. They show the high accuracy and high-performance speed of intelligent clustering and classification methods compared to conventional ones. This analysis would be helpful in performing the required maintenance check and plan for repairs.Originality/valueThe applicability and efficiency of the proposed hybrid K-means-GOA and SVM-GOA models are verified for transformer fault detection using the experimental diverse data set including 266 set of real field test parameters of power transformers.

Heartbeat modeling allows to detect anomalies that reflect the functioning of the heart. Certain approaches face this problem by using Gaussian Mixture Models (GMMs) and other statistical classifiers by extracting the fiducial points provided by the MIT-BIH database. In this work, MIT-BIH database heartbeats are modeled into different heartbeat types from a single subject by using the Gibbs Sampling (GS) algorithm. Firstly, a data pre-processing step is performed; this step involves several tasks such as filtering the raw signals from the MIT-BIH database and reducing the heartbeat types to five. Secondly, the GS is applied to the resulting signals of one subject. Thirdly, the Euclidean distance between each heartbeat type is calculated, and lastly, the Bhattacharyya distance is used to classify heartbeats. The results obtained by the GS algorithm were also compared to results obtained by applying the Expectation Maximization (EM) algorithm to the same data-set. Results allow to conclude that GS is a proper solution for separating each heartbeat type; by providing a significant difference between each heartbeat type which can be used for classification.

Network equivalence is a technique useful for many areas including power systems. In many power system analyses, generation shift factor (GSF)-based bus clustering methods have been widely used to reduce the complexity of the equivalencing problem. GSF captures power flow on a line when power is injected at a node using bus to bus electrical distance. A more appropriate measure is the one which captures what may be called the electrical line distance with respect to a bus termed as relative bus to line distance. With increase in power transactions across different regions, the use of relative bus to line distance becomes appropriate for many analyses. Inspired by the recent trends in network science on the study of network dynamics based on the topological characteristics of a network, in this paper, we present a bus clustering method based on average electrical distance (AED). AED is independent of changes in location of slack bus and is based on the concept of electrical distance introduced in the context of molecular chemistry and pursued later for applications in social and complex networks. AED represents the AED from a bus to buses of the transmission line of interest. We first propose an AED-based method to group the buses into clusters for power systems network equivalence using k-means clustering algorithm integrated with silhouette analysis. One limitation of this method is that despite its speed, sometimes it may yield clusters of inferior quality compared to the optimal solution. To overcome this limitation, we next present our improved clustering method which incorporates a seeding technique that initializes centroids probabilistically. We also incorporate a technique in our method to find the number of clusters, k, to be given as input to our clustering algorithm. The resulting algorithm called AED-based k-means++ clustering method yields a clustering that is O(logk) competitive. Our network equivalence technique is next described. Finally, the efficacy of our new equivalencing technique is demonstrated by evaluating its performance on the IEEE 300-bus system and comparing that to the performance of our AED-based method (Sharma et al. in Power network equivalents: a network science-based k-means clustering method integrated with silhouette analysis. In: Complex networks and their application VI, Lyon, France. p. 78–89, 2017) and the existing GSF-based method.

Performance technology has many analysis models and selecting which to use can be challenging. Arguably, the most prestigious and most used HPT model—a cause analysis model—is Gilbert's behavior engineering model (BEM). However, even this powerful cause analysis model has its limits; although it does examine environmental symptoms in general, it doesn't account for the organizational or environmental levels at which performance problems occur. For data on such levels the practitioner may turn to environmental analysis models such as those developed by Kaufman, Langdon, Rummler & Brache, or Rothwell. But the practitioner who uses both a cause analysis model and an environmental analysis model will be left with two sets of data that do not easily integrate into a useful guide to action. The model presented here—the synchronized analysis model (SAM)— is an effort to remedy this situation. By integrating the cause analysis model of Gilbert's BEM with levels derived from the environmental analysis models, the SAM offers the practitioner an enhanced tool for resolving performance problems.

According to the high-dimensional sparse features of the storage of the textual document, and defects existing in the clustering methods which havealready studied by now and some other problems, an effective text clustering approach (short for TGSOM-FS-FKM) based on tree-structured growingself-organizing maps (TGSOM) and Fuzzy K-Means (FKM) is proposed. It firstly makes preprocess of texts, and filter the majority of noisy words by usingunsupervised feature selection method. Then it used TGSOM to execute the first clustering to get the rough classification of texts, and to get the initial clustering number and each text's category. And then introduced LSA theory to improve the precision of clustering and reduce the dimension of feature vector. After that it used TGSOM to execute the second clustering to get the moreprecise clustering result, and used supervised feature selection method to select feature items. Finally, it used FKM to cluster the result set. In the experiment, it remained the same number of feature items.Experimental results indicate that TGSOM-FS-FKM clustering excels to other clustering method such as DSOM-FS-FCM, and the precision is better thanDSOM-FCM, DFKCN and FDMFC clustering.

Based on the analysis of the improved GSA (gap statistical algorithm) data mining technology, it is applied to the identification of bad data in the power system. Aiming at the shortcomings of this method, a new data preprocessing method and clustering method are proposed. The improved GSA method is a technique to determine the most appropriate number of clusters. The original measurement data is tested by the improved RBF neural network, and the obtained results are clustered by the improved clustering method, and then the optimal number of clusters is judged by using the elbow criterion. Compared to the original algorithm, it can determine the number of clusters faster, greatly reduce the running time, and successfully identify the bad data in the power system.

In this work we present and apply a new Hyperspectral image classification method based on Dirichlet Process Gaussian mixture models (also known as infinite Gaussian mixture models - IGMMs). Since this approach is a non-parametric Bayesian method, we circumvent the problem of model selection which is unavoidable and often difficult when employing traditional parametric Gaussian mixture models (GMM). In order to infer model parameters from observations, we adopt a Gibbs Sampling method to sample the posterior distributions of these parameters. In the preprocessing step, we use Local Fisher's Discriminant Analysis (LFDA) for dimension reduction since we expect it to preserve the multi-modal, non-Gaussian structure of the hyperspectral data. We compared our proposed IGMM based classification method to the existing state-of-the-art classification methods using popular hyperspectral imagery datasets. To simulate challenging real-world sensing environment, we mix every pixel with background and study the performance of these classification approaches. Our experiments show that the proposed LFDA-IGMM method and GMM method have almost the same performance for most cases, given “pure” data - with mixed pixels, LFDA-IGMM outperformed GMM for sub-pixel classification. Both methods outperformed the other commonly used classification approaches when the training sample size is relatively large. Also, the IGMM based method is less sensitive to pixel mixing than other methods.

The paper analyses the advantages, shortcomings and applicability of the enterprise strategic environment analysis model. In addition, it analyses the characteristics of power enterprises which are different from ordinary enterprises. Based on this, the paper proposes the appropriate models which power enterprises should be selected. The study indicates that power companies are suitable to be used the PEST model as the basic method, while using the ecosystem analysis model and the environmental uncertainty analysis model as a supplement, so as to conduct comprehensive application and judgment.

A blind event-based learning algorithm for non-intrusive load disaggregation

The paper presents the theoretical justification and software for an effective method of data identification in intelligent control systems, which is based on the use of the stages of clustering and classification of objects represented by their parametric description. The author proposed a method of data clustering with fixation of cluster boundary points based on the principle of finding their surface tension. The application of this method of cluster analysis allows to increase the efficiency of identification of objects in control systems using the SVM type classifier. The SVM (Support Vector Machine) type classifier is currently the most advanced and, due to the use of generalized decision functions, is able to operate with both linear and non-linearly distributed images of objects in the parameter space. Today, a significant number of cluster analysis methods and algorithms are known. They can be divided into classical, based on the general algorithm of Fu K.S., and the latest, based on taking into account the nature and structure of data and the purpose of their use. Examples of such methods include: k-means, density-based (DBSCAN), Mean-Shift, using Gaussian mixture models (GMM), methods of hierarchical agglomerative clustering (HAC) . Their common drawback is that they form clusters of points in the parameter space, but do not fix their boundary points. The last characteristic (the presence of marked cluster boundary points) is very useful when building classifiers that implement the task of object identification. In this paper, the author proposed and theoretically substantiated the method of clustering with the definition of surface points by analogy with the physical principle of surface tension of a liquid. The work developed an algorithm for the implementation of this method, as well as software in the Python language in the form of the ClusterBorderFinder application. The author's certificate was obtained for the developed program.

In order to have a reliable estimate of wind energy potential of a site, high frequency wind speed and direction data recorded for an extended period of time is required. Weibull distribution function is commonly used to approximate the recorded data distribution for estimation of wind energy. In the present study a comparison of Weibull function and Gaussian mixture model (GMM) as theoretical functions are used. The data set used for the study consists of hourly wind speeds and wind directions of 54 years duration recorded at Ijmuiden wind site located in north of Holland. The entire hourly data set of 54 years is reduced to 12 sets of hourly averaged data corresponding to 12 months. Authenticity of data is assessed by computing descriptive statistics on the entire data set without average and on monthly 12 data sets. Additionally, descriptive statistics show that wind speeds are positively skewed and most of the wind data points are observed to be blowing in south-west direction. Cumulative distribution and probability density function for all data sets are determined for both Weibull function and GMM. Wind power densities on monthly as well as for the entire set are determined from both models using probability density functions of Weibull function and GMM. In order to assess the goodness-of-fit of the fitted Weibull function and GMM, coefficient of determination (R2) and Kolmogorov-Smirnov (K-S) tests are also determined. Although R2 test values for Weibull function are much closer to ‘1’ compared to its values for GMM. Nevertheless, overall performance of GMM is superior to Weibull function in terms of estimated wind power densities using GMM which are in good agreement with the power densities estimated using wind data for the same duration. It is reported that wind power densities for the entire wind data set are 307 W/m2 and 403.96 W/m2 estimated using GMM and Weibull function, respectively.

We present a new framework to detect various types of variable objects within massive astronomical time-series data. Assuming that the dominant population of objects is non-variable, we find outliers from this population by using a non-parametric Bayesian clustering algorithm based on an infinite GaussianMixtureModel (GMM) and the Dirichlet Process. The algorithm extracts information from a given dataset, which is described by six variability indices. The GMM uses those variability indices to recover clusters that are described by six-dimensional multivariate Gaussian distributions, allowing our approach to consider the sampling pattern of time-series data, systematic biases, the number of data points for each light curve, and photometric quality. Using the Northern Sky Variability Survey data, we test our approach and prove that the infinite GMM is useful at detecting variable objects, while providing statistical inference estimation that suppresses false detection. The proposed approach will be effective in the exploration of future surveys such as GAIA, Pan-Starrs, and LSST, which will produce massive time-series data.

Air quality has always been a hot issue of concern to the people, the environmental protection department and the government. Among the massive air quality data, abnormal data can interfere with subsequent experiments and analysis. Therefore, it is necessary to detect abnormal data to improve the accuracy of the data. However, traditional air outlier detection methods require at least one year's data to make inferences about air quality. This paper firstly analyzes the characteristics of air quality big data, and then proposes a framework based on Bayesian non-parametric clustering, namely Dirichlet Process (DP) clustering framework, to realize the outlier detection of air quality. The framework optimizes Gaussian mixture model into infinite Gaussian mixture model according to the results of data analysis, and uses neural network to cluster the data processed by infinite Gaussian mixture model, which effectively improves the clustering accuracy and avoids the need of collecting a large number of training data.

Nonnegative Matrix Factorization (NMF) is valuable in many applications of blind source separation, signal processing and machine learning. A number of algorithms that can infer nonnegative latent factors have been developed, but most of these assume a specific noise kernel. This is insufficient to deal with complex noise in real scenarios. In this paper, we present a hierarchical Dirichlet process nonnegative matrix factorization (DPNMF) model in which the Gaussian mixture model is used to approximate the complex noise distribution. Moreover, the model is cast in the nonparametric Bayesian framework by using Dirichlet process mixture to infer the necessary number of Gaussian components. We derive a mean-field variational inference algorithm for the proposed nonparametric Bayesian model. We first test the model on synthetic data sets contaminated by Gaussian, sparse and mixed noise. We then apply it to extract muscle synergies from the electromyographic (EMG) signal and to select discriminative features for motor imagery single-trial electroencephalogram (EEG) classification. Experimental results demonstrate that DPNMF performs better in extracting the latent nonnegative factors in comparison with state-of-the-art methods.

The aim of this study was to implement a Dirichlet process mixture (DPM) model for automatic tumor edge identification on (18)F-fluorodeoxyglucose positron emission tomography ((18)F-FDG PET) images by optimizing the parameters on which the algorithm depends, to validate it experimentally, and to test its robustness. The DPM model belongs to the class of the Bayesian nonparametric models and uses the Dirichlet process prior for flexible nonparametric mixture modeling, without any preliminary choice of the number of mixture components. The DPM algorithm implemented in the statistical software package R was used in this work. The contouring accuracy was evaluated on several image data sets: on an IEC phantom (spherical inserts with diameter in the range 10-37 mm) acquired by a Philips Gemini Big Bore PET-CT scanner, using 9 different target-to-background ratios (TBRs) from 2.5 to 70; on a digital phantom simulating spherical/uniform lesions and tumors, irregular in shape and activity; and on 20 clinical cases (10 lung and 10 esophageal cancer patients). The influence of the DPM parameters on contour generation was studied in two steps. In the first one, only the IEC spheres having diameters of 22 and 37 mm and a sphere of the digital phantom (41.6 mm diameter) were studied by varying the main parameters until the diameter of the spheres was obtained within 0.2% of the true value. In the second step, the results obtained for this training set were applied to the entire data set to determine DPM based volumes of all available lesions. These volumes were compared to those obtained by applying already known algorithms (Gaussian mixture model and gradient-based) and to true values, when available. Only one parameter was found able to significantly influence segmentation accuracy (ANOVA test). This parameter was linearly connected to the uptake variance of the tested region of interest (ROI). In the first step of the study, a calibration curve was determined to automatically generate the optimal parameter from the variance of the ROI. This "calibration curve" was then applied to contour the whole data set. The accuracy (mean discrepancy between DPM model-based contours and reference contours) of volume estimation was below (1 ± 7)% on the whole data set (1 SD). The overlap between true and automatically segmented contours, measured by the Dice similarity coefficient, was 0.93 with a SD of 0.03. The proposed DPM model was able to accurately reproduce known volumes of FDG concentration, with high overlap between segmented and true volumes. For all the analyzed inserts of the IEC phantom, the algorithm proved to be robust to variations in radius and in TBR. The main advantage of this algorithm was that no setting of DPM parameters was required in advance, since the proper setting of the only parameter that could significantly influence the segmentation results was automatically related to the uptake variance of the chosen ROI. Furthermore, the algorithm did not need any preliminary choice of the optimum number of classes to describe the ROIs within PET images and no assumption about the shape of the lesion and the uptake heterogeneity of the tracer was required.