A Novel Density-based Technique for Outlier Detection of High Dimensional Data Utilizing Full Feature Space

With the rapid development of the Internet, privacy protection in big data has become an important research hotspot in the field of information security. Among them, differential privacy protection is the most widely used big data privacy protection technology today. However, the existing differential privacy protection technology cannot fully and effectively deal with the problem of high-dimensional privacy big data. Although the private data publishing method based on a Bayesian network can effectively deal with the conversion problem of high-dimensional data sets to low-dimensional data sets, this method also has some defects and deficiencies. Therefore, this paper proposes a large data source differential privacy protection algorithm (NIE-PrivBayes algorithm) based on normalized information entropy Bayesian network and proposes three improvements to the original PrivBayes algorithm. First, the original data set is protected by differential privacy on the client side. Secondly, the Bayesian network is constructed by the joint probability distribution obtained by the EM algorithm. Finally, the existing Bayesian network construction algorithm is optimized in the aspects of normalized information entropy. Based on significantly reducing the computational overhead, the effectiveness and availability of publishing high-dimensional data sets are realized under the premise of differential privacy protection.

Outlier detection is a key technique in data ming and machine learning fields. The deviating characters of outliers make huge detrimental effects on the learning tasks. A lot of algorithms are therefore proposed to handle outliers from different perspectives, such as distance, density, angle and so on. Among these approaches, the density-based methods achieve better performance, but also suffer from huge time complexity. Recently, in order to accelerate the speed and improve the performance, the subsampling ensemble method attracts much attention, which has a reasonable theoretical interpretation and high performance. However, existing work only gives the partial picture of outlier detection via row-sampling, the effective portfolio of bi-sampling is still void. In light of this, we propose the general outlier detection framework via bi-sampling, Bi-Sampling Outlier Detection (BSOD) and provide the effective portfolios of the row and column-sampling ratios in a theoretical way. In addition, the benefits of BSOD are fully illustrated in terms of ensemble diversity and divide-and-conquer. Further we employ LOF within BSOD as BI-LOF to conduct extensive experiments. In general, on 30 synthetic and 17 real-world data sets we thoroughly explore the characteristics of BI-LOF with different numbers of instances, features, nearest neighbors, validate the theoretical analysis of BSOD condition on synthetic data sets, and show obvious advantages over other state-of-the-art algorithms in terms of low and high dimensional real-world data sets. And finally we use BI-LOF to conduct image outlier detection and show high quality and stableness of BI-LOF.

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

This paper presents an evolutionary approach for mining diverse and more informative subgroups focused on high dimensional data sets. Subgroup Discovery (SD) is an important tool for knowledge discovery that aims to identify sets of features that distinguish a target group from the others (e.g. successful from unsuccessful treatments). At the same time, to extract information from high dimensional data sets becomes more natural. One of the first and most efficient SD heuristics focused on high dimensional data is the SSDP. However, this model deals superficially with diverse/redundancy in top-k subgroups, which can result in poor information for users. This work presents SSDP+, an extension of the SSDP model to provide diversity in a way that explore the relation between subgroups order to 2enerate a more informative set of patterns.

Outlier detection is an important data mining task for consistency checks, fraud detection, etc. Binary decision making on whether or not an object is an outlier is not appropriate in many applications and moreover hard to parametrize. Thus, recently, methods for outlier ranking have been proposed. Determining the degree of deviation, they do not require setting a decision boundary between outliers and the remaining data. High dimensional and heterogeneous (continuous and categorical attributes) data, however, pose a problem for most outlier ranking algorithms. In this work, we propose our OutRank approach for ranking outliers in heterogeneous high dimensional data. We introduce a consistent model for different attribute types. Our novel scoring functions transform the analyzed structure of the data to a meaningful ranking. Promising results in preliminary experiments show the potential for successful outlier ranking in high dimensional data.

Defining a boundary between inliers and outliers is a major challenge in unsupervised outlier detection. In the absence of labeled data, the true outliers set cannot be evaluated. This lays the burden on both the choice of an efficient outlier detection criterion, and parameter selection. While numerous unsupervised outlier detection criteria, with different parameters, have been proposed, an unsupervised evaluation of outliers is still missing. This work introduces a theoretical basis, and proposes a validity index, to evaluate the quality of outliers. This is not a trivial problem when nothing is known about the structure and density of the data. The proposed index considers the outlierness quality, the deviation between characteristics of outliers and inliers, and the data distortion. Low and high dimensional data sets are used to evaluate the proposed index.

High dimension and complexity of network high-dimensional data lead to poor feature selection effect network high-dimensional data. To effectively solve this problem, feature selection algorithms for high-dimensional network data based on supervised discriminant projection (SDP) have been designed. The sparse representation problem of high-dimensional network data is transformed into an Lp norm optimization problem, and the sparse subspace clustering method is used to cluster high-dimensional network data. Dimensionless processing is carried out for the clustering processing results. Based on the linear projection matrix and the best transformation matrix, the dimensionless processing results are reduced by combining the SDP. The sparse constraint method is used to achieve feature selection of high-dimensional data in the network, and the relevant feature selection results are obtained. The experimental findings demonstrate that the suggested algorithm can effectively cluster seven different types of data and converges when the number of iterations approaches 24. The F1 value, recall, and precision are all kept at high levels. High-dimensional network data feature selection accuracy on average is 96.9%, and feature selection time on average is 65.1 milliseconds. The selection effect for network high-dimensional data features is good.

In recent year, stock investor in Indonesia increased rapidly, so it is required to do analysis about the stock that helps the investor in their investment plan. Clustering is beneficial to select the appropriate stock for investors. Unfortunately, stock prices keep varying from time to time. Consequently, it is not an easy work to select the stock for investment. In addition, stock price time series data are high dimensional data that influenced by many factors. In this study, high dimensional data are obtained by the time frame of each factor. Therefore, it is important to use a suitable technique to cluster high dimensional data. This paper presents High Dimensional Data Clustering (HDDC), a model-based clustering based on Gaussian Mixture Model, using the Expectation-Maximization (EM) algorithm. HDDC via EM algorithm gives a more robust result, and it possible to make an additional assumption. Moreover, this paper combines a high-dimensional clustering technique HDDC via EM algorithm and the most popular feature extraction technique Principal Component Analysis (PCA). This paper comparing methods of clustering technique HDDC and the combination between HDDC and PCA to know the most effective method which gives better result in clustering high-dimensional time series data. The 155 data features are reduced to 7 principal components using PCA analysis. Despite PCA has increased the time efficiency of building the model, clustering technique HDDC via EM algorithm enables to handle the high-dimensional data better than the combination with PCA.

DBFS: An effective Density Based Feature Selection scheme for small sample size and high dimensional imbalanced data sets

High-dimensional data sets are often available in genome-enabled predictions. Such data sets include nonlinear relationships with complex dependence structures. For such situations, vine copula-based (quantile) regression is an important tool. However, the current vine copula-based regression approaches do not scale up to high and ultra-high dimensions. To perform high-dimensional sparse vine copula-based regression, we propose 2 methods. First, we show their superiority regarding computational complexity over the existing methods. Second, we define relevant, irrelevant, and redundant explanatory variables for quantile regression. Then, we show our method's power in selecting relevant variables and prediction accuracy in high-dimensional sparse data sets via simulation studies. Next, we apply the proposed methods to the high-dimensional real data, aiming at the genomic prediction of maize traits. Some data processing and feature extraction steps for the real data are further discussed. Finally, we show the advantage of our methods over linear models and quantile regression forests in simulation studies and real data applications.

Anomaly detection techniques are widely used in a variety of applications, e.g., computer networks, security systems, etc. This paper describes and analyzes an approach to anomaly detection using proximity graphs and the PageRank algorithm. We run a variant of the PageRank algorithm on top of a proximity graph comprised of data points as vertices, which produces a score quantifying the extent to which each data point is anomalous. Previous work in this direction requires first forming a density estimate using the training data, e.g., using kernel methods, and this step is very computationally intensive for high-dimensional data sets. Under mild assumptions and appropriately chosen parameters, we show that PageRank produces point-wise consistent probability density estimates for the data points in an asymptotic sense, and with much less computational effort. As a result, big improvements in terms of running time are witnessed while maintaining similar detection performance. Experiments with synthetic and real-world data sets illustrate that the proposed approach is computationally tractable and scales well to large high-dimensional data sets.

IntroutionThe population multiple correlation coefficient (PMCC) measures the correlation between a given one-dimensional random variable X and multidimensional random vector X .This measure is the maximum correlation between X and any linear combination of components X .Testing the null hypothesis of zero PMCC is the most favourable approach to investigate the existence or non-existence of PMCC between X and X .The classical procedures for testing this hypothesis in high-dimensional data settings are invalid since the sample covariance matrix inverse or sample precision matrix is undefined.To cope with this problem, a simple test is constructed for testing zero PMCC in high-dimensional normal data.A small simulation study was carried out to evaluate the performance of the proposed test in both high-dimensional and low-dimensional normal data sets.Finally, the proposed test is applied to mice tumour volumes data. ateria an ethosThe proposed test includes the following two steps: In the first step, using a plug-in estimator of the sample precision matrix, a simple test statistic, called plug-in test statistic, was derived which uses the EQUALs method (Wang and Jiang, ) to estimate the precision matrix.In the second step, a permutation test is constructed using the proposed plug-in statistic to test the null hypothesis of zero PMCC.Theoretical investigations demonstrate that the proposed testing procedure is a level test. . . . . .

Regarding the magnificent ability of the fuzzy logic in explanation and analyzing the real phenomena, using fuzzy data is growing rapidly. But the tools for doing data mining in a fuzzy data set; especially in high dimension, are not sufficient. Considering the remarkable ability of the LoOP (Local Outlier Probabilities) method in outlier detecting in high dimensional crisp data set, like supplying for each data object, determining outlier probability as scar which is easily explainable competitor on various synthetic and real world data sets, the aim of this article is to present an extended version of this method for using in high dimensional fuzzy database. In order to provide a completely fuzzy strategy, first we propose a new approach to assigning distance between fuzzy high dimension data based on Hausdorff and Vertex distance. After proving that this distance measuring is a metric, since LoOP is a distance based method, we redefined the LoOP algorithm step by step based on the new meter. Then we allocate an outlier score in the range of [0,1] to each data based on LoOP idea, to identify the outliers. The presented numerical tests show the efficiency of the new method, for random cases of knowing and unknowing the outlier and a real case. The results shows also the stability and robustness of the presented method more that its accuracy.

Background/Objectives: This paper aims to develop a novel intelligent recommender system suitable for high dimensional data where multiple factor variables influence on multiple response variables. Methods/Statistical Analysis: This paper suggests that the structure of the cause-and-effect relations among the variables can be represented in a simple form called structured association network model (SANM). Based on the SANM and conventional data mining techniques such as association rule mining and naive Bayesian classifier, the proposed recommender system computes three novel recommendation scores for each response variables, and the variables with high scores can be selected for recommendation. Findings: For illustration, the proposed recommender system has been applied to a mass health examination result data set. Owing to its simple structure, a SANM for a given data set can be easily obtained by simply identifying the factor and the response variables, and the experiment results revealed that the proposed system can identify the recommendable items for the individuals more effectively than the traditional classification techniques such as naive Bayesian classifier. Consequently, we can conclude that the proposed recommender system can deal with high dimensional transaction data set in more effective manner than the traditional approaches where the underlying semantic relationships among the variables are not considered. Applications/Improvements: The proposed recommender system is useful for evaluation of the potential risks of specific diseases. Moreover, the recommendation scores can also be used as a tool for feature construction.

High-dimensional data arise naturally in many domains, and have regularly presented a great challenge for traditional data mining techniques, both in terms of effectiveness and efficiency. Clustering becomes difficult due to the increasing sparsity of such data, as well as the increasing difficulty in distinguishing distances between data points. In this paper, we take a novel perspective on the problem of clustering high-dimensional data. Instead of attempting to avoid the curse of dimensionality by observing a lower dimensional feature subspace, we embrace dimensionality by taking advantage of inherently high-dimensional phenomena. More specifically, we show that hubness, i.e., the tendency of high-dimensional data to contain points (hubs) that frequently occur in k-nearest-neighbor lists of other points, can be successfully exploited in clustering. We validate our hypothesis by demonstrating that hubness is a good measure of point centrality within a high-dimensional data cluster, and by proposing several hubness-based clustering algorithms, showing that major hubs can be used effectively as cluster prototypes or as guides during the search for centroid-based cluster configurations. Experimental results demonstrate good performance of our algorithms in multiple settings, particularly in the presence of large quantities of noise. The proposed methods are tailored mostly for detecting approximately hyperspherical clusters and need to be extended to properly handle clusters of arbitrary shapes.