A Permutation Test for Multiple Correlation Coefficient in High Dimensional Normal Data

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Because of the poor performance of the Random Forest algorithm in processing the classification of high-dimensional unbalanced data, a Hybrid Samping&Feature Selection Random Forest optimization strategy (Hybrid Samping&Feature Selection Random Forest (HF_RF) is proposed in this paper. First, from the data level, the high-dimensional unbalanced data set is preprocessed by SMOTE algorithm combined with random undersampling to achieve balanced unbalanced data. At the same time, the clustering algorithm is combined with SMOTE algorithm to improve the processing ability of the algorithm for negative samples; On the algorithm level, through the Relief F algorithm, different weight values are given to the preprocessed high-dimensional data, irrelevant and redundant features are eliminated, and high-dimensional data is reduced for dimensionality; Finally, the weighted voting principle is used to further elevate the predictive performance of HF_RF. The experimental results show that compared with the traditional algorithm, the proposed algorithm has higher indicators when dealing with high-dimensional unbalanced data, which proves that the HF_RF proposed in this paper is The correctness of the algorithm and its effectiveness in improving the classification performance of high-dimensional unbalanced data.

High-dimensional data sets are often available in genome-enabled predictions. Such data sets include nonlinear relationships with complex dependence structures. For such situations, vine copula-based (quantile) regression is an important tool. However, the current vine copula-based regression approaches do not scale up to high and ultra-high dimensions. To perform high-dimensional sparse vine copula-based regression, we propose 2 methods. First, we show their superiority regarding computational complexity over the existing methods. Second, we define relevant, irrelevant, and redundant explanatory variables for quantile regression. Then, we show our method's power in selecting relevant variables and prediction accuracy in high-dimensional sparse data sets via simulation studies. Next, we apply the proposed methods to the high-dimensional real data, aiming at the genomic prediction of maize traits. Some data processing and feature extraction steps for the real data are further discussed. Finally, we show the advantage of our methods over linear models and quantile regression forests in simulation studies and real data applications.

DBFS: An effective Density Based Feature Selection scheme for small sample size and high dimensional imbalanced data sets

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

BackgroundIdentifying variants associated with complex human traits in high-dimensional data is a central goal of genome-wide association studies. However, complicated etiologies such as gene-gene interactions are ignored by the univariate analysis usually applied in these studies. Random Forests (RF) are a popular data-mining technique that can accommodate a large number of predictor variables and allow for complex models with interactions. RF analysis produces measures of variable importance that can be used to rank the predictor variables. Thus, single nucleotide polymorphism (SNP) analysis using RFs is gaining popularity as a potential filter approach that considers interactions in high-dimensional data. However, the impact of data dimensionality on the power of RF to identify interactions has not been thoroughly explored. We investigate the ability of rankings from variable importance measures to detect gene-gene interaction effects and their potential effectiveness as filters compared to p-values from univariate logistic regression, particularly as the data becomes increasingly high-dimensional.ResultsRF effectively identifies interactions in low dimensional data. As the total number of predictor variables increases, probability of detection declines more rapidly for interacting SNPs than for non-interacting SNPs, indicating that in high-dimensional data the RF variable importance measures are capturing marginal effects rather than capturing the effects of interactions.ConclusionsWhile RF remains a promising data-mining technique that extends univariate methods to condition on multiple variables simultaneously, RF variable importance measures fail to detect interaction effects in high-dimensional data in the absence of a strong marginal component, and therefore may not be useful as a filter technique that allows for interaction effects in genome-wide data.

With the rapid development of hardware and software technology, modern industry has produced a large amount of high-dimensional unlabeled data, such as pictures and videos. As clusters of these data sets may exist in some subspaces, traditional algorithms are no longer applicable. The related algorithms based on sparse subspace can find clusters in the subspace, which solves the problem of high data dimension. However, their clustering process based on only one single feature, which results in their performance being particularly sensitive to this single view. Affected by the integrated algorithm, a large number of multi-view methods began to emerge. These methods improve the clustering performance by integrating the subspace expressions of multiple views, but the problem is that the complementary information of multiple views cannot be fully considered. In addition, the problem of non-uniform distribution in clusters also exists in high-dimensional data sets. In this paper, based on the multi-view subspace clustering method, a clustering algorithm based on tensor low rank expression is proposed to solve the clustering problem of high-dimensional datasets. On the one hand, this paper solves the problem of noise and data corruption by combining the method of $\ell _{2,1} $ norms, which transform the optimization problem of solving multi-view subspace expression into a low-rank expression problem of tensor to fully consider the complementarities between views. On the other hand, the proposed scheme solves the problem of non- uniform distribution in clusters in high-dimensional data by combining with the density peak algorithm based on hierarchical strategy. Experiment results show the effectiveness of the algorithm.

Multiple imputation (MI) is a well-established method for dealing with missing data. MI is computationally intensive when imputing missing covariates with high-dimensional outcome data (e.g., DNA methylation data in epigenome-wide association studies (EWAS)), because every outcome variable must be included in the imputation model to avoid biasing associations towards the null. Instead, EWAS analyses are reduced to only complete cases, limiting statistical power and potentially causing bias. We used simulations to compare 5 MI methods for high-dimensional data under 2 missingness mechanisms. All imputation methods had increased power over complete-case (C-C) analyses. Imputing missing values separately for each variable was computationally inefficient, but dividing sites at random into evenly sized bins improved efficiency and gave low bias. Methods imputing solely using subsets of sites identified by the C-C analysis suffered from bias towards the null. However, if these subsets were added into random bins of sites, this bias was reduced. The optimal methods were applied to an EWAS with missingness in covariates. All methods identified additional sites over the C-C analysis, and many of these sites had been replicated in other studies. These methods are also applicable to other high-dimensional data sets, including the rapidly expanding area of “-omics” studies.

Metric trees are designed for improving efficiency of similarity search in high dimensional data. Searching a metric tree always terminates at leaf nodes, restricting the hierarchical search to a certain leaf level, and the linear search in leafs to a particular leaf size. This may result in performing additional unneeded distance comparisons, which increases the search time as the number of dimensions increases. It is proposed that the search adapts itself to the query in question, so as to avoid unneeded distance comparisons. Thus, the hierarchical search should terminate when no more search- space pruning can be useful, and search then continues linearly through the un-pruned data space. Hierarchical search can thus stop any where above or below the original leaf level of a metric tree. Search rules are proposed to adapt the search to the query parameters. A modification to the metric tree is also suggested to adopt the proposed rules. High dimensional gene expression data sets are used to evaluate the new algorithm, showing speed ups of 55% compared to traditional search.

本文针对高维生物数据特征提出了一种融合T-Rank和Softmax的特征提取算法。该方法比传统特征提取方法在处理高维生物数据更加有效，不仅提取的特征个数较少，而且计算速度快。利用算法本文对高维银屑病基因表达谱数据进行了研究，得到了分类准确率较高的疾病诊断模型。 The paper proposed a new feature extraction algorithm by integrating T-rank and Softmax for the high dimensional biological data sets, which is more effective than traditional method when dealing with high dimensional data. It can not only extract a very few number of features, but also have fast computing speed. By using of this new algorithm, the paper obtains a high accuracy diagnosis model for psoriasis.

Pavement distress images are typical high dimensional nonlinear data. Manifold learning algorithms can find the intrinsic characteristic hidden in the distress images, which helps to better recognize them. Unlike most of manifold learning algorithms, multilayer autoencoders have solved the data reconstructed problem through building a bi-directional mapping between the high dimensional data and the low dimensional data. An automatic pavement distress image recognition method based on multilayer autoencoders was proposed, which combined the image processing method and multilayer autoencoders. In the method, the distress images were firstly processed with the image processing method. Then the images were reduced dimensions and reconstructed with multilayer autoencoders. Lastly, the distress type was recognized through the network. Experiments showed that the recognition accuracy with the proposed method was great higher than that with the BP neural network.

Network intrusion detection system are significant features that contribute to the enterprises and organization network success. In the past decade, Intrusion Detection system (IDS) using several methods have been proposed and implemented to safeguard the networks within an organization to be reliable, secure, and readily available. Although, some of these methods are built utilizing machine learning concepts. IDS utilizing machine learning techniques are accurate, effective and efficient in spotting attacks. However, the machine learning model performance reduces with high dimensional intrusion dataset. Therefore, it is sacrosanct to implement an efficient feature selection technique that can generate a great impact on the classification stage. Also, the feature selection phase is the most serious phase in IDS based on machine learning. This phase is expensive both in time and efforts. Furthermore, most of the proposed machine learning intrusion detection system suffered from low detection accuracy and high false positive rate when the models are experimented on high dimensional dataset. The aim of this paper is to propose a binary firefly algorithm (BFFA) based feature selection for IDS. We first performed normalization in the first stage of the model. Subsequently, the BFFA algorithm was used for feature selection stage. We adopted random forest algorithm for the classification phase. The experiment was performed on high dimensional University of New South Wales-NB 2015 (UNSW-NB15) dataset with seventy five percent of the data used for training the model and twenty percent for testing. The findings showed an accuracy of 99.72%, detection rate of 99.84%, precision of 99.27%, recall of 99.84% and F-score of 99.56%. The results were gauge with the state-of-the-art results and our results were found outstanding.

This paper presents an evolutionary approach for mining diverse and more informative subgroups focused on high dimensional data sets. Subgroup Discovery (SD) is an important tool for knowledge discovery that aims to identify sets of features that distinguish a target group from the others (e.g. successful from unsuccessful treatments). At the same time, to extract information from high dimensional data sets becomes more natural. One of the first and most efficient SD heuristics focused on high dimensional data is the SSDP. However, this model deals superficially with diverse/redundancy in top-k subgroups, which can result in poor information for users. This work presents SSDP+, an extension of the SSDP model to provide diversity in a way that explore the relation between subgroups order to 2enerate a more informative set of patterns.

Radial Visualization technique is a non linear dimensionality reduction method. Radial Visualization projects multivariate data in the 2-dimensional visual space inside the unit circle. Radial Visualization supports display both the samples and the attributes that provides useful information of data structures. In this article, we introduced a new variant of Radial Visualization for visualizing high dimensional data set that named Arc Radial Visualization. The new proposal that modified Radial Visualization supported more space to display high dimensional datasets. Our method provides an improvement in visualizing cluster structures of high dimensional data sets on the Radial Visualization. We present our proposal method with two quality measurements and proved the effectiveness of our approach for several real datasets.

Correlation tests are very important tools for the pathway analysis or graphical modeling of high-dimensional data. In this study, we consider a correlation test under the strongly spiked eigenvalue (SSE) model in high-dimension, low-sample-size scenarios, where the sample size is much smaller than the dimension. High-dimensional data often fit the SSE model. Previously, a high-dimensional test for a correlation matrix under the non-SSE model was constructed using the extended cross-data-matrix (ECDM) methodology. Here, we show that an asymptotic distribution of the test statistic using ECDM methodology under the SSE model can be written using the distribution of the sum of some weighted chi-squared variables when both dimension and sample size reach infinity. We propose a new test procedure using the asymptotic distribution. We also show that the proposed test procedure has preferable properties for power and size. We discuss the performance of the test procedure through simulations and present a demonstration with actual data analyses using a microarray data set.