Pavement Distress Image Recognition Based on Multilayer Autoencoders

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Fuzzy partition based soft subspace clustering and its applications in high dimensional data

BackgroundIdentifying variants associated with complex human traits in high-dimensional data is a central goal of genome-wide association studies. However, complicated etiologies such as gene-gene interactions are ignored by the univariate analysis usually applied in these studies. Random Forests (RF) are a popular data-mining technique that can accommodate a large number of predictor variables and allow for complex models with interactions. RF analysis produces measures of variable importance that can be used to rank the predictor variables. Thus, single nucleotide polymorphism (SNP) analysis using RFs is gaining popularity as a potential filter approach that considers interactions in high-dimensional data. However, the impact of data dimensionality on the power of RF to identify interactions has not been thoroughly explored. We investigate the ability of rankings from variable importance measures to detect gene-gene interaction effects and their potential effectiveness as filters compared to p-values from univariate logistic regression, particularly as the data becomes increasingly high-dimensional.ResultsRF effectively identifies interactions in low dimensional data. As the total number of predictor variables increases, probability of detection declines more rapidly for interacting SNPs than for non-interacting SNPs, indicating that in high-dimensional data the RF variable importance measures are capturing marginal effects rather than capturing the effects of interactions.ConclusionsWhile RF remains a promising data-mining technique that extends univariate methods to condition on multiple variables simultaneously, RF variable importance measures fail to detect interaction effects in high-dimensional data in the absence of a strong marginal component, and therefore may not be useful as a filter technique that allows for interaction effects in genome-wide data.

Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and have been shown to be able to detect anomalies in many practical problems. However, in high dimensional data, conventional outlier detection methods often behave unexpectedly due to a phenomenon called the curse of dimensionality. In this paper, we compare and analyze outlier detection performance in various experimental settings, focusing on text data with dimensions typically in the tens of thousands. Experimental setups were simulated to compare the performance of outlier detection methods in unsupervised versus semi-supervised mode and uni-modal versus multi-modal data distributions. The performance of outlier detection methods based on dimension reduction is compared, and a discussion on using k-NN distance in high dimensional data is also provided. Analysis through experimental comparison in various environments can provide insights into the application of outlier detection methods in high dimensional data.

High dimension and complexity of network high-dimensional data lead to poor feature selection effect network high-dimensional data. To effectively solve this problem, feature selection algorithms for high-dimensional network data based on supervised discriminant projection (SDP) have been designed. The sparse representation problem of high-dimensional network data is transformed into an Lp norm optimization problem, and the sparse subspace clustering method is used to cluster high-dimensional network data. Dimensionless processing is carried out for the clustering processing results. Based on the linear projection matrix and the best transformation matrix, the dimensionless processing results are reduced by combining the SDP. The sparse constraint method is used to achieve feature selection of high-dimensional data in the network, and the relevant feature selection results are obtained. The experimental findings demonstrate that the suggested algorithm can effectively cluster seven different types of data and converges when the number of iterations approaches 24. The F1 value, recall, and precision are all kept at high levels. High-dimensional network data feature selection accuracy on average is 96.9%, and feature selection time on average is 65.1 milliseconds. The selection effect for network high-dimensional data features is good.

Outlier detection is an important data mining task for consistency checks, fraud detection, etc. Binary decision making on whether or not an object is an outlier is not appropriate in many applications and moreover hard to parametrize. Thus, recently, methods for outlier ranking have been proposed. Determining the degree of deviation, they do not require setting a decision boundary between outliers and the remaining data. High dimensional and heterogeneous (continuous and categorical attributes) data, however, pose a problem for most outlier ranking algorithms. In this work, we propose our OutRank approach for ranking outliers in heterogeneous high dimensional data. We introduce a consistent model for different attribute types. Our novel scoring functions transform the analyzed structure of the data to a meaningful ranking. Promising results in preliminary experiments show the potential for successful outlier ranking in high dimensional data.

In order to extract sensitive features of face images from high dimensional image data and facilitate the recognition speed, this paper has proposed a novel method based on the manifold learning and genetic-chaos algorithm optimized kernel extreme learning machine (KELM) for the application of face recognition. The locally linear embedding (LLE) algorithm has been employed to extract distinct features by projecting the original high dimensionality of the face image into a low dimensionality space. Then the KELM is introduced to provide quick and accurate pattern recognition on the extracted features. The only parameter need to be determined in KELM is the neuron number of hidden layer. Literature review indicates that very limited work has addressed the optimization of this parameter. Hence, the genetic-chaos algorithm was used for the first time to optimize the KELM parameter in this paper. A robust KELM structure may be attained after the genetic-chaos optimization. In order to evaluate and verify the proposed method, experiment tests have been carried out using standard face expressions. The experimental analysis results indicate that the performance of the proposed LLE- genetic-chaos-KELM method outperforms its rivals in terms of both recognition accuracy and training speed. Index Terms—face recognition, feature extraction, GA Chaos, KELM.

When we manipulate high dimensional data with Elman neural network, many characteristic variables provide enough information, but too many network inputs go against designing of the hidden-layer of the network and take up plenty of storage space as well as computing time, and in the process interfere the convergence of the training network, even influence the the accuracy of recognition finally. PCA,which is short for principal component analysis is a multivariate analysis method which transforms many characteristic variables to few synthetic variables. Not only can it eliminate relationship among characteristic variables but new variables can also hold most information of the original ones as well. In this paper we make full use of the advantages of PCA and the properties of Elman neural network structures to establish PCA-Elman based on PCA. The new algorithm reduces dimensions of the high dimensional data by PCA, and carry on network training and simulation with low dimensional data that we get, which obviously simplifies the network structure, and in the process, improves the training speed and generalization capacity of the Elman neural network. We prove the effectiveness of the new algorithm by case analysis. The algorithm improves the efficiency in network problems solving and is worth further generalizing.

In recent year, stock investor in Indonesia increased rapidly, so it is required to do analysis about the stock that helps the investor in their investment plan. Clustering is beneficial to select the appropriate stock for investors. Unfortunately, stock prices keep varying from time to time. Consequently, it is not an easy work to select the stock for investment. In addition, stock price time series data are high dimensional data that influenced by many factors. In this study, high dimensional data are obtained by the time frame of each factor. Therefore, it is important to use a suitable technique to cluster high dimensional data. This paper presents High Dimensional Data Clustering (HDDC), a model-based clustering based on Gaussian Mixture Model, using the Expectation-Maximization (EM) algorithm. HDDC via EM algorithm gives a more robust result, and it possible to make an additional assumption. Moreover, this paper combines a high-dimensional clustering technique HDDC via EM algorithm and the most popular feature extraction technique Principal Component Analysis (PCA). This paper comparing methods of clustering technique HDDC and the combination between HDDC and PCA to know the most effective method which gives better result in clustering high-dimensional time series data. The 155 data features are reduced to 7 principal components using PCA analysis. Despite PCA has increased the time efficiency of building the model, clustering technique HDDC via EM algorithm enables to handle the high-dimensional data better than the combination with PCA.

Various machine learning, cloud computing and big data models are most utilized control parameters in software applications. Therefore these systems are most demand at current trending decades. But these models are needs very low data access time, speed for process. Day to day life data storage servers and devices are costly and hardware complex, so dimensionality is increases with rapid manner. Any type of optimization techniques access takes more time consumption for high dimensional data. So, more applications related problems are occurs only at high dimensional data space does not at low space dimensional data. In this research proposes a dimensional reduction technique with Logistic regression (LR) model. This LR model is most helpful for dimensional reduction and clustering problems. LRML method has reduced the dimensional data size and achieved thee efficiency by 95.3% and ratio of reduction by 35.76%.

Along with the rapid advancement of computer science and data engineering, the complexity and dimension of the data considered are bursting. With the increase of the data dimension, the performance of high dimensional index structure drops rapidly. And these high-dimensional data usually contains many redundant, its essence dimension often is much smaller than the original data, so the high-dimensional data processing question boils down to the related dimension reduction method to cut some less relevant data and reduce its dimension. This paper integrates the ameliorated supportive vector machine to propose the novel high-dimensional data dimension reduction algorithm. Our algorithm optimises the traditional methodologies from the three layers: 1) data cleaning and pre-processing steps are added to enhance the feasibility of the captured data; 2) SVM is modified to hold the feature of higher robustness; 3) application scenarios are built to test the performance. The experimental result proves that the proposed method outperforms compared with other state-of-theart algorithms theoretically and numerically.

Recently, many data mining techniques were revisited to cope with the new big data challenges. Nearly all of these algorithms considered the efficiency of the mining algorithm to survive the increasing size of the data. However, as the dimensionality of the data increases, not only the efficiency but also the effectiveness of traditional mining algorithms is compromised. For instance, clusters hidden in some subspaces are hard to be detected using traditional clustering algorithms, as the dimensionality of the data increases. In this paper, we consider both the huge size, and the high dimensionality of big data by providing a novel solution that presents a three-phase model for subspace stream clustering algorithms. Our novel model, overcomes the huge size of the big data in its first phase, by continuously applying a streaming concept over the huge data objects, and summarizing them into micro-clusters. Then, after each certain batch of data, or after upon a user request, the second phase is applied over the data summarized in micro-clusters, to reconstruct the current distribution of the data out of the current summaries. In the third phase, a subspace clustering algorithm is applied to overcome the high dimensionality of the data, and to find hidden clusters within some subspace. An extensive evaluation study over different scenarios that follow our model over a big data set is performed.KeywordsSubspace stream clusteringStreaming big dataReal-time streaming analysis of big data

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

Display and visualization of high dimensional data are typically performed with a well-chosen linear projection of the data or by displaying many linear projections to form an animation. This study presents an evolutionary algorithm for producing nonlinear projections of high dimensional data with cues, in the drawing of the projection, as to the types of distortions introduced. Such projections can provide drawings closer to the true high dimensional distances of the displayed data than any single linear drawing. This permits a researcher to view a good analog to a scatter plot for high dimensional data. The system is demonstrated on a synthetic four dimensional fitness landscape and on distance data derived from RNA folds. Because fitness landscapes often have more dimensions than can be easily visualized it is difficult to gain an intuitive understanding of a fitness landscape. The nonlinear projection algorithm is applied to an abstraction of the fitness landscape called a fitness web. Fitness webs can be used to display the relative quality of optima, the frequency with which they were found by different evolutionary runs, or other factors of interest. In addition to displaying the relative position of optima in a fitness landscape, a graph of the fitness function along the edges a fitness web displays important slices of the fitness landscape. Called fitness morphs these plots can provide intuition about the fitness landscapes as well as direction for subsequent evolutionary searches. The second demonstration of the nonlinear projection algorithm is to data generated from an ad hoc metric on RNA folds. The algorithm yields drawings that permit a researcher to correctly distinguish two different types of folds for iron response elements.