BIGDATA AND MACHINE LEARNING MODELS FOR DIMENTIONALITY REDUCTION PLATFORM

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

Pavement distress images are typical high dimensional nonlinear data. Manifold learning algorithms can find the intrinsic characteristic hidden in the distress images, which helps to better recognize them. Unlike most of manifold learning algorithms, multilayer autoencoders have solved the data reconstructed problem through building a bi-directional mapping between the high dimensional data and the low dimensional data. An automatic pavement distress image recognition method based on multilayer autoencoders was proposed, which combined the image processing method and multilayer autoencoders. In the method, the distress images were firstly processed with the image processing method. Then the images were reduced dimensions and reconstructed with multilayer autoencoders. Lastly, the distress type was recognized through the network. Experiments showed that the recognition accuracy with the proposed method was great higher than that with the BP neural network.

Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and have been shown to be able to detect anomalies in many practical problems. However, in high dimensional data, conventional outlier detection methods often behave unexpectedly due to a phenomenon called the curse of dimensionality. In this paper, we compare and analyze outlier detection performance in various experimental settings, focusing on text data with dimensions typically in the tens of thousands. Experimental setups were simulated to compare the performance of outlier detection methods in unsupervised versus semi-supervised mode and uni-modal versus multi-modal data distributions. The performance of outlier detection methods based on dimension reduction is compared, and a discussion on using k-NN distance in high dimensional data is also provided. Analysis through experimental comparison in various environments can provide insights into the application of outlier detection methods in high dimensional data.

In order to solve the problem that traditional grid-based clustering techniques lack of the capability of dealing with data of high dimensionality, we propose an intersecting grid partition method and a density estimation method. The partition method can greatly reduce the number of grid cells generated in high dimensional data space and make the neighbor-searching easily. On basis of the two methods, we propose grid-based clustering algorithm (GCOD), which merges two intersecting grids according to density estimation. The algorithm requires only one parameter and the time complexity is linear to the size of the input data set or data dimension. The experimental results show that GCOD can discover arbitrary shapes of clusters and scale well.

The sparsity and the problem of the curse of dimensionality of high-dimensional data, which make the most of the traditional clustering algorithm, lose action in high-dimensional space. Therefore, clustering of data in high-dimensional space is becoming the hot research areas. By utilizing the subtractive clustering as initialized method, and combine with the revised clustering validation indices, this paper offers a subspace clustering algorithm for automatically determining the optimal number of clusters on high dimensional data. The experiment results show that the proposed clustering algorithm can get better cluster validation performance than that of conventional indices.

In modern database applications the similarity or dissimilarity of complex objects is examined by performing distance-based queries (DBQs)on data of high dimensionality. The R-tree and its variations are commonly cited multidimensional access methods that can be used for answering such queries. Although, the related algorithms work well for low-dimensional data spaces, their performance degrades as the number of dimensions increases (dimensionality curse). In order to obtain accept- able response time in high-dimensional data spaces, algorithms that ob- tain approximate solutions can be used. Three approximation techniques (α-allowance, N-consider and M-consider) and the respective recursive branch-and-bound algorithms for DBQs are presented and studied in this paper. We investigate the performance of these algorithms for the most representative DBQs (the K-nearest neighbors query and the K-closest pairs query) in high-dimensional data spaces, where the point data sets are indexed by tree-like structures belonging to the R-tree family:R*- trees and X-trees. The searching strategy is tuned according to several parameters, in order to examine the trade-off between cost (I/O activ- ity and response time) and accuracy of the result. The outcome of the experimental evaluation is the derivation of the outperforming DBQ ap- proximate algorithm for large high-dimensional point data sets.

Along with the rapid advancement of computer science and data engineering, the complexity and dimension of the data considered are bursting. With the increase of the data dimension, the performance of high dimensional index structure drops rapidly. And these high-dimensional data usually contains many redundant, its essence dimension often is much smaller than the original data, so the high-dimensional data processing question boils down to the related dimension reduction method to cut some less relevant data and reduce its dimension. This paper integrates the ameliorated supportive vector machine to propose the novel high-dimensional data dimension reduction algorithm. Our algorithm optimises the traditional methodologies from the three layers: 1) data cleaning and pre-processing steps are added to enhance the feasibility of the captured data; 2) SVM is modified to hold the feature of higher robustness; 3) application scenarios are built to test the performance. The experimental result proves that the proposed method outperforms compared with other state-of-theart algorithms theoretically and numerically.

Due to the intricate relationship between different dimensions of high-dimensional data, subspace analysis is often conducted to decompose dimensions and give prominence to certain subsets of dimensions, i.e. subspaces. Exploring and comparing subspaces are important to reveal the underlying features of subspaces, as well as to portray the characteristics of individual dimensions. To date, most of the existing high-dimensional data exploration and analysis approaches rely on dimensionality reduction algorithms (e.g. principal component analysis and multi-dimensional scaling) to project high-dimensional data, or their subspaces, to two-dimensional space and employ scatterplots for visualization. However, the dimensionality reduction algorithms are sometimes difficult to fine-tune and scatterplots are not effective for comparative visualization, making subspace comparison hard to perform. In this article, we aggregate high-dimensional data or their subspaces by computing pair-wise distances between all data items and showing the distances with matrix visualizations to present the original high-dimensional data or subspaces. Our approach enables effective visual comparisons among subspaces, which allows users to further investigate the characteristics of individual dimensions by studying their behaviors in similar subspaces. Through subspace comparisons, we identify dominant, similar, and conforming dimensions in different subspace contexts of synthetic and real-world high-dimensional data sets. Additionally, we present a prototype that integrates parallel coordinates plot and matrix visualization for high-dimensional data exploration and incremental dimensionality analysis, which also allows users to further validate the dimension characterization results derived from the subspace comparisons.

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Digital medical images can be viewed as digital representations of real physical tissue properties that are convenient for handling, storing, transmitting, retrieving, and analyzing image information. In order to perform feature extraction/identification/classification from high-dimensional medical data, we need dimensionality reduction (DR), which can be achieved by identifying the statistical patterns in medical image data with highlighting their similarities and dissimilarities. Here, the dimension of medical images is the total number of pixels in the image. In DR, we try to find an useful representation of reduced dimensionality in high-dimensional data, which minimizes information loss by maximizing local data variance. Given medical image data, the key challenging issue is how can we efficiently extract a low-dimensional latent structure? Various dimension reduction techniques have been developed to process the high-dimensional data where the intrinsic dimensions are assumed to be much lower. In the very ideal case, data can be regressed linearly and DR can be performed by principal component analysis. This chapter explains the theories, principles, and practices of DR techniques.

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

In order to improve the ability of data retrieval in cloud computing environment, a reduced dimension classification algorithm of complex attribute big data in cloud computing based on deep neural network learning is proposed. The complex attribute big data under cloud computing is constructed by low dimensional feature set, and the complex attribute big data under cloud computing is analyzed by linear programming and fitting using grid clustering method. Big data samples of all complex attributes are sampled and trained to extract the associated features of big data, which is a complex attribute under cloud computing. The feature extraction results of complex attribute big data under cloud computing are inputted into the deep neural network learner for data classification, and the complex attribute big data dimensionality reduction classification under cloud computing is realized by combining big data fusion clustering method. The simulation results show that the accuracy of big data dimension reduction classification for complex attributes in cloud computing is high and the error rate is small.

Dimensionality reduction is essential in multidimensional data mining since the dimensionality of real time data could easily reach higher dimensions. Most recent efforts on dimensionality reduction, however, are not adequate to multidimensional data due to lack of scalability. In this paper, we improve the performance of the fractional cuckoo search algorithm by utilizing the FCM function as objective function and we utilize the FCM operator which helps to improve the fitness value of worse solution. The fractional cuckoo search (FCS) algorithm is modified to reduce the dimension and select the best dimension. Once the high dimensional data is reduced in to low dimensional data, then the data is supplied to the clustering algorithm to make the partition easily. Finally, the experimentation is made with synthetic and real datasets and has been we have proved that efficiency of the FCM-FCS algorithm is 1.21% better than FPSO algorithm on iris dataset. The proposed algorithm is 0.66% better than FPSO 4.91% better than FCS on wine dataset and 3.4% better than FPSO 0.1% better than FCS on synthetic dataset.

Massive amount of Earth science data open an unprecedented opportunity to discover potentially valuable information. Earth science data are complex, nonlinear, high-dimensional data, and the sparsity of data in high-dimensional space poses major challenge in clustering of the data. Shared nearest neighbor clustering (SNN) algorithm is one of the well-known and efficient methods to handle high-dimensional spatiotemporal data. The SNN clustering method does not cluster all the data forming rigid boundary selection. This paper reports fuzzy shared nearest neighbor (FSNN) algorithm which is an enhancement of the SNN clustering method that has the capability of handling the data lying in the boundary regions by means of a fuzzy concept. The clusters obtained can be characterized by the cluster centroid, which summarizes the behavior of the ocean points in the cluster. The statistical measure is used to find the significant relation between the cluster centroids and the existing climate indices. In this study, correlation measure is used to find the significant pattern, such as teleconnection or dipole. The experimentation is performed on Indian continent latitude range $$7.5^{\circ }{-}37.5^{\circ }\hbox {N}$$ and longitude range $$67.5^{\circ }{-}97.5^{\circ }\hbox {E}$$ . Extensive experiments are carried out to compare the proposed approach with existing clustering methods such as K-means, fuzzy C-means and SNN. The proposed method, FSNN algorithm, not only handles the data lying in the overlapping region, but it also finds more compact and well-separated clusters. FSNN shows better results in terms of finding a significant correlation between cluster centroids and existing climate indices and validated by ground truth dataset.

Cloud computing and big data: Technologies and applications