A Feature Extraction Method Based on Stacked Denoising Autoencoder for Massive High Dimensional Data

With massive high-dimensional data now commonplace in research and industry, there is a strong and growing demand for more scalable computational techniques for data analysis and knowledge discovery. Key to turning these data into knowledge is the ability to learn statistical models with high interpretability. Current methods for learning statistical models either produce models that are not interpretable or have prohibitive computational costs when applied to massive data. In this paper we address this need by presenting a scalable algorithm for partial least squares regression (PLS), which we call compression-based PLS (cPLS), to learn predictive linear models with a high interpretability from massive high-dimensional data. We propose a novel grammar-compressed representation of data matrices that supports fast row and column access while the data matrix is in a compressed form. The original data matrix is grammar-compressed and then the linear model in PLS is learned on the compressed data matrix, which results in a significant reduction in working space, greatly improving scalability. We experimentally test cPLS on its ability to learn linear models for classification, regression and feature extraction with various massive high-dimensional data, and show that cPLS performs superiorly in terms of prediction accuracy, computational efficiency, and interpretability.

With the increasing demand for location-based services, Wi-Fi-based indoor positioning technology has attracted much attention in recent years because of its ubiquitous deployment and low cost. Considering that Wi-Fi signals fluctuate greatly with time, extracting robust features of Wi-Fi signals is the key point to maintaining good positioning accuracy. To handle the dynamic fluctuation with time and sparsity of Wi-Fi signals, we propose an SDAE (Stacked Denoising Autoencoder)-based feature extraction method, which can obtain a robust and time-independent Wi-Fi fingerprint by learning the reconstruction distribution from a raw Wi-Fi signal and an artificial-noise-added Wi-Fi signal. We also leverage the strong representation ability of MLP (Multi-Layer Perceptron) to build a regression model, which maps the extracted features to the corresponding location. To fully evaluate the performance of our proposed algorithm, three datasets are applied, which represent three different scenarios, namely, spacious area with time interval, no time interval, and complex area with large time interval. The experimental results confirm the validity of our proposed SDAE-based feature extraction method, which can accurately reflect Wi-Fi signals in corresponding locations. Compared with other regression models, our proposed regression model can better map the extracted features to the target position. The average positioning error of our proposed algorithm is 4.24 m when there is a 52-day interval between training dataset and testing dataset. That confirms that the proposed algorithm outperforms other state-of-the-art positioning algorithms when there is a large time interval between training dataset and testing dataset.

Intrusion detection is one of the effective methods to prevent network environment from malicious attack. This paper proposes an intrusion detection method based on CS-SDAE to solve the problem of low detection accuracy caused by low traffic feature extraction accuracy and poor robustness in network intrusion detection. Firstly, the structure optimization algorithm of stacked denoising autoencoder (SDAE) is designed based on Cuckoo Search algorithm, and the number of hidden layers and nodes of each layer is optimized by optimizing the detection accuracy of traffic data. Then, the training data are used to train SDAE to minimize the difference between the reconstructed vector of the noisy data and the original input vector, and the characteristics with strong robustness are obtained. Finally, the extracted features are used to train Softmax to build a classifier to detect malicious attacks. Experimental results show that the proposed method can dynamically adjust the structure of SDAE according to the classification task, and the detection accuracy of SDAE is improved by 20.69% compared with traditional SDAE. It has better detection performance.

Underground metal target detection technology has been widely applied in industrial production, resource exploration, and engineering construction, etc. However, due to the influence of non-negligible noise and high dimensionality in collected data, achieving efficient and accurate underground target classification remains a grand challenge for further applications of underground metal target detection on portable devices with limited computing capability and energy supply. This study aimed to seek out robust and efficient data-based strategies to classify the underground metal targets of different shapes and materials based on electromagnetic induction detection. We investigated thirty-three classification strategies based on eleven dimensionality reduction methods, namely, the least absolute shrinkage and selection operator (LASSO), genetic algorithm-support vector machine (GA-SVM), Pearson correlation coefficient (PCC), mutual information (MI), maximal relevance minimal redundancy Pearson correlation (mRMRP), maximal relevance minimal redundancy mutual information (mRMRMI), statistical features (SF), principal component analysis (PCA), kernel principal component analysis (KPCA), locally linear embedding (LLE), and stacked denoising autoencoder (SDAE), and three machine learning models, namely, artificial neural network (ANN), linear support vector machine (L-SVM), and Gaussian Naïve Bayes (GNB). Several parameters, including classification accuracy, the number of features after dimensionality, the feature type importance, and the time consumption were considered to evaluate the data-based classification strategies. Among the classification strategies investigated and considering the above evaluation parameters, the artificial neural network (ANN) classifier assisted with the kernel principal component analysis (KPCA) feature extraction method yielded the best performance in the material-based classification (accuracy:0.99) and the shape-based classification (accuracy:0.99). The locally linear embedding (LLE) improved the robustness of machine learning classifiers and efficiency of the artificial neural network in the material-based classification (improvement of average accuracy:0.17, reduction of classification time cost:14%) and shape-based classification (improvement of average accuracy:0.16, reduction of classification time cost:22%). Our comparative investigation provides a robust and efficient data-based strategy for underground metal target classification, which is significant for applications of underground metal target detection on portable devices with limited computing capability and energy supply. The cross-combination strategy of dimensionality reduction methods and machine learning models provides a way to find the optimal machine learning model for underground target detection.

Stacked autoencoder (SAE) is hard to achieve satisfactory performance, when input data are complex and non-stationary. Besides, the identification performance of recurrent neural network (RNN) may decrease rapidly under noisy environment. In order to deal with these problems, a novel hybrid deep neural network (DNN) based on stacked denoising autoencoder (SDAE) and gated recurrent unit neural network (GRUNN) is presented. First, the structure of the presented hybrid DNN is given. The hybrid DNN contains a SDAE, a GRUNN, and a softmax classifier. Then, the training algorithm based on action discovery (AD) is proposed to train the presented hybrid DNN. The experimental studies indicate the presented hybrid DNN processes strong anti-noise ability and adaptability to time-varying signals.

With massive high-dimensional data now common-place in research and industry, there is a strong and growing demand for more scalable computational techniques for data analysis and knowledge discovery. In this paper, we review scalable algorithms for learning statistical models on high-dimensional data. Especially, we introduce two techniques of lossless and lossy compressions. The first one is a method using grammar compression. Grammar compression is a lossless compression for texts and has been successfully applied to binary data matrices for scalable learning of statistical models. The second one is a method of lossy compressions named feature maps (FMs). Recently, quite a few number of FMs for kernel approximations have been proposed and have been used in practical applications. Those methods, of which we present a brief survey in this paper, open the door for large-scale analyses of massive and high-dimensional data.

Software defect prediction technology plays an important role in ensuring software quality. The traditional software defect prediction model can only perform “shallow learning” and cannot perform deep mining of data features. Aiming at this problem, we use the stacked denoising auto-encoder (SDAE) to superimpose into deep neural network. First, the deep network model was built through the stacked layers of denoising auto-encoder (DAE), then the unsupervised method was used to train each layer in turn with noised input for more robust expression, characteristics were learnt supervised by back propagation (BP) neural network and the whole net was optimized by using error back propagation. Simulation experiments prove that the prediction accuracy of our SDAE model is significantly improved compared with the traditional SVM and KNN prediction model.

This article presents a new hyperspectral image classification method, which is capable of automatic feature learning while achieving high classification accuracy. The method contains the following two major modules: the spectral classification module and the spatial constraints module. Spectral classification module uses a deep network, called ‘Stacked Denoising Autoencoders’ (SdA), to learn feature representation of the data. Through SdA, the data are projected non-linearly from its original hyperspectral space to some higher-dimensional space, where more compact distribution is obtained. An interesting aspect of this method is that it does not need any prior feature design/extraction process guided by human. The suitable feature for the classification is learnt by the deep network itself. Superpixel is utilized to generate the spatial constraints for the refinement of the spectral classification results. By exploiting the spatial consistency of neighbourhood pixels, the accuracy of classification is further improved by a big margin. Experiments on the public data sets have revealed the superior performance of the proposed method.

To accurately predict the construction costs of foundation pit projects, a model based on the stacked denoising autoencoder (SDAE) is constructed in this work. The influencing factors of foundation pit project construction costs are identified from the four attributes of construction cost management, namely, engineering, the environment, the market, and management. Combined with Chinese national standards and the practice of foundation pit project management, a method of the quantization of the influencing factors is presented. 60 deep foundation pit projects in China are selected to obtain 13 main characteristic factors affecting these project construction cost by using the rough set. Then, considering the advantages of the SDAE in dealing with complex nonlinear problems, a prediction model of foundation pit project construction costs is created. Finally, this paper employs these 60 projects for a case analysis. The case study demonstrates that, compared with the actual construction costs, the calculation error of the proposed method is less than 3%, and the average error is only 1.54%. In addition, three error analysis tools commonly used in machine learning (the determination coefficient, root mean square error, and mean absolute error) emphasize that the calculation accuracy of the proposed method is notably higher than those of other methods (Chinese national code, the multivariate return method, the BP algorithm, the BP model optimized by the genetic algorithm, the support vector machine, and the RBF model). The relevant research results of this paper provide a useful reference for the prediction of the construction costs of foundation pit projects.

The rapid growth of healthcare data in recent years calls for more advanced and efficient analytic techniques. Artificial intelligence facilitates finding insightful patterns in massive high-dimensional data. Considering the latest movements towards using machine learning and deep learning techniques in the medical domain, in this study, we focused on the publications in which researchers employed artificial intelligence techniques for cancer diagnosis and treatment. Using dynamic topic modeling and natural language processing techniques, we analyzed the contents and trends of more than 12,000 scientific publications within the period of 2000 to 2018, extracted from two different sources, i.e., Elsevier’s Scopus and PubMed. While drawing the landscape of cancer research, our results also shed light on the evolution of artificial intelligence techniques and algorithms used for cancer diagnosis and treatment. Our findings confirm that modern computer science algorithms are being widely applied to extract patterns from large-scale medical images to cure different types of cancer with a special focus on deep learning techniques in recent years.

In the past three decades, with the fast development of China's economy, the business data of electric power industry also showed rapid growth and the data size is continual expanding. Thus, the management systems of electric enterprise groups encountered the performance bottlenecks of massive data processing. Hadoop is an open-source software framework developed for reliable, scalable, and efficient distributed computing and storage, and is widely used in massive data processing. Based on the study of the proposed massive business data processing of electric industry, this paper introduces a novel architecture, which is based on Hadoop ecosystem, for management systems of electric enterprise groups to crack the present performance bottlenecks. First, the business logic and processing produce of management system is analyzed. Then new architecture with Hadoop based distributed processing model is introduced in detailed. At last, experimental results with actual business scenarios and data prove the architecture can effectively resolve present performance bottlenecks of massive business data processing, and greatly improve the performance and efficiency of the management system of electric enterprise groups.

The gearbox is one of the key components of many large mechanical transmission devices. Due to the complex working environment, the vibration signal stability of the gear box is poor, the fault feature extraction is difficult, and the fault diagnosis accuracy makes it difficult to meet the expected requirements. To solve this problem, this paper proposes a gearbox fault diagnosis method based on an optimized stacked denoising auto encoder (SDAE) and kernel extreme learning machine (KELM). Firstly, the particle swarm optimization algorithm in adaptive weight (SAPSO) was adopted to optimize the SDAE network structure, and the number of hidden layer nodes, learning rate, noise addition ratio and iteration times were adaptively obtained to make SDAE obtain the best network structure. Then, the best SDAE network structure was used to extract the deep feature information of weak faults in the original signal. Finally, the extracted fault features are fed into KELM for fault classification. Experimental results show that the classification accuracy of the proposed method can reach 97.2% under the condition of low signal-to-noise ratio, which shows the effectiveness and robustness of the proposed method compared with other diagnostic methods.

Sensitive information such as credit card information, username, password and social security number etc, can be stolen using a fake page that imitates trusted website is called phishing. The attacker designs a similar webpage either by copying or making small manipulation to the legitimate page so that the online user cannot distinguish the legitimate and fake websites. A Deep Neural Network (DNN) was introduced to detect the phishing Uniform Resource Locator (URL). Initially, a 30-dimension feature vector was constructed based on URL-based features, Hypertext Markup Language (HTML)-based features and domain-based features. These features were processed in DNN to detect the phishing URL. However, the irrelevant, redundant and noisy features in the dataset increase the complexity of DNN classifier. So the feature selection is required for efficient phishing attack detection. But feature selection is a time-consuming process since it is an independent process. So in this paper, a feature vector is generated by DNN itself using Stacked Denoise Auto Encoder (SDAE). Moreover, the noisy data such as missing features affect the efficiency of phishing detection so the SDAE is trained to reconstruct a clean input feature vector. The initial input feature vector is corrupted by setting some feature vectors as zero. Then the corrupted feature vector is then mapped with basic auto encoder, to a hidden representation from which the input feature vector is reconstructed. The reconstructed features are given as input to DNN which selects the most relevant features and predicts the phishing URL. Hence the sparse feature representation of SDAE increases the classification accuracy of DNN. The experiments are conducted in Ham, Phishing Corpus and Phishload datasets in terms of accuracy, precision, recall and F-measure to prove the effectiveness of DNN-SDAE.

In this paper, we propose a new spectral-spatial classification strategy to enhance the classification performances obtained on hyperspectral images by integrating rotation forests and Markov random fields (MRFs). First, rotation forests are performed to obtain the class probabilities based on spectral information. Rotation forests create diverse base learners using feature extraction and subset features. The feature set is randomly divided into several disjoint subsets; then, feature extraction is performed separately on each subset, and a new set of linear extracted features is obtained. The base learner is trained with this set. An ensemble of classifiers is constructed by repeating these steps several times. The weak classifier of hyperspectral data, classification and regression tree (CART), is selected as the base classifier because it is unstable, fast, and sensitive to rotations of the axes. In this case, small changes in the training data of CART lead to a large change in the results, generating high diversity within the ensemble. Four feature extraction methods, including principal component analysis (PCA), neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA), and linearity preserving projection (LPP), are used in rotation forests. Second, spatial contextual information, which is modeled by MRF prior, is used to refine the classification results obtained from the rotation forests by solving a maximum a posteriori problem using the α-expansion graph cuts optimization method. Experimental results, conducted on three hyperspectral data with different resolutions and different contexts, reveal that rotation forest ensembles are competitive with other strong supervised classification methods, such as support vector machines. Rotation forests with local feature extraction methods, including NPE, LLTSA, and LPP, can lead to higher classification accuracies than that achieved by PCA. With the help of MRF, the proposed algorithms can improve the classification accuracies significantly, confirming the importance of spatial contextual information in hyperspectral spectral-spatial classification.

An efficient deep model for day-ahead electricity load forecasting with stacked denoising auto-encoders