Deep Restricted Kernel Machines Using Conjugate Feature Duality.

Support Vector Machines and Kernel methods have become very popular as methods for learning from examples. In this paper, in order to solve the non-linear and the time of recognition problem, recent advanced techniques are presented. For example, Kernel Principal Components and Proximal Support Vector Machines, Kernel Principal Components Analysis can describe multiple correlations between pixels in the image recognition, and can handle the high order statistics of the original data, at the same time the nonlinear features of the image can be preferably extracted. Proximal Support Vector Machines is an improvement for these characters of Support Vector Machines, combines the proximal algorithm with Support Vector Machines, has a better ability of nonlinear mapping and a stronger generalization capability, and the pace of Proximal Support Vector Machines is very enhanced. In this paper, the computer simulation was progressed based on ORL face database, the effectiveness of the Proximal Support Vector Machines algorithm and Kernel Principal Components Analysis algorithm were shown by experimental results.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

This paper presents a novel algorithm - robust kernel principal component analysis (robust KPCA), on the basis of the research of kernel principal component analysis (KPCA) and robust principal component analysis (RPCA). First, this algorithm sets the radius of the images of the training samples in the feature space using kernel tricks, then determines whether the samples are outliers or not, and finally analyzes the training samples which have eliminated the outliers using KPCA algorithm. The improved KPCA algorithm not only retains the non-linearity property of KPCA algorithm but also gets better robustness. Because the effects of outliers are eliminated, robust KPCA algorithm gets higher accuracy than KPCA algorithm for data analysis. The simulation experiments show that the robust KPCA algorithm developed is better than the KPCA algorithm.

There is a trend in recent machine learning community to construct a nonlinear version of a linear algorithm using the 'kernel method', e.g. Support Vector Machines (SVMs), kernel principal component analysis, kernel fisher discriminant analysis and the recent kernel clustering algorithms. In unsupervised clustering algorithms using kernel method, typically, a nonlinear mapping is used first to map the data into a potentially much higher feature space, where clustering is then performed. A drawback of these kernel clustering algorithms is thatthe clustering prototypes lie in high dimensional feature space and hence lack clear and intuitive descriptions unless using additional projection approximation from the feature to the data space as done in the existing literatures. In this paper, a novel clustering algorithm using the 'kernel method' based on the classical fuzzy clustering algorithm (FCM) is proposed and called as kernel fuzzy c-means algorithm (KFCM). KFCM adopts a new kernel-induced metric in the data space to replace the original Euclidean norm metric in FCM and the clustered prototypes still lie in the data space so that the clustering results can be reformulated and interpreted in the original space. Our analysis shows that KFCM is robust to noise and outliers and also tolerates unequal sized clusters. And finally this property is utilized to cluster incomplete data. Experiments on two artificial and one real datasets show that KFCM hasbetter clustering performance and more robust than several modifications of FCM for incomplete data clustering.

With the rapid development of artificial intelligence technology of deep learning method, it has been applied to many fields, especially to life science. In this paper, a novel approach for the task of voiceprint recognition was proposed. The combination of deep belief network (DBN) and support vector machine (SVM) was used to identify the voiceprint of 10 different individuals. Based on a 24-dimension Mel Frequency Cepstrum Coefficient (MFCC), the authors extracted 256-dimension deep voiceprint features via DBN model that is developed by stacking three layers of Restricted Boltzmann Machine (RBM), and conducted voiceprint recognition by approach of SVM. According to the recognition results, the new approach can significantly improve both of accuracy and efficiency when it was compared with traditional voiceprint features and recognition models. The low dimensional features of voiceprint was extracted into higher dimensions by DBN model, while SVM can avoid the elevation of computation complexity caused by the increases of feature dimension. The combined strengths have been fully expressed by the experimental results.

In kernel methods such as kernel principal component analysis (PCA) and support vector machines, the so called kernel trick is used to avoid direct calculations in a high (virtually infinite) dimensional kernel space. In this brief, based on the fact that the effective dimensionality of a kernel space is less than the number of training samples, we propose an alternative to the kernel trick that explicitly maps the input data into a reduced dimensional kernel space. This is easily obtained by the eigenvalue decomposition of the kernel matrix. The proposed method is named as the nonlinear projection trick in contrast to the kernel trick. With this technique, the applicability of the kernel methods is widened to arbitrary algorithms that do not use the dot product. The equivalence between the kernel trick and the nonlinear projection trick is shown for several conventional kernel methods. In addition, we extend PCA-L1, which uses L1-norm instead of L2-norm (or dot product), into a kernel version and show the effectiveness of the proposed approach.

Cyber security in smart cities: A review of deep learning-based applications and case studies

Expected energy-based restricted Boltzmann machine for classification

Accurate and real-time traffic flow prediction nowadays shows more and more dependence on big transportation data. Deep learning, a powerful method for feature learning, has turned out to be an effective tool to cope with these explosive data. Recently, deep models, especially unsupervised models like deep belief networks (DBN) and stacked autoencoder (SAE), are being employed into the field of traffic research and have shown great prospect. However, there is still a vacancy in the exploration on comparing the performances of different kinds of deep architectures to find an optimal solution. In this paper, we set up two deep-learning-based traffic flow prediction models for feature extraction and performances comparison: One is a deep belief networks (DBN) based on restricted Boltzmann machines (RBMs) that have Gaussian visible units and binary hidden units, and the other is a DBN based on RBMs with all units being binary. A conclusion is drawn where the former one performs better in traffic flow prediction after a series of experiments.

Principal component analysis (PCA) has been proven to be an efficient method in dimensionality reduction, feature extraction and pattern recognition. Kernel principal component analysis (KPCA) can be considered as a natural nonlinear generalization of PCA, which performs linear PCA in a high dimensional space implicitly by using kernel trick. However, both conventional PCA and KPCA suffer from the deficiency of being sensitive to outliers. Existing robust KPCA has to eigen-decompose the gram matrix directly in each step and is much more computationally infeasible due to the large size of the matrix when the number of training samples is large. By extending existing robust PCA algorithm using kernel methods, we present a novel robust adaptive algorithm for calculating the kernel principal components. The proposed method not only preserves the characteristic of capturing underlying nonlinear structure of KPCA but also is robust against outliers by restraining the effect of outlying samples. Compared with existing robust KPCA methods, our method is performed without having to store the kernel matrix, which can reduce significantly the storage burden. In addition, our method shows the potential of expansibility to the incremental learning version. Experimental results on synthetic data indicate that our improved algorithm is effective and promising.

In the past twenty years, progress in intrusion detection has been steady but slow. The biggest challenge is to detect new attacks in real time. In this work, a deep learning approach for anomaly detection using a Restricted Boltzmann Machine (RBM) and a deep belief network are implemented. Our method uses a one-hidden layer RBM to perform unsupervised feature reduction. The resultant weights from this RBM are passed to another RBM producing a deep belief network. The pre-trained weights are passed into a fine tuning layer consisting of a Logistic Regression (LR) classifier with multi-class soft-max. We have implemented the deep learning architecture in C++ in Microsoft Visual Studio 2013 and we use the DARPA KDDCUP'99 dataset to evaluate its performance. Our architecture outperforms previous deep learning methods implemented by Li and Salama in both detection speed and accuracy. We achieve a detection rate of 97.9% on the total 10% KDDCUP'99 test dataset. By improving the training process of the simulation, we are also able to produce a low false negative rate of 2.47%. Although the deficiencies in the KDDCUP'99 dataset are well understood, it still presents machine learning approaches for predicting attacks with a reasonable challenge. Our future work will include applying our machine learning strategy to larger and more challenging datasets, which include larger classes of attacks.

Restricted Boltzmann Machines (RBMs) have been developed for a lot of applications in the past few years, and many of its variants have also appeared. In this paper, RBM model and its learning algorithm with contrastive divergence algorithm will be introduced firstly. Then three important variants of RBM are presented in details, which are sparse RBM, discriminative RBM, and the Deep Boltzmann Machines (DBM). All the variants including original RBM are tested on MNIST handwriting digit dataset for classification task. Our empirical results demonstrate the advantage of RBM models and show that compared with other variants, the DBM is the best one in terms of the classification accuracy.

To overcome the difficulty of automating and intelligently classifying the ground features in remote-sensing hyperspectral images, machine learning methods are gradually introduced into the process of remote-sensing imaging. First, the PaviaU, Botswana, and Cuprite hyperspectral datasets are selected as research subjects in this study, and the objective is to process remote-sensing hyperspectral images via machine learning to realize the automatic and intelligent classification of features. Then, the basic principles of the support vector machine (SVM) and extreme learning machine (ELM) classification algorithms are introduced, and they are applied to the datasets. Next, by adjusting the parameter estimates using a restricted Boltzmann machine (RBM), a new terrain classification model of hyperspectral images that is based on a deep belief network (DBN) is constructed. Next, the SVM, ELM, and DBN classification algorithms for hyperspectral image terrain classification are analysed and compared in terms of accuracy and consistency. The results demonstrate that the average detection accuracies of ELM on the three datasets are 89.54%, 96.14%, and 96.28%, and the Kappa coefficient values are 0.832, 0.963, and 0.924; the average detection accuracies of SVM are 88.90%, 92.11%, and 91.68%, and the Kappa coefficient values are 0.768, 0.913, and 0.944; the average detection accuracies of the DBN classification model are 92.36%, 97.31%, and 98.84%, and the Kappa coefficient values are 0.883, 0.944, and 0.972. The results also demonstrate that the classification accuracy of the DBN algorithm exceeds those of the previous two methods because it fully utilizes the spatial and spectral information of hyperspectral remote-sensing images. In summary, the DBN algorithm that is proposed in this study has high application value in object classification for remote-sensing hyperspectral images.

To study the characteristics and performance of the deep learning in intelligent intrusion detection, two hybrid algorithms, which combine restricted Boltzmann machine (RBM) with support vector machine (SVM) and deep belief network (DBN) respectively, are used to analyze the accuracy, false positive rate, false negative rate and testing time with the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition (KDDCup99). Compared with each other and traditional hybrid intrusion detection algorithm, DBN performs better than the other both in the accuracy and speed, which is attributed to the unsupervised learning of RBM networks and the combination of the neural networks at the bottom.

Deep learning has been successfully applied into pattern recognition due to its deep architecture and effective unsupervised learning, and deep belief network (DBN) is a popular model based on deep learning technique. In this paper, a DBN identification model based on partial least square regression (PLSR), named PLSR-DBN, is proposed for nonlinear system identification. In order to improve the identification accuracy, PLSR is introduced into the supervised fine-tuning of DBN to elimate the overfitting and local minimum resulted from gradients-based learning, and contrastive divergence (CD) algorithm is used in unsupervised pre-training. Finally, the proposed PLSR-DBN is tested on a benchmark nonlinear system. The experiment results show that the proposed PLSR-DBN has a better performance on nonlinear system identification than other similar methods.