Context-aware restricted Boltzmann machine meets collaborative filtering

Restricted Boltzmann Machine (RBM) is a generative, stochastic neural network with two separate layers of hidden and visible units. Training data samples in RBM are usually corrupted by noise. RBM is not robust enough to handle such noises, which leads to uncertainty. In the literature, Fuzzy RBM (FRBM) has already been proposed for enhancing deep learning. In FRBM, the parameters of RBM are modelled as type-1 fuzzy numbers. However, there can be multiple sources of uncertainties such as noises in the data measurements, variations in the environment where they are deployed, etc. Such uncertainties cannot be modelled by type-1 fuzzy sets (T1 FS), since their membership values are themselves crisp in nature. On the other side, IT2 FS can model higher order uncertainties with their fuzzy membership grades. So, we propose to use interval type-2 fuzzy sets (IT2 FS) to model uncertain parameters of RBMs in the learning stage. Since deep neural network (DNN) is pre-trained using stacked RBMs, modeling noises using IT2 FS would demonstrate high performance and low root mean square error (RMSE) while learning. Thus, we propose a new algorithm viz. Interval type-2 fuzzy set based approach for enhanced deep learning (IT2 FS-EDL) in which RBM parameters are modeled as type-2 fuzzy sets in the learning process. Numerical examples and experimentations have been demonstrated to present the suitability of our proposed approach.

The collaborative filtering (CF) technique has been widely used recently in recommendation systems. It needs historical data to give predictions. However, the data sparsity problem still exists. We propose a new item-based restricted Boltzmann machine (RBM) approach for CF and use the deep multilayer RBM network structure, which alleviates the data sparsity problem and has excellent ability to extract features. Each item is treated as a single RBM, and different items share the same weights and biases. The parameters are learned layer by layer in the deep network. The batch gradient descent algorithm with minibatch is used to increase the convergence speed. The new feature vector discovered by the multilayer RBM network structure is very effective in predicting a rating and achieves a better result. Experimental results on the data set of MovieLens show that the item-based multilayer RBM approach achieves the best performance, with a mean absolute error of 0.6424 and a root-mean-square error of 0.7843.

Supply chain flexibility is the processing ability of the enterprize to deal with the uncertain environment of supply and demand. In this paper, we consider the supply side (node interrupt) and demand side (demand fluctuations) under uncertain environment. By using the deep belief network (DBN), which is composed of multilayer Restricted Boltzmann Machine (RBM), it establishes the supply chain flexibility network with optimization of the transfer node and flow. The deep belief network is trained by the data of large manufacturing enterprize, compared with the traditional neural network (MLR, BP and GA). The results show that the deep belief network model overcomes the shortcomings of the traditional neural networks, such as easy to fall into local optimum, long training time and low function fitting degree, and it has higher prediction accuracy. This network model based on the deep belief network can promote the supply chain flexibility more, when supply and demand fluctuations occur.

Accurate and real-time traffic flow prediction nowadays shows more and more dependence on big transportation data. Deep learning, a powerful method for feature learning, has turned out to be an effective tool to cope with these explosive data. Recently, deep models, especially unsupervised models like deep belief networks (DBN) and stacked autoencoder (SAE), are being employed into the field of traffic research and have shown great prospect. However, there is still a vacancy in the exploration on comparing the performances of different kinds of deep architectures to find an optimal solution. In this paper, we set up two deep-learning-based traffic flow prediction models for feature extraction and performances comparison: One is a deep belief networks (DBN) based on restricted Boltzmann machines (RBMs) that have Gaussian visible units and binary hidden units, and the other is a DBN based on RBMs with all units being binary. A conclusion is drawn where the former one performs better in traffic flow prediction after a series of experiments.

There is a current trend towards wearable electrocardiogram (ECG) measurement systems, which enables measurement while the subject performs their normal activities of daily living (e.g., walking, driving, eating). This type of measurement is susceptible to higher levels of contaminants, compared to bedside measurements, due to subject movement and a measurement environment that is not well-controlled. Contaminants in the measured signal (e.g., motion artifact, power line interference) can cause incorrect interpretations, including misdiagnoses. Therefore, prior to ECG interpretations, it is important to have an algorithm capable of automatically classifying the measured ECG based on their signal quality. ECG of low signal quality can undergo additional pre-processing to mitigate the contaminants or the signal can be discarded. This can reduce misdiagnoses, including false-alarms which is a top medical technology hazard. In this paper, we propose an algorithm based on Deep Belief Networks (DBN) which can differentiate between noisy and clean signal measurements. Our algorithm is designed based on a three layer Restricted Boltzmann Machine (RBM) in which the first two RBMs are trained to extract the features and apply them to the third layer of RBM to classify the data. Results, using the MIT-BIH Arrhythmia database, demonstrate that our algorithm can successfully recognize a noisy ECG signal from a clean signal, with a classification accuracy between 75% and 99.5% depending on the level of contaminants. Our algorithm also correctly identifies clean arrhythmic signals and does not misidentify them as noisy. The proposed algorithm is applicable to any ECG measurement systems including wearable and bedside.

Expected energy-based restricted Boltzmann machine for classification

In the past twenty years, progress in intrusion detection has been steady but slow. The biggest challenge is to detect new attacks in real time. In this work, a deep learning approach for anomaly detection using a Restricted Boltzmann Machine (RBM) and a deep belief network are implemented. Our method uses a one-hidden layer RBM to perform unsupervised feature reduction. The resultant weights from this RBM are passed to another RBM producing a deep belief network. The pre-trained weights are passed into a fine tuning layer consisting of a Logistic Regression (LR) classifier with multi-class soft-max. We have implemented the deep learning architecture in C++ in Microsoft Visual Studio 2013 and we use the DARPA KDDCUP'99 dataset to evaluate its performance. Our architecture outperforms previous deep learning methods implemented by Li and Salama in both detection speed and accuracy. We achieve a detection rate of 97.9% on the total 10% KDDCUP'99 test dataset. By improving the training process of the simulation, we are also able to produce a low false negative rate of 2.47%. Although the deficiencies in the KDDCUP'99 dataset are well understood, it still presents machine learning approaches for predicting attacks with a reasonable challenge. Our future work will include applying our machine learning strategy to larger and more challenging datasets, which include larger classes of attacks.

Infant cry recognition is a challenging task as it is hard to determine the speech features that can allow researchers to clearly separate between different types of cries. However, baby cry is treated as a different way of communication of speech. The types of baby cry can be differentiated using Mel-Frequency Cepstral Coefficient (MFCC) with appropriate artificial intelligence model. Stacked restricted Boltzmann machine (RBN) is popular in providing few layers of neural networks to convert the high dimensional data to lower dimensional data to fine tune the input data to a better initialized weight for the neural networks. Usually RBN is used with another deep neural network to form the deep belief networks (DBN), and the studies in this direction is heading towards the convolutional-RBN variant. The study on RBN to pre-train Convolutional neural networks (CNN) without convolution function in the RBN meanwhile is scarce due to the Back propagation and principal component analysis can be applied directly to the CNN. In this paper, we describe the hybrid system between RBN and CNN for learning class specific features for baby cry recognition using the feature of Mel-Frequency Cepstral Coefficient. We archived an 78.6% of accuracy on 5 types of baby cries by validating the proposed model on baby cry recognition.

Deep Belief Networks (DBNs) have emerged as powerful tools for feature learning, representation, and generative modeling. This paper presents a comprehensive exploration of the various training algorithms employed in the training of DBNs. DBNs, composed of multiple layers of stochastic hidden units, have found applications in diverse domains such as computer vision, natural language processing, and bioinformatics. The paper begins by delving into the pre-training phase, where Restricted Boltzmann Machines (RBMs) play a central role. We review the Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) algorithms, shedding light on their strengths and weaknesses in initializing deep networks. Emphasis is placed on their applicability to different data types and scales. Moving to the fine-tuning stage, the paper explores the use of backpropagation with gradient descent, discussing modern optimization techniques, including stochastic gradient descent and adaptive learning rate methods. We also examine regularization techniques like dropout and weight decay to address overfitting concerns. Furthermore, we discuss architectural variants of DBNs, such as Convolutional Deep Belief Networks (CDBNs) for image data and Recurrent DBNs for sequential data. We highlight the adaptation of DBNs for specific tasks, including classification, regression, clustering, and generative modeling.

Aiming at raw vibration signal of rolling bearing with long time series, a fault diagnosis model based on multimodal data fusion and deep belief network is proposed in this paper. First, multimodal data composed of artificial features and model features can be obtained by time-frequency domain analysis and unsupervised learning based on restricted Boltzmann machine (RBM). Second, canonical correlation analysis method is used to extract the typical feature pairs from the multimodal data to realize the feature-level multimodal data fusion. Third, deep belief network is applied to extract deep feature mapping between typical feature pairs and fault types. After greedy layer-wise pre-training and fine-tuning, it is available to achieve the trained model for fault diagnosis of rolling bearing. Typical rolling bearing datasets are used to testify the effectiveness of the proposed method. It is verified that the robustness and accuracy of the proposed method are superior to common methods.

As network technology evolves, cyberattacks are not only increasing in frequency but also becoming more sophisticated. To proactively detect and prevent these cyberattacks, researchers are developing intrusion detection systems (IDSs) leveraging machine learning and deep learning techniques. However, a significant challenge with these advanced models is the increased training time as model complexity grows, and the symmetry between performance and training time must be taken into account. To address this issue, this study proposes a fast-persistent-contrastive-divergence-based deep belief network (FPCD-DBN) that offers both high accuracy and rapid training times. This model combines the efficiency of contrastive divergence with the powerful feature extraction capabilities of deep belief networks. While traditional deep belief networks use a contrastive divergence (CD) algorithm, the FPCD algorithm improves the performance of the model by passing the results of each detection layer to the next layer. In addition, the mix of parameter updates using fast weights and continuous chains makes the model fast and accurate. The performance of the proposed FPCD-DBN model was evaluated on several benchmark datasets, including NSL-KDD, UNSW-NB15, and CIC-IDS-2017. As a result, the proposed method proved to be a viable solution as the model performed well with an accuracy of 89.4% and an F1 score of 89.7%. By achieving superior performance across multiple datasets, the approach shows great potential for enhancing network security and providing a robust defense against evolving cyber threats.

Accurate modulation identification of the received signals is undoubtedly a central component in multiple-input multiple-output (MIMO) communication systems, facilitating the demodulation task. This study presents a flexible and semi-supervised deep learning-driven strategy for automatic modulation identification. To this end, the multiclass classification problem is treated as multiple binary discrimination problems to address modulation identification challenges. Here, we merge the features extraction ability of the Generative Adversarial Network (GAN) model and the semi-supervised anomaly detection scheme, the one-class Support Vector Machine (1SVM). Essentially, a single GAN-based 1SVM detector is trained using training data of each class, with the samples of that class as inlier and all other samples as anomalies (i.e., one-vs.-rest). The 1SVM is trained using the features learned by the GAN model. A dataset consisting of three digital modulations (i.e., BFSK, CPFSK, and PAM4) and three analog modulations (i.e., AM-DSB, AM-SSB, and WB-FM), widely used in wireless communications systems, is employed to demonstrate the performance of the considered deep learning-based methods. Compared to Restricted Boltzmann Machine (RBM) and Deep Belief Network (DBN)-based 1SVM, the conventional GAN, DBN, and RBM with softmax layer as discriminator layer, the proposed GAN-based 1SVM detector offers superior discrimination performance of modulation types by achieving an averaged accuracy of 0.951 and F1-Score of 0.954. Results also showed that the GAN-1SVM detector dominates the state-of-the-art modulation classification techniques.

During exploring the emergence of language, we found that the brain can extract some common features from the same thing in different representations by pattern recognition and association. Consequently, the brain would establish a connection for identical concept from multi-signals. An associative generated model primarily based on Restricted Boltzmann Machine (RBM) and Deep Belief Network (DBN) is set up for multi-signals to simulate the brain's ability. The first step is to use the DBNs for extracting features from multiple input signal sources. The second is using top-level RBM to achieve the goal of associating and generating mutually by fusing each feature. Finally, we verify the feasibility of the model through the realization of generating Arabic digital pictures and Chinese characters digital images reciprocally.

The accuracy of structural damage identification is affected by the uncertainties in the vibration measurements and the finite element modeling. This paper proposes a novel approach based on sparse deep belief network (DBN) for structural damage identification with uncertain and limited data. Vibration characteristics, that is, natural frequencies and mode shapes, are extracted as the input to the network, while the output are the damage locations and severities of the structure. DBN is chosen to train the generated data sets and identify structural damages. Restricted Boltzmann Machines (RBMs) are used as building blocks to composite a DBN. To further enhance the capacity of the RBMs, an arctan-based sparse constraint is utilized to enable the hidden units to become sparse. This is achieved by adding an arctan norm constraint on the whole of the hidden units' activation probabilities. Numerical and experimental studies are conducted to verify the accuracy and performance of the proposed method. Undetermined damage identification is conducted, in which the quantity of input modal data is less than that of the system parameters to be identified. The identification results show that the proposed sparse DBN based on arctan can identify the damage effectively, and its accuracy is better than those obtained by other methods, even when the modeling uncertainty and the measurement noise exist and only limited data is available.

Deep Learning has the hierarchical network architecture to represent the\ncomplicated features of input patterns. Such architecture is well known to\nrepresent higher learning capability compared with some conventional models if\nthe best set of parameters in the optimal network structure is found. We have\nbeen developing the adaptive learning method that can discover the optimal\nnetwork structure in Deep Belief Network (DBN). The learning method can\nconstruct the network structure with the optimal number of hidden neurons in\neach Restricted Boltzmann Machine and with the optimal number of layers in the\nDBN during learning phase. The network structure of the learning method can be\nself-organized according to given input patterns of big data set. In this\npaper, we embed the adaptive learning method into the recurrent temporal RBM\nand the self-generated layer into DBN. In order to verify the effectiveness of\nour proposed method, the experimental results are higher classification\ncapability than the conventional methods in this paper.\n