Image Data Denoising Using the Unsupervised Learning Model of RBM

Restricted Boltzmann Machines (RBMs) have been developed for a lot of applications in the past few years, and many of its variants have also appeared. In this paper, RBM model and its learning algorithm with contrastive divergence algorithm will be introduced firstly. Then three important variants of RBM are presented in details, which are sparse RBM, discriminative RBM, and the Deep Boltzmann Machines (DBM). All the variants including original RBM are tested on MNIST handwriting digit dataset for classification task. Our empirical results demonstrate the advantage of RBM models and show that compared with other variants, the DBM is the best one in terms of the classification accuracy.

Restricted Boltzmann Machine (RBM) is a two layer undirected graph model that capable to represent complex distributions. Recent research has shown RBM-based approach has comparable performance with or even better performance than previous models on many collaborative filtering (CF) tasks. However, the intractable inference makes the training of RBM sophisticated, which prevents it from practical application. We present a novel feedforward neural framework for collaborative filtering called NACF, which is extended from the Neural Autoregressive Distribution Estimator (NADE). Because of the autoregressive feed-forward architecture, NACF can be trained with efficient stochastic gradient descent, instead of slow and hard-to-tune truncated Gibbs sampling for RBM. By introducing linear visible units and dual reversed ordering, NACF show faster convergence and better results than Probabilistic Matrix Factorization (PMF) and corresponding RBM models on MovieLens dataset. Besides, by combining NACF results, the rating prediction of efficientsignificantly improved, showing NACF is an effective and efficient model for collaborative filtering.

Infant cry recognition is a challenging task as it is hard to determine the speech features that can allow researchers to clearly separate between different types of cries. However, baby cry is treated as a different way of communication of speech. The types of baby cry can be differentiated using Mel-Frequency Cepstral Coefficient (MFCC) with appropriate artificial intelligence model. Stacked restricted Boltzmann machine (RBN) is popular in providing few layers of neural networks to convert the high dimensional data to lower dimensional data to fine tune the input data to a better initialized weight for the neural networks. Usually RBN is used with another deep neural network to form the deep belief networks (DBN), and the studies in this direction is heading towards the convolutional-RBN variant. The study on RBN to pre-train Convolutional neural networks (CNN) without convolution function in the RBN meanwhile is scarce due to the Back propagation and principal component analysis can be applied directly to the CNN. In this paper, we describe the hybrid system between RBN and CNN for learning class specific features for baby cry recognition using the feature of Mel-Frequency Cepstral Coefficient. We archived an 78.6% of accuracy on 5 types of baby cries by validating the proposed model on baby cry recognition.

PurposeThe purpose of this paper is to propose an approach to incorporate contextual information into collaborative filtering (CF) based on the restricted Boltzmann machine (RBM) and deep belief networks (DBNs). Traditionally, neither the RBM nor its derivative model has been applied to modeling contextual information. In this work, the authors analyze the RBM and explore how to utilize a user’s occupation information to enhance recommendation accuracy.Design/methodology/approachThe proposed approach is based on the RBM. The authors employ user occupation information as a context to design a context-aware RBM and stack the context-aware RBM to construct DBNs for recommendations.FindingsThe experiments on the MovieLens data sets show that the user occupation-aware RBM outperforms other CF models, and combinations of different context-aware models by mutual information can obtain better accuracy. Moreover, the context-aware DBNs model is superior to baseline methods, indicating that deep networks have more qualifications for extracting preference features.Originality/valueTo improve recommendation accuracy through modeling contextual information, the authors propose context-aware CF approaches based on the RBM. Additionally, the authors attempt to introduce hybrid weights based on information entropy to combine context-aware models. Furthermore, the authors stack the RBM to construct a context-aware multilayer network model. The results of the experiments not only convey that the context-aware RBM has potential in terms of contextual information but also demonstrate that the combination method, the hybrid recommendation and the multilayer neural network extension have significant benefits for the recommendation quality.

Cyber security in smart cities: A review of deep learning-based applications and case studies

Sign language is a language that uses a combination of hand gestures and lip movements for people with hearing impairment to communicate. In Indonesia there are two sign language systems used, Sign System for Indonesian Language known as SIBI (Sistem Isyarat Bahasa Indonesia) recognizes as the official sign language system by the Indonesian Government. This research is focused on the generation process of skeleton sequence; in which represent a SIBI hand gesture excluding the finger joints. The hand skeleton that will be generated from the generation process is limited to root-word gestures only. Some researchers were using a Restricted Boltzmann Machine model and its variant known as Deep Belief Networks (DBN) to solve the sequence modelling problems. One of DBN variants is Sigmoid Belief Network (SBN). An SBN is a Bayesian network that models a binary visible vector. Deep Temporal Sigmoid Belief Network (DTSBN) is a sequence of SBNs (with deep architecture) arranged in such way that at any given time step has a fully generative process capability, where data are readily generated from the model using ancestral sampling. Since, DTSBN performance is quite novel for this particular case, we decided to implement the DTSBN model using the SIBI dataset from the previous research to construct generated hand-skeleton gestures which represent SIBI's root-word gestures. Based on the success of the experimental DTSBN model that has been successfully generated new skeleton sequences, which represent a SIBI hand gesture. Some of the inputs to the model include cartesian coordinates from shoulder joints, elbow joints, and wrist joints and the newly generated data are proven have no significant difference with the actual data set.

For the traditional denoising method, it is necessary to accurately model the signal and noise and optimize and adjust the manual input parameters, which causes the problem that it is difficult to remove the noise of the seismic data. Especially random noise in remote areas has the characteristics of non-stationary, high energy, and serious aliasing of effective signals and random noise in the frequency domain, which brings great difficulty to the recovery of seismic data by conventional denoising methods, but the existence of random noise seriously affects subsequent earthquakes. The processing and interpretation of data makes it difficult to determine the exact location of oil and gas. It is necessary to develop an efficient denoising algorithm, which can retain the complex edge information and rich texture information of the signal as much as possible while removing random noise, so as to restore the seismic data and improve the utilization rate of the seismic data. In this paper, the convolutional neural network is applied to seismic data processing. The algorithm has a good denoising effect in seismic data. While eliminating a large amount of random noise, it retains the texture features in the data, realizes denoising of seismic data, and enhances vision quality.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Riboswitches are structured allosteric RNA molecules that change conformation upon metabolite binding, triggering a regulatory response. Here we focus on the de novo design of riboswitch-like aptamers, the core part of the riboswitch undergoing structural changes. We use Restricted Boltzmann machines (RBM) to learn generative models from homologous sequence data. We first verify, on four different riboswitch families, that RBM-generated sequences correctly capture the conservation, covariation and diversity of natural aptamers. The RBM model is then used to design new SAM-I riboswitch aptamers. To experimentally validate the properties of the structural switch in designed molecules, we resort to chemical probing (SHAPE and DMS), and develop a tailored analysis pipeline adequate for high-throughput tests of diverse sequences. We probe a total of 476 RBM-designed and 201 natural sequences. Designed molecules with high RBM scores, with 20% to 40% divergence from any natural sequence, display ≈ 30% success rate of responding to SAM with a structural switch similar to their natural counterparts. We show how the capability of the designed molecules to switch conformation is connected to fine energetic features of their structural components.

Head pose estimation methods can be broadly classified into learning-based methods and model-based methods. The learning based methods use machine learning techniques to directly predict the pose from image appearance, while the model-based methods link the 2D observation (e.g. facial landmarks) and 3D model through the projection model for pose estimation. However, both methods may have difficulty on images with very low quality (e.g. low resolution, occlusion, and noisy images). For example, there would be limited appearance information to generate accurate landmark detection on lowquality images for reliable face pose estimation. To tackle pose estimation on low-quality images, we propose to combine the learning and model based methods. Specifically, we first build the relationship between facial landmark locations and image appearance using the Restricted Boltzmann Machine (RBM) model. Then, we link the landmark locations and 3D model analytically using the projection model. By combining the RBM model with the projection model, without explicit landmark detection, we predict the head pose with a KL-divergence based method and a gradient-based method. Experimental results demonstrate the effectiveness of the proposed method.

The existing traditional neural network reconstruction models have some questions, including the high training epochs, low recognition rate, and complex structure. The restricted Boltzmann machine (RBM) is an excellent generative learning model for feature extraction, which is a simple model compared to other deep neural networks. The Discriminative Fuzzy Restricted Boltzmann Machine (DFRBM) is proposed by extending its parameters from natural numbers to fuzzy ones. Then we introduced the random permutation (RP) algorithm, with the hidden units random permutation, Discriminative Infinite Fuzzy Restricted Boltzmann Machine (Dis-iFRBM) is proposed. Dis-iFRBM is a better RBM model than DFRBM and Classical RBMs.We further investigate and compare the generative ability of the Dis-iFRBM on image reconstruction. First, we transform the MSTAR SAR piece to HRRPs images. Then the Dis-iFRBM, DFRBM, and Classical RBMs are compared in detail under optimal conditions on the HRRPs images that transformed from MSTAR data sets. Specifically, they can be trained by relatively limited datasets into excellent stand alone classifiers and retain satisfactory generative capability simultaneously. The comparison of experimental images shows that the Dis-iFRBM possesses better generative capability than the Discriminative Fuzzy Restricted Boltzmann Machine (DFRBM). Meanwhile, surveillance images in the city, license plate number recognition, and other scenarios need damaged image restoration. Dis-iFRBM as a model can save computational resources of terminal image devices that deploy in the Urban Internet of Things. The experiment results indicate that the Dis-iFRBM outperforms image restoration. It can achieve smaller average reconstruction errors (AREs) while given a small number of hidden units. Finally, experimentation over several classical RBMs revealed the proposed approach's preferable reconstruction capability.

Restricted Boltzmann Machine (RBM) is a generative, stochastic neural network with two separate layers of hidden and visible units. Training data samples in RBM are usually corrupted by noise. RBM is not robust enough to handle such noises, which leads to uncertainty. In the literature, Fuzzy RBM (FRBM) has already been proposed for enhancing deep learning. In FRBM, the parameters of RBM are modelled as type-1 fuzzy numbers. However, there can be multiple sources of uncertainties such as noises in the data measurements, variations in the environment where they are deployed, etc. Such uncertainties cannot be modelled by type-1 fuzzy sets (T1 FS), since their membership values are themselves crisp in nature. On the other side, IT2 FS can model higher order uncertainties with their fuzzy membership grades. So, we propose to use interval type-2 fuzzy sets (IT2 FS) to model uncertain parameters of RBMs in the learning stage. Since deep neural network (DNN) is pre-trained using stacked RBMs, modeling noises using IT2 FS would demonstrate high performance and low root mean square error (RMSE) while learning. Thus, we propose a new algorithm viz. Interval type-2 fuzzy set based approach for enhanced deep learning (IT2 FS-EDL) in which RBM parameters are modeled as type-2 fuzzy sets in the learning process. Numerical examples and experimentations have been demonstrated to present the suitability of our proposed approach.

Deep Belief Networks (DBNs) have emerged as powerful tools for feature learning, representation, and generative modeling. This paper presents a comprehensive exploration of the various training algorithms employed in the training of DBNs. DBNs, composed of multiple layers of stochastic hidden units, have found applications in diverse domains such as computer vision, natural language processing, and bioinformatics. The paper begins by delving into the pre-training phase, where Restricted Boltzmann Machines (RBMs) play a central role. We review the Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) algorithms, shedding light on their strengths and weaknesses in initializing deep networks. Emphasis is placed on their applicability to different data types and scales. Moving to the fine-tuning stage, the paper explores the use of backpropagation with gradient descent, discussing modern optimization techniques, including stochastic gradient descent and adaptive learning rate methods. We also examine regularization techniques like dropout and weight decay to address overfitting concerns. Furthermore, we discuss architectural variants of DBNs, such as Convolutional Deep Belief Networks (CDBNs) for image data and Recurrent DBNs for sequential data. We highlight the adaptation of DBNs for specific tasks, including classification, regression, clustering, and generative modeling.

Seismic random noise suppression by using deep residual U-Net

Deep learning (DL) applications of seismic reservoir characterization often require the generation of synthetic data to augment available sparse labeled data. An approach for generating synthetic training data consists of specifying probability distributions modeling prior geologic uncertainty on reservoir properties and forward modeling the seismic data. A prior falsification approach is critical to establish the consistency of the synthetic training data distribution with real seismic data. With the help of a real case study of facies classification with convolutional neural networks (CNNs) from an offshore deltaic reservoir, we have highlighted several practical nuances associated with training DL models on synthetic seismic data. We highlight the issue of overfitting of CNNs to the synthetic training data distribution and propose regularization strategies to address it. We demonstrate the efficacy of our proposed strategies by training the CNN on synthetic data and making robust predictions with real 3D partial stack seismic data.