Extracting and inserting knowledge into stacked denoising auto-encoders

Software defect prediction technology plays an important role in ensuring software quality. The traditional software defect prediction model can only perform “shallow learning” and cannot perform deep mining of data features. Aiming at this problem, we use the stacked denoising auto-encoder (SDAE) to superimpose into deep neural network. First, the deep network model was built through the stacked layers of denoising auto-encoder (DAE), then the unsupervised method was used to train each layer in turn with noised input for more robust expression, characteristics were learnt supervised by back propagation (BP) neural network and the whole net was optimized by using error back propagation. Simulation experiments prove that the prediction accuracy of our SDAE model is significantly improved compared with the traditional SVM and KNN prediction model.

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Domain-invariant representation learning using an unsupervised domain adversarial adaptation deep neural network

Heart disease is a prevalent and life-threatening condition that affects a significant portion of the global population. Preventing negative consequences and better health outcomes may be greatly helped by early identification and precise prognosis of cardiac disease. Electrocardiogram (ECG) is the important apparatus which is used for diagnosticians of assessment of cardiogram in the clinical. This will Owing to its ability to learn complicated patterns from massive volumes of data, deep neural networks (DNNs) have recently emerged as strong tools for medical diagnostic and prediction tasks. In this part of research, people investigate the use of deep neural networks for cardiac illness forecasting which is easy way for analysis of cardio problem. We analyse various studies and approaches that have utilized Deep Neural Networks for heart disease prediction. This study gives us whole sheds light on the evolution of methods that use in deep neural networks to forecast cardiac issues. The findings from this survey can guide researchers and practitioners in designing and implementing effective DNN models for heart disease prediction, ultimately contributing to improved clinical decision-making and patient care.In this study, we will examine the various DNN techniques, including CNNs (Convolutional Neural Networks) and Stack Denoising Auto encoder, Deep Belief Network, Recurrent Neural Network (RNN), Long Short term Memory (LSTM) and Gated Recurrent Unit (GRU).

This article presents a new hyperspectral image classification method, which is capable of automatic feature learning while achieving high classification accuracy. The method contains the following two major modules: the spectral classification module and the spatial constraints module. Spectral classification module uses a deep network, called ‘Stacked Denoising Autoencoders’ (SdA), to learn feature representation of the data. Through SdA, the data are projected non-linearly from its original hyperspectral space to some higher-dimensional space, where more compact distribution is obtained. An interesting aspect of this method is that it does not need any prior feature design/extraction process guided by human. The suitable feature for the classification is learnt by the deep network itself. Superpixel is utilized to generate the spatial constraints for the refinement of the spectral classification results. By exploiting the spatial consistency of neighbourhood pixels, the accuracy of classification is further improved by a big margin. Experiments on the public data sets have revealed the superior performance of the proposed method.

DBMiP: A pre-training method for information propagation over deep networks

Deep neural networks have made significant achievements in representation learning of traditionally man-made features, especially in terms of complex objects. Over the decades, this learning process has attracted thousands of researchers and has been widely used in the speech, visual, and text recognition fields. One deep network multi-layer extreme learning machine (ML-ELM) achieves a good performance in representation learning while inheriting the advantages of faster learning and the approximating capability of the extreme learning machine (ELM). However, as with most deep networks, the ML-ELM’s algorithmic performance largely depends on the probability distribution of the training data. In this paper, we propose an improved ML-ELM made via using the local significant regions at the input end to enhance the contributions of these regions according to the idea of the selective attention mechanism. To avoid involving and exploring the complex principle of the attention system and to focus on the clarification of our local regional enhancement idea, the paper only selects two typical attention regions. One is the geometric central region, which is normally the important region to attract human attention due to the focal attention mechanism. The other is the task-driven interest region, with facial recognition as an example. The comprehensive experiments are done on the three public datasets of MNIST, NORB, and ORL. The comparison experiment results demonstrate that our proposed region-enhanced ML-ELM (RE-ML-ELM) achieves performance increases in important feature learning by utilizing the apriori knowledge of attention and has a higher recognition rate than that of the normal ML-ELM and the basic ELM. Moreover, it benefits from the non-iterative parameter training method of other ELMs, and our proposed algorithm outperforms most state-of-the-art deep networks such as deep belief network(DBN), in the aspects of training efficiency. Furthermore, because of the deep structure with fewer hidden nodes at each layer, our proposed RE-ML-ELM achieves a comparable training efficiency to that of the ML-ELM but has a higher training speed with the basic ELM, which is normally the width single network that has more hidden nodes to obtain the similar recognition accuracy with the deep networks. Based on our idea of combining the apriori knowledge of the human selective attention system with the data learning, our proposed region-enhanced ML-ELM increases the image classification performance. We believe that the idea of intentionally combining psychological knowledge with the most algorithms based on data-driven learning has the potential to improve their cognitive computing ability.

Sensitive information such as credit card information, username, password and social security number etc, can be stolen using a fake page that imitates trusted website is called phishing. The attacker designs a similar webpage either by copying or making small manipulation to the legitimate page so that the online user cannot distinguish the legitimate and fake websites. A Deep Neural Network (DNN) was introduced to detect the phishing Uniform Resource Locator (URL). Initially, a 30-dimension feature vector was constructed based on URL-based features, Hypertext Markup Language (HTML)-based features and domain-based features. These features were processed in DNN to detect the phishing URL. However, the irrelevant, redundant and noisy features in the dataset increase the complexity of DNN classifier. So the feature selection is required for efficient phishing attack detection. But feature selection is a time-consuming process since it is an independent process. So in this paper, a feature vector is generated by DNN itself using Stacked Denoise Auto Encoder (SDAE). Moreover, the noisy data such as missing features affect the efficiency of phishing detection so the SDAE is trained to reconstruct a clean input feature vector. The initial input feature vector is corrupted by setting some feature vectors as zero. Then the corrupted feature vector is then mapped with basic auto encoder, to a hidden representation from which the input feature vector is reconstructed. The reconstructed features are given as input to DNN which selects the most relevant features and predicts the phishing URL. Hence the sparse feature representation of SDAE increases the classification accuracy of DNN. The experiments are conducted in Ham, Phishing Corpus and Phishload datasets in terms of accuracy, precision, recall and F-measure to prove the effectiveness of DNN-SDAE.

Accurate pollutant prediction is essential in fields such as meteorology, meteorological disasters, and climate change studies. In this study, long short-term memory (LSTM) and deep neural network (DNN) models were applied to six pollutants and comprehensive air-quality index (CAI) predictions from 2015 to 2020 in Korea. In addition, we used the network method to find the best data sources that provide factors affecting comprehensive air-quality index behaviors. This study had two steps: (1) predicting the six pollutants, including fine dust (PM10), fine particulate matter (PM2.5), ozone (O3), sulfurous acid gas (SO2), nitrogen dioxide (NO2), and carbon monoxide (CO) using the LSTM model; (2) forecasting the CAI using the six predicted pollutants in the first step as predictors of DNNs. The predictive ability of each model for the six pollutants and CAI prediction was evaluated by comparing it with the observed air-quality data. This study showed that combining a DNN model with the network method provided a high predictive power, and this combination could be a remarkable strength in CAI prediction. As the need for disaster management increases, it is anticipated that the LSTM and DNN models with the network method have ample potential to track the dynamics of air pollution behaviors.

Stacked autoencoder (SAE) is hard to achieve satisfactory performance, when input data are complex and non-stationary. Besides, the identification performance of recurrent neural network (RNN) may decrease rapidly under noisy environment. In order to deal with these problems, a novel hybrid deep neural network (DNN) based on stacked denoising autoencoder (SDAE) and gated recurrent unit neural network (GRUNN) is presented. First, the structure of the presented hybrid DNN is given. The hybrid DNN contains a SDAE, a GRUNN, and a softmax classifier. Then, the training algorithm based on action discovery (AD) is proposed to train the presented hybrid DNN. The experimental studies indicate the presented hybrid DNN processes strong anti-noise ability and adaptability to time-varying signals.

By the development of social media, sentiment analysis has changed to one of the most remarkable research topics in the field of natural language processing which tries to dig information from textual data containing users’ opinions or attitudes toward a particular topic. In this regard, deep neural networks have emerged as promising techniques that have been extensively used for this aim in recent years and obtained significant results. Considering the fact that deep neural networks can automatically extract features from data, it can be claimed that intermediate representations extracted from these networks can be also used as appropriate features. While different deep neural networks are able to extract various types of features due to their distinct structures, we decided to combine features extracted from heterogeneous neural networks using multi-view classifiers to enhance the overall performance of document-level sentiment analysis by considering the correlation between them. The proposed multi-view deep network makes use of intermediate features extracted from convolutional and recursive neural networks to perform classification. Based on the results of the experiments, the proposed multi-view deep network not only outperforms single-view deep neural networks but also has superior efficiency and generalization performance.

Purpose: The aim of the article is to develop an algorithm for classifying cracks in the analyzed images using modern methods of deep machine learning and transfer learning based on pretrained convolutional neural network - Inception-ResNet-v2. Design/Methodology/Approach: Transfer learning based on the pretrained convolutional neural network was used to categorize the images. The fully conected layer of the Inception-ResNet-v2 network has been modified. The last layer was trained using a two-class (binary) linear SVM (Support Vector Machine). In total, 20,000 training cases (images) were used to train the fully connected layer within transfer learning process. The research analyzed the possibility of using the deep neural networks for quick and fully automatic identification of cracks / defects on the surface of analyzed parts. Findings: The results indicate that pretrained convolutional neural network using SVM to train a fully connected layer is a very effective solution for visual crack / fault detection. In the analyzed model, a positive classification was obtained at the level of 99.89%. Practical Implications: The model presented in the article can be used in quality control carried out by process monitoring. An effective model for identifying defective parts can be used in both logistics and production processes. Originality/Value: A novelty is the use of a freely available, deep neural network trained to classify 1000 categories of various images for binary categorization of faults (cracks). The algorithm was adjusted by replacing the primary, 1000-output fully connected layer in the Inception-ResNet-v2 network with a binary layer (2 categories). The fully connected layer has been trained using the classification version of the popular SVM learner, but thanks to the combination of this layer with the sophisticated fearure extraction ability of the pre-trained Inception-ResNet-v2 deep network, the resulting predictive model enables the classification of defects with a very high level of accuracy.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

In order to solve the problem of different marine noise signals' classification, a multi-layer neuron networks model, which can be used to learn and analyze different marine noise signals, is built based on DNN (Deep Neural Networks) model in this article. Firstly, let's generate the initialized weight value randomly. Secondly, do linear operation that the input values of each layer multiply the weight values and then add the figures together. Thirdly, function value normalization is achieved by implementing nonlinear sigmoid active function, and we can get error function of actual output and desired output. Fourthly, we can get error coefficient of weight value and minimal value by gradient descent algorithm. In the last, we can get classification weight value which can distinguish different marine noise signals by summing this coefficient and weight value to keep weigh value updated. In the article, a four-layer deep neuron networks is built, of which three layers are hidden. Train the matrix data, test it and the result is that there are four errors among 100 test objects with 94%o accuracy. At the same time, the average accuracy of the 10 test results was 91.7%. It proves that this method can achieve the marine noise signals' classification.

Training very deep neural networks is very difficult because of gradient degradation. However, the incomparable expressiveness of the many deep layers is highly desirable at testing time and usually leads to better performance. Recently, training techniques such as residual networks that enable us to train very deep networks have proved to be a great success. In this paper, we studied the application of the recently proposed deep networks with stochastic depth (DNSD) to train deeper acoustic models for speech recognition. By randomly dropping a subset of layers during training, the studied stochastic depth training method helps reduce the training time substantially, yet the networks trained are much deeper since all the layers are kept during testing. We investigated this approach on the TIMIT data set. Our preliminary experimental results show that when training data are limited, deep networks with stochastic depth helps very little. However, when more training data are available, DNSD significantly improves the recognition accuracy, compared with a conventional deep neural networks.