Generalized Deep Transfer Networks for Knowledge Propagation in Heterogeneous Domains

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Extracting and inserting knowledge into stacked denoising auto-encoders

Mutual calibration training: Training deep neural networks with noisy labels using dual-models

Domain-invariant representation learning using an unsupervised domain adversarial adaptation deep neural network

Training very deep neural networks is very difficult because of gradient degradation. However, the incomparable expressiveness of the many deep layers is highly desirable at testing time and usually leads to better performance. Recently, training techniques such as residual networks that enable us to train very deep networks have proved to be a great success. In this paper, we studied the application of the recently proposed deep networks with stochastic depth (DNSD) to train deeper acoustic models for speech recognition. By randomly dropping a subset of layers during training, the studied stochastic depth training method helps reduce the training time substantially, yet the networks trained are much deeper since all the layers are kept during testing. We investigated this approach on the TIMIT data set. Our preliminary experimental results show that when training data are limited, deep networks with stochastic depth helps very little. However, when more training data are available, DNSD significantly improves the recognition accuracy, compared with a conventional deep neural networks.

Purpose: The aim of the article is to develop an algorithm for classifying cracks in the analyzed images using modern methods of deep machine learning and transfer learning based on pretrained convolutional neural network - Inception-ResNet-v2. Design/Methodology/Approach: Transfer learning based on the pretrained convolutional neural network was used to categorize the images. The fully conected layer of the Inception-ResNet-v2 network has been modified. The last layer was trained using a two-class (binary) linear SVM (Support Vector Machine). In total, 20,000 training cases (images) were used to train the fully connected layer within transfer learning process. The research analyzed the possibility of using the deep neural networks for quick and fully automatic identification of cracks / defects on the surface of analyzed parts. Findings: The results indicate that pretrained convolutional neural network using SVM to train a fully connected layer is a very effective solution for visual crack / fault detection. In the analyzed model, a positive classification was obtained at the level of 99.89%. Practical Implications: The model presented in the article can be used in quality control carried out by process monitoring. An effective model for identifying defective parts can be used in both logistics and production processes. Originality/Value: A novelty is the use of a freely available, deep neural network trained to classify 1000 categories of various images for binary categorization of faults (cracks). The algorithm was adjusted by replacing the primary, 1000-output fully connected layer in the Inception-ResNet-v2 network with a binary layer (2 categories). The fully connected layer has been trained using the classification version of the popular SVM learner, but thanks to the combination of this layer with the sophisticated fearure extraction ability of the pre-trained Inception-ResNet-v2 deep network, the resulting predictive model enables the classification of defects with a very high level of accuracy.

In recent years, deep neural networks (DNNs) have achieved great successes in many areas, such as cognitive computation, pattern recognition, and computer vision. Although many hand-crafted deep networks have been proposed in the literature, designing a well-behaved neural network for a specific application requires high-level expertise yet. Hence, the automatic architecture design of DNNs has become a challenging and important problem. In this paper, we propose a new reinforcement learning method, whose action policy is to select neural blocks and construct deep networks. We define the action search space with three types of neural blocks, i.e., dense block, residual block, and inception-like block. Additionally, we have also designed several variants for the residual and inception-like blocks. The optimal network is automatically learned by a Q-learning agent, which is iteratively trained to generate well-performed deep networks. To evaluate the proposed method, we have conducted experiments on three datasets, MNIST, SVHN, and CIFAR-10, for image classification applications. Compared with existing hand-crafted and auto-generated neural networks, our auto-designed neural network delivers promising results. Moreover, the proposed reinforcement learning algorithm for deep networks design only runs on one GPU, demonstrating much higher efficiency than most of the previous deep network search approaches.

In view of the scene’s complexity and diversity in scene classification, this paper makes full use of the contextual semantic relationships between the objects to describe the visual attention regions of the scenes and combines with the deep convolution neural networks, so that a scene classification model using visual attention and deep networks is constructed. Firstly, the visual attention regions in the scene image are marked by using the context-based saliency detection algorithm. Then, the original image and the visual attention region detection image are superimposed to obtain a visual attention region enhancement image. Furthermore, the deep convolution features of the original image, the visual attention region detection image, and the visual attention region enhancement image are extracted by using the deep convolution neural networks pretrained on the large-scale scene image dataset Places. Finally, the deep visual attention features are constructed by using the multilayer deep convolution features of the deep convolution networks, and a classification model is constructed. In order to verify the effectiveness of the proposed model, the experiments are carried out on four standard scene datasets LabelMe, UIUC-Sports, Scene-15, and MIT67. The results show that the proposed model improves the performance of the classification well and has good adaptability.

Deep neural networks have made significant achievements in representation learning of traditionally man-made features, especially in terms of complex objects. Over the decades, this learning process has attracted thousands of researchers and has been widely used in the speech, visual, and text recognition fields. One deep network multi-layer extreme learning machine (ML-ELM) achieves a good performance in representation learning while inheriting the advantages of faster learning and the approximating capability of the extreme learning machine (ELM). However, as with most deep networks, the ML-ELM’s algorithmic performance largely depends on the probability distribution of the training data. In this paper, we propose an improved ML-ELM made via using the local significant regions at the input end to enhance the contributions of these regions according to the idea of the selective attention mechanism. To avoid involving and exploring the complex principle of the attention system and to focus on the clarification of our local regional enhancement idea, the paper only selects two typical attention regions. One is the geometric central region, which is normally the important region to attract human attention due to the focal attention mechanism. The other is the task-driven interest region, with facial recognition as an example. The comprehensive experiments are done on the three public datasets of MNIST, NORB, and ORL. The comparison experiment results demonstrate that our proposed region-enhanced ML-ELM (RE-ML-ELM) achieves performance increases in important feature learning by utilizing the apriori knowledge of attention and has a higher recognition rate than that of the normal ML-ELM and the basic ELM. Moreover, it benefits from the non-iterative parameter training method of other ELMs, and our proposed algorithm outperforms most state-of-the-art deep networks such as deep belief network(DBN), in the aspects of training efficiency. Furthermore, because of the deep structure with fewer hidden nodes at each layer, our proposed RE-ML-ELM achieves a comparable training efficiency to that of the ML-ELM but has a higher training speed with the basic ELM, which is normally the width single network that has more hidden nodes to obtain the similar recognition accuracy with the deep networks. Based on our idea of combining the apriori knowledge of the human selective attention system with the data learning, our proposed region-enhanced ML-ELM increases the image classification performance. We believe that the idea of intentionally combining psychological knowledge with the most algorithms based on data-driven learning has the potential to improve their cognitive computing ability.

Deep neural network is an important and effective approach to discover patterns inside data. Most existing neural networks, like deep neural network and convolutional neural network, have a fixed network connection among neurons. This will cause overfitting problems and long training time. This article discusses an algorithm to combat these problems, through creating and removing connections during training, leading to faster training and fewer symmetric problems. This enables the network to adapt itself to the proper number of neurons according to the dataset. It greatly speeds up training process, so it is useful for large datasets and deep networks. This method is simple to implement and requires less memory and computing power than existing machine learning algorithm. It also has good results in machine learning problems.

Recently, data-driven algorithms such as deep neural networks have attracted a lot of attention and have become a popular area of research and development. This interest is driven by several factors, such as recent advances in processing power (cheap and powerful hardware), the availability of large datasets (big data), and several small but important algorithmic advances (e.g., convolutional layers). Nowadays, deep neural networks are the state-of-the-art for several computer vision tasks, such as the ones that require high-level understanding of image semantics, e.g., image classification, object segmentation, saliency detection, and also in low level image processing tasks, such as image denoising, inpainting, and super-resolution, among others. Deep learning can handle large volumes of video data by making use of its powerful non-linear mapping and extracting high-level features with very deep networks. Incorporating deep learning into the IoVT can provide radical innovations in video sensing, coding, enhancing, understanding, and evaluation areas making it easier to handle the enormous growth in video data compared to traditional methods.

The raising number of elderly people urges the research of systems able to monitor and support people inside their domestic environment. An automatic system capturing data about the position of a person in the house, through accelerometers and RGBd cameras can monitor the person activities and produce outputs associating the movements to a given tasks or predicting the set of activities that will be executes. We considered, for the task the classification of the activities a Deep Convolutional Neural Network. We compared two different deep network and analyzed their outputs.

DBMiP: A pre-training method for information propagation over deep networks

Recently, the fusion of hyperspectral and light detection and ranging (LiDAR) data has obtained a great attention in the remote sensing community. In this paper, we propose a new feature fusion framework using deep neural network (DNN). The proposed framework employs a novel 3D convolutional neural network (CNN) to extract the spectral-spatial features of hyperspectral data, a deep 2D CNN to extract the elevation features of LiDAR data, and then a fully connected deep neural network to fuse the extracted features in the previous CNNs. Through the aforementioned three deep networks, one can extract the discriminant and invariant features of hyperspectral and LiDAR data. At last, logistic regression is used to produce the final classification results. The experimental results reveal that the proposed deep fusion model provides competitive results. Furthermore, the proposed deep fusion idea opens a new window for future research.

Seeing through disguise: Getting to know you with a deep convolutional neural network