Smart global memory regularization for efficient feature learning in deep neural networks

Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called "REBA" for the unsupervised training of deep neural networks. In contrast to Hinton's conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks.

To construct the risk prediction model of acute exacerbation of chronic obstructive pulmonary disease (AECOPD) and verify its effectiveness based on deep learning and back propagation algorithm neural network (BP neural network). Based on the relevant data of 1 326 patients with chronic obstructive pulmonary disease (COPD) in the team's previous clinical study, the acute exacerbation, and its risk factors during the stable period and 6 months of follow-up were recorded and analyzed. Combined with previous clinical research data and expert questionnaire results, the independent risk factors of AECOPD after screening and optimization by multivariate Logistic regression including gender, body mass index (BMI) classification, number of acute exacerbation, duration of acute exacerbation and forced expiratory volume in one second (FEV1) were used to build the BP neural network by Python 3.6 programming language and Tensorflow 1.12 deep learning framework. The patients were randomly selected according to the ratio of 4:1 to generate the training group and the test group, of which, the training group had 1 061 sample data while the test group had 265 pieces of sample data. The training group was used to establish the prediction model of neural network, and the test group was used for back-substitution test. When using the training group data to construct the neural network model, the training group was randomly divided into training set and verification set according to the ratio of 4:1. There were 849 training samples in the training set and 212 verification samples in the verification set. The optimal model was screened by adjusting the parameters of the neural network and combining the area under the receiver operator characteristic curve (AUC), and the sample data of the test group was substituted into the model for verification. The independent risk factors including gender, BMI classification, number of acute exacerbation, duration of acute exacerbation and FEV1 were collected from the team's previous clinical research, and the AECOPD risk prediction model was constructed based on deep learning and BP neural network. After 10 000 training sessions, the accuracy of the AECOPD risk prediction model in the validation set of the training group was 83.09%. When the number of training times reached 8 000, the accuracy basically tended to be stable and the prediction ability reached the upper limit. The AECOPD risk prediction model trained for 10 000 times was used to predict the risk of the validation set data, and the receiver operator characteristic curve (ROC curve) analysis showed that the AUC was 0.803. When using this model to predict the risk of the data of the test group, the accuracy rate was 81.69%. The risk prediction model based on deep learning and BP neural network has a medium level of prediction efficiency for acute exacerbation within 6 months in COPD patients, which can evaluate the risk of AECOPD and assist the clinic in making accurate treatment decisions.

An energy efficient deep neural network (DNN) learning processor is proposed using direct feedback alignment (DFA). The proposed processor achieves $2.2 \times$ faster learning speed compared with the previous learning processors by the pipelined DFA (PDFA). In order to enhance the energy efficiency by 38.7%, the heterogeneous learning core (LC) architecture is optimized with the 11-stage pipeline data-path. Furthermore, direct error propagation core (DEPC) utilizes random number generators (RNG) to remove external memory access (EMA) caused by error propagation (EP) and improve the energy efficiency by 19.9%. The proposed PDFA based learning processor is evaluated on the object tracking (OT) application, and as a result, it shows 34.4 frames-per-second (FPS) throughput with 1.32 TOPS/W energy efficiency.

This study presents a fault diagnosis system for vehicle heating, ventilation and air conditioning (HVAC) acoustic signal with various feature extractions in deep learning neural network. Traditionally, sound used for fault diagnosis or signal classification is observed the difference of energy in time or frequency domains. Unfortunately, the frequency smearing effect often arises in some critical conditions. In the present study, discrete wavelet transform (DWT) and wavelet packet transform (WPT) are proposed in fault diagnosis. Meanwhile, when using mechanical learning methods, the data are relatively large, in order to reduce the amount of data, DWT and WPT low-frequency decomposition could be used to improve the performance. Furthermore, the signal characteristics more comprehensive, this study attempts to use the feature extraction method of wavelet packet conversion to improve the signal characteristics. In the experiment process, the operation state of the blade blower in the vehicle air conditioner, four different faults were designed, test database was established through sound to classify, and identify the data using deep neural networks to achieve the purpose of blower fault diagnosis. In data analysis, the original signal is presented through wavelet packet decomposition and discrete packet conversion technology, compared with traditional time and frequency domain signals to explore the identification rate, identification speed and related issues. Experimental results show that using WPT combined with deep neural networks have good fault diagnosis and discrimination capabilities, training, and identification time is shorter than time-frequency domain signals.

In this paper, an intelligent multi-microgrid (MMG) energy management method is proposed based on deep neural network (DNN) and model-free reinforcement learning (RL) techniques. In the studied problem, multiple microgrids are connected to a main distribution system and they purchase power from the distribution system to maintain local consumption. From the perspective of the distribution system operator (DSO), the target is to decrease the demand-side peak-to-average ratio (PAR), and to maximize the profit from selling energy. To protect user privacy, DSO learns the MMG response by implementing a DNN without direct access to user's information. Further, the DSO selects its retail pricing strategy via a Monte Carlo method from RL, which optimizes the decision based on prediction. The simulation results from the proposed data-driven deep learning method, as well as comparisons with conventional model-based methods, substantiate the effectiveness of the proposed approach in solving power system problems with partial or uncertain information.

Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.

Algorithmic breakthroughs, the feasibility of collecting huge amount of data, and increasing computational power, contribute to the remarkable achievements of NNs. In particular, since Deep Neural Network (DNN) learning presents astonishing results in speech and image recognition, the amount of sophisticated applications based on it has exploded. However, the increasing number of instances of privacy leakage has been reported, and the corresponding severe consequences have caused great worry in this area. In this article, we focus on privacy issues in NN learning. First, we identify the privacy threats during NN training, and present privacy-preserving training schemes in terms of using centralized and distributed approaches. Second, we consider the privacy of prediction requests, and discuss the privacy-preserving protocols for NN prediction. We also analyze the privacy vulnerabilities of trained models. Three types of attacks on private information embedded in trained NN models are discussed, and a differential privacy-based solution is introduced.

Aim: This study set out to determine how well AI approaches like Artificial Neural Networks (ANNs) and Deep Learning Neural Networks (DLNNs) might be used to forecast rainfall (DNN). These methods of weather prediction were tested and ranked in terms of their efficiency. Substances and Techniques: Group 1 uses a Deep Learning Neural Network (DNN) for analysis, whereas Group 2 uses an Artificial Neural Network (ANN) with a pre-test power (G-power) of 80% and an alpha error rate (Err) of 0.05 for sample test analysis. In all, 20 samples were identified; 10 from each category. According to the outcomes of the MATLAB simulations, the accuracy of the Deep Learning Neural Network (DNN) is 92.59%, while that of the Artificial Neural Network (ANN) is 95.68%. By means of SPSS statistical program, we find that the achieved accuracy ratio is 0.034 (p 0.05). This study concludes that the unique Artificial Neural Network (ANN) algorithm outperforms the Deep Learning Neural Network (DNN) method in predicting rainfall in the context of cutting-edge weather forecasting.

Deep neural networks learn useful representations of data, yet the nature of these representations has not been fully understood. Here, we empirically study the kernels induced by the layer representations during training by analyzing their kernel alignment to the network's target function. We show that representations from earlier to deeper layers increasingly align with the target task for both training and test sets, implying better generalization. We analyze these representations across different architectures, optimization methods and batch sizes. Furthermore, we compare the Neural Tangent Kernel (NTK) of deep neural networks and its alignment with the target during training and find that NTK-target alignment also increases during training.

In this paper we study the effect of target set size on transfer learning in deep learning convolutional neural networks. This is an important problem as labelling is a costly task, or for new or specific classes the number of labelled instances available may simply be too small. We present results for a series of experiments where we either train on a target of classes from scratch, retrain all layers, or subsequently lock more layers in the network, for the Tiny-ImageNet and MiniPlaces2 data sets. Our findings indicate that for smaller target data sets freezing the weights for the initial layers of the network gives better results on the target set classes. We present a simple and easy to implement training heuristic based on these findings.

The main purpose of the various methods of evaluating athlete feature recognition is to monitor the current health of the athletes, thereby providing some feedback on the quality of individual training. Based on deep learning and convolutional neural networks, this paper studies athlete target recognition and proposes a feature vector extraction method based on curvature zero point. Moreover, based on the ideas of deep learning and convolutional neural networks, this paper builds an athlete feature recognition model and optimizes the algorithm. In order to verify the feasibility and efficiency of feature extraction algorithm of the sport athletes proposed by this paper and to facilitate comparison with other algorithms, this paper conducts an algorithm performance test on the sport athlete database. The research results show that the method proposed in this paper has certain advantages in the feature extraction of athletes and can be used in subsequent sports training systems.

We propose a multitask learning (MTL) approach to improve low-resource automatic speech recognition using deep neural networks (DNNs) without requiring additional language resources. We first demonstrate that the performance of the phone models of a single low-resource language can be improved by training its grapheme models in parallel under the MTL framework. If multiple low-resource languages are trained together, we investigate learning a set of universal phones (UPS) as an additional task again in the MTL framework to improve the performance of the phone models of all the involved languages. In both cases, the heuristic guideline is to select a task that may exploit extra information from the training data of the primary task(s). In the first method, the extra information is the phone-to-grapheme mappings, whereas in the second method, the UPS helps to implicitly map the phones of the multiple languages among each other. In a series of experiments using three low-resource South African languages in the Lwazi corpus, the proposed MTL methods obtain significant word recognition gains when compared with single-task learning (STL) of the corresponding DNNs or ROVER that combines results from several STL-trained DNNs.

Feature-learning deep nets progressively collapse data to a regular low-dimensional geometry. How this emerges from the collective action of nonlinearity, noise, learning rate, and other factors, has eluded first-principles theories built from microscopic neuronal dynamics. We exhibit a noise-nonlinearity phase diagram that identifies regimes where shallow or deep layers learn more effectively and propose a macroscopic mechanical theory that reproduces the diagram and links feature learning across layers to generalization.

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.

Background: Retinal fundus images, obtained from routine eye screenings, contain important information, not only concerning microvascular lesions of the retina, but potentially also of the progression and manifestation of various other diabetes complications. Recent advances in the field of machine learning, so-called deep learning neural networks (DNN), make it possible to automatically detect retinal lesions from fundus photos. Our overall aim is to develop a DNN to detect retinal vascular lesions. In this abstract, we describe a pre-processing step consisting of application of a DNN. Methods: For each eye, 5 retinal images of parts of the eye are taken, and collected into a joint mosaic image, used for image grading. There is no automated labelling of the images to identify left or right eyes. To distinguish left- from right eye images, we used a set of 1,916 unique, high-definition retina mosaic fundus images from diabetes patients followed at Steno Diabetes Center Copenhagen (SDCC) between 2003-2017. We trained and validated a DNN, using 964 left-eye (training set: 713, validation: 251) and 952 right-eye images (training set: 706, validation: 246). A DNN is a method that finds the optimal algorithm for relating a large set of input (mosaic fundus images) to output (left or right eye). The DNN used was based on the InceptionV3 architecture and run for 100 epochs with 100 gradient descent steps pr. epoch. Results: The DNN classification obtained a validation AUC-ROC of 0.92 and a validation accuracy of 0.86. Conclusions: The DNN successfully distinguished left-eye from right-eye images. We plan to use the DNN to identify right- and left-eye images in the SDCC image database comprising ∼ 80,000 left-eye and right-eye images. We will subsequently develop a DNN for automated detection of referable diabetic retinopathy of retinal fundus images, for use in the clinic. Disclosure L. Díaz: None. D. Vistisen: Stock/Shareholder; Self; Novo Nordisk A/S. M.E. Jørgensen: Research Support; Self; Amgen, AstraZeneca, Boehringer Ingelheim Pharmaceuticals, Inc., Sanofi-Aventis. Stock/Shareholder; Self; Novo Nordisk A/S. M. Valerius: None. J.N. Hajari: None. H. Lund-Andersen: None. S. Byberg: None.