Efficient Small-Scale Network for Room Layout Estimation

Neural networks are being increasingly used in many applications. While large deep neural networks are quickly advancing, for computing devices that lack powerful processors, it is desirable to use smaller neural networks. In this paper, we focus on smaller neural networks than common deep networks and examine how their size can be made even smaller without sacrificing the performance. We show that for some data types it is possible to make the networks smaller using sparse representation of the input. We have studied time delay neural networks (TDNN) for time series prediction using three different datasets. It is found that sparsifying input to the TDNN, using Discrete Cosine Transform (DCT), or Principal Component Analysis (PCA), can result in performance improvement. The improved performance can be traded off for network size reduction. Therefore, we can make the network smaller and maintain the same performance. It is found that for data that has more randomness and sudden changes in value (higher frequencies present), sparse representation methods using discrete cosine transform or through principal component analysis allow reducing network size by up to 40%.

Nowadays, research, modeling, simulation and realization of brain-like systems to reproduce brain behaviors have become urgent requirements. In this paper, neural bursting and synchronization are imitated by modeling two neural network models based on the Hopfield neural network (HNN). The first neural network model consists of four neurons, which correspond to realizing neural bursting firings. Theoretical analysis and numerical simulation show that the simple neural network can generate abundant bursting dynamics including multiple periodic bursting firings with different spikes per burst, multiple coexisting bursting firings, as well as multiple chaotic bursting firings with different amplitudes. The second neural network model simulates neural synchronization using a coupling neural network composed of two above small neural networks. The synchronization dynamics of the coupling neural network is theoretically proved based on the Lyapunov stability theory. Extensive simulation results show that the coupling neural network can produce different types of synchronous behaviors dependent on synaptic coupling strength, such as anti-phase bursting synchronization, anti-phase spiking synchronization, and complete bursting synchronization. Finally, two neural network circuits are designed and implemented to show the effectiveness and potential of the constructed neural networks.

Neural networks and image pyramids are massively parallel processing structures. In this paper we exploit the similarities as well as the differences between these structures. The general goal is to exchange knowledge between these two fields. After introducing the basic concepts of neural networks and image pyramids we give a translation table of the vocabulary used in image pyramids and those used in neural networks. Image pyramids which store and process numerical information (e.g. grey values of pixels) are very similar to neural networks. Therefore we concentrate on ”symbolic pyramids”. The main idea is to replace a cell of the pyramid by a small neural network, in order to represent and process symbolic information. We will consider local as well as distributed representations for symbolic information. In particular we present a neural implementation of the 2 x 2/2 curve pyramid. We derive some general rules for implementing symbolic pyramids by neural networks. Finally we briefly discuss the role of learning in image pyramids.

This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices. The presented distillation methods are respectively based on Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The two methods basically rely on the successive dimension reduction of a given dense neural network (teacher) hidden features, and the learning of a smaller neural network (student) which solves the initial learning problem along with a mapping problem to the reduced successive features spaces. The presented methods are compared to baselines –learning the student networks from scratch–, and we show that the additional mapping problem significantly improves the performance (accuracy, memory and computing resources) of the student networks.

Memristive devices1 have become a promising candidate for unconventional computing2. In this talk, I will present some of our recent work on unconventional computing experimentally implemented by using memristive devices or crossbar arrays. Using traditional non-volatile memristors with 64 stable analog resistance levels, we have built a dot-product engine based on a 128 x 64 1T1R crossbar array3. Accurate image compression and filtering have been demonstrated with such analog computing accelerator3. In addition, we have demonstrated efficient and self-adaptive in-situ learning in a two-layer neural networks using such memristive arrays4, which is expected to significantly improve the speed and energy efficiency of deep neural networks. Using our newly developed diffusive memristors5 with diffusion dynamics that is critical for neuromorphic functions, we have developed artificial synapses6 and neurons7 to more faithfully emulate their bio-counterparts and more efficiently perform spiking neural network functions. We have further integrated these artificial synapses and neurons into a small neural network, with which pattern classification and unsupervised learning have been demonstrated7. Moreover, the diffusive memristors can be used as true random number generators8 for cybersecurity applications and artificial nociceptors for robotics applications9. 1. Yang, J. J. et al. Memristive switching mechanism for metal/oxide/metal nanodevices. Nature Nanotechnology 3, 429 (2008). 2. Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nature Nanotechnology 8, 13 (2013). 3. Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nature Electronics 1, 52 (2018). 4. Li, C. et al. Efficient and self-adaptive in-situ learning in multilayer memristive neural networks. Nature communications, under revision (2018). 5. Midya, R. et al. Anatomy of Ag/Hafnia-Based Selectors with 1E10 Nonlinearity. Advanced Materials 29, 1604457 (2017). 6. Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nature Materials 16, 101-108 (2017). 7. Wang, Z. et al. Fully memristive neural networks for pattern classification with unsupervised learning. Nature Electronics 1, 137 (2018). 8. Jiang, H. et al. A Novel True Random Number Generator Based on a Stochastic Diffusive Memristor. Nature communications 8, 882 (2017). 9. Yoon, J. H. et al. An artificial nociceptor based on a diffusive memristor. Nature Communications 9, 417 (2018).

This paper presents a method for constructing a size compressed neural network with better or similar accuracy than a given dense neural network, therefore the compressed network requires less memory and computational resources. The presented method relies basically on learning successive mappings between the given dense neural network (teacher) hidden features and the size-compressed neural network (student) hidden features, where the latter is learned also to solve the initial task of the teacher network. The presented method is particularly compared to baselines where we specifically show that the additional learned mappings significantly improve the performance (accuracy and computation) of the student network.

The similarities and differences between the conventional Volterra series techniques and the neural network approach are discussed. The analysis is done from the point of view of representation capabilities for nonlinear systems, and it is shown that a small neural network can represent high-order nonlinear systems, whereas a very large number of terms are required for an equivalent Volterra series representation. This is shown by means of a series expansion of a neural network. Issues common to the two nonlinear modeling approaches are analyzed

Small neural networks (NNs) used for error correction were shown to improve on classic channel codes and to address channel model changes. We extend the code dimension of any such structure by using the same NN under one-hot encoding multiple times, then serially-concatenated with an outer classic code. We design NNs with the same network parameters, where each Reed-Solomon codeword symbol is an input to a different NN. Significant improvements in block error probabilities for an additive Gaussian noise channel as compared to the small neural code are illustrated, as well as robustness to channel model changes.

The quantum theory of atoms in molecules gives access to well-defined local atomic energies. Due to their locality, these energies are potentially interesting in fitting atomistic machine learning models as they inform about physically relevant properties. However, computationally, quantum-mechanically accurate local energies are notoriously difficult to obtain for large systems. Here, we show that by employing semiempirical correlations between different components of the total energy, we can obtain well-defined local energies at a moderate cost. We employ this methodology to investigate energetics in noble liquids or argon, krypton, and their mixture. Instead of using these local energies to fit atomistic models, we show how well these local energies are reproduced by machine-learned models trained on the total energies. The results of our investigation suggest that smaller neural networks, trained only on the total energy of an atomistic system, are more likely to reproduce the underlying local energy partitioning faithfully than larger networks. Furthermore, we demonstrate that networks more capable of this energy decomposition are, in turn, capable of transferring to previously unseen systems. Our results are a step toward understanding how much physics can be learned by neural networks and where this can be applied, particularly how a better understanding of physics aids in the transferability of these neural networks.

In recent years, computational biologists have shown through simulation that small neural networks with fixed connectivity are capable of producing multiple output rhythms in response to transient inputs. It is believed that such networks may play a key role in certain biological behaviors such as dynamic gait control. In this paper, we present a novel method for designing continuous-time recurrent neural networks (CTRNNs) that contain multiple embedded limit cycles, and we show that it is possible to switch the networks between these embedded limit cycles with simple transient inputs. We also describe the design and testing of a fully integrated four-neuron CTRNN chip that is used to implement the neural network pattern generators. We provide two example multipattern generators and show that the measured waveforms from the chip agree well with numerical simulations.

This paper shows a small structured neural network speed estimator for induction motors using a direct torque control (DTC) method. The backpropagation technique was used to train the neural network. The small structure leads to short off line training and processing times. The overall algorithm based on a conventional DTC closed loop is simple and the results are validated through simulation and implementation.

A space-time-sharing optical neural network for implementing a large-scale operation is presented. If the interconnection weight matrix is partitioned into an array of submatrices, a large space-bandwidth pattern can be processed with a smaller neural network. We show that the processing time increases as a square function of the space-bandwidth product of the pattern. To illustrate the space-time-sharing operation, experimental and simulated results are provided.

Score distillation for anomaly detection

Symbolic-neural processing is applied to the 'black box' problem, i.e., the synthesis of a system with a transfer function that adequately matches desired input-output relationships, by using many small neural networks, each capable of experimental training, coupled together through conventional logic paradigms. Explicit knowledge can then be encoded in the structure of the networks as well as in rules and algorithms. Learned knowledge is then input by 'showing' training data to the individual neural networks. The symbolic neural approach is a way of creating models which: (1) are easily completed quickly and have quality that can be improved over time; (2) are adaptive to environments for which they were not specifically programmed; and (3) accept human knowledge both in the form of explicit logic and in the form of experiential training.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Although several new learning rules have been proposed, the space and time complexities of these two bottlenecks remain. The bottlenecks occur because all the existing learning techniques are based on a single large neural structure. In this paper, the authors approach the solution to this problem by analyzing the input vector space and, then, partition this space into subspaces. Each subspace will be learned by a small neural network with a simple learning rule. The authors also speed up the learning process by reducing the number of training vectors down to O(mn) instead of 2/sup n/, where m is the number of vectors having output targets equal to ones and n is the number of input bits. The reduction is based on the concept of guard ring vectors. The experimental results show that the learning can be speeded up by 2-30 times over the non-partition process. However the number of neurons used in the authors' approach is uncontrollable. In some cases, the number is reduced but in some cases it is increased.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>