Forecasting macroscopic dynamics in adaptive Kuramoto network using reservoir computing.

During the last decade reservoir computing (RC) has proven to be a powerful machine learning concept using recurrent networks to process sequential information streams [1]. Lately, the concept of RC was drastically simplified to reduce hardware requirements, thus making machine learning concepts more obtainable to apply in actual physical systems [2]. RC has been implemented in photonic configurations and has efficientiy processed analog optical signals following simple schemes and minimal training overhead. Especially the realization using a semiconductor laser with time-delayed optical feedback has shown the potential of all-optical machine learning, utilizing signals with transient states at the GHz regime [3].

From logical reasoning to mental simulation, biological and artificial neural systems possess an incredible capacity for computation. Such neural computers offer a fundamentally novel computing paradigm by representing data continuously and processing information in a natively parallel and distributed manner. To harness this computation, prior work has developed extensive training techniques to understand existing neural networks. However, the lack of a concrete and low-level machine code for neural networks precludes us from taking full advantage of a neural computing framework. Here we provide such a machine code along with a programming framework by using a recurrent neural network—a reservoir computer—to decompile, code and compile analogue computations. By decompiling the reservoir’s internal representation and dynamics into an analytic basis of its inputs, we define a low-level neural machine code that we use to program the reservoir to solve complex equations and store chaotic dynamical systems as random-access memory. We further provide a fully distributed neural implementation of software virtualization and logical circuits, and even program a playable game of pong inside of a reservoir computer. Importantly, all of these functions are programmed without requiring any example data or sampling of state space. Finally, we demonstrate that we can accurately decompile the analytic, internal representations of a full-rank reservoir computer that has been conventionally trained using data. Taken together, we define an implementation of neural computation that can both decompile computations from existing neural connectivity and compile distributed programs as new connections.

Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (“climate”) can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth (“instability”). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.

In this paper, we investigate online learning-based MIMO-OFDM symbol detection strategies focusing on a special recurrent neural network (RNN) - reservoir computing (RC). We first introduce the Time-Frequency RC to take advantage of the structural information inherent in OFDM signals. Using the time domain RC and the time-frequency RC as building blocks, we provide two extensions of the shallow RC to RCNet: 1) Stacking multiple time domain RCs; 2) Stacking multiple time-frequency RCs into a deep structure. The combination of RNN dynamics, the time-frequency structure of MIMO-OFDM signals, and the deep network enables RCNet to handle the interference and nonlinear distortion of MIMO-OFDM signals to outperform existing methods. Unlike most existing NN-based detection strategies, RCNet is also shown to provide a good generalization performance even with a limited online training set (i.e, similar amount of reference signals/training as standard model-based approaches). Numerical experiments demonstrate that the introduced RCNet can offer a faster learning convergence and as much as 20% gain in bit error rate over a shallow RC structure by compensating for the nonlinear distortion of the MIMO-OFDM signal, such as due to power amplifier compression in the transmitter or due to finite quantization resolution in the receiver.

There is a wave of interest in using physics-informed neural networks for solving differential equations. Most of the existing methods are based on feed-forward networks, while recurrent neural networks solvers have not been extensively explored. We introduce a reservoir computing (RC) architecture, an echo-state recurrent neural network capable of discovering approximate solutions that satisfy ordinary differential equations (ODEs). We suggest an approach to calculate time derivatives of recurrent neural network outputs without using back-propagation. The internal weights of an RC are fixed, while only a linear output layer is trained, yielding efficient training. However, RC performance strongly depends on finding the optimal hyper-parameters, which is a computationally expensive process. We use Bayesian optimization to discover optimal sets in a high-dimensional hyper-parameter space efficiently and numerically show that one set is robust and can be transferred to solve an ODE for different initial conditions and time ranges. A closed-form formula for the optimal output weights is derived to solve first-order linear equations in a one-shot backpropagation-free learning process. We extend the RC approach by solving nonlinear systems of ODEs using a hybrid optimization method consisting of gradient descent and Bayesian optimization. Evaluation of linear and nonlinear systems of equations demonstrates the efficiency of the RC ODE solver.

With the growing use of artificial neural networks (ANNs) in temporal data processing tasks, the cost of training for complex ANNs is an escalating concern. Physical reservoir computing (RC), a variation of recurrent neural networks (RNNs), obviate the need for most data intensive matrix vector multiplication in the recurrent layer by evolving the RC’s internal states with the inherent nonlinear dynamics and short-term memory. In this study, we show that magnetic skyrmion confined in a fixed geometry forming the soft layer of a magnetic tunnel junction (MTJ) can work as a RC and perform autonomous long-term prediction of temporal data. Our proposed skyrmion reservoir allows for manipulation of spin dynamics with ultra energy efficient voltage controlled magnetic anisotropy modulation (VCMA) method. Furthermore, the boundary effect on the skyrmion from the geometric edges provides necessary consistency property of the reservoir. We employ our proposed reservoir for the modeling and prediction of the chaotic time series such as Mackey-Glass and dynamic time-series data, such as household building energy loads. For autonomous run, the predicted output is fed to the input of the reservoir. By comparing our spintronic physical RC approach with energy load forecasting algorithms, such as LSTMs and RNNs, we conclude that the proposed framework presents good performance in achieving high predictions accuracy, while also requiring low memory and energy both of which are at a premium in hardware resource and power constrained edge applications. Further, the proposed approach is shown to require very small training datasets and at the same time being at least 16× energy efficient compared to the sequence to sequence LSTM for accurate household load predictions. Higher endurance, fast processing and well-established technology (i.e., MTJ) to integrate with CMOS technology makes such spintronic reservoir attractive over other emergent memory device-based reservoirs.

This paper provides a comparison between two methods for time series forecasting. The first method is based on traditional recurrent neural networks (RNNs) while the second method is based in Reservoir Computing (RC). Reservoir Computing is a new paradigm that offers an intuitive methodology for using the temporal processing power of RNNs without the inconvenience of training them. So we decided to compare the advantages / disadvantages of using Reservoir Computing and RNNs in the problem of time series forecasting. The first method uses a Nonlinear Autoregressive Network with exogenous inputs (NARX). Optimization was carried out on the NARX architecture through an optimization procedure focused on the best mean squared error (MSE) metrics in the training set. The second method, called RCDESIGN, combines an evolutionary algorithm with Reservoir Computing and simultaneously looks for the best values of parameters, topology and weight matrices without rescaling the reservoir by the spectral radius. Nevertheless RCDESIGN has yielded fast tracking and excellent performance in some benchmark problems including the Narma and Mackey-Glass time-series.

For many years, Evolutionary Algorithms (EAs) have been applied to improve Neural Networks (NNs) architectures. They have been used for solving different problems, such as training the networks (adjusting the weights), designing network topology, optimizing global parameters, and selecting features. Here, we provide a systematic brief survey about applications of the EAs on the specific domain of the recurrent NNs named Reservoir Computing (RC). At the beginning of the 2000s, the RC paradigm appeared as a good option for employing recurrent NNs without dealing with the inconveniences of the training algorithms. RC models use a nonlinear dynamic system, with fixed recurrent neural network named the reservoir, and learning process is restricted to adjusting a linear parametric function. However, an RC model has several hyper-parameters, therefore EAs are helpful tools to figure out optimal RC architectures. We provide an overview of the results on the area, discuss novel advances, and we present our vision regarding the new trends and still open questions.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Real-world applications such as first-person video activity recognition require intelligent edge devices. However, size, weight, and power constraints of the embedded platforms cannot support resource intensive state-of-the-art algorithms. Machine learning lite algorithms, such as reservoir computing, with shallow 3-layer networks are computationally frugal as only the output layer is trained. By reducing network depth and plasticity, reservoir computing minimizes computational power and complexity, making the algorithms optimal for edge devices. However, as a trade-off for their frugal nature, reservoir computing sacrifices computational power compared to state-of-the-art methods. A good compromise between reservoir computing and fully supervised networks are the proposed deep-LSM networks. The deep-LSM is a deep spiking neural network which captures dynamic information over multiple time-scales with a combination of randomly connected layers and unsupervised layers. The deep-LSM processes the captured dynamic information through an attention modulated readout layer to perform classification. We demonstrate that the deep-LSM achieves an average of 84.78% accuracy on the DogCentric video activity recognition task, beating state-of-the-art. The deep-LSM also shows up to 91.13% memory savings and up to 91.55% reduction in synaptic operations when compared to similar recurrent neural network models. Based on these results we claim that the deep-LSM is capable of overcoming limitations of traditional reservoir computing, while maintaining the low computational cost associated with reservoir computing.

Reservoir computing has attracted much attention for its easy training process as well as its ability to deal with temporal data. A reservoir computing system consists of a reservoir part represented as a sparsely connected recurrent neural network and a readout part represented as a simple regression model. In machine learning tasks, the reservoir part is fixed and only the readout part is trained. Although reservoir computing has been mainly applied to time series prediction and recognition, it can be applied to image recognition as well by considering an image data as a sequence of pixel values. However, to achieve a high performance in image recognition with raw image data, a large-scale reservoir including a large number of neurons is required. This is a bottleneck in terms of computer memory and computational cost. To overcome this bottleneck, we propose a new method which combines reservoir computing with untrained convolutional neural networks. We use an untrained convolutional neural network to transform raw image data into a set of smaller feature maps in a preprocessing step of the reservoir computing. We demonstrate that our method achieves a high classification accuracy in an image recognition task with a much smaller number of trainable parameters compared with a previous study.

In this paper, we derive an upper bound on the generalization error of reservoir computing (RC), a special category of recurrent neural networks (RNNs). The particular RC implementation considered in this paper is the echo state network (ESN), and the generalization bound is derived via the empirical Rademacher complexity (ERC) approach. While recent work in deriving risk bounds for RC frameworks makes use of a non-standard empirical Rademacher Complexity (ERC) measure and a direct application of its definition, our work uses the standard ERC measure and tools which allow easier extension to deep RC structures and a fair comparison with other RNN structures including the vanilla RNNs. Finally, we train an ESN for the task of symbol detection, a key component in the receive processing of 5G systems and show with an example how the derived generalization bound can guide the underlying system design. To be specific, we utilize the derived generalization error together with the characterized training loss to analytically identify the optimum number of neurons in the reservoir for the symbol detection task. Simulation results using 3GPP specified channels corroborate our theoretical findings, illustrating the significance and practical relevance of our work.

Reservoir computers (RCs) are biology-inspired computational frameworks for signal processing that are typically implemented using recurrent neural networks. Recent work has shown that Boolean networks (BN) can also be used as reservoirs. We analyze the performance of BN RCs, measuring their flexibility and identifying the factors that determine the effective approximation of Boolean functions applied in a sliding-window fashion over a binary signal, both non-recursively and recursively. We train and test BN RCs of different sizes, signal connectivity, and in-degree to approximate three-bit, five-bit, and three-bit recursive binary functions, respectively. We analyze how BN RC parameters and function average sensitivity, which is a measure of function smoothness, affect approximation accuracy as well as the spread of accuracies for a single reservoir. We found that approximation accuracy and reservoir flexibility are highly dependent on RC parameters. Overall, our results indicate that not all reservoirs are equally flexible, and RC instantiation and training can be more efficient if this is taken into account. The optimum range of RC parameters opens up an angle of exploration for understanding how biological systems might be tuned to balance system restraints with processing capacity.

Reservoir computing (RC), a variant of recurrent neural networks (RNNs), is well-known for its reduced energy consumption through exclusive focus on training the output weight and its superior performance in handling spatiotemporal information. Implementing these networks in hardware requires devices with superior fading memory behavior. Unlike filament-based two-terminal devices, those relying on ferroelectric switching demonstrate improved voltage reliability, while three-terminal transistors provide additional active control. HfO2-based ferroelectric materials such as Hf0.5Zr0.5O2 (HZO), have garnered attention for their scalability and seamless integration with CMOS technology. This study implements a RC hardware based on MoS2-HZO integrated device structure with enhanced spontaneous polarization field. By adjusting the oxygen vacancy concentration, the devices exhibit consistent responses to both identical and nonidentical voltages, making them suitable for diverse RC applications. The high accuracy of MNIST handwritten digits recognition highlights the rich reservoir states of the traditional RC architecture. Additionally, the impact of masks on RC implementation is assessed, showcasing the device's capability for spatiotemporal signal analysis. This development paves the way for implementing energy-efficient and high-performance computing solutions.

Reservoir computing is a machine learning approach that enables us to use recurrent neural networks without involving the complexity of training algorithms and make hardware implementation possible. We present a novel photonic architecture of a reservoir computer that employs a nonlinear node and a resonator to implement a virtual recurrent neural network. This resonator behaves as an echo generator component that substitutes the delay line in delay-based reservoir computers available in the literature. The virtual neural network formed in our implementation is fundamentally different from the delay-based reservoir computers. Different virtual architectures based on the FSR and the Finesse of the resonator are investigated to provide higher performance depending on the task. We test the performance of our proposed reservoir by 3 benchmark tasks, signal classification, nonlinear channel equalization, and memory capacity. Our system enhances the memory capacity of the reservoir compared to the delay-based systems and shows reliable results in signal classification and nonlinear channel equalization.