Transfer learning of chaotic systems.

Reservoir computing is an effective model for learning and predicting nonlinear and chaotic dynamical systems; however, there remains a challenge in achieving a more dependable evolution for such systems. Based on the foundation of Koopman operator theory, considering the effectiveness of the sparse identification of nonlinear dynamics algorithm to construct candidate nonlinear libraries in the application of nonlinear data, an alternative reservoir computing method is proposed, which creates the linear Hilbert space of the nonlinear system by including nonlinear terms in the optimization process of reservoir computing, allowing for the application of linear optimization. We introduce an implementation that incorporates a polynomial transformation of arbitrary order when fitting the readout matrix. Constructing polynomial libraries with reservoir-state vectors as elements enhances the nonlinear representation of reservoir states and more easily captures the complexity of nonlinear systems. The Lorenz-63 system, the Lorenz-96 system, and the Kuramoto-Sivashinsky equation are used to validate the effectiveness of constructing polynomial libraries for reservoir states in the field of state-evolution prediction of nonlinear and chaotic dynamical systems. This study not only promotes the theoretical study of reservoir computing, but also provides a theoretical and practical method for the prediction of nonlinear and chaotic dynamical system evolution.

A scenario frequently encountered in real-world complex systems is the temporary failure of a few components. For spatially extended chaotic systems whose functionality hinges on the collective dynamics of the interacting components, a viable approach to dealing with the failures is replacing the malfunctioning components with their backups, so that the collective dynamics of the systems can be precisely maintained for a duration long enough to resolve the problem. Here, taking the chaotic dynamics of the paradigmatic Kuramoto model as an example and considering the scenario of oscillator failures, we propose substituting the failed oscillators with digital twins trained by the data measured from the system evolution. Specifically, leveraging the technique of an adaptable reservoir computer (RC) in machine learning, we demonstrate that a single, small-size RC can substitute any oscillator in the Kuramoto model such that the time evolution of the synchronization order parameter of the repaired system is identical to that of the original system for a certain time period. The performance of adaptable RC is evaluated in various contexts, and it is found that the sustaining period is influenced by multiple factors, including the size of the training dataset, the overall coupling strength of the system, and the number of substituted oscillators. Additionally, though the synchronization order parameter diverges from the ground truth in the long-term running, the functional networks of the oscillators are still faithfully sustained by the machine substitutions.

Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (“climate”) can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth (“instability”). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.

The aim of this presentation is to show how various ideas coming from the nonlinear stability theory of functional differential systems, stochastic modeling, and machine learning, can be put together in order to create an approximating model that explains the working mechanisms behind a certain type of reservoir computers. Reservoir computing is a recently introduced brain-inspired machine learning paradigm capable of excellent performances in the processing of empirical data. We focus on time-delay based reservoir computers that have been physically implemented using optical and electronic systems and have shown unprecedented data processing rates. Reservoir computing is well-known for the ease of the associated training scheme but also for the problematic sensitivity of its performance to architecture parameters. The reservoir design problem is addressed, which remains the biggest challenge in the applicability of this information processing scheme. Our results use the information available regarding the optimal reservoir working regimes in order to construct a functional link between the reservoir parameters and its performance. This function is used to explore various properties of the device and to choose the optimal reservoir architecture, thus replacing the tedious and time consuming parameter scannings used so far in the literature.

In reservoir computing, dynamical systems are used to drive state-of-the-art machine learning with small training sets and minimal computing resources. Neuromorphic (brain-inspired) computing pose to further improve reservoir computing with energy-efficient spiking neural implementations. Here we propose an analog circuit design for reservoir computing using OZ spiking neurons, STDP (Spike-timing-dependent plasticity) synapses, and learning PES (prescribed error sensitivity) circuitry. We evaluated our design on a small scale using the Iris flower data set, demonstrating the potential application of neuromorphic analog hardware in reservoir computing.

In this paper, we propose and experimentally investigate a novel source separation chaotic optical communication architecture based on optical Reservoir Computing (RC). Benefiting from the RC's powerful high-dimensional mapping capabilities, the well-trained RC can replace the traditional chaotic receiver to achieve high-quality separation decryption without chaos synchronization, which significantly reduces costs and challenges. Specifically, the RC adopted in the proposed scheme has a deep cascade electro-optical feedback (DCEO) structure, which effectively enriches the reservoir state response. Numerical results show that the NMSE of the separated chaotic carrier is 0.0078 and the Q-factor of the separated OOK signal is 9.239, which achieves satisfactory decryption results. Simultaneously, the experimental and numerical conclusions are well verified with each other, which reflects the desirable reproducibility and reliability of the proposed scheme. Furthermore, the DCEO-RC system was compared with the other 3 RCs, and it had the lowest NMSE and the highest Q-factor in both experiments and numerical simulations, which represents the best separation performance. Subsequently, the impact of mask coefficient, SNR, chaotic system configurations, and RC parameters on separation performance was analyzed in detail. Finally, we analyze communication transmission quality. The results show that 20GBaud OOK/QPSK signals can achieve normal communication over 200 km. In summary, the proposed scheme is expected to provide strong support for high-speed chaotic secure communication.

From logical reasoning to mental simulation, biological and artificial neural systems possess an incredible capacity for computation. Such neural computers offer a fundamentally novel computing paradigm by representing data continuously and processing information in a natively parallel and distributed manner. To harness this computation, prior work has developed extensive training techniques to understand existing neural networks. However, the lack of a concrete and low-level machine code for neural networks precludes us from taking full advantage of a neural computing framework. Here we provide such a machine code along with a programming framework by using a recurrent neural network—a reservoir computer—to decompile, code and compile analogue computations. By decompiling the reservoir’s internal representation and dynamics into an analytic basis of its inputs, we define a low-level neural machine code that we use to program the reservoir to solve complex equations and store chaotic dynamical systems as random-access memory. We further provide a fully distributed neural implementation of software virtualization and logical circuits, and even program a playable game of pong inside of a reservoir computer. Importantly, all of these functions are programmed without requiring any example data or sampling of state space. Finally, we demonstrate that we can accurately decompile the analytic, internal representations of a full-rank reservoir computer that has been conventionally trained using data. Taken together, we define an implementation of neural computation that can both decompile computations from existing neural connectivity and compile distributed programs as new connections.

Forecasting a system's behavior is an essential task encountering the complex systems theory. Machine learning offers supervised algorithms, e.g., recurrent neural networks and reservoir computers that predict the behavior of model systems whose states consist of multidimensional time series. In real life, we often have limited information about the behavior of complex systems. The brightest example is the brain neural network described by the electroencephalogram. Forecasting the behavior of these systems is a more challenging task but provides a potential for real-life application. Here, we trained reservoir computer to predict the macroscopic signal produced by the network of phase oscillators. The Lyapunov analysis revealed the chaotic nature of the signal and reservoir computer failed to forecast it. Augmenting the feature space using Takkens' theorem improved the quality of forecasting. RC achieved the best prediction score when the number of signals coincided with the embedding dimension estimated via the nearest false neighbors method. We found that short-time prediction required a large number of features, while long-time prediction utilizes a limited number of features. These results refer to the bias-variance trade-off, an important concept in machine learning.

Real-world applications such as first-person video activity recognition require intelligent edge devices. However, size, weight, and power constraints of the embedded platforms cannot support resource intensive state-of-the-art algorithms. Machine learning lite algorithms, such as reservoir computing, with shallow 3-layer networks are computationally frugal as only the output layer is trained. By reducing network depth and plasticity, reservoir computing minimizes computational power and complexity, making the algorithms optimal for edge devices. However, as a trade-off for their frugal nature, reservoir computing sacrifices computational power compared to state-of-the-art methods. A good compromise between reservoir computing and fully supervised networks are the proposed deep-LSM networks. The deep-LSM is a deep spiking neural network which captures dynamic information over multiple time-scales with a combination of randomly connected layers and unsupervised layers. The deep-LSM processes the captured dynamic information through an attention modulated readout layer to perform classification. We demonstrate that the deep-LSM achieves an average of 84.78% accuracy on the DogCentric video activity recognition task, beating state-of-the-art. The deep-LSM also shows up to 91.13% memory savings and up to 91.55% reduction in synaptic operations when compared to similar recurrent neural network models. Based on these results we claim that the deep-LSM is capable of overcoming limitations of traditional reservoir computing, while maintaining the low computational cost associated with reservoir computing.

Reservoir computing has attracted much attention for its easy training process as well as its ability to deal with temporal data. A reservoir computing system consists of a reservoir part represented as a sparsely connected recurrent neural network and a readout part represented as a simple regression model. In machine learning tasks, the reservoir part is fixed and only the readout part is trained. Although reservoir computing has been mainly applied to time series prediction and recognition, it can be applied to image recognition as well by considering an image data as a sequence of pixel values. However, to achieve a high performance in image recognition with raw image data, a large-scale reservoir including a large number of neurons is required. This is a bottleneck in terms of computer memory and computational cost. To overcome this bottleneck, we propose a new method which combines reservoir computing with untrained convolutional neural networks. We use an untrained convolutional neural network to transform raw image data into a set of smaller feature maps in a preprocessing step of the reservoir computing. We demonstrate that our method achieves a high classification accuracy in an image recognition task with a much smaller number of trainable parameters compared with a previous study.

Reservoir computing systems, a class of recurrent neural networks, have recently been exploited for modelfree, data-based prediction of the state evolution of a variety of chaotic dynamical systems. The prediction horizon demonstrated has been about half dozen Lyapunov time. Is it possible to significantly extend the prediction time beyond what has been achieved so far? We articulate a scheme incorporating time-dependent but sparse data inputs into reservoir computing and demonstrate that such rare "updates" of the actual state practically enable an arbitrarily long prediction horizon for a variety of chaotic systems. A physical understanding based on the theory of temporal synchronization is developed.

During the last decade reservoir computing (RC) has proven to be a powerful machine learning concept using recurrent networks to process sequential information streams [1]. Lately, the concept of RC was drastically simplified to reduce hardware requirements, thus making machine learning concepts more obtainable to apply in actual physical systems [2]. RC has been implemented in photonic configurations and has efficientiy processed analog optical signals following simple schemes and minimal training overhead. Especially the realization using a semiconductor laser with time-delayed optical feedback has shown the potential of all-optical machine learning, utilizing signals with transient states at the GHz regime [3].

Reservoir computing (RC) provides a route to use physical systems for computation and machine learning. Owing to their inherent nonlinearity, nanomechanical systems constitute an interesting technology to serve as reservoir. While RC platforms are built using microelectromechanical systems, the energy efficiency, response time, and footprint of these systems can be significantly improved by using nanoscale devices. Herein, the use of nanoelectromechanical systems (NEMS) is investigated, which can be used in RC, utilizing inherent nonlinearities and the fading memory effect from the transient response of NEMS. The smaller size and higher operating frequencies of NEMS enable faster processing rates compared to micromechanical systems, while their compact footprint, low power consumption, and ability to operate under ambient conditions simplify integration into practical applications. In modified national institute of standards and technology (MNIST) handwritten digit–recognition test, this system achieves 90% accuracy with a 3.3 μs processing time per pixel. Also the effect of driving frequency and amplitude on NEMS classification accuracy is investigated using experiments and simulations and it is shown that no significant dependency in any of the parameters is observed. Herein, an estimate for energy consumption of core NEMS RC system on MNIST data is provided. These results highlight the potential for various applications that require efficient and fast information processing in resource‐constrained environments.

Photonic reservoir computing has been intensively investigated to solve machine learning tasks effectively. A simple learning procedure of output weights is used for reservoir computing. However, the lack of training of input-node and inter-node connection weights limits the performance of reservoir computing. The use of multiple reservoirs can be a solution to overcome this limitation of reservoir computing. In this study, we investigate parallel and deep configurations of delay-based all-optical reservoir computing using semiconductor lasers with optical feedback by combining multiple reservoirs to improve the performance of reservoir computing. Furthermore, we propose a hybrid configuration to maximize the benefits of parallel and deep reservoirs. We perform the chaotic time-series prediction task, nonlinear channel equalization task, and memory capacity measurement. Then, we compare the performance of single, parallel, deep, and hybrid reservoir configurations. We find that deep reservoirs are suitable for a chaotic time-series prediction task, whereas parallel reservoirs are suitable for a nonlinear channel equalization task. Hybrid reservoirs outperform other configurations for all three tasks. We further optimize the number of reservoirs for each reservoir configuration. Multiple reservoirs show great potential for the improvement of reservoir computing, which in turn can be applied for high-performance edge computing.

Reservoir Computing (RC) is a machine learning framework designed to solve time-dependent tasks, but it remains less explored for time-independent tasks. In this work, we adapt RC for image classification by flattening input images into one-dimensional vectors and feeding the entire vector in a single step into an RC. We evaluate our approach on the MNIST and COIL-100 datasets, demonstrating that RC achieves competitive accuracy while requiring significantly shorter training times compared to traditional neural networks. Our experiments show that the difference between the worst and best performance across reservoir network densities is 1–3%. Increasing the reservoir size generally improves accuracy; for instance, increasing the reservoir from 100 to 1000 typically yields a 10–15% improvement, whereas increasing it further from 1000 to 2000 results in only a 3–5% gain. This work highlights RC as an alternative for image classification tasks on standard and moderately sized datasets.