Oscillatory neural networks based on the Kuramoto model for cluster analysis

The standard Kuramoto model has been instrumental in explaining synchronization and desynchronization, two emergent phenomena often observed in biological, neuronal, and physical systems. While the Kuramoto model has turned out effective with one-dimensional oscillators, real-world systems often involve high-dimensional interacting units, such as biological swarms, necessitating a model of multidimensional oscillators. However, existing high-dimensional generalizations of the Kuramoto model commonly rely on a scalar-valued coupling strength, which limits their ability to capture the full complexity of high-dimensional interactions. This work introduces a matrix, A, to couple the interconnected components of the oscillators in a d-dimensional space, leading to a matrix-coupled multidimensional Kuramoto model that approximates a prototypical swarm dynamics by its first-order Fourier harmonics. Moreover, the matrix A introduces an inter-dimensional higher-order interaction that partly accounts for the emergence of 2 d system modes in a d-dimensional population, where each dimension can either be synchronized or desynchronized, represented by a set of almost binary order parameters. The binary system modes capture characteristic swarm behaviors such as fish milling or polarized schooling. Additionally, our findings provides a theoretical analogy to cerebral activity, where the resting state and the activated state coexist unihemispherically. It also suggests a new possibility for information storage in oscillatory neural networks.

The growing number of edge devices in everyday life generates a considerable amount of data that current AI algorithms, like artificial neural networks, cannot handle inside edge devices with limited bandwidth, memory, and energy available. Neuromorphic computing, with low-power oscillatory neural networks (ONNs), is an alternative and attractive solution to solve complex problems at the edge. However, ONN is currently limited with its fully-connected recurrent architecture to solve auto-associative memory problems. In this work, we use an alternative two-layer bidirectional ONN architecture. We introduce a two-layer feedforward ONN architecture to perform image edge detection, using the ONN to replace convolutional filters to scan the image. Using an HNN Matlab emulator and digital ONN design simulations, we report efficient image edge detection from both architectures using various size filters (3 × 3, 5 × 5, and 7 × 7) on black and white images. In contrast, the feedforward architectures can also perform image edge detection on gray scale images. With the digital ONN design, we also assess latency performances and obtain that the bidirectional architecture with a 3 × 3 filter size can perform image edge detection in real-time (camera flow from 25 to 30 images per second) on images with up to 128 × 128 pixels while the feedforward architecture with same 3 × 3 filter size can deal with 170 × 170 pixels, due to its faster computation.

Alternative paradigms to the von Neumann computing scheme are currently arousing huge interest. Oscillatory neural networks (ONNs) using emerging phase-change materials like VO2 constitute an energy-efficient, massively parallel, brain-inspired, in-memory computing approach. The encoding of information in the phase pattern of frequency-locked, weakly coupled oscillators makes it possible to exploit their rich non-linear dynamics and their synchronization phenomena for computing. A single fully connected ONN layer can implement an auto-associative memory comparable to that of a Hopfield network, hence Hebbian learning rule is the most widely adopted method for configuring ONNs for such applications, despite its well-known limitations. An extensive amount of literature is available about learning in Hopfield networks, with information regarding many different learning algorithms that perform better than the Hebbian rule. However, not all of these algorithms are useful for ONN training due to the constraints imposed by their physical implementation. This paper evaluates different learning methods with respect to their suitability for ONNs. It proposes a new approach, which is compared against previous works. The proposed method has been shown to produce competitive results in terms of pattern recognition accuracy with reduced precision in synaptic weights, and to be suitable for online learning.

Oscillatory neural networks are a relatively recent approach for the problem of image segmentation. Inside of this context, the oscillator neuron of Terman-Wang is presented, which is used as base element of an oscillatory network called LEGION (Locally Excitatory Globally Inhibitory Oscillator Network). The continuous version of the LEGION network, based on a set of differential equations, presents high computational complexity and has limited capacity of segmentation, which restricts its practical application, being adequate for implementation in parallel hardware topologies. To reduce the computational complexity in serial computers, an algorithm proposed by Terman and Wang is presented, which implies significant gain of speed in comparison to the continuous version and, in contrast, capacity to discriminate a unlimited number of segments. An interactive version of this algorithm was proposed and the results obtained in segmentation of transmission electron microscopy (TEM) images were evaluated, with the objective of measuring helium bubbles in silicon samples. As final result we found that the LEGION network presents itself as a singular alternative to solve problems of image segmentation, which provides simultaneously both spatial and temporal discrimination of segments.

It is a well-known fact that oscillatory networks can operate as Hopfield-like neural networks, the only difference being that their attractors are limit cycles: one for each memorized pattern. The neuron activities are synchronized on the limit cycles, and neurons oscillate with fixed phase differences (time delays). We prove that this property is a natural attribute of general weakly connected neural networks, and it is relatively independent of the equations that describe the network activity. In particular, we prove an analogue of the Cohen-Grossberg convergence theorem for oscillatory neural networks.

Groups of neurons firing synchronously are hypothesized to underlie many cognitive functions such as attention, associative learning, memory, and sensory selection. Recent theories suggest that transient periods of synchronization and desynchronization provide a mechanism for dynamically integrating and forming coalitions of functionally related neural areas, and that at these times conditions are optimal for information transfer. Oscillating neural populations display a great amount of spectral complexity, with several rhythms temporally coexisting in different structures and interacting with each other. This paper explores inter-band frequency modulation between neural oscillators using models of quadratic integrate-and-fire neurons and Hodgkin-Huxley neurons. We vary the structural connectivity in a network of neural oscillators, assess the spectral complexity, and correlate the inter-band frequency modulation. We contrast this correlation against measures of metastable coalition entropy and synchrony. Our results show that oscillations in different neural populations modulate each other so as to change frequency, and that the interaction of these fluctuating frequencies in the network as a whole is able to drive different neural populations towards episodes of synchrony. Further to this, we locate an area in the connectivity space in which the system directs itself in this way so as to explore a large repertoire of synchronous coalitions. We suggest that such dynamics facilitate versatile exploration, integration, and communication between functionally related neural areas, and thereby supports sophisticated cognitive processing in the brain.

Contents: Preface. Part I: Oscillations in Single Neurons and Local Networks. G. Gross, J.M. Kowalski, B.K. Rhoades, Spontaneous and Evoked Oscillations in Cultured Mammalian Neuronal Networks. D.C. Tam, Detection of Oscillations and Synchronous Firing in Neurons. D.A. Baxter, H.A. Lechner, C.C. Canavier, R.J. Butera, Jr., A.A. DeFraneschi, J.W. Clark, Jr., J.H. Byrne, Coexisting Stable Oscillatory States in Single Cells and Multicellular Neuronal Oscillators. M. Stemmler, M. Usher, C. Koch, Oscillatory Local Field Potentials. G.J. Mpitsos, J.P. Edstrom, Computations Neurons Perform in Networks: Inside Versus Outside and Lessons Learned From a Sixteenth-Century Shoemaker. Part II: Oscillations in Cortical and Cortical/Subcortical Systems. I. Soltesz, The Interplay of Intrinsic and Synaptic Membrane Currents in Delta, Theta and 40-Hz Oscillations. E. Thomas, Dynamics of Low-Frequency Oscillations in a Model Thalamocortical Network. M.E. Jackson, L.J. Cauller, Towards the Function of Reciprocal Corticocortical Connections: Computational Modeling and Electrophysiological Studies. D. Young, An Oscillatory Model of Cortical Neural Processing. G.L. Yuen, Response Synchrony, APG Theory, and Motor Control. Part III: Oscillatory Models in Perception, Memory, and Cognition. D. Horn, I. Opher, Temporal Segmentation and Binding in Oscillatory Neural Systems. A. Grunewald, S. Grossberg, Perceptual Framing and Cortical Synchronization. D. DeMaris, Attention, Depth Gestalts, and Spatially Extended Chaos in the Perception of Ambiguous Figures. G. Borisyuk, R. Borisyuk, Y. Kazanovich, G. Strong, Oscillatory Neural Networks: Modeling Binding and Attention by Synchronization of Neural Activity. B. Baird, T. Troyer, F. Eeckman, Attentional Network Streams of Synchronized 40-Hz Activity in a Cortical Architecture of Coupled Oscillatory Associative Memories. Part IV: Applications of Synchronized and Chaotic Oscillations. G.E. Mobus, P.S. Fisher, Foraging Search at the Edge of Chaos. A.G. Brown, S. Collins, An Oscillatory Associative Memory Analogue Architecture. N.S. Park, D. Robertson, K. Stenning, Symbolic Knowledge Encoding Using a Dynamic Binding Mechanism and an Embedded Inference Mechanism. A. Jagota, X. Wang, Oscillations in Discrete and Continuous Hopfield Networks. S. Wolpert, Modeling Neural Oscillation Using VLSI-Based Neuromimes.

Networks of coupled periodic oscillators (similar to the Kuramoto model) have been proposed as models of associative memory. However, error-free retrieval states of such oscillatory networks are typically unstable, resulting in a near zero capacity. This puts the networks at disadvantage as compared with the classical Hopfield network. Here we propose a simple remedy for this undesirable property and show rigorously that the error-free capacity of our oscillatory, associative-memory networks can be made as high as that of the Hopfield network. They can thus not only provide insights into the origin of biological memory, but can also be potentially useful for applications in information science and engineering.

We study image segmentation on the basis of locally excitatory globally inhibitory oscillator networks (LEGION), whereby the phases of oscillators encode the binding of pixels. We introduce a potential for each oscillator so that only those oscillators with strong connections from their neighborhoods can develop high potentials. Based on this concept, a solution to remove noisy regions in an image is proposed for LEGION, so that it suppresses the oscillators corresponding to noisy regions, without affecting those corresponding to major regions. The network is applied to segmenting real gray-level images and produces reasonable results. LEGION may provide a neurally plausible and effective framework for image segmentation and figure-ground segregation.

We propose an oscillatory neural network implemented with two-dimensional (2-D) tantalum disulfide devices operating in the change density wave regime at room temperature. An elementary cell of the network consists of two 1T-TaS 2 devices connected in series. Such a cell has a constant output and oscillatory states. All cells have the same bias voltage. There is a constant current flowing through the cell in the constant output mode. The oscillations occur at a certain bias voltage due to the electrical-field triggered metal-to-insulator transition owing to the changes in the charge-density-wave phase in the 1T-TaS 2 channel. Two 1T-TaS 2 devices oscillate out-of-phase when one of the devices is in the insulator phase while the other one is in the metallic state. The nearest-neighbor cells are coupled via graphene transistors. The cells are resistively coupled if the graphene transistor is in the on state while they are capacitively coupled if the transistor is in the off state. The operation of the oscillatory neural network is simulated numerically for the 30 × 30 node network. We present the examples of pattern recognition on a template with the fixed coupling among the near-neighbor cells. We also present the results of our numerical modeling mimicking the Game of Life in the network with the time-varying coupling. The 2-D 1T-TaS 2 devices, utilized in the network, offer a unique combination of properties such as scalability, high operational frequency, fast synchronization speed, and radiation hardness, which makes them promising for both consumer electronic and defense applications.

Development of neuromorphic systems based on new nanoelectronics materials and devices is of immediate interest for solving the problems of cognitive technology and cybernetics. Computational modeling of two- and three-oscillator schemes with thermally coupled VO2-switches is used to demonstrate a novel method of pattern storage and recognition in an impulse oscillator neural network (ONN), based on the high-order synchronization effect. The method allows storage of many patterns, and their number depends on the number of synchronous states Ns. The modeling demonstrates attainment of Ns of several orders both for a three-oscillator scheme Ns ~ 650 and for a two-oscillator scheme Ns ~ 260. A number of regularities are obtained, in particular, an optimal strength of oscillator coupling is revealed when Ns has a maximum. Algorithms of vector storage, network training, and test vector recognition are suggested, where the parameter of synchronization effectiveness is used as a degree of match. It is shown that, to reduce the ambiguity of recognition, the number coordinated in each vector should be at least one unit less than the number of oscillators. The demonstrated results are of a general character, and they may be applied in ONNs with various mechanisms and oscillator coupling topology.

Oscillatory Neural Networks (ONNs) are an emerging computing paradigm that encodes information in the phase difference of coupled oscillators. Traditionally, ONNs have been investigated using homogeneous frequency oscillators. However, physical hardware implementations are inherently subject to frequency mismatches, device variability, and nonuniformities, which are typically perceived as undesirable for circuit performance. This work challenges this assumption by inquiring whether frequency discrepancies and multifrequency oscillators can be harnessed for performing computation. First, the frequency variation threshold that oscillators can withstand to compute is derived. Next, for frequency variations beyond such a threshold, a multifrequency ONN computing approach is introduced and validated in both hardware implementation and software simulations by performing pattern retrieval. The results demonstrate computation with nonuniform frequency oscillators and highlight the feasibility of their physical hardware implementation.

Thermal coupling and effect of subharmonic synchronization in a system of two VO2 based oscillators

Networks of phase oscillators have become a widely established paradigmatic model for studying emergent collective behavior across several real-world systems, including neuronal networks, populations of chemical oscillators, and power grids. The Kuramoto model, involving one-dimensional or two-dimensional phase oscillators, demonstrates the potential for networks to showcase exceptional collective dynamics. This encompasses various outcomes such as full, partial, explosive, and asymmetry-induced synchronization, clusters, chimeras, solitary states, and generalized splay states. Notably, increasing all-to-all coupling in the classical Kuramoto model induces full synchronization as the most probable outcome and dominant rhythm. Kuramoto networks with repulsive coupling usually display splay, generalized, and cluster splay states, but the conditions under which a certain rhythm can arise and prevail are not entirely understood. Equally important for connecting Kuramoto networks to practical physical systems is understanding the function of higher-order coupling terms. These terms display a Fourier decomposition of a general 2π-periodic interaction function [1]. Previous studies have demonstrated that the inclusion of higher-order terms in the classical Kuramoto model of oscillators with all-to-all attractive coupling can lead to multiple synchronous states and switching between synchronization clusters. However, the impact of higher-order coupling modes on rhythm generation in repulsive networks remains unexplored. In this work, we present significant progress in addressing the critical issue related to repulsive Kuramoto–Sakaguchi networks of phase oscillators with phase-lagged first-order and higher-order coupling. We demonstrate that weakly repulsive networks of even and odd numbers of oscillators with first-order coupling are dominated by two-cluster and three-cluster splay states, respectively. The three-cluster splay states consist of two distinct coherent clusters and one solitary oscillator. These tripod states can be considered a fusion of a two-body chimera and a solitary state. We have dubbed these patterns of three oscillators as “Cyclops states” in reference to the Greek mythological giant with a single eye. The solitary oscillator and synchronous clusters respectively represent the Cyclops’ eye and shoulders. We present a remarkable discovery that the inclusion of higher-order coupling modes leads to worldwide stability of cyclops states across almost the entire range of the phase-lag parameter controlling repulsion [2]. Beyond the Kuramoto oscillators, we demonstrate the robust presence of this effect in networks of canonical theta-neurons with adaptive coupling. Furthermore, our results provide insight into identifying dominant rhythms within repulsive physical and biological networks.

The Kuramoto model in complex networks