Multi-agent Reinforcement Learning Using Simulated Quantum Annealing

Earthquake rupture forecasting is a critical component of seismic hazard analysis and requires identifying physically plausible rupture paths across complex fault networks. At its core, this task could be formulated as a large-scale combinatorial optimization problem, which involves selecting an optimal subset of fault segments from a set of candidates. Such problems pose significant challenges for traditional algorithms, as the number of admissible multi-fault ruptures grows combinatorially. Current operational workflows rely on locally applied plausibility filters. While computationally efficient, such greedy local heuristics risk excluding globally competitive rupture scenarios, particularly in regions with dense fault connectivity and competing rupture pathways. This work investigates whether quantum annealing hardware can serve as a sampling accelerator for exploring the ensemble of physically plausible ruptures beyond what is currently accessible to classical approaches. Designing a problem formulation that remains scientifically meaningful while respecting the constraints of current quantum hardware (size, noise, etc.) is nontrivial, and naïve encodings often collapse under these limitations. We encode rupture plausibility modeling as low energy solutions to a quadratic unconstrained binary optimization (QUBO) problem defined on a fault-network middle graph. Binary variables represent activated fault segments, while local interaction terms encode Coulomb stress transfer (as a proxy for pairwise rupture compatibility), continuity preferences, and branching behavior. Here, we adopt a classical-quantum hybrid workflow: the quantum annealer is used to sample globally competitive rupture candidates, while classical post-processing implements physical constraints to filter out non-physical ruptures. The workflow is demonstrated on a subset of the fault network used in the 2023 USGS National Seismic Hazard Model including over 200 fault segments, using both simulated thermal and quantum annealing on D-Wave hardware. Results show that our hardware-aware formulation and conditioning enable robust sampling on comparatively large fault-network instances. Views on quantum computing are polarized: some overstate its power, while others dismiss it as impractical. Our results help bridge this by establishing a concrete, testable pathway for integrating quantum annealing into rupture-modeling workflows on existing purpose-built quantum devices.—---This work is supported by the ICSC National Research Centre for High Performance Computing, Big Data and Quantum Computing (CN00000013, CUP D53C22001300005) within the European Union-NextGenerationEU program (National Recovery and Resilience Plan (PNRR) - Mission 4 Component 2 Investment 1.4.)

This paper explores the potential of quantum computing on a WCET(Worst-Case Execution Time (of a program).)-related combinatorial optimization problem applied to a set of several polynomial special cases. We consider the maximization problem of determining the most expensive path in a control flow graph. In these graphs, vertices represent blocks of code whose execution times are fixed and known in advance. We port the considered optimization problem to the quantum framework by expressing it as a QUBO. We then experimentally compare the performances in solving the problem of classic Simulated Annealing (SA), Quantum Annealing (QA), and Quantum Approximate Optimization Algorithm (QAOA). Our experiments suggest that QA represents a fast equivalent of simulated annealing. Indeed, we measured the approximation ratio on the results of QA and SA, showing that their performances are comparable, at least on our set of simplified problems.KeywordsWCETsQuantum computingQUBOCombinatorial optimizationQuantum annealingQAOA

Utilizing present and futuristic Quantum Computers to solve difficult problems in different domains has become one of the main endeavors at this moment. Of course, in arriving at the requisite solution both quantum and classical computers work in conjunction. With the continued popularity of Low Density Parity Check (LDPC) codes and hence their decoding, this paper looks into the latter as a Quadratic Unconstrained Binary Optimization (QUBO) and utilized D-Wave 2000Q Quantum Annealer to solve it. The outputs from the Annealer are classically post-processed using simple minimum distance decoding to further improve the performance. We evaluated and compared this implementation against the decoding performance obtained using Simulated Annealing (SA) and belief propagation (BP) decoding with classical computers. The results show that implementations of annealing (both simulated and quantum) are superior to BP decoding and suggest that the advantage becomes more prominent as block lengths increase. Reduced Bit Error Rate (BER) and Frame Error Rate (FER) are observed for simulated annealing and quantum annealing, at useful SNR range - a trend that persists for various codeword lengths.

Satisfiability filters are a new and promising type of filters for set membership testing. In order to construct satisfiability filters, it is necessary to find disparate solutions to hard random k k -SAT problems. This paper compares simulated annealing, simulated quantum annealing and WalkSAT, an open-source SAT solver, in terms of their ability to find such solutions. The results indicate that solutions found by simulated quantum annealing are generally less disparate than solutions found by the other solvers and therefore less useful for the construction of satisfiability filters.

Probabilistic computing with p-bits is emerging as a computational paradigm for machine learning and for facing combinatorial optimization problems (COPs) with the so-called probabilistic Ising machines (PIMs). From a hardware point of view, the key elements that characterize a PIM are the random number generation, the nonlinearity, the network of coupled probabilistic bits, and the energy-minimization algorithm. Regarding the energy-minimization algorithm in this work we show that PIMs using the simulated quantum annealing (SQA) schedule exhibit better performance as compared to simulated annealing and parallel tempering in solving a number of COPs, such as maximum satisfiability problems, the planted Ising problem, and the traveling salesman problem. Additionally, we design and simulate the architecture of a fully connected CMOS-based PIM that is able to run the SQA algorithm having a spin-update time of 8 ns with a power consumption of 0.22 mW. Our results also show that SQA increases the reliability and the scalability of PIMs by compensating for device variability at an algorithmic level enabling the development of their implementation combining CMOS with different technologies such as spintronics. This work shows that the characteristics of the SQA are hardware agnostic and can be applied in the codesign of any hybrid analog-digital Ising machine implementation. Our results open a promising direction for the implementation of a new generation of reliable and scalable PIMs.

Quantum computing is an information processing paradigm that uses quantum-mechanical properties to speedup computationally hard problems. Gate-based quantum computers and Quantum Annealers (QAs) are two commercially available hardware platforms that are accessible to users today. Although promising, existing gate-based quantum computers consist of only a few dozen qubits and are not large enough for most applications. On the other hand, existing QAs with few thousand qubits have the potential to solve some domain-specific optimization problems. QAs are single instruction machines and to execute a program, the problem is cast to a Hamiltonian, embedded in the hardware, and a single quantum machine instruction (QMI) is run. Unfortunately, noise and imperfections in hardware result in sub-optimal solutions on QAs even if the QMI is run for thousands of trials.Due to the limited programmability of QAs users execute the same QMI for all trials. This subjects all trials to a similar noise profile throughout the execution, resulting in a systematic bias. We observe that systematic bias leads to sub-optimal solutions and cannot be alleviated by executing more trials or using existing error-mitigation schemes. To address this challenge, we propose EQUAL (Ensemble QUantum AnneaLing). EQUAL generates an ensemble of QMIs by adding controlled perturbations to the program QMI. When executed on the QA, the ensemble of QMIs steers the program away from encountering the same bias during all trials and thus, improves the quality of solutions. Our evaluations using the 2041-qubit D-Wave QA show that EQUAL bridges the difference between the baseline and the ideal by an average of 14% (and up to 26%) without requiring any additional trials. EQUAL can be combined with existing error mitigation schemes to further bridge the difference between the baseline and ideal by an average of 55% (and up to 68%).

In edge computing, suppressing data size is a challenge for machine learning models that perform complex tasks such as autonomous driving, in which computational resources (speed, memory size and power) are limited. Efficient lossy compression of matrix data has been introduced by decomposing it into the product of an integer and real matrices. However, its optimisation is difficult as it requires simultaneous optimisation of an integer and real variables. In this paper, we improve this optimisation by utilising recently developed black-box optimisation (BBO) algorithms with an Ising solver for binary variables. In addition, the algorithm can be used to solve mixed-integer programming problems that are linear and non-linear in terms of real and integer variables, respectively. The differences between the choice of Ising solvers (simulated annealing, quantum annealing and simulated quenching) and the strategies of the BBO algorithms (BOCS, FMQA and their variations) are discussed for further development of the BBO techniques.

Optimization methods are critical to radiation therapy. A new technology, quantum annealing (QA), employs novel hardware and software techniques to address various discrete optimization problems in many fields. We report on the first application of quantum annealing to the process of beamlet intensity optimization for IMRT.We apply recently-developed hardware which natively exploits quantum mechanical effects for improved optimization. The new algorithm, called QA, is most similar to simulated annealing, but relies on natural processes to directly minimize a system’s free energy. A simple quantum system is slowly evolved into a classical system representing the objective function. If the evolution is sufficiently slow, there are probabilistic guarantees that a global minimum will be located.To apply QA to IMRT-type optimization, two prostate cases were considered. A reduced number of beamlets were employed, due to the current QA hardware limitations. The beamlet dose matrices were computed using CERR and an objective function was defined based on typical clinical constraints, including dose-volume objectives, which result in a complex non-convex search space. The objective function was discretized and the QA method was compared to two standard optimization methods, simulated annealing and Tabu search, run on a conventional computing cluster.Based on several runs, the average final objective function value achieved by the QA was 16.9 for the first patient, compared with 10.0 for Tabu and 6.7 for the simulated annealing (SA) method. For the second patient, the values were 70.7 for the QA, 120.0 for Tabu and 22.9 for the SA. The QA algorithm required 27–38% of the time required by the other two methods.In this first application of hardware-enabled QA to IMRT optimization, its performance is comparable to Tabu search, but less effective than the SA in terms of final objective function values. However, its speed was 3–4 times faster than the other two methods. This initial experiment suggests that more research into QA-based heuristics may offer significant speedup over conventional clinical optimization methods, as quantum annealing hardware scales to larger sizes.

During recent years, quantum computers have received increasing attention, primarily due to their ability to significantly increase computational performance for specific problems. Computational performance could be improved for mathematical optimization by quantum annealers. This special type of quantum computer can solve quadratic unconstrained binary optimization problems. However, multi-energy systems optimization commonly involves integer and continuous decision variables. Due to their mixed-integer problem structure, quantum annealers cannot be directly used for multi-energy system optimization.To solve multi-energy system optimization problems, we present a hybrid Benders decomposition approach combining optimization on quantum and classical computers. In our approach, the quantum computer solves the master problem, which involves only the integer variables from the original energy system optimization problem. The subproblem includes the continuous variables and is solved by a classical computer. For better performance, we apply improvement techniques to the Benders decomposition. We test the approach on a case study to design a cost-optimal multi-energy system. While we provide a proof of concept that our Benders decomposition approach is applicable for the design of multi-energy systems, the computational time is still higher than for approaches using classical computers only. We therefore estimate the potential improvement of our approach to be expected for larger and fault-tolerant quantum computers.

Quantum computing is a multidisciplinary field at the intersectionof quantum physics, computer science, and mathematics, aiming to har-ness quantum mechanical phenomena for computational advantage. Thisarticle provides a comprehensive review of both the foundational princi-ples and advanced theoretical models of quantum computing. We beginwith the basic framework of quantum computation, introducing the con-cept of qubits as two-level quantum systems, the linear algebra of Hilbertspaces, Dirac’s bra–ket notation, and the postulates of quantum mechan-ics (state superposition, unitary evolution, and projective measurement).We then discuss quantum logic gates and circuits, highlighting how clas-sical reversible computation principles extend to universal quantum gatesets. Next, we survey key quantum algorithms (from Shor’s factoring toGrover’s searching and beyond) and the complexity-theoretic implicationsof quantum computers, contrasting the class BQP with classical complex-ity classes. We cover the theoretical foundations of quantum error cor-rection and fault tolerance, explaining how quantum information can beprotected from decoherence using redundancy and how fault-tolerant pro-tocols can, in principle, allow scalable quantum computation despite noisyhardware. We then explore advanced models and paradigms of quantumcomputation, including measurement-based quantum computing (one-wayquantum computing), topological quantum computation with anyons, adi-abatic quantum computing and quantum annealing, continuous-variablequantum computing, and related approaches. We also provide an overviewof quantum information theory as it relates to computation, discussingentanglement, quantum entropy, and information-theoretic limits like theHolevo bound. Finally, we review the leading physical implementations(quantum hardware models) of quantum computers—trapped ions, super-conducting circuits, photonic systems, solid-state spin qubits, and oth-ers—outlining the challenges and achievements of each approach.

Clustering bike sharing stations using Quantum Machine Learning: A case study of Toronto, Canada

Quantum computations are very important branch of modern cryptology. According to the number of working physical qubits available in general-purpose quantum computers and in quantum annealers, there is no coincidence, that nowadays quantum annealers allow to solve larger problems. In this paper we focus on solving discrete logarithm problem (DLP) over binary fields using quantum annealing. It is worth to note, that however solving DLP over prime fields using quantum annealing has been considered before, no author, until now, has considered DLP over binary fields using quantum annealing. Therefore, in this paper, we aim to bridge this gap. We present a polynomial transformation of the discrete logarithm problem over binary fields to the Quadratic Unconstrained Binary Optimization (QUBO) problem, using approximately 3n2 logical variables for the binary field F2n. In our estimations, we assume the existence of an optimal normal base in the given fields. Such a QUBO instance can then be solved using quantum annealing.

Aim and ApproachLarge cooling water system has great potential of energy saving due to improper operation control. The coupled hydraulic and thermodynamic characteristics among chillers, pumps, and cooling towers lead to the difficulty of cooling water system operation. This study investigates an advanced control approach of multi-agent reinforcement learning (RL) for a large cooling water system. The multiple RL agents interact with each other to deal with the complex coupled characteristics in the cooling water system. The proposed approach is implemented in a real-world cooling water system and validated with experiments.Scientific Innovation and RelevanceIn this study, a multi-agent RL approach is proposed to separately control the chillers, pumps, and cooling towers in the cooling water system. The soft actor critic algorithm is used in the RL agent, as it is an efficient and advanced RL algorithm for continuous and discrete control problems. The agents are designed to have private states, private actions, and shared rewards. The reward functions are carefully designed to balance the agents for global optima.To provide a realistic environment for RL training, a detailed physical modeling platform of a cooling water system is established and validated. This platform is capable of hydraulic and thermodynamic calculation of cooling water systems with customized numbers and characteristics of chillers, pumps, and cooling towers, which is scalable for different cooling water systems.The proposed multi-agent RL controller is implemented in the building automation system of a real cooling water system. A one-month experimental study of the multi-agent RL performance is conducted. The proposed multi-agent RL controller is compared with a single-agent RL controller and a rule-based controller through real-world experiments to demonstrate the energy performance. This is a pilot study of multi-agent RL approach to real-world cooling water system control.Preliminary Results and ConclusionsThe preliminary simulation study is conducted in the established cooling water system model. The electricity consumption simulation error is 4.0% in coefficient of variation of mean absolute error (CVMAE) compared with the measured 15-minute electricity consumption. The 6-month simulation shows that the single-agent RL can save 6.2% energy from the rule-based control, while the multi-agent RL can save 7.2% electricity consumption of the cooling water system. In the following study, the multi-agent RL implementation will be practiced with more experiments and applications for cooling water system control.

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.