Research on Multi-Agent Collaborative Scheduling Planning Method for Time-Triggered Networks

In recent years, Satisfiability Modulo Theories (SMT) solvers have emerged as powerful tools in many formal methods applications, including verification, automated theorem proving, planning and software synthesis. The expressive power of SMT allows problems from many disciplines to be handled in a single unified approach. While SMT solvers are highly effective at handling certain classes of problems due to highly tuned implementations of efficient ground decision procedures, their ability is often limited when reasoning about universally quantified first-order formulas. Since generally this class of problems is undecidable, most SMT solvers use heuristic techniques for answering unsatisfiable when quantified formulas are present. On the other hand, when the problem is satisfiable, solvers using these techniques will either run indefinitely, or give up after some predetermined amount of effort. In a majority of formal methods applications, it is critical that the SMT solver be able to determine when such a formula is satisfiable, especially when it can return some representation of a model for the formula. This dissertation introduces new techniques for finding models for SMT formulas containing quantified first-order formulas. We will focus our attention on finding finite models, that is, models whose domain elements can be represented as a finite set. We give a procedure that is both finite model complete and refutationally complete for a fragment of first-order logic that occurs commonly in practice.

In the future, mixed traffic flow will include two types of vehicles: connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs). CAVs emerge as new solutions to disrupt the traditional transportation system. This new solution shares real-time data with each other and the roadside units (RSU) for network management. Reinforcement learning (RL) is a promising approach for traffic signal management in complex urban areas by leveraging information gathered from CAVs. In particular, coordinating signal management at many intersections is a critical challenge in multi-agent reinforcement learning (MARL). According to this vision, we propose an approach that combines an actor–critic network–based multi-agent deep deterministic policy gradient (MADDPG) model and a rerouting technique (RT) to increase traffic performance in vehicular networks. This algorithm overcomes the inherent non-stationary of Q-learning and the high variance of policy gradient (PG) algorithms. Based on centralized learning with decentralized execution, the MADDPG model employs one actor and one critic for each agent. The actor network uses local information to execute actions, while the critic network is trained with extra information, including the states and actions of other agents. Through a centralized learning process, agents can coordinate with each other, diminishing the influence of an unstable environment. Unlike previous studies, we not only manage traffic light systems but also consider the effect of platooning vehicles on increasing throughput. Experimental results show that our model outperforms other models in terms of traffic performance in different scenarios.

Multiagent reinforcement learning (RL) is widely used and can successfully solve many problems in the real world. In the multiagent RL system, a global critic network is used to guide each agent’s strategy to update to learn the most beneficial strategy for the collective. However, the global critic network also makes the current agent’s learning be affected by other agents’ strategies, which leads to unstable learning. To solve this problem, we propose dynamic decomposed multiagent deep deterministic policy gradient (DD-MADDPG): a new network that considers both global and local evaluations and adaptively adjusts the agent’s attention to the two evaluations. Besides, the use of the experience replay buffer by multiagent deep deterministic policy gradient (MADDPG) produces outdated experience, and the outdated strategies of other agents further affect the learning of the current agent. To reduce the influence of other agents’ outdated experience, we propose TD-Error and Time-based experience sampling (T2-PER) based on DD-MADDPG. We evaluate the proposed algorithm’s performance according to the learning stability and the average return obtained by the agents. We have conducted experiments in the MPE environment. The results show that the proposed method has better stability and higher learning efficiency than MADDPG and has a certain generalization ability.

With increasing travel demand, highway congestion and accidents have become more frequent. As an essential component of intelligent transportation systems (ITS), Hard Shoulder Running (HSR) provides a dynamic strategy to mitigate congestion by temporarily opening shoulder lanes, yet traditional methods often fail to adapt to real-time traffic changes and overlook the shoulder’s critical role in ensuring emergency vehicle access. This study proposes a novel HSR optimization framework based on the Long Short-Term Memory (LSTM) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm, integrated with an improved A* algorithm for EV lane clearing. LSTM is used to extract temporal features from traffic data to support intelligent decision-making in MADDPG, enhanced with prioritized experience replay and importance sampling. The EV lane-clearing task is formulated using graph theory, and the improved A* algorithm determines the optimal path for clearing. A simulation case was developed using Simulation of Urban MObility (SUMO) for a section of the Jinan City Ring Highway, China, evaluating four levels of traffic service. Results show the proposed method reduces total travel time by 14.6%, Time Integrated Time-to-Collision (TIT) by 45.2%, and CO2 emissions by 11.9%. Additionally, with EV intervention, braking times are reduced by up to 376.3% and travel time by 18.1% using the improved A* strategy. These findings demonstrate that the integrated LSTM-MADDPG and A* approach effectively enhances highway traffic efficiency, safety, and sustainability under complex real-world conditions.

This paper describes a method for combining “off-the-shelf” SAT and constraint solvers for building an efficient Satisfiability Modulo Theories (SMT) solver for a wide range of theories. Our method follows the abstraction/refinement approach to simplify the implementation of custom SMT solvers. The expected performance penalty by not using an interweaved combination of SAT and theory solvers is reduced by generalising a Boolean solution of an SMT problem first via assigning don’t care to as many variables as possible. We then use the generalised solution to determine a thereby smaller constraint set to be handed over to the constraint solver for a background theory. We show that for many benchmarks and real-world problems, this optimisation results in considerably smaller and less complex constraint problems. The presented approach is particularly useful for assembling a practically viable SMT solver quickly, when neither a suitable SMT solver nor a corresponding incremental theory solver is available. We have implemented our approach in the ABsolver framework and applied the resulting solver successfully to an industrial case-study: the verification problems arising in verifying an electronic car steering control system impose non-linear arithmetic constraints, which do not fall into the domain of any other available solver.

Satisfiability Modulo Theories (SMT) refers to the problem of deciding the satisfiability of a formula with respect to certain background first-order theories. In this article, we focus on Satisfiablity Modulo Integer Arithmetic, which is referred to as SMT(IA), including both linear and non-linear integer arithmetic theories. Dominant approaches to SMT rely on calling a CDCL-based SAT solver, either in a lazy or eager flavour. Local search, a competitive approach to solving combinatorial problems including SAT, however, has not been well studied for SMT. We develop the first local-search algorithm for SMT(IA) by directly operating on variables, breaking through the traditional framework. We propose a local-search framework by considering the distinctions between Boolean and integer variables. Moreover, we design a novel operator and scoring functions tailored for integer arithmetic, as well as a two-level operation selection heuristic. Putting these together, we develop a local search SMT(IA) solver called LocalSMT. Experiments are carried out to evaluate LocalSMT on benchmark sets from SMT-LIB. The results show that LocalSMT is competitive and complementary with state-of-the-art SMT solvers, and performs particularly well on those formulae with only integer variables. A simple sequential portfolio with Z3 improves the state-of-the-art on satisfiable benchmark sets from SMT-LIB.

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.

Introduction To address the urgent need for enhanced grid flexibility in high-penetration distributed photovoltaic (PV) systems, this paper proposes a novel two-stage framework integrating multi-objective cluster partitioning and multi-agent reinforcement learning (MARL) for cooperative peak shaving. Methods First, a comprehensive multi-objective optimization model defines PV cluster partitioning using three domain-specific metrics: available peak shaving margin (APSM), peak shaving synergy (PSS), and node correlation degree (NCD). This approach replaces conventional clustering algorithms, ensuring partitions balance electrical compactness, regulation capacity, and operational coordination. Subsequently, each partitioned cluster acts as an autonomous agent in a hierarchical MARL framework. Leveraging the multi-agent deep deterministic policy gradient (MADDPG) algorithm, agents collaboratively optimize active/reactive power allocation among clusters while coordinating unit-level control within clusters. Results Validated on the IEEE-33 node system, the strategy achieves a 29.39% reduction in peak-to-valley difference and 54.44% improvement in power balance deviation, significantly smoothing net load curves and enhancing voltage stability. Discussion This work demonstrates how MARL-driven coordination unlocks the inherent flexibility of distributed PV clusters, providing a scalable solution for grid-edge resource participation in system-level peak shaving.

• A grid-aware multi-agent reinforcement learning was developed for EV charging. • Public charging data worked with domestic demand to build a real environment. • A dual-head actor optimised the acceptance and cross-site redirection of EV charging. • A centralised attention-based critic and delayed replay buffer ensured stable learning. • The RL policy reduced peak load variability by 40 % and threshold violations by 37 %. As electric vehicles (EV) become more widespread, managing charging demand is critical for grid stability and efficient resource allocation. However, prior work often optimises single sites or treats demand as exogenous, without explicitly modelling grid capacity constraints or the aggregate effects of redirection, and it rarely offers an integrated capacity-aware view that links domestic and public charging under system-level coordination. This study, therefore, develops a multi-agent reinforcement learning framework with cross-site redirection to optimise EV charging operations. The research area focuses on the Perth and Kinross region of Scotland in the UK, using real public charging records from the council database as well as domestic charging statistics from the UK Department for Transport. Charging sites are modelled as cooperative agents using the Multi-Agent Deep Deterministic Policy Gradient algorithm, trained in a spatiotemporal environment that integrates public charging data and simulated domestic demand. We contribute by unifying public and home charging demand in one learning environment, introducing a cross-site actor trained under centralised training and decentralised execution, and incorporating a delayed replay buffer with short-horizon forecasts so redirection aligns with future congestion. The learned policy reduces peak-hour load standard deviation by up to 40 % and lowers cumulative threshold violations by 37 % compared to the baseline. Distinct weekday and weekend strategies emerge, enabling adaptive coordination under varying demand patterns. The study provides interpretable control for EV networks, balancing peak demand and service quality across sites while addressing the system-level coordination.

Financial investigation decision is a complex affair that highly relies on expert knowledge. In the era of big data, it remains practical to make intelligent financial investigation decisions via computing technology. As a consequence, this work introduces reinforcement learning to search for effective decision schemes that are adaptive to the diverse financial environment. Therefore, an intelligent financial investment decision model based on multi-agent reinforcement learning is proposed in this paper. First, the long short-term memory (LSTM) model is utilized to strengthen financial investment data. Then, the hidden Markov model (HMM) is utilized to make a preliminary judgment on financial investment strategy. Next, a multi-agent deep deterministic policy gradient (MADDPG) is employed to set the goal of agent reinforcement and determine decision indicators based on the financial market environment. Multiple agents (investors) form a cooperative strategy through learning and interaction to achieve the optimal investment decision in the financial market. The experimental results on realistic data indicate that the proposal can play an important role in investment decision-making.

Satisfiability Modulo Theories (SMT) solvers are used in many modern program verification, analysis and testing tools. They owe their scale and efficiency thanks to advances in search algorithms underlying modern SAT solvers and first-order theorem provers. They owe their versatility in software development applications thanks to specialized algorithms supporting theories, such as numbers and algebraic data-types, of relevance for software engineering. This lecture introduces algorithmic principles of SMT solving, taking as basis modern SAT solvers and integration with specialized theory solvers and quantifier reasoning. We detail some of the algorithms used for main theories used in current SMT solvers and survey newer theories and approaches to integrating solvers. The lectures also outline some application scenarios where SMT solvers have found use, including program verification, network analysis, symbolic model checking, test-case generation, and white-box fuzzing.

Multi-agent combat is a combat scenario in multiagent reinforcement learning (MARL). In this combat, agents use reinforcement learning methods to learn optimal policies. Actually, policy may be changed, which leads to a non-stationary environment. In this case, it is difficult to predict opponents' policies. Many reinforcement learning methods try to solve nonstationary problems. Most of the previous works put all agents into a frame and model their policies to deal with non-stationarity of environments. But, in a combat environment, opponents can not be in the same frame as our agents. We group opponents and our agents into two frames, only considering opponents as a part of the environment. In this paper, we focus on the problem of modelling opponents' policies in non-stationary environments. To solve this problem, we propose an algorithm called Additional Opponent Characteristics Multi-agent Deep Deterministic Policy Gradient (AOC-MADDPG) with the following contributions: (1) We propose a new actor-critic framework to deal with nonstationarity of environments in MARL, so that agents can adapt to more complex environments. (2) A model for opponents' policies is built by introducing observations and actions of the opponents into the critic network as additional characteristics. We evaluate our AOC-MADDPG algorithm in two multi-agent combat environments. As a result, our approach significantly outperforms the baseline. Agents trained by our method can get higher rewards in non-stationary environments.

Satisfiability Modulo Theories (SMT) solvers have been successfully applied to solve many problems in formal verification such as bounded model checking (BMC) for many classes of systems from integrated circuits to cyber-physical systems. Typically, BMC is performed by checking satisfiability of a possibly long, but quantifier-free formula. However, BMC problems can naturally be encoded as quantified formulas over the number of BMC steps. In this approach, we then use decision procedures supporting quantifiers to check satisfiability of these quantified formulas. This approach has previously been applied to perform BMC using a Quantified Boolean Formula (QBF) encoding for purely discrete systems, and then discharges the QBF checks using QBF solvers. In this paper, we present a new quantified encoding of BMC for rectangular hybrid automata (RHA), which requires using more general logics due to the real (dense) time and real-valued state variables modeling continuous states. We have implemented a preliminary experimental prototype of the method using the HyST model transformation tool to generate the quantified BMC (QBMC) queries for the Z3 SMT solver. We describe experimental results on several timed and hybrid automata benchmarks, such as the Fischer and Lynch-Shavit mutual exclusion algorithms. We compare our approach to quantifier-free BMC approaches, such as those in the dReach tool that uses the dReal SMT solver, and the HyComp tool built on top of nuXmv that uses the MathSAT SMT solver. Based on our promising experimental results, QBMC may in the future be an effective and scalable analysis approach for RHA and other classes of hybrid automata as further improvements are made in quantifier handling in SMT solvers such as Z3.

Satisfiability Modulo Theories (SMT) solvers have been widely applied in automated software analysis to reason about the queries that encode the essence of program semantics, relieving the heavy burden of manual analysis. Many SMT solving techniques rely on solving Boolean satisfiability problem (SAT), which is an NP-complete problem, so they use heuristic search strategies to seek possible solutions, especially when no known theorem can efficiently reduce the problem. An emerging challenge, named Mixed-Bitwise-Arithmetic (MBA) obfuscation, impedes SMT solving by constructing identity equations with both bitwise operations (and, or, negate) and arithmetic computation (add, minus, multiply). Common math theorems for bitwise or arithmetic computation are inapplicable to simplifying MBA equations, leading to performance bottlenecks in SMT solving. In this paper, we first scrutinize solvers' performance on solving different categories of MBA expressions: linear, polynomial, and non-polynomial. We observe that solvers can handle simple linear MBA expressions, but facing a severe performance slowdown when solving complex linear and non-linear MBA expressions. The root cause is that complex MBA expressions break the reduction laws for pure arithmetic or bitwise computation. To boost solvers' performance, we propose a semantic-preserving transformation to reduce the mixing degree of bitwise and arithmetic operations. We first calculate a signature vector based on the truth table extracted from an MBA expression, which captures the complete MBA semantics. Next, we generate a simpler MBA expression from the signature vector. Our large-scale evaluation on 3000 complex MBA equations shows that our technique significantly boost modern SMT solvers' performance on solving MBA formulas.

We present our extension of ACL2 with Satisfiability Modulo Theories (SMT) solvers using ACL2's trusted clause processor mechanism. We are particularly interested in the verification of physical systems including Analog and Mixed-Signal (AMS) designs. ACL2 offers strong induction abilities for reasoning about sequences and SMT complements deduction methods like ACL2 with fast nonlinear arithmetic solving procedures. While SAT solvers have been integrated into ACL2 in previous work, SMT methods raise new issues because of their support for a broader range of domains including real numbers and uninterpreted functions. This paper presents Smtlink, our clause processor for integrating SMT solvers into ACL2. We describe key design and implementation issues and describe our experience with its use.