A High-Dimensional Dynamic Characteristics-Based Boundary Test Scenario Generation Method for UAVs in Dynamic Target Tracking

When simultaneously addressing the challenges of dynamic target tracking and obstacle avoidance for robots, conventional control and control only based on reinforcement learning cannot deal with the complex scenarios effectively. The purpose of this study is to design a robot control algorithm that combines deep reinforcement learning (Soft Actor-Critic, SAC) with PID to achieve real-time tracking of a moving object and effectively avoid single or multiple obstacles. The control of the robot is divided into two key components: initially, the first joint of the 6-degree-of-freedom robot is controlled by PID algorithm, which makes the working plane (the plane coincident with the axis of the first joint and parallel to the linkage) quickly approach the target until it overlaps. Subsequently, the task of reinforcement learning is simplified to control the planar robot to track the target projection in working plane while avoiding the obstacle projection, ultimately achieve target tracking and obstacle avoidance in 3D space. The simulation and experiment results show that the proposed method has good efficiency and convergence speed. The SAC-PID strategy effectively controls the Universal-Robots UR5 to complete dynamic target tracking while accomplishing obstacle avoidance in both virtual and real-world environments.

Autonomous control of group of unmanned aerial vehicles based on task allocation mechanisms shows great potential for ground tactical mission support. We introduce experimental simulation system combining flexible mission control of ground assets in urban environment and autonomous aerial support utilizing multi-agent problem solving techniques. Two case-studies are presented for evaluation - cooperative area surveillance and dynamic target tracking with undervalued number of assets. We show the strength and benefits of multi-agent task allocation and delegation mechanisms in such dynamic scenarios mainly in case of limited number of assets.

With the large-scale integration of distributed power sources, distribution network planning is undergoing significant transformations. To further enhance the efficiency and practicality of distribution network planning, it is essential to model the uncertainties in source–load dynamic scenarios. However, traditional scenario generation methods struggle with high-dimensional variables and complex spatiotemporal characteristics, posing severe challenges for distribution network planning. To address these issues, this paper proposes a multi-time scale source–load scenario generation method based on temporal convolutional networks and multi-head attention mechanisms within a temporal generative adversarial network framework. This algorithm not only enhances the richness and robustness of source–load scenarios in distribution networks but also serves as a valuable reference for medium-long-term analysis and planning. Finally, the results present a set of daily, weekly, and monthly multi-time scale source–load scenarios, and multiple evaluation indicators are utilized to evaluate the quality of the generated scenarios; the accuracy of the generated scenarios is increased by about 2%.

Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning

In recent years, there has been a growing trend of applying reinforcement learning (RL) in financial applications. This approach has shown great potential for decision-making tasks in finance. In this review, we present a comprehensive study of the applications of RL in finance and conduct a series of meta-analyses to investigate the common themes in the literature, such as the factors that most significantly affect RL's performance compared with traditional methods. Moreover, we identify challenges, including explainability, Markov decision process modeling, and robustness, that hinder the broader utilization of RL in the financial industry and discuss recent advancements in overcoming these challenges. Finally, we propose future research directions, such as benchmarking, contextual RL, multi-agent RL, and model-based RL, to address these challenges and to further enhance the implementation of RL in finance.

Small target detection and dynamic target continuous tracking are difficulties in the field of target detection and tracking respectively. The experimental results show that the smaller the target size is, the lower the detection accuracy is, the more dynamic the target is, and the lower the tracking accuracy is. This paper presents a high dynamic small target detection and tracking system which is composed of a yolov4 network, KCF tracker, and Kalman filter prediction algorithm. In this system, the detection network is divided into global and local levels, in which the global network is used for initial detection and re-detection after tracking failure, while the local detection network and tracker are used for real-time tracking of small targets, a more robust system is constructed. Compared with the original target detection network, this method weakens the influence of external factors such as the change of target shape and the fast-moving speed on the detection accuracy. The recognition rate of the small target is more than 96%, and the detection accuracy achieves 78%.

Simulation-based testing has emerged as an essential tool for verifying and validating autonomous vehicles (AVs). However, contemporary methodologies, such as deterministic and imitation learning-based driver models, struggle to capture the variability of human-like driving behavior. Given these challenges, we propose HAD-Gen, a general framework for realistic traffic scenario generation that simulates diverse human-like driving behaviors. The framework first clusters the vehicle trajectory data into different driving styles according to safety features. It then employs maximum entropy inverse reinforcement learning on each of the clusters to learn the reward function corresponding to each driving style. Using these reward functions, the method integrates offline reinforcement learning pre-training and multi-agent reinforcement learning algorithms to obtain general and robust driving policies. Multi-perspective simulation results in highway scenarios show that our proposed scenario generation framework can generate diverse, human-like driving behaviors with strong generalization capability. The proposed framework achieved a 90.96% goal-reaching rate, an off-road rate of 2.08%, and a collision rate of 6.91% in new unseen driving scenarios, outperforming prior approaches by over 20% in goal-reaching performance. The source code is released at https://github.com/RoboSafe-Lab/HAD-Gen.

The short-term wind power scenarios have a significant impact on the operation cost and power system reliability due to the stochastic generation scheduling of wind-integrated power systems. In order to obtain the scenarios containing the information of forecast error distribution and fluctuation distribution for short-term wind power, a scenario generation method is proposed. This paper characterizes forecast error via empirical distributions of a set of forecast bins and assumes that wind power fluctuations over unit interval follow t location-scale distribution. An inverse transform sampling from a multivariate normal distribution is adopted to generate a large number of wind power scenarios. The covariance matrix of the multivariate normal distribution is estimated to fit the distribution of historical wind power fluctuations. The proposed scenario generation method is applied to the actual aggregate wind power data in the whole regions of Ireland's Power System. The results indicate that the variability of wind power scenarios can be adjusted by estimating the key range parameter in the exponential covariance structure of a multivariate normal distribution.

On scenario construction for stochastic shortest path problems in real road networks

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

Reinforcement Learning with Temporal Logic Constraints

Stable optimisation-based scenario generation via game theoretic approach

This paper presents the development of a motivated learning (ML) agent with symbolic I/O. Our earlier work on the ML agent was enhanced, giving it autonomy for interaction with other agents. Specifically, we equipped the agent with drives and pains that establish its motivations to learn how to respond to desired and undesired events and create related abstract goals. The purpose of this paper is to explore the autonomous development of motivations and memory in agents within a simulated environment. The ML agent has been implemented in a virtual environment created within the NeoAxis game engine. Additionally, to illustrate the benefits of an ML-based agent, we compared the performance of our algorithm against various reinforcement learning (RL) algorithms in a dynamic test scenario, and demonstrated that our ML agent learns better than any of the tested RL agents.This paper presents the development of a motivated learning (ML) agent with symbolic I/O. Our earlier work on the ML agent was enhanced, giving it autonomy for interaction with other agents. Specifically, we equipped the agent with drives and pains that establish its motivations to learn how to respond to desired and undesired events and create related abstract goals. The purpose of this paper is to explore the autonomous development of motivations and memory in agents within a simulated environment. The ML agent has been implemented in a virtual environment created within the NeoAxis game engine. Additionally, to illustrate the benefits of an ML-based agent, we compared the performance of our algorithm against various reinforcement learning (RL) algorithms in a dynamic test scenario, and demonstrated that our ML agent learns better than any of the tested RL agents.

For implementing dynamic target tracking, two distributed CCD (charge-coupled device) cameras are set up to capture the poses of the tracking and target cars, which have the front-wheel for the steering orientation and the rear-wheel for the forward-backward motion. Based on the control authority of these two CCD cameras, a suitable reference command for the proposed controller of the tracking car is planned on a personal computer and then transmitted to the tracking car by a wireless device. Only the information of the upper bound of system knowledge is required to select the suitable scaling factors and coefficients of sliding surface for the proposed controller so that an acceptable performance is achieved. Since the target car is dynamic and the tracking car possesses dynamics, a grey prediction for the pose of the target car is employed to plan an effective reference command for the tracking car to enhance the performance of target tracking. Finally, a sequence of experiments is carried out to confirm the usefulness of the proposed control system.

Underwater dynamic target tracking technology has a wide application prospect in marine resource exploration, underwater engineering operations, naval battlefield monitoring, and underwater precision guidance. Aiming at the underwater dynamic target tracking problem, an autonomous underwater vehicle tracking control method based on trajectory prediction is studied. First, a deep learning-based target detection algorithm is developed. For the image collected by the multibeam forward-looking sonar image, this algorithm uses the YOLO v3 network to determine the target in a sonar image and obtain the position of the target. Then, a time profit Elman neural network (TPENN) is constructed to predict the trajectory information of the dynamic target. Compared with an ordinary Elman neural network, its accuracy of dynamic target prediction is increased. Finally, underwater tracking of the dynamic target is realized using the model predictive controller (MPC), and the tracking result is stable and reliable. Through simulations and experiment, the proposed underwater dynamic target tracking control method is demonstrated to be effective and feasible.