Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration

Dynamic economic dispatch of integrated energy system based on generative adversarial imitation learning

Analyzing the handwriting generation process is an important issue and has been tackled by various generation models, such as kinematics based models and stochastic models. In this study, we use a reinforcement learning (RL) framework to realize handwriting generation with the careful future planning ability. In fact, the handwriting process of human beings is also supported by their future planning ability; for example, the ability is necessary to generate a closed trajectory like `0' because any shortsighted model, such as a Markovian model, cannot generate it. For the algorithm, we employ generative adversarial imitation learning (GAIL). Typical RL algorithms require the manual definition of the reward function, which is very crucial to control the generation process. In contrast, GAIL trains the reward function along with the other modules of the framework. In other words, through GAIL, we can understand the reward of the handwriting generation process from handwriting examples. Our experimental results qualitatively and quantitatively show that the learned reward catches the trends in handwriting generation and thus GAIL is well suited for the acquisition of handwriting behavior.

Generative adversarial imitation learning (GAIL) has shown good results in several research areas by taking advantage of generative adversarial networks. However, GAIL lacks a reward mechanism and usually adopts a model‐free approach based on stochastic policies, which is not ideal for solving complex, dynamically uncertain population intelligence problems, especially in the face of autonomous driving environments. In this paper, a policy framework is shaped by combining the human knowledge with GAIL (HKGAIL). HKGAIL embeds human decision models into the learning process to infer the underlying structure of expert demonstrations. The skills learned from expert demonstrations can directly guide the actions (policies) of the learning process of the agents, and the policies can be optimized through the feedback function of the discriminator. Experiments on both driving and landing tasks show that HKGAIL was able to better fit the policy close to the expert, and was 16.2% safer than GAIL for the driving task.

The agent in imitation learning (IL) is expected to mimic the behavior of the expert. Its performance relies highly on the quality of given expert demonstrations. However, the assumption that collected demonstrations are optimal cannot always hold in real-world tasks, which would seriously influence the performance of the learned agent. In this paper, we propose a robust method within the framework of Generative Adversarial Imitation Learning (GAIL) to address imperfect demonstration issue, in which good demonstrations can be adaptively selected for training while bad demonstrations are abandoned. Specifically, a binary weight is assigned to each expert demonstration to indicate whether to select it for training. The reward function in GAIL is employed to determine this weight (i.e. higher reward results in higher weight). Compared to some existing solutions that require some auxiliary information about this weight, we set up the connection between weight and model so that we can jointly optimize GAIL and learn the latent weight. Besides hard binary weighting, we also propose a soft weighting scheme. Experiments in the Mujoco demonstrate the proposed method outperforms other GAIL-based methods when dealing with imperfect demonstrations.

TrajGAIL: Generating urban vehicle trajectories using generative adversarial imitation learning

Deterministic generative adversarial imitation learning

Imitation learning (IL) is a popular paradigm for training policies in robotic systems when specifying the reward function is difficult. However, despite the success of IL algorithms, they impose the somewhat unrealistic requirement that the expert demonstrations must come from the same domain in which a new imitator policy is to be learned. We consider a practical setting, where (i) state-only expert demonstrations from the real (deployment) environment are given to the learner, (ii) the imitation learner policy is trained in a simulation (training) environment whose transition dynamics is slightly different from the real environment, and (iii) the learner does not have any access to the real environment during the training phase beyond the batch of demonstrations given. Most of the current IL methods, such as generative adversarial imitation learning and its state-only variants, fail to imitate the optimal expert behavior under the above setting. By leveraging insights from the Robust reinforcement learning (RL) literature and building on recent adversarial imitation approaches, we propose a robust IL algorithm to learn policies that can effectively transfer to the real environment without fine-tuning. Furthermore, we empirically demonstrate on continuous-control benchmarks that our method outperforms the state-of-the-art state-only IL method in terms of the zero-shot transfer performance in the real environment and robust performance under different testing conditions.

The fuel economy of hybrid electric vehicles is inextricably linked to the energy management strategy (EMS). In this study, a practicality-oriented learning-based EMS for a power-split hybrid electric bus (HEB) is presented, which combines the generative adversarial imitation learning (GAIL) and deep reinforcement learning (DRL). Considering the regular and fixed route of the HEB, optimal control samples that are not affected by the cost function can be obtained by the boundary-line dynamic programing (B-DP) method. On this basis, the samples are dynamically fitted using the GAIL method to inverse derive the reward function that can explain the B-DP control behavior. Concurrently, the proximal policy optimization DRL algorithm will regulate the energy distribution of the vehicle in real time and continuously optimize the energy management capability based on the reward feedback from GAIL. Finally, the feasibility and effectiveness of the proposed EMS is verified by simulation. The results show that the proposed strategy exhibits near-optimal control performance under both the China heavy-duty commercial vehicle cycle-bus and China city bus cycle.

Biological evolution produces complexity through genetic variations based on randomness. In conventional communication or language simulation models, genetic variations based on randomness and fitness function rewarding task achievements play an important role in evolving communication signals to complex ones. However, it is known that not only genetic variations evolve communication but also imitative learning during developmental processes contributes to the evolution of communication. What we investigated here was to find a different principle of generating complexity which does not rely on the randomness or external environmental complexity but only on the learning processes in communication. Our hypothesis is that the contradictory learning mechanism we call the adversarial imitation learning can work to increase the complexity without relying on the random processes. To investigate our hypothesis, we implemented the adversarial imitation learning on a simulation where two agents interact with and imitate each other. Our results showed that the adversarial imitation learning causes chaotic dynamics and investigating the learning results in different types of interaction between the two; it was clarified that the adversarial imitation learning is necessary for the emergence of the chaotic time series.

Prior research has extensively examined supervised and unsupervised methods for search tasks, focusing primarily on individual model accuracy and performance. However, there’s a gap in understanding how these models collaborate during complex tasks. This study’s primary objective was to evaluate a simulated robot employing Proximal Policy Optimization (PPO) and another using Generative Adversarial Imitation Learning (GAIL) algorithms, working together for a search-and-retrieve task. The environment, built in Unity3D, contained 56 distractors (e.g., walls, tables, and chairs) with negative points and 112 targets (e.g., pistol, laptop, and armour) with positive points during the gameplay. A PPO robot searched for the targets within an environment with a GAIL robot player. The results demonstrated that the inclusion of the GAIL robot along with the PPO robot led to superior performance of the multi-robot team in the search-and-retrieve task. The GAIL robot outperformed the PPO robot when both players performed individually in the search-and-retrieve task. The PPO robots demonstrated comparatively poorer performance when performing alone in a task without GAIL players. Our findings highlight the importance of collaborative multi-robot search involving generative imitation learning within a simulated environment.

With the development of artificial intelligence technology, deep learning methods with big data analysis capabilities provide great potential for the safe operation of power grids. In this paper, based on the combined load data of wind, light and electricity in a northwest region of China, the generator network formulates the unit output plan after the fault occurs. Based on the information of actual power load and actual new energy output before and after the fault, aiming at minimizing the system power generation cost, and considering the constraints of safe operation of the system, an expert system generation fault recovery strategy for this grid fault is formed. It is found that the fault recovery strategy in the early stage of training is quite different from the fault recovery strategy generated by expert system, and the error value is large. With the complete training of the generative adversarial network, the fault recovery auxiliary decision-making scheme approaching the fault recovery expert system can be given in various load and new energy output scenarios, and the error between the two is maintained within 5 %. The research on power grid fault recovery strategy based on generative adversarial imitation learning network model shows the autonomous and safe fault recovery ability of force control system.

Addressing the pain points of difficulty in obtaining expert strategies and insufficient generalisation of imitation learning in dynamic industrial edge environments, this paper proposes computational task offloading framework driven by multimodal generative adversarial imitation learning (GAIL-TO). GAIL-TO learns optimal strategy features from suboptimal historical logs through the adversarial mechanism of generator-discriminator, designs a spatiotemporal feature fusion encoder; constructs a lightweight adversarial training architecture, adapting to resource-constrained devices; develops an edge collaborative training mechanism, utilising server computing power to perform adversarial training, with only lightweight generators deployed at the terminal. It demonstrates superior performance compared to standard IL and MADDPG.

We present a signal and thermal integrity (SI/TI) cooptimization framework for multi-chiplet placement and routing using generative adversarial imitation learning (GAIL). Design of modern 2.5-D integrated circuits face the critical challenge simultaneously optimizing TI and SI. GAIL goes beyond simply mimicking expert strategies by learning policies that generate consistent optimal behaviors across diverse state distributions, making it particularly well-suited for chiplet placement problems requiring simultaneous consideration of conflicting TI and SI objectives. Our approach leverages GAIL to learn placement strategies that effectively balance these competing objectives, incorporating both eye diagram-based signal quality metrics with turn penalties and power-aware thermal modeling. Experimental results demonstrate that our framework successfully achieves balanced optimization between thermal and SI constraints, enabling adaptive placement decisions for heterogeneous chiplet systems without requiring objective-specific retraining.

Automated sperm sample analysis using computer vision techniques has gained increasing interest due to the tedious and time-consuming nature of manual evaluation. Deep learning models have been applied for sperm detection, tracking, motility analysis, and morphology recognition. However, the lack of labeled data hinders their adoption in laboratories. In this work, we propose a method to generate synthetic spermatozoa video sequences using Generative Adversarial Imitation Learning (GAIL). Our approach uses a parametric model based on Bezier splines to generate frames of a single spermatozoon. We evaluate our method against U-net and GAN-based approaches, and demonstrate its superior performance.

Recent studies have shown that reinforcement learning (RL) can provide state-of-the-art performance at learning sophisticated heuristics by exploiting the shared internal structure combinatorial optimization instances in the data. However, existing RL-based methods require too much trial-and-error reliant on sophisticated reward engineering, which is laborious and inefficient for practical applications. This paper proposes a novel framework (RAIL) that combines RL and generative adversarial imitation learning (GAIL) to meet the challenge by searching in branch-and-bound algorithms. RAIL has a policy architecture with dual decoders, corresponding to the sequence decoding of RL and the edge decoding of GAIL, respectively. The two complement each other and restrict each other to improve the learned policy and reward function iteratively.