Active control for enhancing vortex induced vibration of a circular cylinder based on deep reinforcement learning

We present the first application of an artificial neural network trained through a deep reinforcement learning agent to perform active flow control. It is shown that, in a two-dimensional simulation of the Kármán vortex street at moderate Reynolds number ($Re=100$), our artificial neural network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder. By interacting with the unsteady wake, the artificial neural network successfully stabilizes the vortex alley and reduces drag by approximately 8 %. This is performed while using small mass flow rates for the actuation, of the order of 0.5 % of the mass flow rate intersecting the cylinder cross-section once a new pseudo-periodic shedding regime is found. This opens the way to a new class of methods for performing active flow control.

The present study applies a Deep Reinforcement Learning (DRL) algorithm to Active Flow Control (AFC) of a two-dimensional flow around a confined square cylinder. Specifically, the Soft Actor-Critic (SAC) algorithm is employed to modulate the flow of a pair of synthetic jets placed on the upper and lower surfaces of the confined squared cylinder in flow configurations characterized by Re of 100, 200, 300, and 400. The investigation starts with an analysis of the baseline flow in the absence of active control. It is observed that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics. Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features. With the application of the SAC algorithm, we demonstrate the capability of DRL-based control in effectively suppressing vortex shedding, while significantly diminishing drag and fluctuations in lift. Quantitatively, the data-driven active control strategy results in a drag reduction of approximately 14.4%, 26.4%, 38.9%, and 47.0% for Re = 100, 200, 300, and 400, respectively. To understand the underlying control mechanism, we also present detailed flow field comparisons, which showcase the adaptability of DRL in devising distinct control strategies tailored to the dynamic conditions at varying Re. These findings substantiate the ability of DRL to control chaotic, multi-frequency dominated vortex shedding phenomena, underscoring the robustness of DRL in complex AFC problems.

This study employs deep reinforcement learning for active flow control in a turbulent flow field of high Reynolds numbers at Re = 274 000. That is, an agent is trained to obtain a control strategy that can reduce the drag of a cylinder while also minimizing the oscillations of the lift. Probes are placed only around the surface of the cylinder, and a proximal policy optimization (PPO) agent controls nine zero-net mass flux jets on the downstream side of the cylinder. The trained PPO agent effectively reduces drag by 29% and decreases lift oscillations by 18% of amplitude, with the control effect demonstrating good repeatability. Control tests of this agent within the Reynolds number range of Re = 260 000 to 288 000 show that the agent's control strategy possesses a certain degree of robustness, with very similar drag reduction effects under different Reynolds numbers. Analysis using power spectral energy reveals that the agent learns specific flow frequencies in the flow field and effectively suppresses low-frequency, large-scale structures. Graphically visualizing the policy, combined with pressure, vorticity, and turbulent kinetic energy contours, reveals the mechanism by which jets achieve drag reduction by influencing reattachment vortices. This study successfully implements robust active flow control in realistically significant high Reynolds number turbulent flows, minimizing time costs (using two-dimensional geometrical models and turbulence models) and maximally considering the feasibility of future experimental implementation.

Fatigue life calculations have been made for suspended spans of offshore pipelines exposed to steady flow and undergoing vortex-induced vibrations. The effect of the proximity of the sea bottom was taken into account by utilizing the results of related model tests. It was found that the effect of the sea-bottom proximity is to increase the fatigue life of a suspended span of given length or equivalently to increase the safe length of a suspended span for a given fatigue life. Furthermore, the effect of the pipe-to-wall gap to pipe diameter ratio on the fatigue life of such spans has been determined. INTRODUCTION A steady flow about a bluff body, such as a pipeline with a circular cross section, will separate and form a wake. Because of the instability and mutual interaction of the separated shear layers, vortices form and periodically shed from alternate sides of the body. The result is steady and unsteady drag forces in line with the flow and unsteady lift forces? in the transverse, or cross flow direction. If the pipeline is flexible and/or flexibly supported, it begins to vibrate under the action of these periodic forces. The vibrations of the pipeline modify the flow and lead to a nonlinear interaction between the elastic and the fluid systems. The maximum amplitude of inline vortex-induced vibrations is much smaller than the maximum amplitude of transverse vortex induced vibrations, so their contribution to the fatigue life of a pipeline is small. Consequently, attention is focused on the transverse vibrations which will be referred to as "vortex induced vibrations" hereafter. In the last ten years a number of comprehensive overviews of this important class of fluid-structure interaction problems have appeared [1-4]. Experience with unburied offshore pipelines laid in strong current areas has shown that such pipelines may develop unsupported spans due to sea bottom being scoured out from under the pipe due to current action. In turn such unsupported spans exposed to currents may undergo vortex induced vibrations which may affect the fatigue life of the pipeline. Due to lack of data for the vortex-induced vibrations of a flexible cylinder in close proximity to a plane boundary, fatigue life calculations of suspended pipeline spans that determine the maximum safe length of such spans have been based, out of necessity, on data for the vortex-induced vibrations of an isolated flexible cylinder, i.e., no wall effect. In a recent experimental study, Tsahalis and Jones [5] determined the effect of the proximity of a plane boundary on the vortex-induced vibrations of a flexible pipe. They found that the proximity of the plane boundary has the following effects on the response of the pipe as compared to the response of the isolated pipe, i.e., no wall effect (see also Figure 1). First, the increase of the response frequency with the flow velocity is less steep. Second, the first perceptible vibrations take place at a higher flow velocity. Fourth, the maximum amplitude is achieved at a higher flow velocity. Fifth, the maximum amplitude is reduced. Sixth, once the maximum amplitude is achieved it appears that it remains constant for higher flow velocities.

The vortex-induced vibration (VIV) is a classical problem in ocean engineering. Intensive research on this field for flow around a circular cylinder has been observed, due to practical application, mainly the design of risers, cables and pipelines with free span. The relevance of this phenomenon is related to the structure failure, consequence of large displacement or fatigue. In the present study the influence of initial condition on the vortex induced vibration (VIV) of a circular cylinder with two degree of freedom is investigated by the numerical solution of the slightly compressible formulation of Reynolds Average Navier-Stokes equations. An upwind and Total Variation Diminishing (TVD) conservative scheme is used to solve the governing equations written in curvilinear coordinates. The k–ε turbulence model is used to simulate the turbulent flow in the wake of the cylinder. Two different initial conditions have been tested, free-stream and continuous reduced velocity increase (using the previous reduced velocity as initial condition for the next value). Results for the phase angle, amplitude, frequency, and lift coefficient are presented. The numerical results have been compared with experimental data of Jauvtis and Williamson [1]. The results indicate that the history of cylinder movement has a important impact in the amplitude oscillation observed in-line and cross-flow, principally in the reduced velocity range associated with the upper branch. Results obtained for the initial and lower branch seems to be independent of the initial condition. Further investigation are necessary to understand the difference observed such as the absence of the jump in the cross-flow oscillation between the initial and upper branch and the absence of in-line oscillation for reduced velocity in the range of 1–4 and the peak of in-line oscillation at reduced velocity 6.0.

In late 2003 BP contracted 2H to manage and run a series of forced oscillation tests at the Marin test basin, Netherlands. The tests examined the effect of vortex induced vibration (VIV) and galloping response on a full size section model of a single pipe riser and a non-symmetric riser bundle. The tests used a 200mm diameter pipe at a Reynolds number of 39,600. This paper presents the results (lift and added mass coefficient with increasing amplitude and reduced velocity) from the single pipe forced oscillation tests. A method is developed to convert the test data into parabolic lift curves and damping coefficients for Shear7 VIV analysis. The parabolic life curves created from the single pipe forced oscillation test data are shown. The method is then extended for use with the test data from the non-symmetric riser bundle forced oscillation tests, and accounts for differences in peak reduced velocity in the test data by calculating effective vortex shedding diameters. Sensitivity analysis is conducted using the parabolic lift curves with Shear7 on an example deepwater riser. This shows that the fatigue lives predicted using the parabolic lift curves are generally lower than the default Shear7 lift curves. Introduction Forced oscillation VIV tests on a 200mm diameter single rough pipe were conducted using the MARIN forced oscillation test rig. Further tests were conducted using a nonsymmetric riser bundle. 2H Offshore worked with BP and MARIN to establish a strategy for both single pipe and the non-symmetric riser bundle tests. 2H managed the tests for BP and provided post analysis. The objectives of this paper is to present:Lift coefficients obtained from the large scale single rough pipe testsParabolic lift curves created from the single rough pipe test data.The method used to convert test data into lift curves accounting for the non-symmetric riser bundle.Sensitivity of VIV fatigue life to the different lift curves. Further tests using the single pipe at a range of critical and trans-critical Reynolds numbers with and without the rough coating were conducted. These are not detailed in this paper. In this paper the term 'lift coefficient' refers to the lift coefficient in phase with velocity, CLV, which indicates the flow induced lift (vortex induced lift) as opposed to the lift coefficient in phase with acceleration. Forced Oscillation Tests for VIV Forces Forced Oscillation Test Rig The tests were conducted using a 200mm diameter × 3.4m long aluminium pipe that was artificially roughened by coating the pipe with 500 micron sand grains. The roughness is greater than may be seen on real risers, but was chosen to provide a definite rough pipe benchmark. Moreover ocean conditions would be more turbulent than would be the case in the tank, and the higher roughness would make up for this by forcing laminar-turbulent boundary layer transition. The test rig used was able to test in both forwards and reverse directions at speeds of up to 2m/s.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

A computational study was performed of a flow-excited Helmholtz resonator. This setup was used to model the phenomenon that occurs when a sunroof is open, or rear passenger compartment window lowered, in an automobile. The results of the computational study were validated through comparison to experimental and numerical results found in literature. After validating the setup of the computational study, the research then focused on using methods of active flow control to reduce the sound pressure levels and fluctuations within the resonator. Specifically, this was accomplished through the use of a small elongated jet placed upstream of the resonator opening. After seeing changes in the pressure levels using this active flow control technique, the velocity of the jet was altered to study its effect on the level of pressure reduction. The final results from this computational study show the desired outcome of a reduction in pressure and it was concluded that air from the jet that is added to the boundary layer produces an effect similar to that of running the simulation at a higher free stream velocity, but without active flow control. Thus, using the jet for active flow control will provide the ability to reduce pressure levels across a spectrum of free stream velocities.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

This paper discusses an autonomous driving control strategy based on Deep Reinforcement Learning (DRL), which aims to improve the decision-making ability of autonomous driving system in complex traffic environments. Deep reinforcement learning has a wide range of applications in many fields, such as robotics and medicine. Autonomous driving has emerged as a significant research focus in recent years. By combining deep learning and reinforcement learning, the model is able to autonomously learn and optimize driving behavior under dynamically changing road conditions. The DRL-based control strategy performs well in vehicle obstacle avoidance, pedestrian recognition, and traffic rule compliance in the face of complex environments such as city streets, intersections, and congested road sections, significantly improving the safety and efficiency of autonomous driving. This article will first introduce deep reinforcement learning. Then, the autonomous driving control strategy based on deep reinforcement learning is introduced. This research provides valuable insights for developing and implementing DRL-based autonomous driving systems.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

Numerical simulations of vortex-induced vibration of a three-dimensional flexible Cylinder under uniform turbulent flow are calculated when Reynolds number is 1.35×104. In order to achieve the vortex-induced vibration, the three-dimensional unsteady, viscous, incompressible Navier-Stokes equation and LES turbulence model are solved with the finite volume approach, the Cylinder is discretized according to the finite element theory, and its dynamic equilibrium equations are solved by the Newmark method. The fluid-Cylinder interaction is realized by utilizing the diffusion-based smooth dynamic mesh method. Considering VIV system, the variety trends of lift coefficient, drag coefficient, displacement, vertex shedding frequency, phase difference angle of Cylinder are analyzed under different frequency ratios. The nonlinear phenomena of locked-in, phase-switch are captured successfully. Meanwhile, the limit cycle and bifurcation of lift coefficient and displacement are analyzed using trajectory, phase portrait and Poincare sections. The results reveal that: when drag coefficient reaches its minimum value, the transverse amplitude reaches its maximum and the “lock-in” begins simultaneously. In the range of “lock-in”, amplitude decreases gradually with increasing of frequency ratio. When lift coefficient reaches its minimum value, the phase difference, between lift coefficient and lateral displacement, undergoes a suddenly change from the “out-of-phase” to the “in-phase” mode. There is no bifurcation of lift coefficient and lateral displacement occurred in three dimensional flexible Cylinder submitted to uniform turbulent flow.