Deep Reinforcement Learning on FPGA for Self-Healing Cryogenic Power Amplifier Control

With the high interest in digital modulation techniques which are very sensitive to the PA nonlinearity, modern wireless communication systems require the usage of linearization techniques to improve the linear behavior of the RF power amplifier. The powerful and cheap digital processing technology makes the digital predistortion (DPD) a competitive candidate for the linearization of the PA. This thesis introduces the basic principle of DPD, its implementation on FPGA and the adaptive DPD system. The linearization of 4 PAs with DPD technique has been introduced: for the hybrid class AB PA operating at 2.6 GHz with a WiMAX testing signal, 33.7 dBm average power, 29.6 % drain efficiency, 13 dB ACPR and 9 dB NMSE improvement have been obtained; for the hybrid Doherty PA operating at 3.4 GHz with an I/Q testing signal, 35.0 dBm average power, 36.8 % drain efficiency, 12 dB ACPR and 13 dB NMSE improvement have been obtained; for the MMIC class AB PA operating at 7 GHz with an I/Q testing signal, 29.4 dBm average power, 25.7 % drain efficiency, 12 dB ACPR and 12 dB NMSE improvement have been obtained; for the two-stage PA operating at 24 GHz with an I/Q testing signal, 23.5 dBm average power, more than 14.0 % drain efficiency, 11 dB ACPR and 11 dB NMSE improvement have been obtained. The DPD algorithm has been implemented on FPGA with two methods based on LUT and a direct structure with only adders and multipliers. The block RAM on the FPGA board is chosen as the table in the LUT methods. The linearization performance for these three methods is similar. The test PA is the hybrid Doherty PA mentioned above and the test signal is the I/Q signal with 7.4 dB PAPR. 35.1 dBm average power, 36.8 % efficiency, 11 dB ACPR and 11 dB NMSE improvement have been obtained. The cost of logic resources for the direct structure method is the largest with 1,172 flip-flops, while the number of flip-flops for the two LUT methods are 263 and 583, respectively. A new adaptive algorithm has been proposed in this thesis for the adaptive DPD system. This new algorithm improves the performance in extracting the model parameters in complex number domain. With the experimental data from a combined class AB PA, the final accuracy of the model extracted by the new algorithm has been improved from -20 dB to about -40 dB and the converge speed is faster

The drives of power amplifiers of magnetic levitation servo spindle applied to micro EDM (Electron discharge machining) based on FPGA (Field programmable gate array) are developed. The drive module that produces the PWM (Pulse width modulation) waves that conform to the requirements of spindle power amplifiers involving three-level half bridge switch power amplifier, three-level full bridge switch power amplifier and motor power amplifier, according to input of action values from magnetic levitation servo spindle controller DSP (Digital signal processor) is designed. Multi-PWM producing algorithm by single timer with multi comparing nodes is proposed. After being applied to control system of magnetic levitation servo spindle, the drives accomplish conversion from DSP action values to PWM waves. The PWMs that drive radial coil current power amplifier have phase difference of 180°. The PWMs that drive axial coil current power amplifier have dead zone time and their wave shapes are complementary. The PWMs that drive the motor power amplifier conform to requirements of power amplifier chip LMD18200. Using FPGA to realize the drive function of spindle power amplifier makes the drive design more flexible, resource distribution of DSP more optimized, the task of algorithms running more centralized, and the drives module gets simpler and more reliable because the phase shift circuit and dead zone circuit are cancelled but the function of phase shift and dead zone is reserved.

Reinforcement learning (RL), a promising framework for data-driven decision making in an uncertain environment, has successfully been applied in many real-world operation and control problems. However, the application of RL in a large-scale decentralized multi-agent environment remains a challenging problem due to the partial observability and limited communications between agents. In this paper, we develop a model-based kernel RL approach and a model-free deep RL approach for learning a decentralized, shared policy among homogeneous agents. By leveraging the strengths of both these methods, we further propose a novel deep ensemble multi-agent reinforcement learning (MARL) method that efficiently learns to arbitrate between the decisions of the local kernel-based RL model and the wider-reaching deep RL model. We validate the proposed deep ensemble method on a highly challenging real-world air traffic control problem, where the goal is to provide effective guidance to aircraft to avoid air traffic congestion, conflicting situations, and to improve arrival timeliness, by dynamically recommending adjustments of aircraft speeds in real-time. Extensive empirical results from an open-source air traffic management simulation model, developed by Eurocontrol and built on a real-world data set including thousands of aircrafts, demonstrate that our proposed deep ensemble MARL method significantly outperforms three state-of-the-art benchmark approaches.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

Most existing behavioral models of the radio frequency (RF) power amplifiers assume that power amplifiers are memoryless devices. But in broadband communication systems, memory effects of the RF power amplifiers are significantly observed. The traditional memoryless models can no longer accurately depict the input-output relationship of power amplifiers. This paper investigates power amplifiers memory effects and a pre-distortion method is proposed for linearizing RF power amplifiers with memory effects. A valid behavioral model based on the Volterra series is suggested to treat memory effects. Since the characteristics of amplifiers change during transmission time, a recursive least squares algorithm with size- fixed observation matrices is developed to update the parameters of the proposed power amplifiers model. The identification algorithm can decrease computational complexity and data storage space and facilitate real-time online identification. Simultaneously, the pre-distortion model is constructed to simulate. Simulation results show that the proposed method can effectively compensate for the nonlinear distortion and memory effects of RF power amplifiers. The model's accuracy is used to evaluate power amplifiers memory effects by the enormalized mean square error (NMSE). In the end, simulation and evaluation of this pre-distortion system proceeded, and the power spectral density estimation was also applied to calculate the adjacent channel power ratio.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

The need for self-adaptive systems is increasing with the growth in automation to improve accuracy and performance. Such self-adaptive systems can be built using Reinforcement Learning (RL) models with the help of simulated and recorded data. Deep Reinforcement Learning (DRL) is proposed by infusing NN into RL models and can be used to improve performance in use cases with sparse reward distributions. Traditional Actor-Critic DRL models act with a single critic and single actor network in case of on-policy execution. In contrast, the performance of the agent which makes decisions in the environment improves when considering the information exchange from multiple critics. In this work, the proposed Teacher Tutor Trainee a version of Multi Critic architecture provides the impact of information exchange from multiple critics for a channel estimation use case. Channel Estimation is a method of calculating the impact of the propagation medium and other external factors that deteriorate a signal and its optimization. The performance improvement can be observed from the increase in the explained variance. On implementation, it is observed that the Proximal Policy Optimization model has shown better performance than the Advantage Actor-Critic model.

Intrusion detection has been a vast-surveyed topic for many decades as network attacks are tremendously growing. This has heightened the need for security in networks as web-based communication systems are advanced nowadays. The proposed work introduces an intelligent semi-supervised intrusion detection system based on different algorithms to classify the network attacks accurately. Initially, the pre-processing is accomplished using null value dropping and standard scaler normalization. After pre-processing, an enhanced Deep Reinforcement Learning (EDRL) model is employed to extract high-level representations and learn complex patterns from data by means of interaction with the environment. The enhancement of deep reinforcement learning is made by associating a deep autoencoder (AE) and an improved flamingo search algorithm (IFSA) to approximate the Q-function and optimal policy selection. After feature representations, a support vector machine (SVM) classifier, which discriminates the input into normal and attack instances, is employed for classification. The presented model is simulated in the Python platform and evaluated using the UNSW-NB15, CICIDS2017, and NSL-KDD datasets. The overall classification accuracy is 99.6%, 99.93%, and 99.42% using UNSW-NB15, CICIDS2017, and NSL-KDD datasets, which is higher than the existing detection frameworks.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

Reinforcement Learning (RL) is a successful and established Artificial Intelligence (AI) method, particularly with recent groundbreaking progress in Deep Reinforcement Learning (DRL). Reinforcement learning is very well suited for sequential decision-making tasks, wherein a learned agent learns an optimal policy after many interactions with an environment. The present paper examines the application of reinforcement learning for automating screening of literature in academic research, particularly in the fields of computer science and e-learning. Keyword filtering techniques, while predominantly applied, are found to be inflexible as well as unable to capture the dynamic nature of research themes. To overcome such constraints, we present a Deep Q-Network (DQN)-based reinforcement learning model that combines reinforcement learning with the Semantic Scholar API to enhance research paper classification based on dynamically acquired decision rules. The proposed reinforcement learning model was trained and tested with a dataset of 8,934 research papers, accessed by systematic searching. The agent filtered and picked 11 effective papers depending on improved selection criteria like publication date, keyword relevance, and scholarly topic provided. The model iteratively optimizes the decision-making process through reward-based learning and therefore maximizes categorization accuracy over time. Test experiments demonstrate utilization of RL-based suggested framework yields classification accuracy at 91.5%, recall at 86.3%, and precision at 89.7%. A comparison test demonstrates that the approach performs 12.5% better on recall and 9.8% better on accuracy compared to traditional keyword-filtering methods. The finding confirms the power of the model in minimizing false positives and false negatives for screening literature, hence proving the scalability and adaptability of reinforcement learning in managing high academic data. This work offers a scalable, cognitive approach to conducting systematic reviews of literature through the application of reinforcement learning to programmatically execute work in academic research. The work shows the promise of reinforcement learning to further enhance research methodology, make literature reviews more effective, and facilitate more knowledgeable decision-making in fast-changing scientific disciplines. Further research will be focused on incorporating hybrid AI models with multi-agent systems of reinforcement learning for responsiveness and classification enhancements.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

This paper presents an exhaustive description of a field programmable gate-array (FPGA) based set-up for envelope tracking and dynamic biasing of RF power amplifiers (PA). The system includes digital adaptive predistortion (DAPD) and gain compensation deployed in a FPGA device. For testing purposes a GaN HEMT RF PA operating at 3.5 GHz was considered. Preliminary experimental results are provided showing the DAPD compensate for the steady-state PA nonlinear distortion arising as a consequence of varying the power supply.

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

Aiming at a comprehensive and concise tutorial survey, recap of variational inference and reinforcement learning with Probabilistic Graphical Models are given with detailed derivations. Reviews and comparisons on recent advances in deep reinforcement learning are made from various aspects. We offer detailed derivations to a taxonomy of Probabilistic Graphical Model and Variational Inference methods in deep reinforcement learning, which serves as a complementary material on top of the original contributions.