Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Path planning in an unknown environment is a basic task for mobile robots to complete tasks. As a typical deep reinforcement learning, deep Q-network (DQN) algorithm has gained wide popularity in path planning tasks due to its self-learning and adaptability to complex environment. However, most of path planning algorithms based on DQN spend plenty of time for model training and the learned model policy depends only on the information observed by sensors. It will cause poor generalization capability for the new task and time waste for model retraining. Therefore, a new deep reinforcement learning method combining DQN with prior knowledge is proposed to reduce training time and enhance generalization capability. In this method, a fuzzy logic controller is designed to avoid the obstacles and help the robot avoid blind exploration for reducing the training time. A target-driven approach is used to address the lack of generalization, in which the learned policy depends on the fusion of observed information and target information. Extensive experiments show that the proposed algorithm converges faster than DQN algorithm in path planning tasks and the target can be reached without retraining when the path planning task changes.

In this paper, we investigate the user pairing and power allocation scheme for multiple cellular users (CUs) under the downlink non-orthogonal multiple access (NOMA) system. To maximize the sum-rate of all CUs, we formulate the resource allocation issue through optimizing the user pairing relationship and power allocation method. However, due to the nonconvex property of the formulated problem, the original problem is decoupled into two subproblems. First, the optimal power allocation scheme with a given subchannel assignment is obtained via a closed-form solution. Furthermore, based on the obtained optimal allocation scheme, a classical deep reinforcement learning (DRL) method called Deep Q-Network (DQN) algorithm is adopted to find the optimal user pairing scheme, where the DQN algorithm is characterized by higher learning efficiency and better performance of the features extraction ability compared with traditional reinforcement learning (RL) schemes. Simulation results validate the effectiveness of our proposed resource allocation, as compared against the RL based scheme and conventional orthogonal multiple access (OMA) method.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

As the basic problem of unmanned vehicle navigation control, path planning has been widely studied. Reinforcement learning (RL) has been found an effective way of path optimization for the highly nonlinear and unmodeled dynamics. However, the RL based methods suffer from the "dimension disaster" under the high-dimension state spaces. In this paper, the path planning of an unmanned vehicle with collision avoidance is considered, and an improved Deep Q-Network (DQN) algorithm is proposed to reduce the computation load in the high-dimension state space. First, the states, actions and rewards are determined based on the task requirement, and a smoothing function is defined as an additional penalty term to modify the basic reward function. Then, the two-dimension grid of the state space is mapped to a gray image, which is applied as the input of a neural network, i.e., the Q-Network. Finally, simulation results show that the modified DQN algorithm is more stable and the fluctuation frequency is significantly reduced.

Numerous studies conducted on crowd evacuation have illustrated the effectiveness of using evacuation guidance systems with visual evacuation signage. The efficient arrangement of signage on the premises is crucial for the optimal use of these systems; however, to determine the optimal assignment of visual signs is challenging because of several factors, such as complex crowd behaviors, multiple conflicting objectives in evacuations, randomness, and situation uncertainties. Blackbox optimization techniques with multiagent simulations as objective functions have been applied to evaluate the outcome of visual sign arrangements in evacuation studies. They explore an efficient arrangement of evacuation signs achieving a swift and safe crowd evacuation. However, the randomness and complexity involved in the problem causes significant variances in the outcomes of the solutions, increasing the risk that the satisfactory solution explored during the design phase may result in significantly different severe consequences during actual implementation. Here, we formulated this problem as a stochastic multi-objective optimization problem with two objectives: maximizing the number of agents who select the correct exit, and minimizing the total evacuation time. The average value at risk (AVaR) used as a risk measure in finance and economics was employed to evaluate the risks involved in the solutions. NSGA-II algorithms incorporating simulations with the evacuation decision model have been used to explor solutions satisfying both expected values and AVaRs.KeywordsEvolutionary multi-objective optimizationBlack-box optimizationVisual evacuation signage assignment problemAverage values at riskNSGA-II

To solve problems involving sequential decision-making, Deep Reinforcement Learning (DRL) combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL).These fusions are skilled at training agents for video games because they use neural networks to approximate nonlinear functions.The Deep Q Network (DQN), a key algorithm in this field, tends to overestimate Q-values despite its effectiveness.The Double Deep Q Network (Double DQN) was presented to address this.The Kera's framework in PyCharm is used in this study to examine the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game.To speed up its training, improvements were also made to the DQN model.The enhanced DQN performed better than the traditional DQN but less well than the Double DQN, according to the results.This study delves into the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game, utilizing the Kera's framework in PyCharm.Additionally, enhancements were made to the DQN model to expedite its training.Results affirmed that the enhanced DQN outperformed the conventional DQN but lagged the Double DQN.The training loss trajectory further substantiated Double DQN's superiority in mitigating overestimation issues.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

In order to better understand state-of-the-art reinforcement learning agent, deep Q-network, a brief review of reinforcement learning and Q-learning are first described. Then recent advances of deep Q-network are presented, and double deep Q-network and dueling deep Q-network that go beyond deep Q-network are also given.

The Micro-Electro-Mechanical Systems Fabry-Perot (MEMS-FP) filter is a pivotal photosensitive component within optical instrumentation, serving an integral role across various optical applications, including frequency selection spectroscopy and spectral analysis. The advent of MEMS-FP technology has significantly contributed to advancements in the optical field, underscoring the necessity for precise pre-fabrication theoretical predictions of device properties through finite element analysis (FEA). Nonetheless, the complexity inherent in three-dimensional modelling of MEMS-FP structures, coupled with the intricate nature of finite element simulations, presents substantial challenges that impede progress in MEMS-FP research. To navigate these challenges, this paper introduces a novel approach employing reinforcement learning (RL) for the automated design of MEMS-FP devices. This method encompasses the development of a Deep Q-Network (DQN) algorithm, the formulation of a tailored reward function, and an innovative strategy for updating the status of MEMS-FP devices. Various designs schemes of MEMS-FP devices can be acquired using MEMS-FP state search. The disparity between the MEMS-FP device design scheme and performance index can be assessed using a reward function. The DQN algorithm is crafted to update the network responsible for predicting Q-values. The agent trained by our method can rapidly calculate the MEMS-FP device design scheme that fulfils a specified performance index when provided with the performance index of the MEMS-FP device. By amalgamating reinforcement learning with MEMS-FP device research, this approach adeptly identifies optimal design configurations that meet specified performance criteria with enhanced precision and efficiency and promote the advancement of MEMS-FP devices.

In this paper, an underlay cognitive radio network with energy harvesting is considered which works in slotted mode. SU decides to transmit data or harvest energy from environment according to the available energy of the battery and the channel state in each slot. Considering the highly dynamic characteristics of channel occupancy, channel gain and energy arrival, a deep Q-network (DQN) algorithm in deep reinforcement learning is proposed. By setting the corresponding reward function under different channel occupancy states, the agent can select the appropriate working mode and value of transmit power according to the change of cognitive radio environment. After a period of learning, agent obtains the optimal strategy in finite time slots. Simulation results show that the proposed method can converge and perform better than other baseline strategies.

Against the backdrop of the rapid development of big data technology, the phenomenon of price discrimination, colloquially known as "big data-enabled price gouging," has sparked widespread social concern and a crisis of consumer trust. To investigate this issue and seek solutions, this study combines traditional empirical analysis with cutting-edge machine learning simulation. First, through questionnaire surveys and regression analysis, and based on equity theory and price sensitivity theory, we verify the direct negative impact of big data-driven price discrimination on customer loyalty, revealing the mediating role of perceived unfairness and the moderating role of price sensitivity. Second, based on these empirical results, we construct a high-fidelity market simulation environment and employ the Deep Q-Network (DQN) algorithm from Deep Reinforcement Learning (DRL) to train a dynamic pricing agent. The agent's objective is to maximize long-term profit while maintaining customer loyalty. The simulation results show that the trained reinforcement learning strategy significantly outperforms both myopic greedy pricing and fixed-price strategies in terms of total revenue and customer retention. The final policy learned by the agent confirms that to maintain long-term customer relationships, it is necessary to offer discounts to price-sensitive, low-loyalty users while adopting moderate pricing for high-loyalty users, thereby effectively validating our initial theoretical hypotheses. This study not only reveals the internal mechanism through which price discrimination affects customer loyalty but also provides a scientific basis and practical path for enterprises to optimize pricing and achieve sustainable development using advanced algorithms.

Antenna selection is often used for physical layer security to implement secure communications. However, due to the rapid changes of the main channel and the feedback delay of the channel state information (CSI), the transmitter obtains outdated CSI, and the outdated CSI leads to the outdated optimal transmit antenna. In order to improve the security of the system based on outdated CSI, in this paper, we propose a deep reinforcement learning framework of Deep Q Network (DQN) to predict the optimal transmit antenna in the multiple input multiple output (MIMO) wiretap channel. The legitimate receiver receives the pilot signals from each transmitting antenna, and the signal-to-noise ratio (SNR) of the pilot signals transmitted by each transmitting antenna can be obtained through maximal ratio combining. And then the legitimate receiver uses the DQN to predict the transmitting antenna at the next moment according to these SNRs. The simulation results show that DQN algorithm can effectively predict the optimal antenna at the next moment, and reduce the secrecy outage probability of MIMO wiretap system, compared with the traditional algorithm.

Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment. Reinforcement Learning methods enable a state action function in mobile robots suited to their environment. During trial-and-error interaction with its surroundings, it helps a robot to find an ideal behavior on its own. The Deep Q Network (DQN) algorithm is used in TurtleBot 3 (TB3) to achieve the goal by successfully avoiding the obstacles. But it requires a large number of training iterations. This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field (APF) algorithms. First, a TB3 Waffle Pi DQN is built and trained to reach the goal. Then the APF shortest path algorithm is incorporated into the DQN algorithm. The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System (ROS). The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algorithm. The performance improvement rate of the proposed DQN + APF in comparison with DQN in terms of the number of successful targets is attained by 88%. The performance of the proposed DQN + APF in comparison with DQN in terms of average time is achieved by 0.331 s. The performance of the proposed DQN + APF in comparison with DQN average rewards in which the positive goal is attained by 85% and the negative goal is attained by −90%.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.