Artificial Potential Field Incorporated Deep-Q-Network Algorithm for Mobile Robot Path Prediction

Path planning is a key task in mobile robots, and the application of Deep Q Network (DQN) algorithm for mobile robot path planning has become a hotspot and challenge in current research. In order to solve the obstacle avoidance limitations faced by the DQN algorithm in indoor robot path planning, this paper proposes a solution based on an improved DQN algorithm. In view of the low learning efficiency of the DQN algorithm, the Duel DQN structure is introduced to enhance the performance and combined with a Prioritized Experience Replay (PER) mechanism to ensure the stability of the robot during the learning process. In addition, the idea of Munchausen Deep Q Network (M-DQN) is incorporated to guide the robot to learn the optimal policy more effectively. Based on the above improvements, the PER-D2MQN algorithm is proposed in this paper. In order to validate the effectiveness of the proposed algorithm, we conducted multidimensional simulation comparison experiments of the PER-D2MQN algorithm with DQN, Duel DQN, and the existing methodology PMR-DQN in the Gazebo simulation environment and examined the cumulative and average rewards for reaching the goal point, the number of convergent execution steps, and the time consumed by the robot in reaching the goal point. The simulation results show that the PER-D2MQN algorithm obtains the highest reward in both static and complex environments, exhibits the best convergence, and finds the goal point with the lowest average number of steps and the shortest elapsed time.

Path planning is a fundamental task in autonomous mobile robot navigation and one of the most computationally intensive tasks. In this work, a parallel version of the bacterial potential field (BPF) method for path planning in mobile robots is presented. The BPF is a hybrid algorithm, which makes use of a bacterial evolutionary algorithm (BEA) with the artificial potential field (APF) method, to take advantage of intelligent and classical methods. The parallel bacterial potential field (parallel-BPF) algorithm is implemented on a graphics processing unit (GPU) to speed up the path planning computation in mobile robot navigation. Simulation results to validate the analysis and implementation are provided; the experiments were specially designed to show the effectiveness and the efficiency of the parallel-BPF algorithm.

Path planning in an unknown environment is a basic task for mobile robots to complete tasks. As a typical deep reinforcement learning, deep Q-network (DQN) algorithm has gained wide popularity in path planning tasks due to its self-learning and adaptability to complex environment. However, most of path planning algorithms based on DQN spend plenty of time for model training and the learned model policy depends only on the information observed by sensors. It will cause poor generalization capability for the new task and time waste for model retraining. Therefore, a new deep reinforcement learning method combining DQN with prior knowledge is proposed to reduce training time and enhance generalization capability. In this method, a fuzzy logic controller is designed to avoid the obstacles and help the robot avoid blind exploration for reducing the training time. A target-driven approach is used to address the lack of generalization, in which the learned policy depends on the fusion of observed information and target information. Extensive experiments show that the proposed algorithm converges faster than DQN algorithm in path planning tasks and the target can be reached without retraining when the path planning task changes.

Developing an autonomous indoor mobile robot navigation system from scratch is very difficult and it takes a long time. It requires a series of complex tasks such as handling given sensor inputs, calculating all the information needed for autonomous navigation, and controlling actuators required for movement. In this paper, an autonomous navigation system for indoor mobile robots is introduced mainly based on open source provided by the robot operating system. The presented system is capable of autonomously navigating an unstructured indoor environment avoiding collision with static or dynamic objects. To this end, the system consists of three main modules: mapping, localization, and planning. The mapping module builds a global map for an unknown environment by means of a simultaneous localization and mapping algorithm based on laser scanner data. The localization module estimates the mobile robot’s pose within the prebuilt map by way of an adaptive Monte Carlo localization approach. The planning module builds a local cost map for collision avoidance, generates collision-free trajectories to reach a goal pose based on the cost map, and produces control commands to follow the trajectories. The presented system has been tested not only in simulation environments built in the Gazebo simulator but also in real environments utilizing the Jackal mobile robot, to validate its performance for autonomous navigation including collision avoidance.

The obstacle avoidance and navigation are important tasks for a mobile robot in applications such as industry, military, exploration and automated vehicles. This paper presents an implementation of mapping and navigation of autonomous mobile robot using Robot Operating System (ROS) and Orbbec Astra camera. In recent days, Lidar is used for the mapping and navigation problem by sensing the objects which are above floor level by using only a single horizontal scanning line and this may result in inaccurate generation of map leads to collision during autonomous navigation of mobile robots. In order to overcome limitation of Lidar, Orbbec Astra (Kinect) camera is used for mapping of indoor environment which can detect objects which are above and below the floor level and navigation of mobile robot from start to target location without colliding with obstacles. Experimental results show the map of the indoor environment generated using Orbbec Astra camera is matched with real indoor environment, and generated map is tested experimentally to navigate the mobile robot from start to target location without collision with obstacles.KeywordsMobile robotsMappingNavigationOrbbec Astra cameraROS

Autonomous exploration and mapping is an important part of autonomous navigation of mobile robots in an unknown environment. This paper proposes a new 3D exploration strategy based on the wavefront algorithm. RGB-D camera is used to obtain environmental information. The diffusion process of wavefront algorithm is used to find frontier points. Firstly, selection function determines the next frontier point to be explored. Then, the mobile robot moves to the frontier point according to the path planned by the way of wavefront algorithm. When the mobile robot reaches the frontier point and completes the mapping, the mobile robot continues to search for the next frontier point. Finally, the exploration strategy is tested using the Robot Operating System (ROS). Simulational experiment shows that the 3D exploration based on the wavefront algorithm can choose the right frontier point and complete the exploration task quickly.

In the field of intelligent manufacturing, robot grasping and sorting is important content. However, there are some disadvantages in the traditional single-view-based manipulator grasping methods by using a 2D camera, where the efficiency and the accuracy of grasping are both low when facing the scene of stacking and occlusion for the reason that there is information missing by single-view 2D camera-based methods while acquiring scene information, and the methods of grasping only can’t change the difficult-to-grasp scene which is stack and occluded. Regarding the issue above, a pushing-grasping collaborative method based on the deep Q-network in dual viewpoints is proposed in this paper. This method in this paper adopts an improved deep Q-network algorithm, with an RGB-D camera to obtain the information of objects’ RGB images and point clouds from two viewpoints, which solved the problem of lack of information missing. What’s more, it combines the pushing and grasping actions with the deep Q-network, which make it have the ability of active exploration, so that the trained manipulator can make the scenes less stacking and occlusion, and with the help of that, it can perform well in more complicated grasping scenes. In addition, we improved the reward function of the deep Q-network and propose the piecewise reward function to speed up the convergence of the deep Q-network. We trained different models and tried different methods in the V-REP simulation environment, and it drew a conclusion that the method proposed in this paper converges quickly and the success rate of grasping objects in unstructured scenes raises up to 83.5%. Besides, it shows the generalization ability and well performance when novel objects appear in the scenes that the manipulator has never grasped before.

The reinforcement learning problem of complex action control in a multi-player wargame has been a hot research topic in recent years. In this paper, a game system based on turn-based confrontation is designed and implemented with state-of-the-art deep reinforcement learning models. Specifically, we first design a Q-learning algorithm to achieve intelligent decision-making, which is based on the DQN (Deep Q Network) to model complex game behaviors. Then, an a priori knowledge-based algorithm PK-DQN (Prior Knowledge-Deep Q Network) is introduced to improve the DQN algorithm, which accelerates the convergence speed and stability of the algorithm. The experiments demonstrate the correctness of the PK-DQN algorithm, it is validated, and its performance surpasses the conventional DQN algorithm. Furthermore, the PK-DQN algorithm shows effectiveness in defeating the high level of rule-based opponents, which provides promising results for the exploration of the field of smart chess and intelligent game deduction.

Antenna selection is often used for physical layer security to implement secure communications. However, due to the rapid changes of the main channel and the feedback delay of the channel state information (CSI), the transmitter obtains outdated CSI, and the outdated CSI leads to the outdated optimal transmit antenna. In order to improve the security of the system based on outdated CSI, in this paper, we propose a deep reinforcement learning framework of Deep Q Network (DQN) to predict the optimal transmit antenna in the multiple input multiple output (MIMO) wiretap channel. The legitimate receiver receives the pilot signals from each transmitting antenna, and the signal-to-noise ratio (SNR) of the pilot signals transmitted by each transmitting antenna can be obtained through maximal ratio combining. And then the legitimate receiver uses the DQN to predict the transmitting antenna at the next moment according to these SNRs. The simulation results show that DQN algorithm can effectively predict the optimal antenna at the next moment, and reduce the secrecy outage probability of MIMO wiretap system, compared with the traditional algorithm.

Mobile Robot Path Planning problem (MRPPP) is the most prominent research area employed in different real-time environments. In this article, a hybrid strategy is employed and validated with different environments for the path planning of mobile robots. The Artificial Potential Field (APF) algorithm and Multi-objective Genetic Algorithm (MOGA) are hybridized to solve an MRPPP. In the proposed hybrid methodology, the implementation is accomplished in three phases. To generate the initial population, an Artificial Potential Field (APF) algorithm is applied to discover all viable routes in an environment between the start and destination points. The collision-free paths are generated by determining the artificial force produced by obstacles and the target. The population-based evolutionary algorithm is employed to derive an optimal solution path from the initial population consisting of candidate paths. The selection operator is used to select the qualified subpopulation to the next generation to converge into the optimal solution. The genetic operators such as two-point crossover and mid-point mutation are exercised which are specific to the path planning problem.

In this paper, we investigate the user pairing and power allocation scheme for multiple cellular users (CUs) under the downlink non-orthogonal multiple access (NOMA) system. To maximize the sum-rate of all CUs, we formulate the resource allocation issue through optimizing the user pairing relationship and power allocation method. However, due to the nonconvex property of the formulated problem, the original problem is decoupled into two subproblems. First, the optimal power allocation scheme with a given subchannel assignment is obtained via a closed-form solution. Furthermore, based on the obtained optimal allocation scheme, a classical deep reinforcement learning (DRL) method called Deep Q-Network (DQN) algorithm is adopted to find the optimal user pairing scheme, where the DQN algorithm is characterized by higher learning efficiency and better performance of the features extraction ability compared with traditional reinforcement learning (RL) schemes. Simulation results validate the effectiveness of our proposed resource allocation, as compared against the RL based scheme and conventional orthogonal multiple access (OMA) method.

The Micro-Electro-Mechanical Systems Fabry-Perot (MEMS-FP) filter is a pivotal photosensitive component within optical instrumentation, serving an integral role across various optical applications, including frequency selection spectroscopy and spectral analysis. The advent of MEMS-FP technology has significantly contributed to advancements in the optical field, underscoring the necessity for precise pre-fabrication theoretical predictions of device properties through finite element analysis (FEA). Nonetheless, the complexity inherent in three-dimensional modelling of MEMS-FP structures, coupled with the intricate nature of finite element simulations, presents substantial challenges that impede progress in MEMS-FP research. To navigate these challenges, this paper introduces a novel approach employing reinforcement learning (RL) for the automated design of MEMS-FP devices. This method encompasses the development of a Deep Q-Network (DQN) algorithm, the formulation of a tailored reward function, and an innovative strategy for updating the status of MEMS-FP devices. Various designs schemes of MEMS-FP devices can be acquired using MEMS-FP state search. The disparity between the MEMS-FP device design scheme and performance index can be assessed using a reward function. The DQN algorithm is crafted to update the network responsible for predicting Q-values. The agent trained by our method can rapidly calculate the MEMS-FP device design scheme that fulfils a specified performance index when provided with the performance index of the MEMS-FP device. By amalgamating reinforcement learning with MEMS-FP device research, this approach adeptly identifies optimal design configurations that meet specified performance criteria with enhanced precision and efficiency and promote the advancement of MEMS-FP devices.

The semiconductor industry is a resource-intensive sector that heavily relies on energy, water, chemicals, and raw materials. Within the semiconductor manufacturing process, the diffusion furnace, ion implantation machine, and plasma etching machine exhibit high energy demands or operate at extremely high temperatures, resulting in significant electricity consumption, which is usually carbon-intensive. To address energy conservation concerns, the industry adopts batch production technology, which allows for the simultaneous processing of multiple products. The energy-efficient parallel batch scheduling problem arises from the need to optimise product grouping and sequencing. In contrast to existing heuristics, meta-heuristics, and exact algorithms, this paper introduces the Deep Q-Network (DQN) algorithm as a novel approach to address the proposed problem. The DQN algorithm is built upon the agent’s systematic learning of scheduling rules, thereby enabling it to offer guidance for online decision-making regarding the grouping and sequencing of products. The efficacy of the algorithm is substantiated through extensive computational experiments.

In the field of unmanned aerial vehicle (UAV) path planning, the conventional deep Q-network (DQN) algorithm encounters the issue of action space discretization, which results in the generation of unsmooth and inefficient planned paths. To address this issue, we introduce the particle swarm optimization (PSO) algorithm into DQN to convert the discrete action space into a continuous one. This method divides the agent’s surrounding space into discrete and continuous action spaces. The PSO algorithm performs a global search in the continuous space to obtain a continuous candidate solution, while DQN learns a policy in the discrete space to obtain a discrete candidate solution. Then, the two candidate solutions are combined using a weighted vector method to determine a direction that balances global search and policy learning. Additionally, we introduce a novel feature matrix as the state space for DQN, providing more accurate environmental and positional representations. Furthermore, we incorporate a mechanism into the base prioritized experience replay (PER) and N-step updates, which combines the current temporal difference error (TD-error) with historical priorities and includes a policy entropy penalty term, thereby enhancing DQN’s ability to learn long-term dependencies. The performance of the PSO-DQN model is further improved through an enhanced ε-greedy policy and learning rate decay strategy. Simulation results and experiments using the Flightmare simulator demonstrate that the proposed method generates smoother and more efficient paths for drones, exhibiting strong robustness in complex environments.

PhotoVoltaic (PV) panels have been increasingly favored by residential users in recent years, due to noticeable reductions in their costs. The PV systems become more effective when combined with battery packages, which store the energy produced by the PV systems for later use. This way, the PV systems are able to provide flexible and reliable services even when the peak demand for electricity misalign with the window of most efficient PV power generation. In this study, we develop an integrated charge/discharge scheme for lithium-ion batteries to maximize their total expected benefit. Specifically, we develop a Markov Decision Process (MDP) model to maximize the battery utilization, subject to uncertainty in weather conditions and electricity demands, while accounting for battery degradation due to calendar aging and charging/discharging cycles. Due to the extremely slow rate of degradation in batteries, the state space of the MDP is excessively large. To solve the problem efficiently, we establish structural properties of the MDP and exploit them to solve the problem. We improve the backward induction algorithm with established structural properties. We further improve the Deep Q-network (DQN) algorithm by proposing two novel algorithms, the augmented DQN (ADQN) algorithm and the stochastic augmented DQN (SADQN) algorithm. Computational results indicate that ADQN and SADQN solve the problem much faster than DQN, with better solution qualities. The ADQN and SADQN algorithms provide flexibility for practitioners in real-world implementations.