DQN-Based Optimization Framework for Secure Sharded Blockchain Systems

Deep reinforcement learning (DRL) has been extensively used to address portfolio optimization problems. DRL agents acquire knowledge and make decisions through unsupervised interactions with their environment without requiring explicit knowledge of the joint dynamics of portfolio assets. Among these DRL algorithms, the combination of actor-critic algorithms and deep function approximators is the most widely used DRL algorithm. Here, we find that training the DRL agent using the actor-critic algorithm and deep function approximators may lead to scenarios where the improvement in the DRL agent's risk-adjusted profitability is insignificant. We argue that such situations primarily arise from the following two problems: sparsity in positive reward and the curse of dimensionality. These limitations prevent DRL agents from comprehensively learning asset price change patterns in the training environment. As a result, the DRL agents cannot effectively explore the dynamic portfolio optimization policy to improve the risk-adjusted profitability in the training process. To address these problems, we propose a novel multi-agent learning system based on the hierarchical deep reinforcement learning (HDRL) algorithmic framework in this research. Under this framework, the agents work together as a learning system for portfolio optimization. Specifically, by designing an auxiliary agent that works together with the executive agent for optimal policy exploration, the learning system can focus on exploring the policy with higher risk-adjusted return in the action space with positive return and low variance. The performance of the proposed learning system is evaluated using a portfolio of 29 stocks from the Dow Jones index in four different experiments. In the training process, the objective functions of the actor and critic both ultimately achieve stable convergence in the training process. The risk-adjusted profitability of our learning system in the training environment is significantly improved. Hence, we prove that the policies executed by our learning system in out-sample experiments originate from the DRL agents' comprehensive learning of asset price change patterns in the training environment. Furthermore, we find that adopting the auxiliary agent and HDRL training algorithm can efficiently overcome the issue of the curse of dimensionality and improve the training efficiency in the positive reward sparse environment. In each back-test experiment, the proposed learning system is compared to sixteen traditional strategies and ten strategies based on machine learning algorithms in the performance of profitability and risk control ability. The empirical results in the four evaluation experiments demonstrate the efficacy of our learning system, which outperforms all other strategies by at least 8.2% in terms of Sharpe ratio, Sorino ratio, and Calmar ratio. This indicates that the policies learned in the training environment can exhibit excellent generalization ability in the back-testing experiments.

Virtual network embedding (VNE) refers to the process of mapping virtual networks onto physical networks, which can improve the utilization and flexibility of network resources. However, due to the complexity of VNE problems and the requirement for network security, how to efficiently complete VNE and ensure network security is important. In this article, we analyze the characteristics of the cloud–edge–terminal collaborative vehicular network and design a resource allocation mechanism based on VNEVNE, abbreviated as DRLS-VNE. First, we establish a multidimensional heterogeneous network model and design a five-layer neural network as the deep reinforcement learning (DRL) agent. The DRL agent can adaptively extract network feature attributes, thereby improving the performance of DRLS-VNE. Second, we introduce a dynamic trust evaluation mechanism, which can evaluate the trustworthiness of each node in the virtual network in real time and set embedding constraints based on the evaluation results. Finally, we conduct experiments to verify the practicality and effectiveness of DRLS-VNE. The experimental results show that our solution can significantly enhance the performance of the VNE solution.

Can machine learning help us make better decisions about a changing planet? In this paper, we illustrate and discuss the potential of a promising corner of machine learning known as deep reinforcement learning (RL) to help tackle the most challenging conservation decision problems. We provide a conceptual and technical introduction to deep RL as well as annotated code so that researchers can adopt, evaluate and extend these approaches. RL explicitly focuses on designing an agent who interacts with an environment that is dynamic and uncertain. Deep RL is the subfield of RL that incorporates deep neural networks into the agent. We train deep RL agents to solve sequential decision‐making problems in setting fisheries quotas and managing ecological tipping points. We show that a deep RL agent is able to learn a nearly optimal solution for the fisheries management problem. For the tipping point problem, we show that a deep RL agent can outperform a sensible rule‐of‐thumb strategy. Our results demonstrate that deep RL has the potential to solve challenging decision problems in conservation. While this potential may be compelling, the challenges involved in successfully deploying RL‐based management to realistic scenarios are formidable—the required expertise and computational cost may place these applications beyond the reach of all but large, international technology firms. Ecologists must establish a better understanding of how these algorithms work and fail if we are to realize this potential and avoid the pitfalls such a transition would bring. We ultimately set forth a research framework based on well‐posed, public challenges so that ecologists and computer scientists can collaborate towards solving hard decision‐making problems in conservation.

In general, the Internet of Things (IoT) relies on centralized servers due to limited computing power and storage capacity. These server-based architectures have vulnerabilities such as DDoS attacks, single-point errors, and data forgery, and cannot guarantee stability and reliability. Blockchain technology can guarantee reliability and stability with a P2P network-based consensus algorithm and distributed ledger technology. However, it requires the high storage capacity of the existing blockchain and the computational power of the consensus algorithm. Therefore, blockchain nodes for IoT data management are maintained through an external cloud, an edge node. As a result, the vulnerability of the existing centralized structure cannot be guaranteed, and reliability cannot be guaranteed in the process of storing IoT data on the blockchain. In this paper, we propose a multi-level blockchain structure and consensus algorithm to solve the vulnerability. A multi-level blockchain operates on IoT devices, and there is an IoT chain layer that stores sensor data to ensure reliability. In addition, there is a hyperledger fabric-based monitoring chain layer that operates the access control for the metadata and data of the IoT chain to lighten the weight. We propose an export consensus method between the two blockchains, the Schnorr signature method, and a random-based lightweight consensus algorithm within the IoT-Chain. Experiments to measure the blockchain size, propagation time, consensus delay time, and transactions per second (TPS) were conducted using IoT. The blockchain did not exceed a certain size, and the delay time was reduced by 96% to 99% on average compared to the existing consensus algorithm. In the throughput tests, the maximum was 1701 TPS and the minimum was 1024 TPS.

Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However most current DRL video-game agents learn end-to-end from the video-output of the game, which is superfluous for many applications and creates a number of additional problems. More importantly, directly working on pixel-based raw video data is substantially distinct from what a human player does. In this paper, we present a novel method which enables DRL agents to learn directly from object information. This is obtained via use of an object embedding network (OEN) that compresses a set of object feature vectors of different lengths into a single fixed-length unified feature vector representing the current game-state and fulfills the DRL simultaneously. We evaluate our OEN-based DRL agent by comparing to several state-of-the-art approaches on a selection of games from the GVG-AI Competition. Experimental results suggest that our object-based DRL agent yields performance comparable to that of those approaches used in our comparative study.

In this work, we proposed a vulnerability-aware network adaptation framework that can generate a robust network topology against epidemic attacks by leveraging deep reinforcement learning (DRL) to build a resilient network. We call our proposed framework DREVAN, representing Deep REinforcement Learning-based Vulnerability-Aware Network Adaptations. The goal of the proposed DREVAN is to minimize security vulnerability caused by epidemic attacks exploiting the software monoculture for node compromise while maximizing network connectivity in terms of the size of the giant component. To be specific, the DREVAN aims to autonomously identify a pair of network adaptation budgets for adding or removing edges (i.e., how many edges to add or remove) by leveraging DRL. In this work, we tackle the inherent challenge of using DRL in reducing the learning curve of a DRL agent by proposing two algorithms. First, we proposed a vulnerability ranking algorithm of edges and nodes, namely VREN, for the DRL agent to select which edges to add or remove based on the lowest expected vulnerability between two nodes or the highest vulnerability of edges, respectively. Second, we also developed a Fractal-based Solution Search algorithm (FSS) to effectively direct the DRL agent towards the effective samples to visit and quickly identify and converge to an optimal solution (i.e., budget sizes of edge additions and removals). Via our extensive comparative performance analysis of the six different schemes, we demonstrated the outperformance of our proposed DREVAN-based schemes over counterpart and baseline schemes.

Internet of Things (IoT) expects to incorporate massive machine-type (MCT) devices, such as vehicles, sensors, and wearable devices, which brings a large number of application tasks that need to be processed. Additionally, data collected from various devices needs to be executed and processed in a timely, reliable, and efficient manner. In this paper, we introduce the multi-access edge computing (MEC) service into the IoT system such that it can assist the computation offloading and resource allocation for different application tasks. Further, we define a total cost function as the weighted sum of task delay and energy consumption and propose a novel deep reinforcement learning (DRL)-based decision-making algorithm to cooperatively optimize the task offloading and resource allocation. More specifically, we transform the initial problem into a convex optimization problem with the aid of our proposed atom action generation technique (2AGT) and adaptive aggregation parameter update strategy (2APUS). Additionally, the DRL agent will be retrained periodically to adapt to different network environments. Finally, numerical simulation results validate that the proposed algorithm is nearly close to optimal performance compared with some cutting-edge schemes.

Cloud radio access networks (CRANs) have become a key enabling technique for the next generation wireless communications. Resource allocation in CRANs still needs to be further improved to reach the objective of minimizing power consumption and meeting demands of wireless users over a long period. Inspired by the success of Deep Reinforcement Learning (DRL) on solving complicated control problems, we present a novel framework, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL</i> , for power-efficient resource allocation in CRANs with deep reinforcement learning. Specifically, we define the state space, action space and reward function for the DRL agent, apply a deep neural network (DNN) to approximating the action-value function, and formally formulate the resource allocation problem (in each decision epoch) as a convex optimization problem. Under ReCARL, we propose two different DRL agents: one has a regular DNN structure trained with the basic deep Q-learning method ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL-Basic</i> ); while the other has a context-aware DNN structure trained with a hybrid deep Q-learning method ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL-Hybrid</i> ). We evaluated the performance of ReCARL along with the two DRL agents by comparing them with two widely-used baselines via extensive simulation. The simulation results show that ReCARL achieves significant power savings while meeting user demands, and it can well handle highly dynamic cases.

In this paper, we propose a deep reinforcement learning (DRL) based predictive control scheme for reducing the energy consumption and energy cost of pumping systems in wastewater treatment plants (WWTP), in which the pumps are operated in a binary mode, using on/off signals. As global energy consumption increases, the efficient operation of energy-intensive facilities has also become important. A WWTP in Busan, Republic of Korea is used as the target of this study. This WWTP is a large energy-consuming facility, and the pumping station accounts for a significant portion of the energy consumption of the WWTP. The framework of the proposed scheme consists of a deep neural network (DNN) model for forecasting wastewater inflow and a DRL agent for controlling the on/off signals of the pumping system, where proximal policy optimization (PPO) and deep Q-neural network (DQN) are employed as the DRL agents. To implement smart control with DRL, a reward function is designed to consider the energy consumption amount and electricity price information. In particular, new features and penalty factors for pump switching, which are essential for preventing pump wear, are also considered. The performance of our designed DRL agents is compared with those of WWTP experts and conventional approaches such as scheduling method and model predictive control (MPC), in which integer linear programming (ILP) optimization is employed. Results show that the designed agents outperform the other approaches in terms of compliance with operating rules and reducing energy costs.

ABSTRACTReal‐time Hybrid Simulation (RTHS) is a cyber‐physical technique that studies the dynamic behavior of a system by combining physical and numerical components that are coupled through a boundary condition enforcer. In structural engineering, the numerical components are subjected to environmental loads that become dynamic displacements of the physical substructure applied through an actuator. However, the dynamics of the coupling between components and the complexities of the system lead to synchronization challenges that affect the accuracy of the simulation. Thus, requiring tracking controllers to ensure the fidelity of the simulation. This paper studies deep reinforcement learning (DRL) as a novel alternative to designing the tracking controller. Three controllers are designed: a DRL agent combined with a conventional time delay compensation, a conventional feedback controller combined with a DRL agent, and a DRL agent with a complex neural network architecture. The proposed approaches are tested using a virtual RTHS benchmark problem, and the results are compared with an optimized controller that has a proportional‐integral‐derivative controller and phase‐lead compensation. The results show that DRL can address the synchronization challenges of RTHS with a model‐free approach and simple neural network architectures. The work shown in this study is a critical step toward model‐free control methodologies that can transform and further develop the RTHS method. The proposed methodology can be used to address important challenges related to RTHS, including nonlinearities and uncertainties of the physical substructure, complex boundary conditions, and the computational efficiency when physical structures with complex dynamics are present.

This paper investigates the use of Deep Reinforcement Learning (DRL) to control a profit-seeking storage device trading in the European Continuous Intra-day Electricity Market (CIM). The main objective is to study whether model-free DRL can profitably trade on the CIM. Two DRL agents are compared: Twin Delayed Deep Deterministic Policy Gradients (TD3), and TD3 with behavior cloning. The agents are trained and evaluated in a simulated CIM environment, which uses historical market data to simulate other market participants. A Rolling Intrinsic (RI) algorithm is used as a benchmark. Results indicate that the agents are profitable and occasionally outperform RI, in one instance obtaining 162.03% of RI profit. However, none of the agents consistently outperforms the baseline. These results suggest that DRL has the potential to increase profitability considerably compared to RI, but that the observation provided to the agent is not descriptive enough of the CIM to learn a robust policy. Future research could use additional features in the observation, or model-based DRL to improve performance.

The service provisioning in multi-domain software-defined elastic optical networks (SD-EONs) is an interesting but difficult problem to tackle, because the basic problem of lightpath provisioning, i.e., the routing and spectrum assignment (RSA), is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {NP}$ </tex-math></inline-formula> -hard, and each domain is owned and operated by a different carrier. Therefore, even though numerous RSA heuristics have been proposed, there does not exist a universal winner that can always achieve the lowest blocking probability in all the scenarios of a multi-domain SD-EON. This motivates us to revisit the inter-domain provisioning problem in this paper by leveraging deep reinforcement learning (DRL). Specifically, we propose DeepCoop, which is an inter-domain service framework that uses multiple cooperative DRL agents to achieve scalable network automation in a multi-domain SD-EON. DeepCoop employs a DRL agent in each domain to optimize intra-domain service provisioning, while a domain-level path computation element (PCE) is introduced to obtain the sequence of the domains to go through for each lightpath request. By sharing a restricted amount of information among each other, the DRL agents can make their decisions distributedly. To ensure scalability and universality, we design the action space of each DRL agent based on well-known RSA heuristics, and architect the agents based on the soft actor-critic (SAC) scenario. We run extensive simulations to evaluate DeepCoop, and the results show that DeepCoop can adapt to the dynamic environment in a multi-domain SD-EON to always select the best RSA heuristic for minimizing blocking probability, and it outperforms the existing algorithms on inter-domain provisioning in various scenarios. Moreover, we verify that the distributed training implemented in DeepCoop ensures its universality and scalability (i.e., its training and operation do not depend on the topology of the SD-EON).

Geologic carbon storage (GCS) must be safe and profitable. To achieve these goals for gigaton-scale GCS operations, decision-making in the presence of uncertainty is required. Geophysical monitoring methods can inform such decisions, given their sensitivity to the spatiotemporal changes in the subsurface during and after injection. We investigate a novel framework for the optimal control of GCS operations using geophysical monitoring. We refer to this decision-making tool as “geophysical control” and develop sequential decision-making models trained using digital twins of GCS operations and the corresponding geophysical monitoring signals. In particular, we obtain these models via deep reinforcement learning (DRL) and specifically focus on two types of uncertainty: geophysical noise and uncertainty in the subsurface petrophysical model. Our objective is to demonstrate how each source of stochasticity affects the decision-making process when one seeks to maximize profit while minimizing the risk of induced seismicity through an optimal policy that determines the annual target CO2 injection rate. We train a suite of DRL agents with different geophysical observations (surface time-lapse gravity, surface seismic amplitude-variation-with-offset (AVO), and combined gravity and AVO surveys), different signal-to-noise ratio levels, and with/without petrophysical model uncertainties. A comparison of the learning behavior of these independent DRL agents shows that (1) the DRL framework has the capacity to learn optimal CO2 injection policies; (2) training performance degrades with increasing geophysical noise (especially more in the seismic AVO case); and (3) the combination of AVO and gravity enhances decision-making, especially in the presence of geophysical noise. Our results show that the use of multigeophysical measurements and the incorporation of subsurface model uncertainties are critical in developing robust injection control agents using DRL.

This work focuses on using Deep Reinforcement Learning (DRL) to control an autonomous vehicle in the hyper-realistic urban simulation LGSVL. Classical control systems such as MPC maneuver vehicles based on a given trajectory, current velocity, position, distances, and more. Our approach does not pass this information to the DRL agent but only images provided by the camera. Current DRL efforts also exploit similar approaches for autonomous driving, but they are only suitable for small, simple tasks using simple simulations. Our approach consists of two differently trained neural networks (NN), a perceptual NN for representation learning and an actor NN for selecting the correct action. The perception NN will be trained via representation and self-supervised learning to strengthen our DRL agent's understanding of the scene. It can recognize temporal information and the dynamics of a complex environment. This work shows the importance of decoupling the perception and decision (actor) model for autonomous driving. All in all, we could drive autonomously in a hyper-realistic urban simulation using our modular DRL framework. Moreover, our approach also provides a solution for other similar tasks in the field of robotics based on images.

Deep reinforcement learning (DRL) has achieved significant results in many machine learning (ML) benchmarks. In this short survey, we provide an overview of DRL applied to trading on financial markets with the purpose of unravelling common structures used in the trading community using DRL, as well as discovering common issues and limitations of such approaches. We include also a short corpus summarization using Google Scholar. Moreover, we discuss how one can use hierarchy for dividing the problem space, as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market; the results of this survey are two-fold: firstly, to find the most promising directions for further research and secondly, to show how a lack of consistency in the community can significantly impede research and the development of DRL agents for trading.