Risk-Sensitive Portfolio Management by Using C51 Algorithm

In recent years, investment strategies on financial markets using deep learning have attracted a significant amount of research attention. The objective of these studies is to obtain investment action that has a low risk and increases profit. On the other hand, Distributional Reinforcement Learning (DRL) expands the action-value function to a discrete distribution in reinforcement learning, which can control risk. However, DRL has not yet been used to learn investment action. In this study, we construct a low-risk investment trading model using DRL. This model is backtested on Nikkei 225 dataset and compared with Deep Q Network (DQN). We evaluate performance in terms of final asset amounts, their standard deviation, and the Sharpe ratio. The experimental results show that the proposed DRL-based method can learn low-risk actions with increasing profit, outperforming the compared method DQN.

Despite active research on trading systems based on reinforcement learning, the development and performance of research methods require improvements. This study proposes a new action-specialized expert ensemble method consisting of action-specialized expert models designed specifically for each reinforcement learning action: buy, hold, and sell. Models are constructed by examining and defining different reward values that correlate with each action under specific conditions, and investment behavior is reflected with each expert model. To verify the performance of this technique, profits of the proposed system are compared to those of single trading and common ensemble systems. To verify robustness and account for the extension of discrete action space, we compared and analyzed changes in profits of the three actions to our model's results. Furthermore, we checked for sensitivity with three different reward functions: profit, Sharpe ratio, and Sortino ratio. All experiments were conducted with S&P500, Hang Seng Index, and Eurostoxx50 data. The model was 39.1% and 21.6% more efficient than single and common ensemble models, respectively. Considering the extended discrete action space, the 3-action space was extended to 11- and 21-action spaces, and the cumulative returns increased by 427.2% and 856.7%, respectively. Results on reward functions indicated that our models are well trained; results of the Sharpe and Sortino ratios were better than the implementation of profit only, as in the single-model cases. The Sortino ratio was slightly better than the Sharpe ratio.

Developing an excellent quantitative trading strategy to obtain a high Sharpe ratio requires optimizing several parameters at the same time. Example parameters include the window length of a moving average sequence, the choice of trading instruments, and the thresholds used to generate trading signals. Simultaneously optimizing all these parameters to seek a high Sharpe ratio is a daunting and time-consuming task, partly because of the unknown mechanism determining the Sharpe ratio. This article proposes using Bayesian optimization to systematically search for the optimal parameter configuration that leads to a high Sharpe ratio. The author shows that the proposed intelligent search strategy performs better than manual search, a common practice that proves to be inefficient. The author’s framework also can easily be extended to other parameter selection tasks in portfolio optimization and risk management.

To solve problems involving sequential decision-making, Deep Reinforcement Learning (DRL) combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL).These fusions are skilled at training agents for video games because they use neural networks to approximate nonlinear functions.The Deep Q Network (DQN), a key algorithm in this field, tends to overestimate Q-values despite its effectiveness.The Double Deep Q Network (Double DQN) was presented to address this.The Kera's framework in PyCharm is used in this study to examine the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game.To speed up its training, improvements were also made to the DQN model.The enhanced DQN performed better than the traditional DQN but less well than the Double DQN, according to the results.This study delves into the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game, utilizing the Kera's framework in PyCharm.Additionally, enhancements were made to the DQN model to expedite its training.Results affirmed that the enhanced DQN outperformed the conventional DQN but lagged the Double DQN.The training loss trajectory further substantiated Double DQN's superiority in mitigating overestimation issues.

Reinforcement Learning (RL) is increasingly being applied to complex decision-making tasks such as financial trading. However, designing effective reward functions remains a significant challenge. Traditional static reward functions often fail to adapt to dynamic environments, leading to inefficiencies in learning. This paper presents a novel approach, called Self-Rewarding Deep Reinforcement Learning (SRDRL), which integrates a self-rewarding network within the RL framework. The SRDRL mechanism operates in two primary phases: First, supervised learning techniques are used to learn from expert knowledge by employing advanced time-series feature extraction models, including TimesNet and WFTNet. This step refines the self-rewarding network parameters by comparing predicted rewards with expert-labeled rewards, which are based on metrics such as Min-Max, Sharpe Ratio, and Return. In the second phase, the model selects the higher value between the expert-labeled and predicted rewards as the RL reward, storing it in the replay buffer. This combination of expert knowledge and predicted rewards enhances the performance of trading strategies. The proposed implementation, called Self-Rewarding Double DQN (SRDDQN), demonstrates that the self-rewarding mechanism improves learning and optimizes trading decisions. Experiments conducted on datasets including DJI, IXIC, and SP500 show that SRDDQN achieves a cumulative return of 1124.23% on the IXIC dataset, significantly outperforming the next best method, Fire (DQN-HER), which achieved 51.87%. SRDDQN also enhances the stability and efficiency of trading strategies, providing notable improvements over traditional RL methods. The integration of a self-rewarding mechanism within RL addresses a critical limitation in reward function design and offers a scalable, adaptable solution for complex, dynamic trading environments.

In order to better understand state-of-the-art reinforcement learning agent, deep Q-network, a brief review of reinforcement learning and Q-learning are first described. Then recent advances of deep Q-network are presented, and double deep Q-network and dueling deep Q-network that go beyond deep Q-network are also given.

Humans have to respond to everyday tasks with goal-directed actions in complex and time-continuous environments. However, modeling human behavior in such environments has been challenging. Deep Q-networks (DQNs), an application of deep learning used in reinforcement learning (RL), enable the investigation of how humans transform high-dimensional, time-continuous visual stimuli into appropriate motor responses. While recent advances in DQNs have led to significant performance improvements, it has remained unclear whether these advancements translate into improved modeling of human behavior. Here, we recorded motor responses in human participants (N = 23) while playing three distinct arcade games. We used stimulus features generated by a DQN as predictors for human data by fitting the DQN’s response probabilities to human motor responses using a linear model. We hypothesized that advancements in RL models would lead to better prediction of human motor responses. Therefore, we used features from two recently developed DQN models (Ape-X and SEED) and a third baseline DQN to compare prediction accuracy. Compared to the baseline DQN, Ape-X and SEED involved additional structures, such as dueling and double Q-learning, and a long short-term memory, which considerably improved their performances when playing arcade games. Since the experimental tasks were time-continuous, we also analyzed the effect of temporal resolution on prediction accuracy by smoothing the model and human data to varying degrees. We found that all three models predict human behavior significantly above chance level. SEED, the most complex model, outperformed the others in prediction accuracy of human behavior across all three games. These results suggest that advances in deep RL can improve our capability to model human behavior in complex, time-continuous experimental tasks at a fine-grained temporal scale, thereby opening an interesting avenue for future research that complements the conventional experimental approach, characterized by its trial structure and use of low-dimensional stimuli.

This paper presents a hybrid framework for multi-stock trading that combines the decision-making ability of Deep Q-Networks (DQN) with the allocation precision of portfolio optimization models. Realistic markets are noisy and non-stationary, and complex action spaces can hinder reinforcement learning (RL) performance. The DQN generates buy/sell signals based on market conditions. The framework passes buy-listed assets to an optimizer, which computes portfolio weights. Five allocation strategies are examined: naïve 1/N, Markowitz Mean–Variance, Global Minimum Variance, Risk Parity, and Sharpe Ratio Maximization. Empirical evaluations on emerging-market exchange-traded funds (ETFs), as well as additional tests on U.S. equities, show that even the baseline DQN outperforms traditional technical indicators. Furthermore, integrating any of the optimization approaches with DQN yields measurable improvements in return-risk performance metrics. Among the hybrid frameworks, DQN combined with Sharpe Ratio Maximization delivers the most consistent gains. The findings highlight the value of decomposing stock selection from capital allocation and demonstrate the effectiveness of the proposed DQN-optimization framework on our testbed.

Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment. Reinforcement Learning methods enable a state action function in mobile robots suited to their environment. During trial-and-error interaction with its surroundings, it helps a robot to find an ideal behavior on its own. The Deep Q Network (DQN) algorithm is used in TurtleBot 3 (TB3) to achieve the goal by successfully avoiding the obstacles. But it requires a large number of training iterations. This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field (APF) algorithms. First, a TB3 Waffle Pi DQN is built and trained to reach the goal. Then the APF shortest path algorithm is incorporated into the DQN algorithm. The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System (ROS). The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algorithm. The performance improvement rate of the proposed DQN + APF in comparison with DQN in terms of the number of successful targets is attained by 88%. The performance of the proposed DQN + APF in comparison with DQN in terms of average time is achieved by 0.331 s. The performance of the proposed DQN + APF in comparison with DQN average rewards in which the positive goal is attained by 85% and the negative goal is attained by −90%.

In-Sample and Out-of-Sample Sharpe Ratios of Multi-Factor Asset Pricing Models

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

In the modern logistics system, the multi-objective cold chain vehicle routing problem has attracted much attention due to its key role in ensuring food and drug safety, reducing operating costs, and responding to global sustainable development initiatives. This problem is essentially an NP hard combinatorial optimization problem, whose complexity is further exacerbated by the need to simultaneously balance multiple conflicting objectives such as transportation costs, temperature control costs, customer time window penalties, and carbon emissions. Although traditional heuristic algorithms are effective in specific scenarios, they often lack adaptability to dynamic environments; However, emerging pure deep reinforcement learning methods face challenges of low sample efficiency and insufficient generalization ability when dealing with complex constraints and large-scale instances. To address these challenges, this paper proposes an innovative hybrid reinforcement learning vehicle routing problem (HRL-VRP) framework. The core of this framework is an intelligent agent based on deep Q-networks, which learns the optimal strategy of a dynamic selection and scheduling heuristic operator through interaction with the optimization environment. These operators constitute a feature rich action space, covering genetic algorithm operators for global exploration, optimized A * search operators for local refinement, and temperature geographic dual dimensional clustering operators for problem decomposition. Through extensive experimental verification in the cold chain distribution scenario in the suburbs of Shanghai, the HRL-VRP framework has demonstrated excellent performance. The main contribution of this study is the proposal and validation of an adaptive optimization paradigm that can effectively integrate classical operations research algorithms with modern reinforcement learning techniques.

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

Machine learning has been attracting significant attention again since the potential of deep learning was recognized. Not only has machine learning been improved, but it has also been integrated with “reinforcement learning,” revealing other potential applications, e.g., deep Q-networks (DQN) and AlphaGO proposed by Google DeepMind. It is against this background that this special issue, “Cutting Edge of Reinforcement Learning and its Hybrid Methods,” focuses on both reinforcement learning and its hybrid methods, including reinforcement learning with deep learning or evolutionary computation, to explore new potentials of reinforcement learning.Of the many contributions received, we finally selected 13 works for publication. The first three propose hybrids of deep learning and reinforcement learning for single agent environments, which include the latest research results in the areas of convolutional neural networks and DQN. The fourth through seventh works are related to the Learning Classifier System, which integrates evolutionary computation and reinforcement learning to develop the rule discovery mechanism. The eighth and ninth works address problems related to goal design or the reward, an issue that is particularly important to the application of reinforcement learning. The last four contributions deal with multiagent environments.These works cover a wide range of studies, from the expansion of techniques incorporating simultaneous learning to applications in multiagent environments. All works are on the cutting edge of reinforcement learning and its hybrid methods. We hope that this special issue constitutes a large contribution to the development of the reinforcement learning field.

Artificial intelligence (AI) agent created with Deep Q-Networks (DQN) can defeat human agents in video games. Despite its high performance, DQN often exhibits odd behaviors, which could be immersion-breaking against the purpose of creating game AI. Moreover, DQN is capable of reacting to the game environment much faster than humans, making itself invincible (thus not fun to play with) in certain types of games. On the other hand, supervised learning framework trains an AI agent using historical play data of human agents as training data. Supervised learning agent exhibits a more human-like behavior than reinforcement learning agents because of imitating training data. However, its performance is often no better than human agents. The ultimate purpose of AI agents is to entertain human players. A good performance and a humanlike behavior are important factors of the AI agents, and both of them should be achieved simultaneously. This study proposes frameworks combining reinforcement learning and supervised learning and we call then separated network model and shared network model. We evaluated their performances by the game scores and behaviors by Turing test. The experimental results demonstrate that the proposed frameworks develop an AI agent of better performance than human agent and natural behavior than reinforcement learning agents.