Congestion Control in SDN-Based Networks via Multi-Task Deep Reinforcement Learning

Market making (MM) is a trading activity by an individual market participant or a member firm of an exchange that buys and sells same securities with the primary goal of profiting on the bid-ask spread, which contributes to the market liquidity. Reinforcement learning (RL) is emerging as a quite popular method for automated market making, in addition to many other financial problems. The current state of the art in MM based on RL includes two recent benchmarks which use temporal-difference learning with Tile-Codings and Deep Q Networks (DQN). These two benchmark approaches focus on single-asset modelling, limiting their applicability in realistic scenarios, where the MM agents are required to trade on a collection of assets. Moreover, the Multi-Asset trading reduces the risk associated with the returns. Therefore, we design a Multi-Asset Market Making (MAMM) model, known as MTDRLMM, based on Multi-Task Deep RL. From a Multi-Task Learning perspective, multiple assets are considered as multiple tasks of the same nature. These assets share common characteristics among them, along with their individual traits. The experimental results show that the MAMM is more profitable than Single-Asset MM, in general. Moreover, the MTDRLMM model achieves the state-of-the-art in terms of investment return in a collection of assets.KeywordsMulti-Asset Market MakingMulti-Task Deep Reinforcement LearningTile-CodingsDeep Q Networks

Multi-task reinforcement learning endeavors to accomplish a set of different tasks with a single policy. To enhance data efficiency by sharing parameters across multiple tasks, a common practice segments the network into distinct modules and trains a routing network to recombine these modules into task-specific policies. However, existing routing approaches employ a fixed number of modules for all tasks, neglecting that tasks with varying difficulties commonly require varying amounts of knowledge. This work presents a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of certain intermediate modules, thereby flexibly choosing different numbers of modules for each task. Under this framework, we further introduce a ResRouting method to address the issue of disparate routing paths between behavior and target policies during off-policy training. In addition, we design an automatic route-balancing mechanism to encourage continued routing exploration for unmastered tasks without disturbing the routing of mastered ones. We conduct extensive experiments on various robotics manipulation tasks in the Meta-World benchmark, where D2R achieves state-of-the-art performance with significantly improved learning efficiency.

Magnetotelluric (MT) forward modeling is a key technique in magnetotelluric sounding, and deep learning has been widely applied to MT forward modeling. In three-dimensional (3-D) problems, although existing methods can predict forward modeling results with high accuracy, they often use multiple networks to simulate multiple forward modeling parameters, resulting in low efficiency. We apply multi-task learning (MTL) to 3-D MT forward modeling to achieve simultaneous inference of apparent resistivity and impedance phase, effectively improving overall efficiency. Furthermore, through comparative analysis of feature map differences in various decoder layers of the network, we identify the optimal branching point for multi-task learning decoders. This enhances the feature extraction capabilities of the network and improves the prediction accuracy of forward modeling parameters. Additionally, we introduce an uncertainty-based loss function to dynamically balance the learning weights between tasks, addressing the shortcomings of traditional loss functions. Experiments demonstrate that compared with single-task networks and existing multi-task networks, the proposed network (MT-FeatureNet) achieves the best results in terms of Structural Similarity Index Measure (SSIM), Mean Relative Error (MRE), and Mean Absolute Error (MAE). The proposed multi-task learning model not only improves the efficiency and accuracy of 3-D MT forward modeling but also provides a novel approach to the design of multi-task learning network structures.

In this paper, we aim to propose a novel transmission control protocol (TCP) congestion control method from a cross-layer-based perspective and present a deep reinforcement learning (DRL)-driven method called DRL-3R (DRL for congestion control with Radio access network information and Reward Redistribution) so as to learn the TCP congestion control policy in a superior manner. In particular, we incorporate the RAN information to timely grasp the dynamics of RAN, and empower DRL to learn from the delayed RAN information feedback potentially induced by several consecutive actions. Meanwhile, we relax the implicit assumption [that the feedback to one specific action returns at a round-trip-time (RTT) after the action is applied] in previous researches, by redistributing the rewards and evaluating the merits of actions more accurately. Experiment results show that besides maintaining a reasonable fairness, DRL-3R significantly outperforms classical congestion control methods (e.g., TCP Reno, Westwood, Cubic, BBR and DRL-CC) on network utility by achieving a higher throughput while reducing delay in various network environments.

Purpose To develop and evaluate a novel multitask deep learning framework for automated detection and localization of endoleaks at aortic digital subtraction angiography (DSA) performed during real-world endovascular aneurysm repair (EVAR) procedures for abdominal aortic aneurysm. Materials and Methods This retrospective study analyzed intraoperative aortic DSA images from patients undergoing EVAR (January 2017-December 2021). An expert panel assessed each sequence for endoleaks. Each sequence was processed into three input channels: peak density, time to peak, and area under the time-density curve, generating three two-dimensional perfusion maps per patient. These maps served as input into a convolutional neural network for binary detection (classification) and localization (regression) of endoleaks through multitask learning. Fivefold cross-validation was performed, with patients split 80:20 into training and testing datasets for each fold. Performance metrics included area under the receiver operating characteristic curve, F1 score, precision, and recall and were compared with human experts. Results The study included 220 patients (median age, 74 years [IQR, 68-79]; 181 male). Endoleaks were visible in 111 of 220 (50.5%) patients. The model identified and localized endoleaks with an area under the receiver operating characteristic curve of 0.85 ± 0.0031 (SD), F1 score of 0.78 ± 0.21, 95% precision, and 73% recall. Compared with the procedural team (94% precision, 63% recall), it had higher values in both metrics, with an F1 score within the human observer range (0.75-0.85). Balancing regression and classification by multitask learning delivered optimal results. The interobserver agreement among human experts was moderate (Fleiss κ = 0.404). Conclusion A novel, fully automated deep learning method accurately detected and localized endoleaks at DSA imaging from EVAR procedures. Keywords: Endoleaks, Deep Learning, Digital Subtraction Angiography, Endovascular Aneurysm Repair, Abdominal Aortic Aneurysm, Interventional Suite, Hybrid Operating Room Supplemental material is available for this article. © RSNA, 2025.

Congestion control is a fundamental mechanism for TCP protocol, which has been extensively studied in the past three decades. However, our experimental evaluations show that the state-of-art congestion control algorithms such as Cubic and BBR are far from optimal: they have unresolved issues such as insufficient usage of available bandwidth, inadaptable to dynamic bandwidth variants, and compromising on one or more performance dimensions. To address these challenges, we propose a novel congestion control mechanism called NeuRoc that coordinately uses online changepoint detection and deep reinforcement learning (DRL) technique to generate the optimal congestion control policy, which allows TCP operating at Kleinrock ’s optimal operation point to achieve fully bandwidth usage and low latency. To address the practical issues of deploying the deep learning based congestion control mechanism, we propose a cold-started training and deployment framework to reduce the cost of bootstrap. We implement NeuRoc on an emulation platform which connects to the Linux network protocol stack through virtual network interfaces. Extensive experiments show that NeuRoc achieves the best throughput-latency tradeoff compared with the state-of-the-arts in a variety of scenarios.

In machine learning, the relatedness across multiple tasks is usually complex and entangled. Due to dataset bias, the relatedness among tasks might be distorted and mislead the training of the models with solid learning ability, such as the multi-task neural networks. In this paper, we propose the idea of Relatedness Refinement Multi-Task Learning (RRMTDL) by introducing adversarial learning in the multi-task deep neural network to tackle the problem. The RRMTDL deep learning model restrains the misleading relatedness task by adversarial training and extracts information sharing across tasks with valuable relatedness. With RRMTDL, multi-task deep learning can enhance the task-specific representation for the major tasks by excluding the misleading relatedness. We design tests with various combinations of task-relatedness to validate the proposed model. Experimental results show that the RRMTDL model can effectively refine the task relatedness and prominently outperform other multi-task deep learning models in datasets with entangled task labels.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

Colorectal cancer (CRC) is the third most commonly diagnosed cancer and the second most lethal cancer in the world. The segmentation and staging of tumours in CT images are important for CRC diagnosis and treatment. Using deep learning to fulfil medical image diagnosis is a promising approach. Most current methods solve segmentation and staging tasks separately, while ignoring the associated information between tasks. Multi-task deep learning (MTL) can solve the two tasks simultaneously. In this study, we proposed a CRC_MTL framework applying multi-task deep learning to complete two tasks jointly for analysing CRC tumours in abdominal CT images. The CRC_MTL framework not only improves the accuracy of the classification task from 91.40% to 93.52% but also improves the dice coefficient of the segmentation task from 0.6900 to 0.7379.

Early detection and timely surgical intervention are crucial in reducing mortality rates associated with clinically significant prostate cancer (csPCa). Currently, clinical diagnostics primarily depend on magnetic resonance imaging (MRI) and nuclear medicine, with the potential diagnostic value of abdominal computed tomography (CT) remaining underexplored. This study aims to evaluate the effectiveness of multi-task deep learning neural networks in identifying early-stage prostate cancer using CT scans. In this study, we enrolled 539 patients from the Department of Radiology (N=461) and Nuclear Medicine (N=78). We utilized a multi-task deep learning network model (MTDL), based on the 3DUnet architecture, to segment and analyze the collected abdominal plain CT images. The predictive performance of this model was compared with a radiomics model and a single-task deep learning model using ResNet18. A diagnostic nomogram was then developed using the multi-task deep learning approach, incorporating prediction results and PSAD, age. The diagnostic performance of the different models was evaluated using the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The 461 patients from the Department of Radiology were divided into training and test sets at a ratio of 6:4, while the patients from the Department of Nuclear Medicine formed the validation set. Our MTDL nomogram demonstrated AUCs of 0.941 (95% confidence interval [CI]: 0.905valceedi 0.912 (95% CI: 0.904valceedi and 0.932 (95% CI: 0.883valceed in the training, test, and validation cohorts, respectively. This study indicates that combining abdominal CT with a multi-task neural network model effectively diagnoses csPCa, offering superior diagnostic performance compared to clinical models. Additionally, the multi-task neural network model outperformed both the single-task neural network model and the radiomics model in diagnostic accuracy. Our study demonstrated that the MTDL nomogram can accurately predict the presence of prostate cancer using abdominal CT scans, offering significant value for the early diagnosis of prostate cancer.

To develop and validate a multi-task deep learning (MTDL) model using multiphase contrast-enhanced CT (CECT) for simultaneously assessing histological subtypes, clinical stages, and anatomical complexity grades of solid malignant renal tumors. This two-center retrospective study included patients with solid malignant renal tumors and their preoperative kidney CECT images. A progressive layered extraction (PLE)-based MTDL model was trained and externally tested. Model performance was evaluated using the area under the receiver operating characteristic curve (AUC) and decision curve analysis (DCA), compared with the results of five radiologists. Among 798 patients (mean age, 54 ± 12 years; 279 females; Center A: n = 620, Center B: n = 178), 597 (74.8%) had clear cell renal cell carcinomas (ccRCC), 150 (18.8%) were clinical staging III/IV, and 187 (23.4%) had high anatomical complexity. On the external test set, the MTDL model achieved AUCs of 0.89 (95% CI: 0.82, 0.94) for distinguishing ccRCC from non-ccRCC, 0.87 (95% CI: 0.81, 0.93) for clinical staging (I/II vs. III/IV), and 0.87 (95% CI: 0.82, 0.92) for anatomical complexity grading (low-intermediate vs. high). The MTDL model outperformed single-task deep learning (STDL) in clinical staging (AUC: 0.87 vs. 0.82, p = 0.022), showed higher net benefit on DCA, and demonstrated better diagnostic performance than junior radiologists in histological subtyping and clinical staging. Additionally, it used 68% less memory and was 60% faster than STDL models. The CECT-based MTDL model demonstrated robust performance in simultaneously predicting histological subtypes, clinical stages, and anatomical complexity grades of malignant renal tumors. Question Accurate preoperative description of the histological subtyping, clinical staging, and anatomical complexity of malignant renal tumors is crucial for treatment decision-making. Findings By sharing features, the multi-task deep learning algorithm model enhances clinical staging performance and significantly improves computational efficiency in predicting all three tasks simultaneously. Clinical relevance The multi-task deep learning algorithm model enables rapid and accurate comprehensive preoperative evaluation of renal tumors, which assists surgeons in optimizing surgical plans and promotes the advancement of renal tumor management toward precision and efficiency.

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

The exponential growth in mobile data demand, fueled by the proliferation of smart devices, IoT applications, and multimedia services, has posed significant challenges to the performance andscalability of 5G networks. Traditional network traffic management techniques are increasingly insufficient to meet the dynamic and complex requirements of modern wireless communication systems. In response, deep learning (DL) has emerged as a powerful tool to enable intelligent, adaptive, and realtime optimization of mobile network traffic in 5G environments. This review provides a comprehensive overview of the integration of deep learning algorithms into various layers of the 5G architecture, including radio access networks (RAN), core networks, and edge computing platforms. Key DL models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM), deep reinforcement learning (DRL), and graph neural networks (GNNs) are analyzed in terms of their application to traffic prediction, congestion control, resource allocation, and quality of service (QoS) enhancement. The article also explores existing challenges such as data scarcity, model interpretability, latency constraints, and deployment complexity. Furthermore, it discusses emerging trends like federated learning, transfer learning, and edge intelligence as promising directions for future research. By synthesizing state-of-the-art contributions, this review highlights the transformative potential of DL in building efficient, resilient, and autonomous 5G networks.