A Generalized Load Balancing Policy With Multi-Teacher Reinforcement Learning

The COVID-19 pandemic is recognized as one of the most serious global health problems, and many countries implemented lockdown measures to mitigate the effects of the crisis caused by this respiratory infectious disease. In this study, we investigated the relationship between social distancing policies and changes in traffic volume in Sinchon Station, South Korea. We used an official COVID-19 report provided by the Korea Disease Control and Prevention Agency (KCDA) and Seoul Metropolitan Government (SMG) to review social distancing policies, and the changes in traffic patterns before and during the COVID-19 pandemic between January 2020 and November 2021 were analyzed. Our study reveals that the changes in the overall traffic patterns from acceleration phases to deceleration phases of COVID-19 were related to the alert levels of social distancing policies implemented to tackle the situation resulting from the COVID-19 pandemic. Herein, we found that a significant decline in traffic volume took place from August to September 2020 (13.5–19.7%, weekday; 19.4–31.7%, weekend), from December 2020 to January 2021 (20.0%−26.6%, weekday; 26.8–34.0%, weekend), and from July to September 2021 (3.2–13.1%, weekday; 38.3–44.7%, weekend) when compared to the corresponding periods in 2019 (paired t-test; p < 0.001). The results of this study provide strong support for the effectiveness of Seoul’s preemptive measures, namely, the central government’s intensive social distancing campaign, in managing and reducing the impact of the pandemic situation based on the precise analysis of 10 types of facilities.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Concept drifting poses a real challenge for network models which depends on statistical heuristics learned from the data stream, for example Anomaly Based Detection/Prevention Systems. These models tend to become inconsistent over a period of time as the underlying data stream like network traffic tends to change and get affected by evolution of concept drift. Change in network traffic pattern is inevitable, it impacts the enterprises which are dynamic in nature especially cloud-centric enterprises. These changes in the network pattern can be of short time period or they can be persistent for longer time duration. Change in network traffic pattern is not always because of malicious activity, changes can be benign and thus impacting the performance of the IDS/IPS model. There is a need to quantify concept drifts and incorporate them in the model. In this paper we have proposed a supervised learning model to quantify the concept drift in the network traffic. The proposed model uses adaptive learning strategies with fixed training window to constantly evolve the model. Classification of data is done by Naive Bayes Classifier. ROC curve generated from Naive Bayes classifiers has been used as a de facto method for identifying concept drift. Classifications have been carried out on entire dataset and also on specific flow attributes like source ip, destination ip, source port, destination port, flags and protocols. In this paper we demonstrate the capabilities of the proposed model to identify drift in the network pattern and also which flow attributes have contributed in concept drifting using ROC curve.

AQROM: A quality of service aware routing optimization mechanism based on asynchronous advantage actor-critic in software-defined networks

Development of network cybersecurity technology needs IDS on network technology because modern networks face against dynamic attacks which display various attack methods. To address this problem, we present a different combination of a hybrid ideal solution for an adaptive intrusion detection framework that is incorporated with various analytical techniques approach to achieve accuracy and reliability in detecting threats. This proposed ideal design combines Random Forest based noise-handling classifier and recognizing structured and also static traffic behaviour, while the LSTM neural model helps to analyze traffic behavior by understanding temporal patterns. Security professionals can track network-based attack behavior through their ability to monitor traffic pattern changes which occur over time. The forecast detector system detects unusual network traffic patterns which the fusion model layer uses to create precise threat responses. The framework receives validation through testing with the CICIDS2017 is the dataset which contains various type of traffic patterns from both normal and attack activities. It performs comprehensive data preprocessing, feature selection and normalization to get better input data quality. The model becomes more adaptable through pseudo-labeling which enables it to analyze and learn from unclassified network traffic while utilizing this each set of information for subsequent training cycles to detect new traffic patterns. The system performs continuous network behavior monitoring through its clustering with retraining process. The method demonstrates that uniting statistical analysis with temporal data and adaptive learning into one system framework produces better intrusion detection results which also enable detecting intrusion for network at large scales.

A reliable and accurate short-term traffic forecasting system is crucial for the successful deployment of any intelligent transportation system. A lot of forecasting models have been developed in recent years but none of them could consistently outperform the others. In real-world applications, traffic forecasting accuracy can be affected by a lot of factors. Impacts of long-term changes to traffic patterns to short-term traffic forecasting are profound and this can easily make an existing forecasting system outdated. Therefore, it is very important for forecasting systems to detect long-term changes in traffic patterns and make updates accordingly. This paper presents a new forecasting mechanism, in which a dynamic hybrid approach is taken and self-learning ability is enhanced. Results of a case study show the proposed approach is feasible in enhancing the adaptability of traffic forecasting systems.

In this paper we study the problem of dynamic load balancing in broadcast WDM networks by retuning a subset of transceivers in response to changes in the overall traffic pattern. Assuming an existing wavelength assignment and some information regarding the new traffic demands, we present two approaches to obtaining a new wavelength assignment such that (a) the new traffic load is balanced across the channels, and (b) the number of transceivers that need to be retuned is minimized. The latter objective is motivated by the fact that tunable transceivers take a non-negligible amount of time to switch between wavelengths during which parts of the network are unavailable for normal operation. Our main contribution is a new approximation algorithm for the load balancing problem that provides for tradeoff selection, using a single parameter, between the two conflicting goals. This algorithm leads to a scalable approach to reconfiguring the network since, in addition to providing guarantees in terms of load balancing, the expected number of retunings scales with the number of channels, not the number of nodes in the network.

Wireless devices operating under IEEE 802.11 x standards have the ability to transmit at various data-rates. However, there is no specification in the standard for dynamically selecting a suitable value of data-rate. Numerous solutions are available, mainly from device manufacturers and research community; focusing on a particular algorithm and its suitability for the overall system. However, none of the previous designs were flexible to change their behavior with changes in traffic nature and pattern. Not all traffic types have the same requirements and so should be the behavior of underlying algorithms for data-rate control. This paper highlights various data-rate control algorithms; classifying them on the basis of several parameters and discusses their suitability for different traffic patterns. It also proposes an algorithm, specifically designed for real time services. The cross-layer framework monitors traffic nature and patterns at the transport layer and activates one of three available rate control algorithms which suits the requirements of a particular type of traffic. According to the specific design of a selected algorithm, it then selects data-rate values at runtime. The results verify the tremendous difference among the three algorithms in terms of throughput and delay and signify their usability for specific traffic types.

5G resources should be properly planned for users to have a good quality of service. This planning includes defining the most suitable numerology indexes and best spectrum allocation considering the requirements of current traffic, which may change over time. Furthermore, when accounting for changes in traffic pattern, any necessary reconfigurations should be minimized. Here in this article, an optimization model is developed for the planning of spectrum allocation to the best mix numerology. The model considers adapting the operating numerology mix according to the current presence (or not) of traffic requirements. The model also works under any wireless communication scenario in 5G, and under any traffic pattern.

BackgroundThe Democratic National Convention (DNC) in Boston, Massachusetts in 2004 provided an opportunity to evaluate the impacts of a localized and short-term but potentially significant change in traffic patterns on air quality, and to determine the optimal monitoring approach to address events of this nature. It was anticipated that the road closures associated with the DNC would both influence the overall air pollution level and the distribution of concentrations across the city, through shifts in traffic patterns.MethodsTo capture these effects, we placed passive nitrogen dioxide badges at 40 sites around metropolitan Boston before, during, and after the DNC, with the goal of capturing the array of hypothesized impacts. In addition, we continuously measured elemental carbon at three sites, and gathered continuous air pollution data from US EPA fixed-site monitors and traffic count data from the Massachusetts Highway Department.ResultsThere were significant reductions in traffic volume on the highway with closures north of Boston, with relatively little change along other highways, indicating a more isolated traffic reduction rather than an across-the-board decrease. For our nitrogen dioxide samples, while there was a relatively small change in mean concentrations, there was significant heterogeneity across sites, which corresponded with our a priori classifications of road segments. The median ratio of nitrogen dioxide concentrations during the DNC relative to non-DNC sampling periods was 0.58 at sites with hypothesized traffic reductions, versus 0.88 for sites with no changes hypothesized and 1.15 for sites with hypothesized traffic increases. Continuous monitors measured slightly lower concentrations of elemental carbon and nitrogen dioxide during road closure periods at monitors proximate to closed highway segments, but not for PM2.5 or further from major highways.ConclusionWe conclude that there was a small but measurable influence of DNC-related road closures on air quality patterns in the Boston area, and that a low-cost monitoring study combining passive badges for spatial heterogeneity and continuous monitors for temporal heterogeneity can provide useful insight for community air quality assessments.

The widespread deployment of smart meters and communication technologies brings opportunities to improve the adaptability and flexibility of future manufacturing systems under growing complexity and uncertainty. As a result, data-driven approaches, especially Reinforcement Learning (RL) methods, are gaining wide attention in recent years. Although in the current literature, RL approaches show superiority over traditional methods in many applications, a comprehensive review and comparison of different RL methods and their use in discrete manufacturing system control under changing environments have not yet been established. This paper first provides a literature review of RL algorithms for decision-making in discrete manufacturing systems, and then systematically discusses the underlying mechanisms of four most commonly used RL algorithms. In addition, the performance of these RL algorithms is compared by solving a production control problem, which aims to maximize the profit under time-varying production costs. The comparison results show that the single-agent RL methods with discrete action space can provide satisfactory results in small-scale systems, while multi-agent RL algorithms are more suitable to solve problems in complex, large-scale systems to remit the curse of dimensionality. This research sheds light on the impacts of RL method selection on the search for high-quality solutions in manufacturing systems, which is envisioned to drive future research on RL-supported manufacturing systems.

Operating room (OR) human traffic has been implicated as a cause of surgical site infection. We first observed the normal human traffic pattern in our Pediatric Orthopedic ORs, and then examined the effect of surveillance on that traffic pattern. This study consisted of 2 phases: phase I sought to observe the OR traffic pattern (number of door swings, maximum and minimum number of OR personnel, number of OR personnel at 30-minute intervals, or changes in nursing, anesthesia, or surgeon staff) during surgical cases without OR personnel being notified, and for phase II, the same traffic pattern was monitored with their knowledge. Two thousand four hundred forty-two minutes of surgical time were observed in phase I, and 1908 minutes were observed in phase II. There was no difference (P=0.06) in the time between door swings between phase I (1.39 min) and phase II (1.70 min), no difference (P=1.000) in the maximum number of people in the OR between phase I (11.5 people, range: 7-15 people) and phase II (11.5 people, range: 8-20 people), and no difference (P=1.000) in the minimum number of people in the OR between phase I (4.67 people, range: 4-6 people) and phase II (4.71 people, range: 3-6 people). There was a difference in the time between door swings (P=0.03) and maximum number of people in the OR (P=0.005) based on the length of the surgery (less or more than 120 min). There was no difference in the time between door swings (P=0.11), but there was a difference in the maximum number of people in the OR (P=0.002) based on type of surgery (spine vs. others). There was no role of surveillance of human traffic in the OR. To achieve any change in the OR traffic pattern, monitoring alone may not be sufficient; other novel techniques or incentives may need to be considered.

Neural networks may be useful alternatives for statistical classification techniques when the available data is incomplete. This paper discusses the results obtained from a statistical technique called hierarchical grouping and the Kohonen neural network for classifying traffic patterns. The Kohonen neural network is shown to be a reasonable approximation of the hierarchical-grouping technique. It is suggested that hierarchical grouping of a small sample of typical traffic patterns may be a useful first step in setting up a Kohonen neural network for traffic-pattern classification. The Kohonen neural network can be used for classifying a large number of complete as well as incomplete traffic patterns as they become available. The neural network can also adapt the classification process to the change in the typical traffic patterns over time.

This paper focuses on the capacity of wireless ad hoc networks and analyzes the effect of key factors viz. network size, traffic patterns and detailed local radio interactions on the capacity of such networks. The capacity is evaluated with several different network layouts and traffic patterns through simulations. To demonstrate the impact of these factors, the capacity evaluation starts with a simple case of a chain of evenly spaced nodes in a network environment and progresses to a network with random traffic and randomly spaced nodes. Initially, capacity of static nodes is evaluated for various network layouts and traffic patterns. Since, in most scenarios, nodes do not travel significant distances during packet transmissions. As an enhancement, mobility of nodes is introduced into the network scenario and the performance is again evaluated. The simulations are carried out using OPNET modeler and the results obtained are presented in this report. The results are analyzed to understand the impact of these factors on the capacity and consequently suggest measures to increase the same. This work shows that the achievable capacity of ad hoc network depends on network size, traffic pattern and mobility. In a single cell topology, it is found that there is a 50% reduction in network throughput, if the node size increases from 2 to 10 nodes, whereas there is a 74% reduction in the throughput for chain topology for the same increase in node size. In a lattice topology with horizontal traffic, there is a 46 % reduction in network throughput when the lattice size increases from 4X4 to 5x5.The same percentage of reduction is observed when both horizontal and network traffic is introduced. In a random network topology with random traffic, there is an 80 % reduction in network throughput when the node size increases from 150 to 750 nodes. However, for the same scenario with the introduction of mobility to the nodes, a slight improvement is achieved with an overall 75% reduction in network throughput.

JUE insight: Demand for transportation and spatial pattern of economic activity during the pandemic