Beyond performance: A POMDP-based machine learning framework for expert cognition.

Sepsis is a life-threatening condition, caused by the body’s extreme response to an infection. In the United States, 1.7 million cases of sepsis occur annually, resulting in 265,000 deaths. Delayed diagnosis and treatment are associated with higher mortality rates. An exponential rise in the availability of medical data has allowed for the development of sophisticated machine learning algorithms to predict sepsis earlier than the onset. However, these models often underperform, as the training data are retrospective and do not fully capture the uncertain future. In this study, we develop a novel framework, which we refer to as MLePOMDP, to leverage and combine the underlying, high-level knowledge about sepsis progression and machine learning (ML) for classification. Specifically, we use a hidden Markov model to describe sepsis development at a high level, where the ML model makes the higher-order “observations” from temporal data. Consequently, a partially observable Markov decision process (POMDP) model is developed to make classification decisions. We analytically establish that the optimal policy is of threshold-type, which we exploit to efficiently optimize MLePOMDP. MLePOMDP is calibrated and tested using high-frequency physiological data collected from bedside monitors. Different from past POMDP-based frameworks, MLePOMDP is developed for a prediction task using a very small state definition, produces highly interpretable results, and accounts for a novel and clinically meaningful action space. Our results show that MLePOMDP outperforms machine learning–based benchmarks by up to 8% in precision. Importantly, MLePOMDP is able to reduce false alarms by up to 28%. An additional experiment is conducted to show the generalizability of MLePOMDP to different patient cohorts. Summary of Contribution: This study develops a novel real-time decision support framework for early sepsis prediction by integrating well-known machine learning models (random forest and neural networks) with a well-established sequential decision-making model, namely, a partially observable Markov decision process (POMDP). The structural properties of the optimal policy are further explored and a threshold-type structure is established, which is then leveraged to develop a customized algorithm to solve the problem more efficiently. The resulting framework demonstrates the benefit of applying POMDPs to augment machine learning outputs. Specifically, the framework results in the reduction of false alarms in sepsis predictions where decisions are made in real time, hence improving the overall prediction precision.

This paper proposes a continuous state partially observable Markov decision process (POMDP) model for the corrosion maintenance of oil and gas pipelines. The maintenance operations include complex and extensive activities to detect the corrosion type, determine its severity, predict the deterioration rate, and plan future inspection (monitoring) schemes and maintenance policy. A POMDP model is formulated as a decision-making support tool to effectively handle partially observed corrosion defect levels. It formulates states as the pipeline’s degradation level using a probability distribution. Inline inspection (ILI) methods estimate the latest state of the pipeline, which also defines the initial state of the optimization process. The set of actions comprises corrosion mitigation operations. The errors associated with the ILI method are used to construct the observation function for the model. The sum of inspection, maintenance operations, and failure costs for a given state and action are formulated as rewards. Numerical experiments are made based on data collected from the literature. The results showed that different policies, whether derived from solvers (theoretical) or determined from practical experience, can be compared to identify the best maintenance alternative using the model. It was also observed that the choice of the solvers is important since they affect the discounted rewards and the run time to obtain them. The model approximates the parameters and uncertainty associated with the propagation of corrosion, proficiency of inspection methods, and implementation of maintenance policies. Overall, it can be applied to improve the maintenance decision-making process for the oil and gas pipeline as it incorporates the stochastic features of the operation.

Aiming to reduce energy consumption, minimize transmission latency, and optimize the long-term performance of Wireless Sensor Networks (WSN), this paper proposes an improved routing model of Directed Diffusion. We formulated the routing issue as a partially observable Markov decision process (POMDP), such that the optimal nodes selection policy yielded by maximizing the expected reward. Since the solution of POMDP is ready-made, the attention is concentrated only on the formulation of the routing issue into a POMDP model, more specifically, the definition of the tuple which contains states, actions, observations, transition probabilities, observation probabilities, and reward function. Our experimental study shows that the POMDP-based Directed Diffusion routing outperforms other routing techniques in terms of three widely used metrics including average dissipated energy, average delay, and average lifetime.

During the process of disease diagnosis, overdiagnosis can lead to potential health loss and unnecessary anxiety for patients as well as increased medical costs, while underdiagnosis can result in patients not being treated on time. To deal with these problems, we construct a partially observable Markov decision process (POMDP) model of chronic diseases to study optimal diagnostic policies, which takes into account individual characteristics of patients. The objective of our model is to maximize a patient’s total expected quality-adjusted life years (QALYs). We also derive some structural properties, including the existence of the diagnostic threshold and the optimal diagnosis age for chronic diseases. The resulting optimization is applied to the management of coronary heart disease (CHD). Based on clinical data, we validate our model, demonstrate how the quantitative tool can provide actionable insights for physicians and decision makers in health-related fields, and compare optimal policies with actual clinical decisions. The results indicate that the diagnostic threshold first decreases and then increases as the patient’s age increases, which contradicts the intuitive non-decreasing thresholds. Moreover, diagnostic thresholds were higher for women than for men, especially at younger ages.

We study synthesis problems with constraints in partially observable Markov decision processes (POMDPs), where the objective is to compute a strategy for an agent that is guaranteed to satisfy certain safety and performance specifications. Verification and strategy synthesis for POMDPs are, however, computationally intractable in general. We alleviate this difficulty by focusing on planning applications and exploiting typical structural properties of such scenarios; for instance, we assume that the agent has the ability to observe its own position inside an environment. We propose an abstraction refinement framework, which turns such a POMDP model into a (fully observable) probabilistic two-player game (PG). For the obtained PGs, efficient verification and synthesis tools allow to determine strategies with optimal safety and performance measures, which approximate optimal schedulers on the POMDP. If the approximation is too coarse to satisfy the given specifications, a refinement scheme improves the computed strategies. As a running example, we use planning problems where an agent moves inside an environment with randomly moving obstacles and restricted observability. We demonstrate that the proposed method advances the state of the art by solving problems several orders of magnitude larger than those that can be handled by existing POMDP solvers. Furthermore, this method gives guarantees on safety constraints, which is not supported by the majority of the existing solvers.

The Industrial IoT era has seen an outburst of areas benefiting from collecting more data. This includes Industry 4.0 and predictive maintenance, which have benefited from advancements in edge and fog computing. Predictive maintenance aims to minimize the downtime due to maintenance of machinery, while simultaneously minimizing the risk of unforeseen failures. This paper proposes a method to aid industries to make maintenance scheduling decisions that can be adopted in a distributed factory environment. The Partially Observable Markov Decision Process (POMDP) approach is used to determine the optimal time for maintenance for a machine. We first put forward an offline method for learning the Markov model parameters using historical sensor data. To allow for continual learning, an algorithm based on particle filters is proposed to provide online estimation of parameters of a Partially Observable MDP model. The particle filter algorithm allows the framework to adapt uniquely to each machine. The relative benefits of the POMDP model over a standard MDP model in the presence of noisy sensor data are evaluated through simulations which show significant improvements in revenue and reduced downtime. The POMDP and particle filter computations are executed on GPU-accelerated edge devices which achieve a speed-up of around 4 times compared to the CPU implementation.

Breast cancer is the most common nonskin cancer and the second leading cause of cancer death in U.S. women. Although mammography is the most effective modality for breast cancer screening, it has several potential risks, including high false-positive rates. Therefore, the balance of benefits and risks, which depend on personal characteristics, is critical in designing a mammography screening schedule. In contrast to prior research and existing guidelines that consider population-based screening recommendations, we propose a personalized mammography screening policy based on the prior screening history and personal risk characteristics of women. We formulate a finite-horizon, partially observable Markov decision process (POMDP) model for this problem. Our POMDP model incorporates two methods of detection (self or screen), age-specific unobservable disease progression, and age-specific mammography test characteristics. We solve this POMDP optimally after setting transition probabilities to values estimated from a validated microsimulation model. Additional published data is used to specify other model inputs such as sensitivity and specificity of test results. Our results show that our proposed personalized screening schedules outperform the existing guidelines with respect to the total expected quality-adjusted life years, while significantly decreasing the number of mammograms and false-positives. We also report the lifetime risk of developing undetected invasive cancer associated with each screening scenario.

For over a decade, the Hidden Markov Model (HMM) has been the primary tool used for acoustic modeling in the field of speech recognition. In this paper we examine a more general approach using a Partially Observable Markov Decision Process (POMDP) to mo del the base phonetic unit. We introduce the concept of multiple phonetic context classes, one for each of the infinite possible contexts a phoneme can be in, and show how a POMDP can be used to represent such a model. Much the same way that tying mixtures at the state level across phonemes sharing linguistic properties is used to fill in gaps in the model space due to lack of data, the POMDP model can fill in additional gaps, in effect adding a second level of clustering driven by the data itself.

Foraging mammals exhibit a familiar yet poorly characterized phenomenon, ‘alternation’, a pause to sniff in the air preceded by the animal rearing on its hind legs or raising its head. Rodents spontaneously alternate in the presence of airflow, suggesting that alternation serves an important role during plume-tracking. To test this hypothesis, we combine fully resolved simulations of turbulent odor transport and Bellman optimization methods for decision-making under partial observability. We show that an agent trained to minimize search time in a realistic odor plume exhibits extensive alternation together with the characteristic cast-and-surge behavior observed in insects. Alternation is linked with casting and occurs more frequently far downwind of the source, where the likelihood of detecting airborne cues is higher relative to ground cues. Casting and alternation emerge as complementary tools for effective exploration with sparse cues. A model based on marginal value theory captures the interplay between casting, surging, and alternation.

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation‐based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution‐based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

The proposed research is aimed towards developing models for control and assessment of child-robot interaction within a robot-assisted autism spectrum disorder diagnostic protocol. The robot-assisted protocol contains several tasks which consist of robot actions to elicit the interaction and observations through which the robot perceives the behaviour of the child. Tasks of the protocol are modelled using the partially observable Markov decision process (POMDP) model, which enables the robot to autonomously choose actions and assess the interaction. Assessment is based on the robot belief state related to the unobservable state of the child. The robot-assisted diagnostic protocol is modelled using hierarchical POMDP model enabling the robot to autonomously select the sequence of tasks within the protocol. The effectiveness of the developed models will be experimentally evaluated through examinations with multiple children in clinical settings.

Partially observable Markov decision processes (POMDPs) are models for sequential decision-making under uncertainty and incomplete information. Machine learning methods typically train recurrent neural networks (RNN) as effective representations of POMDP policies that can efficiently process sequential data. However, it is hard to verify whether the POMDP driven by such RNN-based policies satisfies safety constraints, for instance, given by temporal logic specifications. We propose a novel method that combines techniques from machine learning with the field of formal methods: training an RNN-based policy and then automatically extracting a so-called finite-state controller (FSC) from the RNN. Such FSCs offer a convenient way to verify temporal logic constraints. Implemented on a POMDP, they induce a Markov chain, and probabilistic verification methods can efficiently check whether this induced Markov chain satisfies a temporal logic specification. Using such methods, if the Markov chain does not satisfy the specification, a byproduct of verification is diagnostic information about the states in the POMDP that are critical for the specification. The method exploits this diagnostic information to either adjust the complexity of the extracted FSC or improve the policy by performing focused retraining of the RNN. The method synthesizes policies that satisfy temporal logic specifications for POMDPs with up to millions of states, which are three orders of magnitude larger than comparable approaches.

Partially observable Markov decision processes (POMDPs) have recently become popular among many AI researchers because they serve as a natural model for planning under uncertainty. Value iteration ...

Maintenance Planning under Imperfect Monitoring: an Efficient POMDP Model Using Interpolated Value Function