Predictive Maintenance Using GPU-Accelerated Partially Observable Markov Decision Process

Aiming to reduce energy consumption, minimize transmission latency, and optimize the long-term performance of Wireless Sensor Networks (WSN), this paper proposes an improved routing model of Directed Diffusion. We formulated the routing issue as a partially observable Markov decision process (POMDP), such that the optimal nodes selection policy yielded by maximizing the expected reward. Since the solution of POMDP is ready-made, the attention is concentrated only on the formulation of the routing issue into a POMDP model, more specifically, the definition of the tuple which contains states, actions, observations, transition probabilities, observation probabilities, and reward function. Our experimental study shows that the POMDP-based Directed Diffusion routing outperforms other routing techniques in terms of three widely used metrics including average dissipated energy, average delay, and average lifetime.

We study synthesis problems with constraints in partially observable Markov decision processes (POMDPs), where the objective is to compute a strategy for an agent that is guaranteed to satisfy certain safety and performance specifications. Verification and strategy synthesis for POMDPs are, however, computationally intractable in general. We alleviate this difficulty by focusing on planning applications and exploiting typical structural properties of such scenarios; for instance, we assume that the agent has the ability to observe its own position inside an environment. We propose an abstraction refinement framework, which turns such a POMDP model into a (fully observable) probabilistic two-player game (PG). For the obtained PGs, efficient verification and synthesis tools allow to determine strategies with optimal safety and performance measures, which approximate optimal schedulers on the POMDP. If the approximation is too coarse to satisfy the given specifications, a refinement scheme improves the computed strategies. As a running example, we use planning problems where an agent moves inside an environment with randomly moving obstacles and restricted observability. We demonstrate that the proposed method advances the state of the art by solving problems several orders of magnitude larger than those that can be handled by existing POMDP solvers. Furthermore, this method gives guarantees on safety constraints, which is not supported by the majority of the existing solvers.

This paper proposes a continuous state partially observable Markov decision process (POMDP) model for the corrosion maintenance of oil and gas pipelines. The maintenance operations include complex and extensive activities to detect the corrosion type, determine its severity, predict the deterioration rate, and plan future inspection (monitoring) schemes and maintenance policy. A POMDP model is formulated as a decision-making support tool to effectively handle partially observed corrosion defect levels. It formulates states as the pipeline’s degradation level using a probability distribution. Inline inspection (ILI) methods estimate the latest state of the pipeline, which also defines the initial state of the optimization process. The set of actions comprises corrosion mitigation operations. The errors associated with the ILI method are used to construct the observation function for the model. The sum of inspection, maintenance operations, and failure costs for a given state and action are formulated as rewards. Numerical experiments are made based on data collected from the literature. The results showed that different policies, whether derived from solvers (theoretical) or determined from practical experience, can be compared to identify the best maintenance alternative using the model. It was also observed that the choice of the solvers is important since they affect the discounted rewards and the run time to obtain them. The model approximates the parameters and uncertainty associated with the propagation of corrosion, proficiency of inspection methods, and implementation of maintenance policies. Overall, it can be applied to improve the maintenance decision-making process for the oil and gas pipeline as it incorporates the stochastic features of the operation.

For over a decade, the Hidden Markov Model (HMM) has been the primary tool used for acoustic modeling in the field of speech recognition. In this paper we examine a more general approach using a Partially Observable Markov Decision Process (POMDP) to mo del the base phonetic unit. We introduce the concept of multiple phonetic context classes, one for each of the infinite possible contexts a phoneme can be in, and show how a POMDP can be used to represent such a model. Much the same way that tying mixtures at the state level across phonemes sharing linguistic properties is used to fill in gaps in the model space due to lack of data, the POMDP model can fill in additional gaps, in effect adding a second level of clustering driven by the data itself.

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

Optimizing active surveillance for prostate cancer using partially observable Markov decision processes

Partially observable Markov decision processes (POMDPs) are a general mathematical model for sequential decision-making in stochastic environments under state uncertainty. POMDPs are often solved online, which enables the algorithm to adapt to new information in real time. Online solvers typically use bootstrap particle filters based on importance resampling for updating the belief distribution. Since directly sampling from the ideal state distribution given the latest observation and previous state is infeasible, particle filters approximate the posterior belief distribution by propagating states and adjusting weights through prediction and resampling steps. However, in practice, the importance resampling technique often leads to particle degeneracy and sample impoverishment when the state transition model poorly aligns with the posterior belief distribution, especially when the received observation is noisy. We propose an approach that constructs a sequence of bridge distributions between the state-transition and optimal distributions through iterative Monte Carlo steps, better accommodating noisy observations in online POMDP solvers. Our algorithm demonstrates significantly superior performance compared to state-of-the-art methods when evaluated across multiple challenging POMDP domains.

Partially observable Markov decision processes (POMDPs) provide a principled framework for sequential planning in uncertain single agent settings. An extension of POMDPs to multiagent settings, called interactive POMDPs (I-POMDPs), replaces POMDP belief spaces with interactive hierarchical belief systems which represent an agent’s belief about the physical world, about beliefs of other agents, and about their beliefs about others’ beliefs. This modification makes the difficulties of obtaining solutions due to complexity of the belief and policy spaces even more acute. We describe a general method for obtaining approximate solutions of I-POMDPs based on particle filtering (PF). We introduce the interactive PF, which descends the levels of the interactive belief hierarchies and samples and propagates beliefs at each level. The interactive PF is able to mitigate the belief space complexity, but it does not address the policy space complexity. To mitigate the policy space complexity – sometimes also called the curse of history – we utilize a complementary method based on sampling likely observations while building the look ahead reachability tree. While this approach does not completely address the curse of history, it beats back the curse’s impact substantially. We provide experimental results and chart future work.

During the process of disease diagnosis, overdiagnosis can lead to potential health loss and unnecessary anxiety for patients as well as increased medical costs, while underdiagnosis can result in patients not being treated on time. To deal with these problems, we construct a partially observable Markov decision process (POMDP) model of chronic diseases to study optimal diagnostic policies, which takes into account individual characteristics of patients. The objective of our model is to maximize a patient’s total expected quality-adjusted life years (QALYs). We also derive some structural properties, including the existence of the diagnostic threshold and the optimal diagnosis age for chronic diseases. The resulting optimization is applied to the management of coronary heart disease (CHD). Based on clinical data, we validate our model, demonstrate how the quantitative tool can provide actionable insights for physicians and decision makers in health-related fields, and compare optimal policies with actual clinical decisions. The results indicate that the diagnostic threshold first decreases and then increases as the patient’s age increases, which contradicts the intuitive non-decreasing thresholds. Moreover, diagnostic thresholds were higher for women than for men, especially at younger ages.

This study explores expert-novice differences in anticipation under uncertainty by combining partially observable Markov decision process (POMDP) modeling with machine learning classification. Forty-eight participants (24 experts, 24 novices) completed a basketball pass/shot anticipation task. Through POMDP modeling, two cognitive parameters-sensory precision (SP) and prior belief (pB)-were extracted to capture internal decision processes. Results showed that experts fit the POMDP model more closely, requiring more iterations for parameter convergence and achieving higher pseudo R2 values than novices. Experts demonstrated significantly higher SP, indicating superior ability to filter key cues under noisy conditions. Their pB values remained closer to neutral, suggesting flexible reliance on prior knowledge. In contrast, novices exhibited more biased priors and a lower, more dispersed SP. Machine learning analyses revealed that SP and pB jointly formed distinct clusters for experts and novices in a two-dimensional parameter space, with classification accuracies exceeding 90% across multiple methods. These findings indicate that expertise entails both enhanced perceptual precision and adaptive prior calibration, reflecting deeper cognitive reorganization rather than simple skill increments. Our dual-parameter approach offers a model-based perspective on expert cognition and may inform future research on the multifaceted nature of expertise.

The existing procedures for Autism Spectrum Disorder (ASD) diagnosis are often time consuming and tiresome both for highly-trained human evaluators and children. In addition, prospective human evaluators need to undergo a rigorous and lengthy training process that may not be accessible or affordable to all interested individuals. Hence, this paper proposes a framework for robot-assisted ASD evaluation based on Partially Observable Markov Decision Process (POMDP) modelling. POMDP is broadly used for modelling optimal sequential decision making tasks under uncertainty. Spurred by the widely accepted Autism Diagnostic Observation Schedule (ADOS), we start off with emulating ADOS. In other words, our POMDP model explicitly takes into account the ADOS stratification into several modules, ongoing task informativeness and robotic sensor deficiencies. Relying only on imperfect sensor observations, the robot provides an assessment of the child’s ASD-relevant functioning level (which is partially observable) within a particular task. Finally, we demonstrate that the proposed POMDP framework provides fine-grained outcome quantification, which could also increase the appeal of robot-assisted diagnostic protocols in the future.

The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that provably adheres to one or more specifications. Yet, the general problem is undecidable, and policies require full (and thus potentially unbounded) traces of execution history. To provide good approximations of such policies, POMDP agents often employ randomization over action choices. We consider the problem of computing simpler policies for POMDPs, and provide several approaches to still ensure their expressiveness. Key aspects are (1) the combination of an arbitrary number of specifications the policies need to adhere to, (2) a restricted form of randomization, and (3) a light-weight preprocessing of the POMDP model to encode memory. We provide a novel encoding as a mixed-integer linear program as baseline to solve the underlying problems. Our experiments demonstrate that the policies we obtain are more robust, smaller, and easier to implement for an engineer than those obtained from state-of-the-art POMDP solvers.

Standard value function approaches to finding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.

Breast cancer is the most common nonskin cancer and the second leading cause of cancer death in U.S. women. Although mammography is the most effective modality for breast cancer screening, it has several potential risks, including high false-positive rates. Therefore, the balance of benefits and risks, which depend on personal characteristics, is critical in designing a mammography screening schedule. In contrast to prior research and existing guidelines that consider population-based screening recommendations, we propose a personalized mammography screening policy based on the prior screening history and personal risk characteristics of women. We formulate a finite-horizon, partially observable Markov decision process (POMDP) model for this problem. Our POMDP model incorporates two methods of detection (self or screen), age-specific unobservable disease progression, and age-specific mammography test characteristics. We solve this POMDP optimally after setting transition probabilities to values estimated from a validated microsimulation model. Additional published data is used to specify other model inputs such as sensitivity and specificity of test results. Our results show that our proposed personalized screening schedules outperform the existing guidelines with respect to the total expected quality-adjusted life years, while significantly decreasing the number of mammograms and false-positives. We also report the lifetime risk of developing undetected invasive cancer associated with each screening scenario.

This thesis presents a recent research on the problem of environmental modeling for both localization and map building for wheel-based, differential driven, fully autonomous and self-contained mobile robots. The robots behave in an indoor office environment. They have a multi-sensor setup where the encoders are used for odometry and two exteroperceptive sensors, a 360° laser scanner and a monocular vision system, are employed to perceive the surrounding. The whole approach is feature based meaning that instead of directly using the raw data from the sensor features are firstly extracted. This allows the filtering of noise from the sensors and permits taking account of the dynamics in the environment. Furthermore, a properly chosen feature extraction has the characteristic of better isolating informative patterns. When describing these features care has to be taken that the uncertainty from the measurements is taken into account. The representation of the environment is crucial for mobile robot navigation. The model defines which perception capabilities are required and also which navigation technique is allowed to be used. The presented environmental model is both metric and topological. By coherently combining the two paradigms the advantages of both methods are added in order to face the drawbacks of a single approach. The capabilities of the hybrid approach are exploited to model an indoor office environment where metric information is used locally in structures (rooms, offices), which are naturally defined by the environment itself while the topology of the whole environment is resumed separately thus avoiding the need of global metric consistency. The hybrid model permits the use of two different and complementary approaches for localization, map building and planning. This combination permits the grouping of all the characteristics which enables the following goals to be met: Precision, robustness and practicability. Metric approaches are, per definition, precise. The use of an Extended Kalman Filter (EKF) permits to have a precision which is just bounded by the quality of the sensor data. Topological approaches can easily handle large environments because they do not heavily rely on dead reckoning. Global consistency can, therefore, be maintained for large environments. Consistent mapping, which handle large environments, is achieved by choosing a topological localization approach, based on a Partially Observable Markov Decision Process (POMDP), which is extended to simultaneous localization and map building. The theory can be mathematically proven by making some assumptions. However, as stated during the whole work, at the end the robot itself has to show how good the theory is when used in the real world. For this extensive experimentation for a total of more than 9 km is performed with fully autonomous self-contained robots. These experiments are then carefully analyzed. With the metric approach precision with error bounds of about 1 cm and less than 1 degree is further confirmed by ground truth measurements with a mean error of less than 1 cm. The topological approach is successfully tested by simultaneous localization and map building where the automatically created maps turned out to work better than the a priori maps. Relocation and closing the loop are also successfully tested.