Improving Human Behavior Using POMDPs with Gestures and Speech Recognition

A Partially Observable Markov Decision Processes (POMDP) framework has been developed for Electrohydrodynamic (EHD) printing to facilitate decisionmaking under uncertainty and automate adjustments in additive manufacturing. The POMDP system utilizes its reinforcement learning structure to optimize decisions by interacting with the environment. It processes feedback from a high-speed camera imaging system that monitors the printed line width (in-pixels) during printing which represents the state space. Image processing and unsupervised learning algorithms including K-means, Gaussian mixture models (GMM), and Agglomerative clustering are implemented to analyze this state space and create the transition model. This state space is partially observed and linked to the recorded highspeed images, through the observational model. As the difference between the first and last groups of the region of interest (ROI) grayscale profile of 200 images received from the stream images represents the observational space of the POMDP. Then the observed space is discretized to develop a finite set of observations. This model facilitates the dynamic adjustment of printing parameters such as plotting speed and standoff distance, ensuring optimal printing quality within the desired specification. It applies different solution methods and compares their performance to select the best for the EHD settings. Results show that the point-based value iteration method is the best approximation solution that can be used in this POMDP EHD settings framework with moderate complexity compared to other solution approaches.

Sepsis is a life-threatening condition, caused by the body’s extreme response to an infection. In the United States, 1.7 million cases of sepsis occur annually, resulting in 265,000 deaths. Delayed diagnosis and treatment are associated with higher mortality rates. An exponential rise in the availability of medical data has allowed for the development of sophisticated machine learning algorithms to predict sepsis earlier than the onset. However, these models often underperform, as the training data are retrospective and do not fully capture the uncertain future. In this study, we develop a novel framework, which we refer to as MLePOMDP, to leverage and combine the underlying, high-level knowledge about sepsis progression and machine learning (ML) for classification. Specifically, we use a hidden Markov model to describe sepsis development at a high level, where the ML model makes the higher-order “observations” from temporal data. Consequently, a partially observable Markov decision process (POMDP) model is developed to make classification decisions. We analytically establish that the optimal policy is of threshold-type, which we exploit to efficiently optimize MLePOMDP. MLePOMDP is calibrated and tested using high-frequency physiological data collected from bedside monitors. Different from past POMDP-based frameworks, MLePOMDP is developed for a prediction task using a very small state definition, produces highly interpretable results, and accounts for a novel and clinically meaningful action space. Our results show that MLePOMDP outperforms machine learning–based benchmarks by up to 8% in precision. Importantly, MLePOMDP is able to reduce false alarms by up to 28%. An additional experiment is conducted to show the generalizability of MLePOMDP to different patient cohorts. Summary of Contribution: This study develops a novel real-time decision support framework for early sepsis prediction by integrating well-known machine learning models (random forest and neural networks) with a well-established sequential decision-making model, namely, a partially observable Markov decision process (POMDP). The structural properties of the optimal policy are further explored and a threshold-type structure is established, which is then leveraged to develop a customized algorithm to solve the problem more efficiently. The resulting framework demonstrates the benefit of applying POMDPs to augment machine learning outputs. Specifically, the framework results in the reduction of false alarms in sepsis predictions where decisions are made in real time, hence improving the overall prediction precision.

Aiming to reduce energy consumption, minimize transmission latency, and optimize the long-term performance of Wireless Sensor Networks (WSN), this paper proposes an improved routing model of Directed Diffusion. We formulated the routing issue as a partially observable Markov decision process (POMDP), such that the optimal nodes selection policy yielded by maximizing the expected reward. Since the solution of POMDP is ready-made, the attention is concentrated only on the formulation of the routing issue into a POMDP model, more specifically, the definition of the tuple which contains states, actions, observations, transition probabilities, observation probabilities, and reward function. Our experimental study shows that the POMDP-based Directed Diffusion routing outperforms other routing techniques in terms of three widely used metrics including average dissipated energy, average delay, and average lifetime.

This paper considers the optimization problem for partially observable Markov decision processes (POMDPs) with the continuous state, observation, and action spaces. POMDPs with the discrete spaces have emerged as a promising approach to the decision systems with imperfect state information. However, in recent applications of POMDPs, there are many problems that have continuous states, observations, and actions. For such problems, due to the infinite dimensionality of the belief space, the existing studies usually discretize the continuous spaces with the sufficient or nonsufficient statistics, which may cause the curse of dimensionality and performance degradation. In this paper, based on the sensitivity analysis of the performance criteria, we have developed a simulation-based policy iteration algorithm to find the local optimal observation-based policy for POMDPs with the continuous spaces. The proposed algorithm needs none of the specific assumptions and prior information, and has a low computational complexity. One numerical example of the complicated multiple-input multiple-output beamforming problem shows that the algorithm has a significant performance improvement.

We show that the optimal decision policy for several types of Bayesian sequential detection problems has a threshold switching curve structure on the space of posterior distributions. This is established by using lattice programming and stochastic orders in a partially observed Markov decision process (POMDP) framework. A stochastic gradient algorithm is presented to estimate the optimal linear approximation to this threshold curve. We illustrate these results by first considering quickest time detection with phase-type distributed change time and a variance stopping penalty. Then it is proved that the threshold switching curve also arises in several other Bayesian decision problems such as quickest transient detection, exponential delay (risk-sensitive) penalties, stopping time problems in social learning, and multi-agent scheduling in a changing world. Using Blackwell dominance, it is shown that for dynamic decision making problems, the optimal decision policy is lower bounded by a myopic policy. Finally, it is shown how the achievable cost of the optimal decision policy varies with change time distribution by imposing a partial order on transition matrices.

Partially Observable Markov Decision Processes (POMDP) provides piecewise-linear a natural and principled framework for sequential decision-making under uncertainty. However, large-scale POMDP suffers from the exponential growth of the belief points and policy trees space. We present a new point-based incremental pruning algorithm based on the piecewise linearity and convexity of the value function. Instead of reasoning about the whole belief space when pruning the cross-sums in POMDP policy construction, our algorithm uses belief points to perform approximate pruning by generating policy trees, and get the optimal policy in real-time belief states. The empirical results indicate that point-based incremental pruning for heuristic search methods can handle large POMDP domains efficiently.

Performance of machine tool tends to deteriorate in the production process. This deterioration increases the processing energy consumption and leads to more defectives and corresponding energy waste. Maintenance can be taken to restore the performance of machine tool and improve the energy efficiency, which has a significant impact on the total energy consumption and productivity. This paper proposes an approach to improve the energy efficiency of the production process through scheduling the maintenance actions of the machine tool, taking into account productivity, product quality, and energy consumption. The deteriorating machine tool is modeled as a discrete-time, discrete-state Markov process. Partially observable Markov decision process (POMDP) framework is applied to develop the maintenance decision-making model, where the joint observation of processing energy consumption and quality of manufactured workpiece is used to infer the status of the machine tool. An optimal maintenance policy maximizing the total expected reward about energy efficiency over a finite horizon is obtained, which consists of a sequence of decision rules corresponding to the optimal action for each belief vector. The characteristics of the optimal policy are illustrated through a numerical example and the effects of parameters on the policy are analyzed.

Maintenance can be the factor of either increasing or decreasing system's availability, so it is valuable work to evaluate a maintenance policy from cost and availability point of view, simultaneously and according to decision maker's priorities. This study proposes a Partially Observable Markov Decision Process (POMDP) framework for a partially observable and stochastically deteriorating system in which inspection and maintenance optimal policies of Condition Based Maintenance (CBM) must be determined to maximize the average profit and availability of the system simultaneously. A recent exact method named Accelerated Vector Pruning method (AVP) and some other popular estimating and exact methods are applied and compared in solving such problems.

Optimal policies for partially observed Markov decision processes (POMDPs) are history-dependent: Decisions are made based on the entire history of observations. Memoryless policies, which take decisions based on the last observation only, are generally considered useless in the literature because we can construct POMDP instances for which optimal memoryless policies are arbitrarily worse than history-dependent ones. Our purpose is to challenge this belief. We show that optimal memoryless policies can be computed efficiently using mixed integer linear programming (MILP), and perform reasonably well on a wide range of instances from the literature. When strengthened with valid inequalities, the linear relaxation of this MILP provides high quality upper-bounds on the value of an optimal history dependent policy. Furthermore, when used with a finite horizon POMDP problem with memoryless policies as rolling optimization problem, a model predictive control approach leads to an efficient history-dependent policy, which we call the short memory in the future (SMF) policy. Basically, the SMF policy leverages these memoryless policies to build an approximation of the Bellman value function. Numerical experiments show the efficiency of our approach on benchmark instances from the literature.

Partially observable Markov decision processes (POMDPs) provide a principled, general framework for robot motion planning in uncertain and dynamic environments. They have been applied to various robotic tasks. However, solving POMDPs exactly is computationally intractable. A major challenge is to scale up POMDP algorithms for complex robotic tasks. Robotic systems often have mixed observability : even when a robot’s state is not fully observable, some components of the state may still be so. We use a factored model to represent separately the fully and partially observable components of a robot’s state and derive a compact lower-dimensional representation of its belief space. This factored representation can be combined with any point-based algorithm to compute approximate POMDP solutions. Experimental results show that on standard test problems, our approach improves the performance of a leading point-based POMDP algorithm by many times.

Entrepreneurship thrives in an environment of profound uncertainty, where feedback emerges as a critical tool for mitigating ambiguity and informing adaptive decision-making. This study explores how entrepreneurs leverage feedback to navigate uncertainty, focusing on its dynamic evolution across two crucial stages of venture development: opportunity identification and exploitation. By analyzing 308 feedback events from 12 venture episodes of Zhejiang entrepreneurs in Europe, this research unpacks the interplay between feedback content, sources, and entrepreneurial actions. Using a Partially Observable Markov Decision Process (POMDP) framework, the study conceptualizes feedback as a probabilistic and iterative process influenced by entrepreneurial actions within uncertain conditions. The findings indicate that feedback evolves as entrepreneurial goals and contexts shift, and actions with entrepreneurs' cognitive reflection result in superior feedback utilization compared to reactive approaches. This research extends entrepreneurship theory by conceptualizing feedback as an evolving process, introducing entrepreneurs' cognitive reflection as a means of leveraging feedback, and demonstrating the applicability of POMDP for modelling adaptive decision-making. Practically, it provides strategies for entrepreneurs and advisors to tailor feedback-seeking behaviours to specific venture goals and stages, fostering resilience and success.

Hierarchical modeling of systems with similar components: A framework for adaptive monitoring and control

Partially observable Markov decision processes (POMDPs) provide a general framework for AI planning, but they lack the structure for representing real world planning problems in a convenient and efficient way. Representations built on logic allow for problems to be specified in a compact and transparent manner. Moreover, decision making algorithms can assume and exploit structure found in the state space, actions, observations, and success criteria, and can solve with relative efficiency problems with large state spaces. In recent years researchers have sought to combine the benefits of logic with the expressiveness of POMDPs. In this paper, we show how to build upon and extend the results in this fusing of logic and decision theory. In particular, we present a compact representation of POMDPs and a method to update beliefs after actions and observations. The key contribution is our compact representation of belief states and of the operations used to update them. We then use heuristic search to find optimal plans that maximize expected total reward given an initial belief state

We consider the general problem of optimal stochastic control of a dedicated-platform that processes one primary function or task (target-task). The dedicated-platform has two modes of action at each period of time: it can attempt to process the target-task at the given period of time, or suspend the target-task for later completion. We formulate the optimal trade-off between the processing cost and the latency in completion of the target-task as a Partially Observable Markov Decision Process (POMDP). By reformulating this POMDP as a Markovian search problem, we prove that the optimal control policies are threshold in nature. Threshold policies are computationally efficient and inexpensive to implement in real time systems. Numerical results demonstrate the effectiveness of these threshold based operating algorithms as compared to non-optimal heuristic algorithms.KeywordsPartially Observable Markov Decision Process (POMDP)optimal threshold policiesdynamic programmingBellman equationtwo-state Markov chainoptimal searchoverlook probabilitysufficient statisticsHidden Markov Model (HMM)

Standard value function approaches to finding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.