Probabilistic decision graphs for optimization under uncertainty

Influence diagrams are a powerful graphic representation for decision models, complementary to decision trees. Influence diagrams and decision trees are different graphic representations for the same underlying mathematical model and operations. This article describes the elements of an influence diagram, and shows several familiar decision problems represented as decision trees and as influence diagrams. The authors also contrast the information highlighted in each graphic representation, demonstrate how to calculate the expected utilities of decision alternatives modeled with an influence diagram, provide an overview of the conceptual basis of the solution algorithms that have been developed for influence diagrams, discuss the strengths and limitations of influence diagrams relative to decision trees, and describe the mathematical operations that are used to evaluate both decision trees and influence diagrams. They use clinical examples to illustrate the mathematical operations of the influence-diagram-evaluation algorithm; these operations are arc reversal, chance node removal by averaging, and decision node removal by policy determination. Influence diagrams may be helpful when problems have a high degree of conditional independence, when large models are needed, when communication of the probabilistic relationships is important, or when the analysis requires extensive Bayesian updating. The choice of graphic representation should be governed by convenience, and will depend on the problem being analyzed, on the experience of the analyst, and on the background of the consumers of the analysis.

A computational theory of decision networks

Building an influence diagram in a knowledge-based decision system

Given a set of geometric objects each associated with a time value, we wish to determine whether a given property is true for a subset of those objects whose time values fall within a query time window. We call such problems time-windowed decision problems, and they have been the subject of much recent attention, for instance studied by Bokal, Cabello, and Eppstein [SoCG 2015]. In this paper, we present new approaches to this class of problems that are conceptually simpler than Bokal et al.'s, and also lead to faster algorithms. For instance, we present algorithms for preprocessing for the time-windowed 2D diameter decision problem in O(n log n) time and the time-windowed 2D convex hull area decision problem in O(n alpha(n) log n) time (where alpha is the inverse Ackermann function), improving Bokal et al.'s O(n log^2 n) and O(n log n loglog n) solutions respectively. Our first approach is to reduce time-windowed decision problems to a generalized range successor problem, which we solve using a novel way to search range trees. Our other approach is to use dynamic data structures directly, taking advantage of a new observation that the total number of combinatorial changes to a planar convex hull is near linear for any FIFO update sequence, in which deletions occur in the same order as insertions. We also apply these approaches to obtain the first O(n polylog n) algorithms for the time-windowed 3D diameter decision and 2D orthogonal segment intersection detection problems.

This chapter starts with the formulation of a case containing enough information to illustrate the construction of the decision matrix and problem solving. Probably the most valuable feature of this example is the thorough analysis performed when using several objective functions. The idea is to demonstrate the wealth of infor­mation that can be extracted from the model and how it can detect if there have been shortcomings in establishing the mathematical model. It also reveals how the information provided can help the DM in making clear that some concepts are worth reviewing in the light of information that cast a doubt about early stage concepts. It finalizes with analysis of how weights assigned to the diverse objectives may affect the solution.

We introduce the notion of LImited Memory Influence Diagram (LIMID) to describe multistage decision problems in which the traditional assumption of no forgetting is relaxed. This can be relevant in situations with multiple decision makers or when decisions must be prescribed under memory constraints, such as in partially observed Markov decision processes (POMDPs). We give an algorithm for improving any given strategy by local computation of single policy updates and investigate conditions for the resulting strategy to be optimal.

LImited Memory Influence Diagrams (LIMIDs) are general models of decision problems for representing limited memory policies (Lauritzen and Nilsson (2001)). The evaluation of LIMIDs can be done by Single Policy Updating that produces a local maximum strategy in which no single policy modification can increase the expected utility. This paper examines the quality of the obtained local maximum strategy and proposes three different methods for evaluating LIMIDs. The first algorithm, Temporal Policy Updating, resembles Single Policy Updating. The second algorithm, Greedy Search, successively updates the policy that gives the highest expected utility improvement. The final algorithm, Simulating Annealing, differs from the two preceeding by allowing the search to take some downhill steps to escape a local maximum. A careful comparison of the algorithms is provided both in terms of the quality of the obtained strategies, and in terms of implementation of the algorithms including some considerations of the computational complexity.

Following the Bayesian paradigm, we derive the likelihood of inspection data conditional on the parameter of interest-the lifetime. The inspection data collect,ed in service thus provides sequential updating of our prediction. The prior distribution of the lifetime parameters can be constructed using laboratory dat,a. The updating mecl~anism, in connection with the decision problems in maintenance policy, is graphically represented as an influence diagram.

Unconstrained influence diagrams extend the language of influence diagrams to cope with decision problems in which the order of the decisions is unspecified. Thus, when solving an unconstrained influence diagram, we not only look for an optimal policy for each decision but also for a so-called step policy specifying the next decision given the observations made so far. However, due to the complexity of the problem, temporal constraints can force the decision maker to act before the solution algorithm has finished and, in particular, before an optimal policy for the first decision has been computed. This paper addresses this problem by proposing an anytime algorithm that at any time provides a qualified recommendation for the first decisions of the problem. The algorithm performs a heuristic-based search in a decision tree representation of the problem. We provide a framework for analyzing the performance of the algorithm, and experiments based on this framework indicate that the proposed algorithm performs significantly better under time constraints than dynamic programming.

Using influence diagrams for data worth analysis

Given a set of geometric objects each associated with a time value, we wish to determine whether a given property is true for a subset of those objects whose time values fall within a query time window. We call such problems time-windowed decision problems, and they have been the subject of much recent attention, for instance studied by Bokal et al. (in: Proceedings of the 31st International Symposium on Computational Geometry (SoCG), pp 240–254, 2015). In this paper, we present new approaches to this class of problems that are conceptually simpler than Bokal et al. ’s, and also lead to faster algorithms. For instance, we present algorithms for preprocessing for both the time-windowed 2D diameter decision problem and the time-windowed 2D convex hull area decision problem in $$O(n \log n)$$ time, improving Bokal et al. ’s $$O(n \log ^2 n)$$ and $$O(n \log n \log \log n)$$ solutions respectively. Our first approach is to reduce time-windowed decision problems to a generalized range successor problem, which we solve using a novel way to search range trees. Our other approach is to use dynamic data structures directly, taking advantage of a new observation that the total number of combinatorial changes to a planar convex hull is linear for any FIFO update sequence, in which deletions occur in the same order as insertions. We also apply these approaches to obtain the first $$O(n\, \mathrm{polylog}\, n)$$ algorithms for the time-windowed 3D diameter decision and 2D orthogonal segment intersection detection problems.

We develop new graphical representations for the problem of sequential decision making in partially observable multiagent environments, as formalized by interactive partially observable Markov decision processes (I-POMDPs). The graphical models called interactive influence diagrams (I-IDs) and their dynamic counterparts, interactive dynamic influence diagrams (I-DIDs), seek to explicitly model the structure that is often present in real-world problems by decomposing the situation into chance and decision variables, and the dependencies between the variables. I-DIDs generalize DIDs, which may be viewed as graphical representations of POMDPs, to multiagent settings in the same way that I-POMDPs generalize POMDPs. I-DIDs may be used to compute the policy of an agent given its belief as the agent acts and observes in a setting that is populated by other interacting agents. Using several examples, we show how I-IDs and I-DIDs may be applied and demonstrate their usefulness. We also show how the models may be solved using the standard algorithms that are applicable to DIDs. Solving I-DIDs exactly involves knowing the solutions of possible models of the other agents. The space of models grows exponentially with the number of time steps. We present a method of solving I-DIDs approximately by limiting the number of other agents’ candidate models at each time step to a constant. We do this by clustering models that are likely to be behaviorally equivalent and selecting a representative set from the clusters. We discuss the error bound of the approximation technique and demonstrate its empirical performance.

We study the stochastic versions of a broad class of combinatorial problems where the weights of the elements in the input dataset are uncertain. The class of problems that we study includes shortest paths, minimum weight spanning trees, and minimum weight matchings over probabilistic graphs, and other combinatorial problems like knapsack. We observe that the expected value is inadequate in capturing different types of risk averse or risk-prone behaviors, and instead we consider a more general objective which is to maximize the expected utility of the solution for some given utility function, rather than the expected weight (expected weight becomes a special case). We show that we can obtain a polynomial time approximation algorithm with additive error ϵ for any ϵ >; 0, if there is a pseudopolynomial time algorithm for the exact version of the problem (This is true for the problems mentioned above) and the maximum value of the utility function is bounded by a constant. Our result generalizes several prior results on stochastic shortest path, stochastic spanning tree, and stochastic knapsack. Our algorithm for utility maximization makes use of the separability of exponential utility and a technique to decompose a general utility function into exponential utility functions, which may be useful in other stochastic optimization problems.

This paper studies a class of renewable resource allocation problems for the processing of dynamically arriving tasks with deterministic deadlines. This class of problems has many applications, however, it conforms to neither the standard resource allocation model nor the standard optimal control model. A new problem formulation has to be developed and analyzed to provide a satisfactory solution. In this paper, a new formulation is presented. By augmenting the state variabls, the problem is converted into a Markovian decision problem. It can then be treated, at least in principle, by using the stochastic dynamic programming (SDP) method. However, since the system dynamics involves the evolution of sets (of tasks and resources), the implementation of the dynamic programming equation is by no means straightforward. For a problem with infinite planning horizon, the optimal strategy is shown to be stationary under mild conditions. An SDP algorithm based on a successive approximation technique is developed to obtain the optimal stationary strategy. The implementation of the algorithm employs a special coding scheme to handle set variables, and utilizes a dominance property for computational efficiency. Effects of key system parameters on optimal decisions are investigated and analyzed through numerical examples. As the computational complexity of the algorithm is of exponential increase, practical applications of the algorithm is limited to problems of moderate size. Two heuristic rules are therefore investigated and compared to the optimal policy.

Decisions are usually taken sequentially in climate change policy: every certain amount of years, new agreements and promises are made about greenhouse gas emission reduction etc. In the intersection of decision theory and climate science, sequential decision problems can be formulated and solved, to find optimal sequences of policies and support policy makers with some advice.There are, however, many uncertainties affecting the outcome of these optimisations. Since these decision problems tend to be very simple in comparison with the complexity of the real world, knowing how different uncertainties affect optimal policies might be more important than what the optimal policy comes out to be. In this work, we explore how some uncertainties affect optimal policies and the possible trajectories associated with those optimal policies.   For this aim we formulate a sequential decision problem with a single "global" policy maker. The decision problem starts with the world state in 2020 and decisions are taken every 10 years till 2100. The policy maker has options regarding CO2 emissions reduction, geoengineering in the form of solar radiation modification and carbon dioxide removal.We simulate the effects of the decisions on the world’s state with SURFER. SURFER is a simple and fast model featuring a carbon cycle responsive to positive and negative emissions, it allows for geoengineering and accounts for sea level rise from ice sheets (containing tipping points) and from ocean expansion and glacier melt. SURFER has been shown to reproduce the globally averaged behavior of earth system models and models of intermediate complexity from decades to millennia. As opposed to some optimal decision problems in the context of climate change which use integrated assessment models of the climate and the economy, here, with the aim of transparency and simplicity, we consider only a climate model. We define a modular and transparent cost function that contains what the policy maker cares about. This function is a linear sum of costs associated with: green transition, geoengineering use and risks, temperature and ocean acidification damages and long term sea level rise commitments.Using this decision problem we investigate how different kinds of uncertainties affect the sequence of optimal policies obtained and the optimal trajectories associated with those optimal policies. We consider three different kinds of uncertainties: uncertainties in the priorities of the decision maker (i.e., in the reward, cost or utility function), uncertainties on some physical parameters (in particular, climate sensitivity and ice sheet tipping points) and political uncertainty (policymaker’s decisions may not be implemented).