Balancing Project Schedule, Cost, and Value under Uncertainty: A Reinforcement Learning Approach

We suggest a new approach to calculating a project's net present value, termed the displaced equity method. Based on a straightforward formula, it analyzes a project partially financed with debt from the perspective that every year the amount of outstanding debt displaces an equivalent amount of equity that otherwise would be tied up in the project. Although they represent distinct shareholders' perspectives, the displaced equity method and the equity residual method yield identical net present values and internal rates of return. Every year, the project's value calculated with the displaced equity method is equal to the sum of the project's debt and equity values. In practice, when the schedule of expected outstanding debt amounts is known, using the displaced equity method is an easy way to estimate the project's net present value.

THE PURPOSE OF [1] was to define conditions, consistent with the expected utility maxim, under which a project's present value could be calculated by application of the income-component discounting notion first introduced in [7]. The anlaysis focused on an individual decision-maker and unfortunately did not consider the corporation with stock trading in a perfectly competitive market. Before turning to the implications of market environment for my results, let us first consider Turnbull's certain world counter-example from which he reasons that my additivity condition does not necessarily hold. One can see that if Turnbull were true to his analysis or to my assumption, that any individual may borrow or lend at a riskless rate without affecting that rate (see [1], p. 995, second paragraph), the U(C1, C2) preference function and his corresponding analysis are irrelevant to this discussion. For if one does allow borrowing and lending at a single interest rate in a certain world-the perfect market assumption, then clearly for any preference function that gives rise to convex indifference curves in (C1, C2) then Pz = Px + Py. To see this draw, the Fisher diagram (Figure 1) where the absolute slope of the market opportunity line is one plus the interest rate, i. Then starting at (C*, C*) it is clear that if the individual is promised an additional Z dollars in period 2 then the maximum price, Pz, that he would be willing to pay in period 1 is Z/(l + i), for he could obtain that trade-off in the market place. Identical reasoning leads to Px= X/(l + i) and Py = Y/(l + i), consequently Pz = Px + Py. My interpretation of the remainder of Turnbull's comment is that the calculation of a project's net present value should simply yield the additions to a firm's future equilibrium stock price and if the market values of the stock's risky component income streams are additive, as they are when equilibrium is represented by a complete set of Arrow-Debreu securities, or in a mean-variance setting with a riskless asset and homogeneous expectations, then additivity holds for the present values or prices of the project's risky income streams. These observations by Turnbull are clearly valid. The additivity principle in the context of a competitive stock market also holds true for heterogeneous expectations, given, of course, equal and costless access to all information by all potential investors. (c.f. [5], and [6].) The common thread connecting these results is that in perfectly competitive security markets all participants achieve efficient portfolio diversification, and therefore equilibrium prices will consequently reflect all possible gains from risk reduction. Since I was negligent in considering the impact of market conditions in deriving my conclusions, some statement should have been made regarding the implications for my results of that other type of market, an imperfect one. If a merger affects

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Cannibalization Risk and Limited Liability: Implications for Firm Valuation and Capital Budgeting

Biomass gasification based power plants can play an important role in power sector in Malaysia with her abundant agricultural and forest resources. In this research energy and economic feasibility, and environmental impact of biomass gasification power plant has been analyzed and assessed for sustainable power generation in Malaysia, the plant being a clean development mechanism supported project. Gasification based power generation with a dual fuel internal combustion engine is found more biomass combustion power plant. But, the annual fuel cost is significantly higher with gasification plant claiming MYR 10 million/y for a generation of 67,500 MWh. The net present value, internal rate of return, and PBP for a 10-MW gasification based dual fuel internal combustion engine power plant without loan financing using empty fruit bunch pellet as biomass; with CER issued at USD16/tCO 2 ; and without loan financing provides net present value, internal rate of return, and PBP of MYR 19.68 million, 11.90%, and 6.05 y, respectively. With a loan financing at 3% per year with CER still be effective the net present value, internal rate of return, and PBP is estimated to be MYR 7.03 million, 3.97%, and 11.78 y, respectively. However, if a 60:40 debt equity is applied then the project net present value, internal rate of return, and payback period seen to improve slightly. The net present value attains a positive value of MYR 3.6 million. Internal rate of return and PBP values are found to be 7.08% and 8.27 y, respectively. The emission reduction achieved with the plant is 17,863 t CO 2 /y, 206 t SO 2 /y, 90 t NOx/y, and 10.37 t CO/y, respectively. Energy efficiency and tariff policy have been found to have highest impact on economic profitability of gasification based power generation units.

Introduction The development of a hydrocarbon resource should be planned to maximize the net present value (NPV) of the project, subject to any imposed constraints. Maximizing the NPV of a thermal heavy-oil project can be complex because of the interplay of individual-well production and injection profiles with field-level production and injection constraints imposed by a central processing facility (CPF). In addition, for thermal heavy-oil-recovery methods such as cyclic-steam stimulation (CSS), the scheduling of the production, soak, and injection cycles of the wells has a significant impact on the overall project NPV. This study presents the results of a novel study to maximize the NPV of a greenfield CSS project by incorporating a newly developed analytical horizontal CSS model coupled to a field-production aggregation and scheduling model, which was in turn coupled to an economic-evaluation model. The close integration of these models allowed for the optimization of input parameters to be achieved simultaneously across all three models to maximize the NPV of the entire project. The integrated model work flow and the resulting optimized case will be summarized and discussed in detail. The significance of the work flow developed in this study is that it demonstrates that the key design parameters (such as the CPF capacity and schedule) of a thermal heavy-oil-exploitation scheme can be calculated and optimized on the basis of the economics of the entire project by use of an integrated model.

A Workflow and Estimate for the Economic Viability of Offshore Wind Projects

Assessing investment value of privately-owned public rental housing projects with multiple options

Project value is highly correlated with decisions made during concept comparison and selection. Decisions as such are made amid uncertainty, putting value at risk. This study details a methodology for determining the loss in project value when inaccurate estimates are used during concept comparison and selection. The difference between net present values (NPVs) based on inaccurate estimates and those based on an alternate hypothesis that is assumed to represent the truth determines the magnitude of loss. The value of the information required to reduce uncertainty can then be obtained.

ABSTRACTSmallholder vegetable farmers involved in agricultural production are confronted with numerous challenges which can adversely affect performance. Farmers would prefer to adopt the most profitable vegetable production systems. A cost-benefit analysis of integrated crop management (ICM) and conventional method (CM) systems for vegetable production was conducted. Primary data were solicited from 120 vegetable farmers using questionnaires. Descriptive statistics were used to describe levels of awareness and extent to which farmers understood use of the ICM system. A cash flow projection was done on a 0.4 ha size of land for 5 years. Net present value (NPV) and cost-benefit ratio (CBR) analyses were performed for farmers operating under the ICM and CM production systems. The NPV analysis indicated production of vegetables under both systems was viable. The incremental NPV for cabbage (Brassica oleracea var. capitata L.) and onion (Allium cepa L.) production, and the whole farm enterprise were all positive, indicating the ICM system was more financially viable than the CM system. The NPV increase was GHS 2563.58 (GHS is Ghanaian currency, 1$US = 3.5 GHS in January 2015 when data were collected) for cabbage and GHS 3949.43 for onion, and of the whole farm enterprise, i.e., combined cabbage and onion production, was GHS 6162.75. The CBR analysis indicated that vegetable production under the two systems was viable, confirming results of the NPV analyses. The CBR for cabbage production was 1.58 for the CM and 2.08 for the ICM systems; the CBR for onion production was 2.69 for the CM and 4.36 for the ICM systems. The CBR for whole farm enterprise was 2.42 for the CM and 3.93 for the ICM systems. Sensitivity analyses, under the assumption of a 5% reduction in yield and a 10% cost over-run, indicated positive NPVs for both production systems for cabbage and onion production, and the whole farm enterprise. The NPVs from use of the ICM system were higher than under the CM system. Overall, vegetable production is profitable under both the ICM and CM systems, and awareness is a factor influencing practice of the ICM system.

Active screening is a common approach in controlling the spread of recurring infectious diseases such as tuberculosis and influenza. In this approach, health workers periodically select a subset of population for screening. However, given the limited number of health workers, only a small subset of the population can be visited in any given time period. Given the recurrent nature of the disease and rapid spreading, the goal is to minimize the number of infections over a long time horizon. Active screening can be formalized as a sequential combinatorial optimization over the network of people and their connections. The main computational challenges in this formalization arise from i) the combinatorial nature of the problem, ii) the need of sequential planning and iii) the uncertainties in the infectiousness states of the population. Previous works on active screening fail to scale to large time horizon while fully considering the future effect of current interventions. In this paper, we propose a novel reinforcement learning (RL) approach based on Deep Q-Networks (DQN), with several innovative adaptations that are designed to address the above challenges. First, we use graph convolutional networks (GCNs) to represent the Q-function that exploit the node correlations of the underlying contact network. Second, to avoid solving a combinatorial optimization problem in each time period, we decompose the node set selection as a sub-sequence of decisions, and further design a two-level RL framework that solves the problem in a hierarchical way. Finally, to speed-up the slow convergence of RL which arises from reward sparseness, we incorporate ideas from curriculum learning into our hierarchical RL approach. We evaluate our RL algorithm on several real-world networks. Results show that our RL algorithm can scale up to 10 times the problem size of state-of-the-art (the variant that considers the effect of future interventions but unscalable) in terms of planning time horizon. Meanwhile, it outperforms state-of-the-art (the variant that scales up but does not consider the effect of future interventions) by up to 33% in solution quality.

SummaryThe exploration and production (E&P) industry is facing a net-present-value (NPV) paradox. Despite the fact that the NPV method is widely criticized by practitioners and academics alike, the NPV method remains the cornerstone of E&P project valuation. We posit that this contradiction, which we labeled the NPV paradox, is likely to be caused by a combination of limitations of the method, a lack of theoretical understanding, and ambiguity regarding the implementation of the NPV method.Even though the NPV method has been described in numerous papers and textbooks, rigorous and succinct guidance on how to determine risk premiums for systematic risk is not available. We demonstrate that risk-adjusted discount rates are very sensitive to the choice of the length of the periods over which returns are determined (daily, weekly, or monthly), length of the time horizons considered (such as 10 or 25 years), and start date (such as 1965 or 1990). We discuss the fundamental implications and rationale of choices and their effects on the variables that underpin the risk-adjusted discount rate: risk-free rate, company β, market-risk premium, and the cost of debt.Although not entirely satisfactory, we argue for a moderate downward revision of discount rates for projects with timelines exceeding 20 years. This recommendation is dependent on recent advancements in public finance and the reality that the exposure to systematic risk in the long run is significantly less in many real-life E&P projects than the capital-assessment-pricing model (CAPM) implies. The inflated discount rate that is currently used, combined with the extended investment horizons that are common in the upstream sector, will for example result in an underweighting of decommissioning and future legacy costs.In addition to a set of widely recognized shortcomings of the NPV model, there are also lesser well-known issues. For example, the failure of CAPM to capture bankruptcy risk has a bearing on the project-risk premium. Also, the application of the NPV model implies a set of probabilistic assumptions around market risks that are likely to be invalidated when evaluating a set of market scenarios or using a series of probability-weighted market scenarios.

Reinforcement learning for a class of continuous-time input constrained optimal control problems

Hydrocarbon field (re-)development projects require the evaluation of a large number of development options under uncertainty. Furthermore, information of data gathering programs might result in narrowing parameter ranges and change the choice of the preferred development option. The large number of development options (and decisions accordingly) which have to be taken under uncertainty leads to the necessity to determine the impact of the decisions on the (re-)development project objective. Knowing the sensitivity of the decisions on the project objective (e.g. NPV) allows for resource and data acquisition planning. The impact of decisions on the project value can be determined by performing a Generalized Sensitivity Analysis. This analysis does not replace Value of Information but facilitates planning and allows focusing on important decisions. To further improve Decision Analysis and focus on important parameters, a Generalized Sensitivity Analysis of uncertain parameters on decisions can be performed. The advantage of such an investigation over sensitivity analysis on Oil Originally in Place (OOIP) or Net Present Value (NPV) is that it includes parameter interactions. Furthermore, it covers the impact of a parameter on the decisions directly rather than indirectly when OOIP or NPV sensitivity is used. The analysis is shown at an example project in a Decision Analysis framework. The use of decision impact evaluation and parameter assessment on decisions might lead to more focused and faster hydrocarbon field (re-)development project execution.

Green roofs are considered to be one of the optimal tools for saving energy and protecting the environment in developed countries. In this paper, an analysis of the possible application of green roofs on existing residential buildings with flat roofs is presented. In the economic analysis, models of existing buildings in Belgrade, with two different types of green roofs, are studied. A key indicator of investment profitability in this investigation is the net present value (NPV) of the green roof project. Besides the private economic impact, other aspects of green roof applications, significant for sustainable development, have been highlighted. The values of the reductions in the annual energy needed for heating and cooling are compared for different scenarios. A maximum energy saving of 22% in the heating season is determined in the building energy simulation program for the model with an intensive green roof. Life cycle profit analysis was based on the probabilistic approach. The corresponding variance-based sensitivity analysis determined the impact of various parameters on the final result. In all models, the first order sensitivity index, which measures the impact of the number of residential units on the NPV, ranges from 12.2% to 63.6%. Sensitivity analysis showed that the benefit of property value increase has the highest influence on the calculated NPV in scenarios that account for this benefit. The obtained results in those scenarios indicate that the most probable NPV at the end of the life cycle is EUR 43/m2 and EUR 82/m2 for extensive and intensive green roofs, respectively.