Applying unweighted least‐squares based techniques to stochastic dynamic programming: theory and application

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

This research is about the development of a dynamic programming model for solving fuzzy linear programming problems. Initially, fuzzy dynamic linear programming model FDLP is developed. This research revises the established dynamic programming model for solving linear programming problems in a crisp environment. The mentioned approach is upgraded to address the problem in an uncertain environment. Dynamic programming model can either be passing forward or backward. In the proposed approach backward dynamic programming approach is adopted to address the problem. It is then followed by implementing the proposed method on the education system of Pakistan. The education system of Pakistan comprises of the Primary, Middle, Secondary, and Tertiary education stages. The problem is to maximize the efficiency of the education system while achieving the targets with minimum usage of the constrained resources. Likewise the model tries to maximize the enrollment in the Primary, Middle, Secondary and Tertiary educational categories, subject to the total available resources in a fuzzy uncertain environment. The solution proposes that the enrollment can be increased by an amount 9997130, by increasing the enrollment in the Middle and Tertiary educational categories. Thus the proposed method contributes to increase the objective function value by 30%. Moreover, the proposed solutions violate none of the constraints. In other words, the problem of resources allocation in education system is efficiently managed to increase efficiency while remaining in the available constrained resources. The motivation behind using the dynamic programming methodology is that it always possesses a numerical solution, unlike the other approaches having no solution at certain times. The proposed fuzzy model takes into account uncertainty in the linear programming modeling process and is more robust, flexible and practicable.

Photo by Josh Riemer on Unsplash
 Introduction
 With the rapid advancements in neurotechnological machinery and improved analytical insights from machine learning in neuroscience, the availability of big brain data has increased tremendously. Neurological health research is done using digitized brain data.[1] There must be adequate data governance to secure the privacy of subjects participating in brain research and treatments. If not properly regulated, the research methods could lead to significant breaches of the subject’s autonomy and privacy. This paper will address the necessity for neuroprotection laws, which effectively govern the use of big brain data to ensure respect for patient privacy and autonomy.
 Background
 Artificial intelligence and machine learning can be integrated with neuroscience big brain data to drive research studies. This integrative technology allows patterns of electrical activity in neurons to be studied in detail.[2]Specifically, it uses a robotic system which can reason, plan, and exhibit biologically intelligent behavior. Machine learning is a method of computer programming where the code can adapt its behavior based on big brain data.[3] The big brain data is the collection of large amounts of information for the purpose of deciphering patterns through computer analysis using machine learning.[4] The information that these technologies provide is extensive enough to allow a researcher to read a patient’s mind. AI and machine learning technologies work by finding the underlying structure of brain data, which is then described by patterns known as latent factors, eventually resulting in an understanding of the brain’s temporal dynamics.[5]
 Through these technologies, researchers are able to decipher how the human brain computes its performances and thoughts. However, due to the extensive and complex nature of the data processed through AI and machine learning, researchers may gain access to personal information a patient may not wish to reveal. From a bioethical lens, tensions arise in the realm of patient autonomy. Patients are not able to control the transmission of data from their brains that is analyzed by researchers. Governing brain data through laws may enhance the extent of patient privacy in the case where brain data is being used through AI technologies.[6] A responsible approach to governing brain data would require a sophisticated legal structure.
 Analysis
 Impact on Patient Autonomy and Privacy 
 In research pertaining to big brain data, the consent forms do not fully cover the vast amounts of information that is collected. According to research, personal data has become the most sought out commodity to provide content to corporations and the web-based service industry. Unfortunately, data leaks that release private information frequently occur.[7] The storage of an individual’s data on technologies accessible on the internet during research studies makes it vulnerable to leaks, jeopardizing an individual’s privacy. These data leaks may cause the patient to be identified easily, as the degree of information provided by AI technologies are personalized and may be decoded through brain fingerprinting methods.[8]
 There has been an extensive growth in the development and use of AI. It is efficient in providing information to radiologists who diagnose various diseases including brain cancer and psychiatric disease, and AI assists in the delivery of telemedicine.[9] However, the ethical pitfall of reduced patient autonomy must be addressed by analyzing current AI technologies and creating more options for patient preference in how the data may be used. For instance, facial recognition technology[10] commonly used in health care produces more information than listed in common consent forms, threatening to undermine informed consent. Facial recognition software collects extensive data and may disclose more information than a person would prefer to provide despite being a useful tool for diagnosing medical and genetic conditions.[11] In addition, people may not be aware that their images are being used to generate more clinical data for other purposes. It is difficult to guarantee the data is anonymized. Consent requirements must include informing people about the complexity of the potential uses of the data; software developers should maximize patient privacy.[12] Furthermore, there is a “human element” in the use of AI technologies as medical providers control the use and the extent to which data is captured or accessed through the AI technologies.[13] People must understand the scope of the technology and have clear communication with the physician or health care provider about how the medical information will be used. 
 Existing Laws for Brain Data Governance 
 A strict system of defined legal responsibilities of medical providers will ensure a higher degree of patient privacy and autonomy when AI technologies and data from machine learning are used. Governing specific algorithmic data is crucial in safeguarding a patient’s privacy and developing a gold standard treatment protocol following the procurement of the information.[14] Certain AI technologies provide more data than others, and legal boundaries should be established to ensure strong performance, quality control, and scope for patient privacy and autonomy. For instance, currently AI technologies are being used in the realm of intensive neurological care. However, there is a significant level of patient uncertainty about how much control patients have over the data’s uses.[15] Calibrated legal and ethical standards will allow important brain data to be securely governed and monitored.
 Once brain signals are recorded and processed from one individual, the data may be merged with other data in Brain Computer Interface Technology (BCI).[16] To ensure a right and ability to retrieve personal data or pull it from the collection, specific regulations for varying types of data are needed.[17] The importance of consent and patient privacy must be considered through giving patients a transparent view of how brain data is governed.[18] The legal system must address discriminatory issues and risks to patients whose data is used in studies. Laws like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Protection Act (CCPA) can serve as effective models to protect aggregated data. These laws govern consumer information and ensure the compliance when personal data is collected.[19] California voters recently approved expansion of the CCPA to health data. The Washington Privacy Act, which would have provided rights to access, change, and withdraw personal data, failed to pass. Other states should improve privacy as well,[20] although a federal bill would be preferable. Scientists at the Heidelberg Academy of Sciences argue for data security to be governed in a manner that balances patient privacy and autonomy with the commercial interests of researchers.[21] The balance could be achieved through privacy protections like those in the Washington Privacy Act. Although the Health Insurance Portability and Accountability Act (HIPAA) provides an overall framework to deter the likelihood of dangers to patient protection and privacy, more thorough laws are warranted to combat pervasive data transfer and analysis that technology has brought to the health care industry.[22] Breaches of patient privacy under current HIPAA regulations include releasing patient information to a reporter without their consent and sending HIV data to a patient’s employer without consent.[23] HIPAA does not cover information being shared with outside contractors who do not have an agreement with technology companies to keep patient data confidential. HIPAA regulations also do not always address blatant breaches on patient data confidentiality.[24] Patients must be provided with methods to monitor the data being analyzed to be able to view the extent of private information being generated via AI technologies. In health research, the medical purposes of better diagnosis, earlier detection of diseases, or prevention are ethical justifications for the use of the data if it was collected with permission, the person understood and approved the uses of the data, and the data was deidentified.
 A standard governance framework is required in providing the fairest system of care to patients who allow their brain data to be examined. Informed consent in the neuroscience field could reaffirm the privacy and autonomy of patients by ensuring that they understand the type of information collected. Laws also could protect data after a patient’s death. Malpractice in the scope of brain data could give people a cause of action critical in safeguarding patient’s rights. Data breach lawsuits will become common but generally do not cover deidentified data that becomes part of big data collection. A more synchronized approach to the collection and consent process will encourage an understanding of how big data is used to diagnose and treat patients. Some altruistic people may even be more likely to consent if they know the largescale data collection is helpful to treat and diagnose people. Others should have the ability to opt out of sharing neurological data, especially when there is not certainty surrounding deidentification.[25]
 Conclusion
 Artificial intelligence and machine learning technologies have the potential to aid in the diagnosis and treatment of people globally by extracting and aggregating brain data specific to individuals. However, the secure use of the data is necessary to build trust between care providers and patients, as well as in balancing the bioethical principles of beneficence and patient autonomy. We must ensure the highest quality of care to patients, while protecting their privacy, informed consent, and clinical trust. More sophis

This paper examines long-term optimal operation using dynamic programming for a large hydropower system of 10 reservoirs in Northeast China. Besides considering flow and hydraulic head, the optimization explicitly includes time-varying electricity market prices to maximize benefit. Two techniques are used to reduce the 'curse of dimensionality' of dynamic programming with many reservoirs. Discrete differential dynamic programming (DDDP) reduces the search space and computer memory needed. Object-oriented programming (OOP) and the ability to dynamically allocate and release memory with the C++ language greatly reduces the cumulative effect of computer memory for solving multi-dimensional dynamic programming models. The case study shows that the model can reduce the 'curse of dimensionality' and achieve satisfactory results.

Open AccessOpen Access licenseAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail Go to SectionOpen AccessOpen Access license HomeStochastic SystemsVol. 3, No. 2 Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct SearchMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluPublished Online:15 Nov 2013https://doi.org/10.1287/10-SSY020AbstractIn this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation. Back to Top Next FiguresReferencesRelatedInformationCited byInpatient Overflow: An Approximate Dynamic Programming ApproachJ. G. Dai, Pengyi Shi16 May 2019 | Manufacturing & Service Operations Management, Vol. 21, No. 4A Unified Framework for Optimization Under UncertaintyWarren B. Powell4 November 2016The Minimum Expected Penalty Relocation Problem for the Computation of Compliance Tables for Ambulance VehiclesThije van Barneveld21 April 2016 | INFORMS Journal on Computing, Vol. 28, No. 2Discrete Optimization Models for Homeland Security and Disaster ManagementLaura Albert McLay26 October 2015Simulation-Based Approximate Policy Iteration with Generalized Logistic FunctionsAntoine Sauré, Jonathan Patrick, Martin L. Puterman28 September 2015 | INFORMS Journal on Computing, Vol. 27, No. 3Technical Note—Trading Off Quick versus Slow Actions in Optimal SearchSteven M. Shechter, Farhad Ghassemi, Yasin Gocgun, Martin L. Puterman6 March 2015 | Operations Research, Vol. 63, No. 2A Bound on the Performance of an Optimal Ambulance Redeployment PolicyMatthew S. Maxwell, Eric Cao Ni, Chaoxu Tong, Shane G. Henderson, Huseyin Topaloglu, Susan R. Hunter16 July 2014 | Operations Research, Vol. 62, No. 5 Volume 3, Issue 2December 2013Pages 322-633 Article Information Metrics Information Received:November 01, 2010Published Online:November 15, 2013 Copyright © 2013, The author(s)Cite asMatthew S. Maxwell, Shane G. Henderson, Huseyin Topaloglu (2013) Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search. Stochastic Systems 3(2):322-361. https://doi.org/10.1287/10-SSY020 PDF download

Chapter 22 Duality Theory and Approximate Dynamic Programming for Pricing American Options and Portfolio Optimization

Approximate dynamic programming based approach to process control and scheduling

With the development of marine science, aeronautics and astronautics, energy, chemical industry, biomedicine and management science, many complex systems face the problem of optimization and control. Approximate dynamic programming solves the curse of dimensionality problem of dynamic programming, and it is a new kind of approximate optimization solution that emerges in recent years. Based on the analysis of optimization system, this paper proposes a nonlinear multi-input multi-output, online learning, and data-driven approximate dynamic programming structure and its learning algorithm. The method is achieved from the following three aspects: 1) the critic function of multi-dimensional input critic module of the approximate dynamic programming is approximated with a data-driven k-nearest neighbor method; 2) the multi-output policy iteration of the approximate dynamic programming actor module is calculated with an exponential convergence performance; 3) The critic and actor modules are learned synchronously, and achieve the online optimal and control effect. The optimal control for the longitudinal motion of a thermal underwater glider is used to show the effect of the proposed method. This work can lay a foundation for the theory and application of a nonlinear data-driven multi-input multi-output approximate dynamic programming method. It's also the consensus needs in optimization control and artificial intelligence of many scientific and engineering fields, such as energy conservation, emission reduction, decision support and operational management etc.

We study the approximate dynamic programming approach to revenue management in the context of attended home delivery. We draw on results from dynamic programming theory for Markov decision problems, convex optimisation and discrete convex analysis to show that the underlying dynamic programming operator has a unique fixed point. Moreover, we also show that - under certain assumptions - for all time steps in the dynamic program, the value function admits a continuous extension, which is a finite-valued, concave function of its state variables. This result opens the road for achieving scalable implementations of the proposed formulation, as it allows making informed choices of basis functions in an approximate dynamic programming context. We illustrate our findings using a simple numerical example and conclude with suggestions on how our results can be exploited in future work to obtain closer approximations of the value function.

Optimal Exercise Patterns and Valuation of Employee Stock Options: An Approximate Dynamic Programming Approach

A Least-Squares Temporal Difference based method for solving resource allocation problems

This paper advances a neural-network-based approximate dynamic programming control mechanism that can be applied to complex control problems such as helicopter flight control design. Based on direct neural dynamic programming (DNDP), an approximate dynamic programming methodology, the control system is tailored to learn to maneuver a helicopter. The paper consists of a comprehensive treatise of this DNDP-based tracking control framework and extensive simulation studies for an Apache helicopter. A trim network is developed and seamlessly integrated into the neural dynamic programming (NDP) controller as part of a baseline structure for controlling complex nonlinear systems such as a helicopter. Design robustness is addressed by performing simulations under various disturbance conditions. All designs are tested using FLYRT, a sophisticated industrial scale nonlinear validated model of the Apache helicopter. This is probably the first time that an approximate dynamic programming methodology has been systematically applied to, and evaluated on, a complex, continuous state, multiple-input multiple-output nonlinear system with uncertainty. Though illustrated for helicopters, the DNDP control system framework should be applicable to general purpose tracking control.

Many continuous process industries are required to operate multiproduct supply chains under significant uncertainty in market demands and prices, and mathematical programming formulations have been developed for the deterministic version of the problem. However, available extensions to the stochastic case cannot be readily used to obtain an optimal operating policy in practice because the uncertainty results in an explosion of the scenarios to be considered over an extended time horizon. In this article we represent the uncertainty through Markov chains and use an approach based on stochastic dynamic programming (DP), which can generate a dynamic operating policy that incorporates information about the uncertainty in the problem at each time step. DP is computationally infeasible because of the size of the state and action space for realistic problems. We propose a novel stochastic optimization algorithmic framework called “DP in a heuristically restricted state space.” The confined state space is generated by simulating various potential scenarios under centralized dynamic inventory and production policy generated by combining static local inventory policy heuristics. The resulting DP policy responds to the time‐varying demand for products by appropriately stitching together decisions made by the local static policies. This is a new formulation for effectively combining heuristic policies. © 2006 American Institute of Chemical Engineers AIChE J, 2006

In any complex or large scale sequential decision making problem, there is a crucial need to use function approximation to represent the relevant functions such as the value function or the policy. The Dynamic Programming (DP) and Reinforcement Learning (RL) methods introduced in previous chapters make the implicit assumption that the value function can be perfectly represented (i.e. kept in memory), for example by using a look-up table (with a finite number of entries) assigning a value to all possible states (assumed to be finite) of the system. Those methods are called exact because they provide an exact computation of the optimal solution of the considered problem (or at least, enable the computations to converge to this optimal solution). However, such methods often apply to toy problems only, since in most interesting applications, the number of possible states is so large (and possibly infinite if we consider continuous spaces) that a perfect representation of the function at all states is impossible. It becomes necessary to approximate the function by using a moderate number of coefficients (which can be stored in a computer), and therefore extend the range of DP and RL to methods using such approximate representations. These approximate methods combine DP and RL methods with function approximation tools.

Today, the digital currency market represented by bitcoin has become an important part of the capital market. As the COVID-19 pandemic has increased volatility in capital markets in the past two years, effective trend forecasting and risk-averse investment strategies can bring substantial profits to companies and individuals and reduce the possibility of financial risks. To achieve the above objectives, we established the following models: (1)price prediction models of gold and bitcoin; (2)trading market risk judgment model; (3)revenue valuation model based on dynamic programming. This paper uses ARIMA and LSTM neural network to establish the prediction model. After checking the historical data, the best parameter fitting values of ARIMA model were obtained by using the exhaustive method according to BIC criterion, which were ARIMA (4,1,3) (gold) and ARIMA (2,1,2) (bitcoin) respectively. Then LSTM recurrent neural network is used to modify the forecast results, and the price forecast graph with high fitting degree with the real data is obtained. Then, this paper combines the technical indicators commonly used in the financial market to form a market risk judgment model. On the basis of the eight laws of buying and selling, the weight index is introduced to put forward the bull and bear market judgment rules of gold and bitcoin trading, and the tendency distribution map of the market environment in the past five years is obtained. At the same time, the buying risk coefficient is quantified by combining the maximum retraction rate. On the basis of the first two sub-models, this paper establishes a revenue valuation model based on the idea of dynamic programming. By setting the policy trigger threshold to control the risk, the problem that local optimal solution may appear in the model is improved.