- Research Article
19
- 10.1016/j.artint.2014.07.001
Depth-based short-sighted stochastic shortest path problems
- Jul 14, 2014
- Artificial Intelligence
- Felipe W Trevizan + 1 more +1
Depth-based short-sighted stochastic shortest path problems
Extended dominance and a stochastic shortest path problem
Depth-based short-sighted stochastic shortest path problems
Depth-based short-sighted stochastic shortest path problems
Maximizing Cumulative User Engagement in Sequential Recommendation
To maximize cumulative user engagement (e.g. cumulative clicks) in sequential recommendation, it is often needed to tradeoff two potentially conflicting objectives, that is, pursuing higher immediate user engagement (e.g., click-through rate) and encouraging user browsing (i.e., more items exposured). Existing works often study these two tasks separately, thus tend to result in sub-optimal results. In this paper, we study this problem from an online optimization perspective, and propose a flexible and practical framework to explicitly tradeoff longer user browsing length and high immediate user engagement. Specifically, by considering items as actions, user's requests as states and user leaving as an absorbing state, we formulate each user's behavior as a personalized Markov decision process (MDP), and the problem of maximizing cumulative user engagement is reduced to a stochastic shortest path (SSP) problem. Meanwhile, with immediate user engagement and quit probability estimation, it is shown that the SSP problem can be efficiently solved via dynamic programming. Experiments on real-world datasets demonstrate the effectiveness of the proposed approach. Moreover, this approach is deployed at a large E-commerce platform, achieved over 7% improvement of cumulative clicks.
Read moreUtilization of a dependency-tracking language to reduce computational time during multidisciplinary design optimization
Utilization of a dependency-tracking language to reduce computational time during multidisciplinary design optimization
Solving in real-time the dynamic and stochastic shortest path problem for electric vehicles by a prognostic decision making strategy
Solving in real-time the dynamic and stochastic shortest path problem for electric vehicles by a prognostic decision making strategy
Read moreOptimal hyperpaths with non-additive link costs
Optimal hyperpaths with non-additive link costs
POWER OPTIMIZATION OF MULTI-FLUID TRANSPORTATION SYSTEMS
POWER OPTIMIZATION OF MULTI-FLUID TRANSPORTATION SYSTEMS
Path integral control and bounded rationality
Path integral methods have recently been shown to be applicable to a very general class of optimal control problems. Here we examine the path integral formalism from a decision-theoretic point of view, since an optimal controller can always be regarded as an instance of a perfectly rational decision-maker that chooses its actions so as to maximize its expected utility. The problem with perfect rationality is, however, that finding optimal actions is often very difficult due to prohibitive computational resource costs that are not taken into account. In contrast, a bounded rational decision-maker has only limited resources and therefore needs to strike some compromise between the desired utility and the required resource costs. In particular, we suggest an information-theoretic measure of resource costs that can be derived axiomatically. As a consequence we obtain a variational principle for choice probabilities that trades off maximizing a given utility criterion and avoiding resource costs that arise due to deviating from initially given default choice probabilities. The resulting bounded rational policies are in general probabilistic. We show that the solutions found by the path integral formalism are such bounded rational policies. Furthermore, we show that the same formalism generalizes to discrete control problems, leading to linearly solvable bounded rational control policies in the case of Markov systems. Importantly, Bellman's optimality principle is not presupposed by this variational principle, but it can be derived as a limit case. This suggests that the information-theoretic formalization of bounded rationality might serve as a general principle in control design that unifies a number of recently reported approximate optimal control methods both in the continuous and discrete domain.
Read moreData‐driven optimal event‐triggered consensus control for unknown nonlinear multiagent systems with control constraints
SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.
Read moreRobust Controllability of Boolean Control Networks via Dynamic Programming.
This article presents a novel dynamic programming approach to determine the robust controllability of Boolean control networks (BCNs) subject to stochastic disturbances. By applying Bellman's optimality principle, we derive the recurrence relation for computing the optimal time matrix, a crucial concept characterizing robust reachability between two arbitrary states. We develop a finite-termination dynamic programming algorithm to calculate the optimal time matrix exactly and efficiently, with a rigorously certified iteration count. Sufficient and necessary conditions for robust controllability are then established based on the optimal time matrix. Furthermore, for any pair of reachable states, we construct time-optimal state feedback control laws to steer the system from the initial state to the target state, regardless of disturbances. Finally, extensive numerical experiments with biological networks validate the effectiveness of the proposed approach, showing significant improvements in computational efficiency. Additionally, we introduce a Q-learning-based algorithm and compare its performance, highlighting the advantages of our dynamic programming approach in terms of both efficiency and solution quality.
Read moreIncorporated Multi-Stage Nash Equilibriums for the Generation Allocation Problem Considering Ramp Rate Effects
This paper presents a novel method to find the profit-maximizing Nash equilibriums in allocating generation amounts with consideration of ramp-rates under competitive market environment. In order to find the Nash equilibriums it is necessary to search all the feasible combinations of generators' outputs which satisfy various constraints. The procedure to eliminate the dominated strategies can be formulated using Bellman's optimality principle of dynamic programming problem and hence the backward or forward search algorithm of dynamic programming can be easily applied. Therefore, the Nash equilibriums are found using dynamic programming method and we found that there exist several Nash equilibriums in the generation allocation problem. Individual generators participate in a game to maximize its profit through competitions and play aa game with bidding strategies of its generation quantities in a spot market. The ramp-rate physically or technically limits generators to increase or decrease outputs in its range and restricts the number of bidding strategies of each generator. We suggest the dynamic programming to find the Nash equilibriums while removing the dominated strategies in each stage (or each time). In the case studies, we analyzed the generation allocation game for a 12-hour multi-stage and compared it with the results of dynamic economic dispatch. Both of the two cases were considered generator's ramp-rate effects.
Read moreOn Boundedness of Q-Learning Iterates for Stochastic Shortest Path Problems
We consider a totally asynchronous stochastic approximation algorithm, Q-learning, for solving finite space stochastic shortest path (SSP) problems, which are undiscounted, total cost Markov decision processes with an absorbing and cost-free state. For the most commonly used SSP models, existing convergence proofs assume that the sequence of Q-learning iterates is bounded with probability one, or some other condition that guarantees boundedness. We prove that the sequence of iterates is naturally bounded with probability one, thus furnishing the boundedness condition in the convergence proof by Tsitsiklis [Tsitsiklis JN (1994) Asynchronous stochastic approximation and Q-learning. Machine Learn. 16:185–202] and establishing completely the convergence of Q-learning for these SSP models.
Read moreSimplicial Label Correcting Algorithms for continuous stochastic shortest path problems
The problem of optimal feedback planning under prediction uncertainties among static obstacles is considered. A discrete-time stochastic state transition model is defined over a continuous state space. A relation to a continuous “nearby” deterministic model is proven for small time steps; the cost-to-go function of the stochastic model is approximated with that of the deterministic model, and the approximation error is found to be proportional to the time step. This motivates using numerical methods, which are vastly available for solving deterministic problems, to approximate the original stochastic problem. We demonstrate this application using a Simplicial Label Correcting Algorithm. This algorithms uses a piecewise linear discretization to compute the shortest-path plan on a simplicial complex. Additionally, the theoretical error bound between the approximate solution and the exact solution is derived and confirmed during numerical experiments. This paper provides a rigorous analysis as well as algorithmic and implementation details of the proposed model for the stochastic shortest path problem in continuous spaces with obstacles.
Read moreNeighborhood reduction strategy for tabu search implementation in asymmetric traveling salesman problem
The Traveling Salesman Problem (TSP) is one of the most widely discussed problems in combinatorial optimization. It has many practical applications in fields of distribution and logistics management, scheduling problems etc. Since these problems are hard, in addition to exact algorithms, research has focused on heuristic techniques to solve TSPs. Computational time is a major concern while solving large TSPs. This problem intensifies further if the graph becomes asymmetric (ATSP). Metaheuristics like tabu search are widely used to find a reasonably good tour fast. Given the practical relevance of ATSPs the lack of literature on it is surprising. The primary objective of our work is to implement tabu search for large ATSPs to obtain good tours in reasonable time. To do that, we make the underlying graph sparse by developing an elite tour based preprocessing scheme. Tabu search is implemented on this reduced graph which results in a reduction of computational time. We also create diversified initial tours suitable for multi-start tabu search in this process. We present our computational experiences both on randomly generated instances and benchmark instances.
Read morePredicting Primary Sequence-Based Protein-Protein Interactions Using a Mercer Series Representation of Nonlinear Support Vector Machine
The prediction of protein-protein interactions (PPIs) is essential to understand the cellular processes from a medical perspective. Among the various machine learning techniques, kernel-based Support Vector Machine (SVM) has been commonly employed to discriminate between interacting and non-interacting protein pairs. The main drawback of employing the kernel-based SVM to datasets with many features, such as the primary sequence-based protein-protein dataset, is the significant increase in computational time of training stage. This increase in computational time is mainly due to the presence of the kernel in solving the quadratic optimisation problem (QOP) involved in nonlinear SVM. In order to fix this issue, we propose a novel and efficient computational algorithm by approximating the kernel-based SVM using a low-rank truncated Mercer series as well as desired. As a result, the QOP for the approximated kernel-based SVM will be very tractable in the sense that there is a significant reduction in computational time of training and validating stages. We illustrate the novelty of the proposed method by predicting the PPIs of “S. Cerevisiae” where the protein features extracted using the multiscale local descriptor (MLD), and then we compare the predictive performance of the proposed low-rank approximation with the existing methods. Finally, the new method results in significant reduction in computational time for predicting PPIs with almost as accuracy as kernel-based SVM.
Read moreHeterogeneous HW/SW FPGA-Based Embedded System for Database Sequencing Applications
Database sequencing applications including sequence comparison, searching, and analysis are considered among the most computation power and time consumers. Heuristic algorithms suffer from sensitivity while traditional sequencing methods, require searching the whole database to find the most matched sequences, which requires high computation power and time. This paper introduces a dynamic programming technique based-on a measure of similarity between two sequential objects in the database using two components, namely frequency and mean. Additionally, database sequences that have the lowest scores in the comparison process were excluded such that the similarity algorithm between a query sequence and other database sequences is applied to meaningful parts of the database. The proposed technique was implemented and validated using a heterogeneous HW/SW FPGA-based embedded system platform. The implementation was partitioned into (1) hardware part (running on logic gates of FPGA) and (2) software part (running on ARM processor of FPGA). The validation study showed a significant reduction in computation time by accelerating the database sequencing processes by 60% comparing to traditional known methods.
Read more