Reinforcement learning-based optimal false data injection attack against control signals in probabilistic Boolean control networks

One of the major issues in systems biology is developing control theory for gene regulatory networks (GRNs). Particularly, an important objective is to develop therapeutic intervention strategies to alter regulatory dynamics of GRNs to avoid undesired or diseased states. Several optimal control strategies have been developed to find therapeutic intervention strategies for small (or medium) sized GRNs modeled as probabilistic Boolean control networks (PBCNs). However, due to the humongous nature of GRNs, we require an optimal control strategy that scales to large systems without posing any constraints on the network dynamics. In this article, we formulate the infinite horizon discounted cost optimal control problem by leveraging the Markov decision process (MDP) based PBCN structure to model GRNs. Further, we design a stationary optimal control strategy, thereby avoiding the states associated with a disease. An augmented state space is defined to facilitate the control design method. By exploiting Kullback–Leibler (KL) divergence and the log transformation of the cost function referred to as the desirability function, the optimal control is devised using the path integral (PI) approach. We propose a sampling‐based technique for the approximation of the PI and hence the optimal control of PBCNs. The sampling‐based strategy is amicable to parallel implementation, thereby addressing the optimal control problem of large‐scale PBCNs. Finally, the viability of the results in the article is shown by some illustrative examples.

State-flipped control design for the stabilization of probabilistic Boolean control networks.

A genetic regulatory network (GRN) is a dynamic system to describe interactions among a large number of different substances in a living biological cell. There have been numerous attempts to model the dynamical behaviour of genetic regulatory networks. Boolean models are a class of discrete models for modelling genetic regulatory networks, which are conditioned on the premise that genes interact with each other through Boolean logic. As mentioned previously, Boolean models include Boolean Networks, Instantaneously Random Probabilistic Boolean Networks and Context-Sensitive Boolean Networks. There are a large number of issues surrounding Boolean models, including Boolean Networks and Probabilistic Boolean Networks. The model inference is the procedure of determining whether the modelled networks are consistent with the given data sample, and choosing one model, from among many, that makes more sense for the data. This paper has reviewed the research into modelling gene regulatory networks, especially upon Boolean models.

In this paper, we propose a novel random forest (RF) Q-learning hybrid with experience replay for feedback stabilization of probabilistic Boolean control networks (PBCNs). In particular, by resorting to a model-free reinforcement learning (RL) framework, we present a random forest Q-learning (QLRF) algorithm to design optimal state feedback controllers, thereby stabilizing PBCNs to a given equilibrium point. In reference to better the process of learning the Q-table by replacing it with a function approximator, we substitute the existent neural network (NN) architecture by a RF. We provide insights on the overall computational complexity between the two ways of solving the same problem, proving RF better than its NN counterparts for such applications. The simulations performed on some of the standard examples in the literature demonstrates the effectiveness of the proposed idea.

This paper investigates the finite-time observability of probabilistic Boolean multiplex control networks (PBMCNs). Firstly, the finite-time observability of the PBMCNs is converted into the set reachability issue according to the parallel interconnection technique (a minor modification of the weighted pair graph method in the literature). Secondly, the necessary and sufficient condition for the finite-time observability of PBMCNs is presented based on the set reachability. Finally, the main conclusions are substantiated by providing illustrative examples.

Intervention in a gene regulatory network is used to help it avoid undesirable states, such as those associated with a disease. Several types of intervention have been studied in the framework of a probabilistic Boolean network (PBN), which is essentially a finite collection of Boolean networks in which at any discrete time point the gene state vector transitions according to the rules of one of the constituent networks. For an instantaneously random PBN, the governing Boolean network is randomly chosen at each time point. For a context-sensitive PBN, the governing Boolean network remains fixed for an interval of time until a binary random variable determines a switch. The theory of automatic control has been previously applied to find optimal strategies for manipulating external (control) variables that affect the transition probabilities of an instantaneously random PBN to desirably affect its dynamic evolution over a finite time horizon. This paper extends the methods of external control to context-sensitive PBNs. This paper treats intervention via external control variables in context-sensitive PBNs by extending the results for instantaneously random PBNs in several directions. First, and most importantly, whereas an instantaneously random PBN yields a Markov chain whose state space is composed of gene vectors, each state of the Markov chain corresponding to a context-sensitive PBN is composed of a pair, the current gene vector occupied by the network and the current constituent Boolean network. Second, the analysis is applied to PBNs with perturbation, meaning that random gene perturbation is permitted at each instant with some probability. Third, the (mathematical) influence of genes within the network is used to choose the particular gene with which to intervene. Lastly, PBNs are designed from data using a recently proposed inference procedure that takes steady-state considerations into account. The results are applied to a context-sensitive PBN derived from gene-expression data collected in a study of metastatic melanoma, the intent being to devise a control strategy that reduces the WNT5A gene's action in affecting biological regulation, since the available data suggest that disruption of this influence could reduce the chance of a melanoma metastasizing.

Driven by the desire to understand genomic functions through the interactions among genes and gene products, the research in gene regulatory networks has become a heated area in genomic signal processing. Among the most studied mathematical models are Boolean networks and probabilistic Boolean networks, which are rule-based dynamic systems. This tutorial provides an introduction to the essential concepts of these two Boolean models, and presents the up-to-date analysis and simulation methods developed for them. In the Analysis section, we will show that Boolean models are Markov chains, based on which we present a Markovian steady-state analysis on attractors, and also reveal the relationship between probabilistic Boolean networks and dynamic Bayesian networks (another popular genetic network model), again via Markov analysis; we dedicate the last subsection to structural analysis, which opens a door to other topics such as network control. The Simulation section will start from the basic tasks of creating state transition diagrams and finding attractors, proceed to the simulation of network dynamics and obtaining the steady-state distributions, and finally come to an algorithm of generating artificial Boolean networks with prescribed attractors. The contents are arranged in a roughly logical order, such that the Markov chain analysis lays the basis for the most part of Analysis section, and also prepares the readers to the topics in Simulation section.

Robust control invariance (RCI) is crucial for ensuring that the state of the system remains within the predefined set under disturbances. In this paper, we study the robust control invariance issues of probabilistic Boolean control networks using state‐flipped control (SFC). Employing the semi‐tensor product method, we propose a matrix‐based testing condition to determine whether a given set can become a robust SFC invariance set under a specified SFC scheme. Additionally, we derive a necessary and sufficient condition to test the existence of feasible SFC schemes. To optimize the process, we present an algorithm that calculates feasible SFC schemes with the minimal flip set, ensuring the given set becomes a robust SFC invariance set. The idea of the algorithm is to identify all beneficial states for achieving RCI and direct the system towards these states, thereby avoiding the need to explore all possible SFC schemes and significantly reducing time complexity. Compared with the event‐triggered approach, our method reduces the time complexity of designing feasible control schemes from to . Finally, a biological simulation is presented to demonstrate the effectiveness of the theoretical results.

A probabilistic Boolean network (PBN) is a discrete-time system composed of a collection of Boolean networks between which the PBN switches in a stochastic manner. This article focuses on the study of quotients of PBNs. Given a PBN and an equivalence relation on its state set, we consider a probabilistic transition system that is generated by the PBN; the resulting quotient transition system then automatically captures the quotient behavior of this PBN. We therefore describe a method for obtaining a probabilistic Boolean system that generates the transitions of the quotient transition system. Applications of this quotient description are discussed, and it is shown that for PBNs, controller synthesis can be performed easily by first controlling a quotient system and then lifting the control law back to the original network. A biological example is given to show the usefulness of the developed results.

An integer programming approach to optimal control problems in context-sensitive probabilistic Boolean networks

In this paper, we consider stabilization and reconstruction of sampled-data Boolean control networks (BCNs) under noisy sampling interval. A sampled-data BCN under noisy sampling interval is first converted into a probabilistic Boolean network (PBN). We then obtain some necessary and sufficient conditions for global stochastic stability of the considered sampled-data BCN under two types of noisy sampling intervals. However, in analyzing the stochastic stability of large-scale sampled-data BCNs under noisy sampling interval, using the above necessary and sufficient conditions leads to huge computational cost. Therefore, for a large-scale sampled-data BCN, we have to transform it into a size-reduced probabilistic logical network. Then by studying the stochastic stability of the probabilistic logical network, some sufficient conditions for global stochastic stability of the large-scale sampled-data BCN are obtained. Moreover, based on the given steady-state probabilities of the transformed PBN, the reconstruction problem of sampled-data BCNs under noisy sampling interval can be well solved as a linear programming (LP) problem.

A probabilistic Boolean network (PBN) is a collection of Boolean networks endowed with a probability structure describing the likelihood with which a constituent network is active at each time step. This paper proposes a definition of bisimulation for PBNs. The notion is inspired by the analogous notions for probabilistic chains and for stochastic linear systems. Necessary and sufficient conditions to check the proposed notion are derived, model reduction of PBNs via bisimulation is addressed, and a discussion on the use of probabilistic bisimulation for optimal control design is given. The present results extend the theory of bisimulation known for deterministic Boolean networks to a stochastic setting.

In this paper, we study control problems of Boolean Networks (BNs) and Probabilistic Boolean Networks (PBNs). For BN CONTROL, by applying external control, we propose to derive the network to the desired state within a few time steps. For PBN CONTROL, we propose to find a control sequence such that the network will terminate in the desired state with a maximum probability. Also, we propose to minimize the maximum cost of the terminal state to which the network will enter. Integer linear programming and dynamic programming in conjunction with hard constraints are then employed to solve the above problems. Numerical experiments are given to demonstrate the effectiveness of our algorithms. We also present a hardness result suggesting that PBN CONTROL is harder than BN CONTROL.

Intervention in a gene regulatory network is used to avoid undesirable states, such as those associated with a disease. Several types of intervention have been studied in the framework of a probabilistic Boolean network (PBN), which is a collection of Boolean networks in which the gene state vector transitions according to the rules of one of the constituent networks and where network choice is governed by a selection distribution. The theory of automatic control has been applied to find optimal strategies for manipulating external control variables that affect the transition probabilities to desirably affect dynamic evolution over a finite time horizon. In this paper we treat a case in which we lack the governing probability structure for Boolean network selection, so we simply have a family of Boolean networks, but where these networks possess a common attractor structure. This corresponds to the situation in which network construction is treated as an ill-posed inverse problem in which there are many Boolean networks created from the data under the constraint that they all possess attractor structures matching the data states, which are assumed to arise from sampling the steady state of the real biological network. Given a family of Boolean networks possessing a common attractor structure composed of singleton attractors, a control algorithm is derived by minimizing a composite finite-horizon cost function that is a weighted average over all the individual networks, the idea being that we desire a control policy that on average suits the networks because these are viewed as equivalent relative to the data. The weighting for each network at any time point is taken to be proportional to the instantaneous estimated probability of that network being the underlying network governing the state transition. The results are applied to a family of Boolean networks derived from gene-expression data collected in a study of metastatic melanoma, the intent being to devise a control strategy that reduces the WNT5A gene's action in affecting biological regulation

Boolean networks, in spite of their structural simplicity, seem to be able to simulate the dynamics of complex biological and nonbiological systems. Learning algorithms in neural networks have shown to be a very promising approach to some problems connected to artificial intelligence. Positive feedback has been successfully used by the genetic algorithm and the ant system. In this paper we propose an adaptive Boolean network that takes advantage of all these properties. >