Polynomial-Time Probabilistic Reasoning with Partial Observations via Implicit Learning in Probability Logics

Reinforcement learning, a machine learning technique, has already shown advancement in solving complex sequential decision-making problems. Many tasks involve multiple agents and require sequential decision-making policies to achieve common goals, such as warehouse automation, autonomous driving, and game-playing. To derive the policies for all agents, these problems can be modelled as multi-agent systems and addressed by multi-agent reinforcement learning (MARL). However, optimizing policies in multi-agent scenarios presents significant challenges due to the intricate behaviours of multiple agents and the non-stationary nature of the environments’ complex dynamics. Firstly, optimizing policies in multi-agent scenarios presents significant challenges due to the complexity of multi-agent behaviours, especially in partially observable environments. Also, the dynamic nature of agents’ behaviours and their interactions with other agents lead to changes in the environment’s states and agents’ observations over time, which is more complicated in open environments. Furthermore, the need to balance individual and collective objectives in certain real-world multi-agent environments also complicates the decision-making process. This doctoral thesis aims to tackle the following three fundamental multi-agent research problems and propose a solution for each. Our research covers from theoretical analysis to practical application. We start by studying the problem of learning an efficient policy under partially observable environments. We notice that a group of agents cooperates to com- pete against another group of agents (named opponents), with the limitation that information about the opponents is inaccessible due to partial observability. To address this issue, we propose a novel multi-agent distributional actor-critic algorithm to achieve speculative opponent modeling with purely local information (i.e., the controlled agent’s observations, actions, and rewards). The actor maintains a speculated belief of the opponents, which we call the speculative opponent models, to predict opponents’ actions using local observations and make decisions accordingly. The distributional critic models the return distribution of the policy. It reflects the quality of the actor and thus can guide the training of the imaginary opponent model that the actor relies on. Extensive experiments confirm that our method successfully models opponents’ behaviours without their data and delivers superior performance against baseline methods with a faster convergence speed. Furthermore, in some environments, teammates’ numbers and policies change with the market, requiring workers to adapt to perform different sets of tasks across time. To solve this issue, we propose an RL-based method that allows the controlled agent to collaborate with dynamic teammates in open environments. The controlled agent maintains a dual teamwork situation inference model to capture the current teamwork state and facilitate reasonable decision-making under partial observability. Considering the dynamic types of teammates, we first leverage the Chinese Restaurant Process-based model to categorize versatile teammate policies into distinct clusters, which improves the efficiency of identifying current teamwork situations. Next, to model the heterogeneous relationships among agents and accommodate the varying number of teammates, we apply heterogeneous graph attention neural networks to learn the representation of the teamwork situation. Extensive experiments confirm that our method outperforms state-of-the-art baselines with faster convergence in various ad hoc teamwork tasks. Lastly, in certain real-world applications, such as routing problems and warehouse management, decision-makers must balance overall benefits with individual fairness among agents. Achieving both learning efficiency and fairness concurrently presents a complex, multi-objective, joint-policy optimization challenge. Moreover, existing approaches are predominantly confined to simulation environments. To address the above issues, we present a pioneering MARL approach to balance individual and collective objectives for agents collaboration. Extensive experiments on synthetic and real-world datasets demonstrate that our method not only surpasses the state-of-the-art DRL methods, but also exhibits advantages over established heuristics, notably in achieving significantly faster optimization speeds. This method underlines the essential integration of fairness into real-world applications, representing a major advancement in creating equitable and efficient logistics solutions. To conclude, this doctoral thesis investigates three fundamental multi-agent decision-making research problems that are ubiquitous and unsolved. The proposed three MARL methodology solutions achieve efficient policy training and performance for agents in multi-agent environments with uncertainties raised by the partially observable, the open environment, and the individual-collective objectives of MARL. This thesis delves into novel designs of various critical components of MARL, including MDP formulations, policy networks, training algorithms, and inference methods. These contributions significantly elevate the effectiveness and efficiency of cooperative MARL, establishing new performance benchmarks.

Hyperproperties are properties that describe the correctness of a system as a relation between multiple executions. Hyperproperties generalize trace properties and include information-flow security requirements, like noninterference, as well as requirements like symmetry, partial observation, robustness, and fault tolerance. We initiate the study of the specification and verification of hyperproperties of Markov decision processes (MDPs). We introduce the temporal logic PHL (Probabilistic Hyper Logic), which extends classic probabilistic logics with quantification over schedulers and traces. PHL can express a wide range of hyperproperties for probabilistic systems, including both classical applications, such as probabilistic noninterference, and novel applications in areas such as robotics and planning. While the model checking problem for PHL is in general undecidable, we provide methods both for proving and for refuting formulas from a fragment of the logic. The fragment includes many probabilistic hyperproperties of interest.

Learning nonlinear turbulent dynamics from partial observations via analytically solvable conditional statistics

We study the problem of verifying and enforcing the privacy of a partially observable stochastic system in the presence of an eavesdropping intruder. The intruder has the knowledge of the system model. It also has partial observation to system states and actions based on which it updates the belief which is a probability distribution over the system states. The intruder’s objective is to learn the system’s secrets in terms of whether the system is currently in some critical or sensitive states. We model the system as a partially observable Markov decision process and propose a notion of privacy with respect to the intruder ’s belief. Our key observation is that the evolution of the intruder’s belief can be represented as a discrete-time switched system. The privacy verification and enforcement can thus be cast into a reachability problem with respect to the unsafe subset of beliefs defined by the privacy requirement. We show how to apply control theory to verify and enforce the system’s privacy with two proposed approaches.

The increasing penetration of distributed energy resources has led to the emergence of Virtual Power Plants (VPPs) as a key mechanism for enhancing grid flexibility and market participation. However, the concurrent operation of multiple VPPs introduces a dynamic and strategic decision-making environment due to their interdependence through shared market prices and system constraints. Current models often rely on exogenous price assumptions or centralized coordination, which limits their ability to capture endogenous price formation and strategic coupling among agents. To this end, this paper proposes a multi-agent reinforcement learning method for the joint scheduling of multiple VPPs in dynamic markets. The scheduling problem is set up as a Markov game, where each VPP acts as an independent agent that learns the best bidding and dispatch strategies under partial observability and evolving market conditions. By continuously adapting to market feedback and peer behaviors, agents learn to coordinate effectively without requiring centralized control or complete system information. Simulation results demonstrate improvements in economic efficiency and system-wide coordination. This work offers a scalable and adaptive solution for multi-VPP operation.

Probability distribution model can affect the extreme value of standard ice thickness on conductors of transmission lines for different return periods. This paper discusses the impact of three probability distribution models on calculation of standard ice thickness, which are Pearson III distribution model (P-III), generalized extreme value distribution model (GEV), and generalized Pareto distribution model (GPD), respectively. The historic icing data have been collected in Lvcongpo Mountain from Hubei province, including icing data from north-south direction, west-east direction, bigger data from two directions, and smaller data from two directions. The analyzing results indicate that GPD is more suitable than P-III and GEV for the icing data collected from Lvcongpo Mountain, which is helpful to reasonably determine the design ice thickness of conductors of transmission lines.

We introduce a new type of fuzzy set called Fuzzy Distribution Set (FDS). Fuzzy distribution sets are fuzzy sets defined on a finite domain subject to a sum of membership values equal to 1. Such fuzzy sets can serve as models of subjective probability distributions and subjective weight distributions. Considering these distributions as fuzzy sets gives a possibility to extend on such distributions the operations of fuzzy sets and, more generally, the calculus of fuzzy restrictions developed during the last decades. Recently Yager introduced the concept of negation of probability distributions. In our works, we studied several classes of such negations. Here we consider an involutive negation of probability distributions as a complement of FDS. We introduce the operations of union and intersection of fuzzy distribution sets. These basic operations on FDS can serve as a basis for the application of fuzzy logic methods to subjective probability and weight distributions. These operations can be used for the development of reasoning models with subjective probability distributions and subjective weighting functions. Weight distributions can be used in multi-criteria, multi-person, and multi-attribute decision-making models.

In this study, probability analysis was performed on peak daily rainfall data in order to predict rainfall interval values and to determine the best fit functions in some parts of Nigeria. The selected towns are Kaduna, Kano, Yola, Jos, Damaturu and Maiduguri. The obtained peak daily rainfall values were subjected to Gumbel, Log-Gumbel, Normal, Log-Normal, Pearson and Log-Pearson probability distributions. Mathematical equation for probability distribution functions were established for each town and used to predict peak rainfall. The predicted values were subjected to goodness of fit tests such as Chi-square, Correlation Coefficient, Coefficient of Determination and Errors of Estimates to determine how best the fits are. The model that satisfies the tests adequately was selected as the best fit model. The study revealed that the peak rainfall at Kaduna, Jos, Kano, Yola and Damaturu are best fitted by log-Gumbel, while log- Pearson distribution is suitable for predicting peak rainfall in Maiduguri. The result also shows that the occurrences of peak daily rainfall depth of 100 mm and above are rare in the selected areas. Keywords: Peak rainfall, Probability distribution models, Return period, Probability interval and goodness of fit tests

A Bayesian adaptive control approach to the combined optimal investment/reinsurance problem of an insurance company is studied. The insurance company invests in a money market and a capital market index with an unknown appreciation rate, or “drift”. Using a Bayesian approach, the unknown drift is described by an unobservable random variable with a known (prior) probability distribution. We assume that the risk process of the company is governed by a diffusion approximation to the compound Poisson risk process. The company also purchases reinsurance. The combined optimal investment/reinsurance problem is formulated as a stochastic optimal control problem with partial observations. We employ filtering theory to transform the problem into one with complete observations. The control problem is then solved by the dynamic programming Hamilton–Jacobi–Bellman (HJB) approach. Semi-analytical solutions are obtained for the exponential utility case.

Next-generation wireless networks have been designed to transport integrated multimedia services based on a cellular extension of a packet-switched architecture using variants of the Internet protocol (IP). Each call, arriving to or active within the network, carries demand for one or more services in parallel, where each service type has a guaranteed quality of service (QoS). Admission of new calls to the wireless IP network (WIN) from the gateway of a wired network or from a mobile subscriber (MS) is allowed by call admission control (CAC) procedures. MS roaming among the nodes of the WIN is controlled by handoff procedures between base stations (BSs), or BS controllers (BSCs), and the MSs. Both handoff and CAC procedures are typically embedded in the media access control (MAC) and logical link control (LLC) layers of the WIN protocol stack. Performance metrics, such as, probabilities of call blocking and handoff, control procedures within the WIN protocol. Earlier published results by the author present a method for the performance analysis of wireless multimedia networks based on multivariate point-process (MVPP) models and their semi-martingale representations. The models describe the finite-horizon, transient behavior of the packet flows in integrated multimedia traffic. This paper extends the earlier work to focus on the optimization of real-time control procedures embedded in the WIN protocol stack. The point processes corresponding to the packet flow of each service type and to collateral random network events are decomposed into right-continuous, pure jump processes and predictable, integrated random rate processes. Control of the set of point processes describing WIN behavior is implemented through the construction of an absolutely continuous change of a reference probability measure on network events to a controlled measure. With respect to the constructed controlled measure, the integrated random rates in the semi-martingale representations of the network point processes explicitly depend on the parameters of the protocol mechanisms for packet access, routing, switching, handoff, power control, and other resource allocations. The MVPP control approach via constructed probability measure also supports predictive models of packet flows that incorporate measurement-based estimates of the probability distributions for voice, video, data, and other Internet traffic as well as radio path distortions. Optimization of the performance metrics, represented in terms of the MVPPs and the control parameters of candidate WIN protocols for multimedia services, are discussed for both partial and complete observations of network events. The models are further used to develop optimal, recursive stochastic filters of network state, based on partial or incomplete observations of packet flow dynamics.

: With better natural language semantic representations, computers can do more applications more efficiently as a result of better understanding of natural text. However, no single semantic representation at this time fulfills all requirements needed for a satisfactory representation. Logic-based representations like first-order logic capture many of the linguistic phenomena using logical constructs, and they come with standardized inference mechanisms, but standard first-order logic fails to capture the graded aspect of meaning in languages. Distributional models use contextual similarity to predict the graded semantic similarity of words and phrases but they do not adequately capture logical structure. In addition there are a few recent attempts to combine both representations either on the logic side (still, not a graded representation), or in the distribution side (not full logic). We propose using probabilistic logic to represent natural language semantics combining the expressivity and the automated inference of logic, and the gradedness of distributional representations. We evaluate this semantic representation on two tasks, Recognizing Textual Entailment (RTE) and Semantic Textual Similarity (STS). Doing RTE and STS better is an indication of a better semantic understanding. Our system has three main components, 1. Parsing and Task Representation, 2. Knowledge Base Construction, and 3. Inference. The input natural sentences of the RTE/STS task are mapped to logical form using Boxer which is a rule based system built on top of a CCG parser, then they are used to formulate the RTE/STS problem in probabilistic logic. Then, a knowledge base is represented as weighted inference rules collected from different sources like WordNet and on-the-fly lexical rules from distributional semantics.

Sagar Island, setting on the continental shelf of Bay of Bengal, is one of the most vulnerable deltas to the occurrence of extreme rainfall-driven climatic hazards. Information on probability of occurrence of maximum daily rainfall will be useful in devising risk management for sustaining rainfed agrarian economy vis-a-vis food and livelihood security. Using six probability distribution models and long-term (1982–2010) daily rainfall data, we studied the probability of occurrence of annual, seasonal and monthly maximum daily rainfall (MDR) in the island. To select the best fit distribution models for annual, seasonal and monthly time series based on maximum rank with minimum value of test statistics, three statistical goodness of fit tests, viz. Kolmogorove–Smirnov test (K-S), Anderson Darling test (A 2 ) and Chi-Square test (X 2) were employed. The fourth probability distribution was identified from the highest overall score obtained from the three goodness of fit tests. Results revealed that normal probability distribution was best fitted for annual, post-monsoon and summer seasons MDR, while Lognormal, Weibull and Pearson 5 were best fitted for pre-monsoon, monsoon and winter seasons, respectively. The estimated annual MDR were 50, 69, 86, 106 and 114 mm for return periods of 2, 5, 10, 20 and 25 years, respectively. The probability of getting an annual MDR of >50, >100, >150, >200 and >250 mm were estimated as 99, 85, 40, 12 and 03 % level of exceedance, respectively. The monsoon, summer and winter seasons exhibited comparatively higher probabilities (78 to 85 %) for MDR of >100 mm and moderate probabilities (37 to 46 %) for >150 mm. For different recurrence intervals, the percent probability of MDR varied widely across intra- and inter-annual periods. In the island, rainfall anomaly can pose a climatic threat to the sustainability of agricultural production and thus needs adequate adaptation and mitigation measures.

2. INTRODUCTION Since the inception of online social networks, the amount of social data that is being published on a daily basis has been increasing at an unprecedented rate. Smart, GPS-enabled, always-connected personal devices have taken the data generation to a new level by making it tremendously easy to generate and share social content like check-in information, likes, microblogs (e.g., Twitter), multimedia data, and so on. There is an enormous value in reasoning about such streaming data and deriving meaningful insights from it in real-time. Examples of potential applications include advertising, sentiment analysis, detecting natural disasters, social recommendations, personalized trends, spam detection, to name a few. There is thus an increasing need to build scalable systems to support such applications. Complex nature of social networks and their rapid evolution, coupled with the huge volume of streaming social data and the need for real-time processing, raise many computational challenges that have not been addressed in prior work. Social network data comprises of two major components. First, there is a network (linkage) component that captures the underlying interconnection structure among the entities in the social network. Second, there is content data that is typically associated with the nodes and the edges in the social network. The social network data stream contains updates to both these components. The structure of the network may itself change rapidly in many cases, especially when things like webpages and user tags (e.g., Twitter hashtags) are treated as nodes of the network. However, most of the social network data stream consists of updates to the data associated with the nodes and the edges, e.g., status updates and other content uploaded by the users, communication among the users, and so on. There is interest in performing a wide variety of queries and analytics over such data streams in real time. The queries can range from simple publish-subscribe queries, where a user is interested in being notified when something happens in his friend circle, to complex anomaly detection queries, where the goal is to identify anomalous behavior as early as possible. In this paper, we present an introduction to this new research area of stream querying and reasoning over social data. This area combines aspects from several well-studied research areas, chief among them, social network analysis, graph databases, and data streams. We provide a formal definition of the problem, survey the related prior work, and discuss some of the key research challenges that need to be addressed (and some of the solutions that have been proposed). We note that we use the term stream reasoning in this paper to encompass a broad range of tasks including various types of analytics, probabilistic reasoning, statistical inference, and logical reasoning. We contrast our use of this term with the recent work by Valle et al. [104, 105] who define this term more specifically to refer to integration of logical reasoning systems with data streams in the context of the Semantic Web. Given the vast amount of work on this and related topics, it is not our intention to be comprehensive in this brief overview. Rather we aim to cover some of the key ideas and representative work.

A new approach in evaluating the slope failure probability under earthquake condition was proposed in this study. Unlike the use of a deterministic seismic coefficient in a pseudostatic analysis, the uncertainties of earthquake magnitude, location, frequency, and seismic-wave attenuation were taken into account in the new approach. The probability distributions of the earthquake parameters follow those described in probabilistic seismic hazard analysis (PSHA). Along with the considerations of the uncertainties of slope parameters, such as slope angle, slope height, and soil/rock properties, the slope failure probability can be estimated by a probabilistic analysis. In particular, the new approach used Monte Carlo simulation (MCS) in the analysis, in which random parameters were generated with prescribed probability distributions and statistics. With an n-trial MCS being performed, slope failure probability is equal to the ratio between the trial of slope failure and total trials. In this study, the approach was also demonstrated by a benchmark PSHA example integrated with a hypothetical infinite slope. For such a slope under the setup of seismicity, its failure probability increased to 8.3% in a 50-year condition, from 0.16% in a one-year condition. The increase in slope failure probability resulted from a higher earthquake frequency with respect to a longer duration of interest.

In this paper, the problems of basic fuzzy event space and of probability fuzzy space are studied. Firstly, the concepts of basic fuzzy event, fuzzy event and basic fuzzy event space are defined, related properties are investigated, and some results that will be used in the next study of probability fuzzy space are obtained. Then, the definitions of the probability function for fuzzy events and probability fuzzy space are given, some properties of the defined probability function are obtained. In addition, some models of probability distribution of probability fuzzy space based on a known probability space are proposed, and some examples are given to show the usability of the proposed models of probability distribution.