Leveraging Generative and Probabilistic Models for Diagnostics of Cyber-Physical Systems

While Search-Based Software Testing (SBST) has improved significantly in the last decade we propose that more flexible, probabilistic models can be leveraged to improve it further. Rather than searching for an individual, or even sets of, test case(s) or datum(s) that fulfil specific needs the goal can be to learn a generative model tuned to output a useful family of values. Such generative models can naturally be decomposed into a structured generator and a probabilistic model that determines how to make non-deterministic choices during generation. While the former constrains the generation process to produce valid values the latter allows learning and tuning to specific goals. SBST techniques differ in their level of integration of the two but, regardless of how close it is, we argue that the flexibility and power of the probabilistic model will be a main determinant of success. In this short paper, we present how some existing SBST techniques can be viewed from this perspective and then propose additional techniques for flexible generative modelling the community should consider. In particular, Probabilistic Programming languages (PPLs) and Genetic Programming (GP) should be investigated since they allow for very flexible probabilistic modelling. Benefits could range from utilising the multiple program executions that SBST techniques typically require to allowing the encoding of high-level test strategies.

Coherence Therapy is an empirically derived experiential psychotherapy based on Psychological Constructivism. Symptoms are viewed as necessary output from an implicit model of the world. The therapist curates experiences and directs attention toward discovering the model. Rendered explicit, the model is juxtaposed with contradictory knowledge driving memory re-consolidation with resolution of the symptom. The Bayesian Brain views perception and action as inferential processes. Prior beliefs are combined in a generative model to explain the hidden causes of sensations through a process of Active Inference. Prior beliefs that are poor fits to the real world are suboptimal. Suboptimal priors with optimal inference produce Bayes Optimal Pathology with behavioral symptoms. The Active Inference Model of Coherence Therapy posits that Coherence Therapy is a dyadic act of therapist guided Active Inference that renders the (probable) hidden causes of a client’s behavior conscious. The therapist’s sustained attention on the goal of inference helps to overcome memory control bias against retrieval of the affectively charged suboptimal prior. Serial experiences cue memory retrieval and re-instantiation of the physiological/affective state that necessitates production of the symptom in a particular context. As this process continues there is a break in modularity with assimilation into broader networks of experience. Typically, the symptom produced by optimal inference with the suboptimal prior is experienced as unnecessary/inappropriate when taken out of the particular context. The implicit construct has been re-represented and rendered consciously accessible, by a more complex but more accurate model in which the symptom is necessary in some contexts but not others. There is an experience of agency and control in symptom creation, accompanied by the spontaneous production of context appropriate behavior. The capacity for inference has been restored. The Active Inference Model of Coherence Therapy provides a framework for Coherence Therapy as a computational process which can serve as the basis for new therapeutic interventions and experimental designs integrating biological, cognitive, behavioral, and environmental factors.

Deep generative models are promising in detecting novel cyber-physical attacks, mitigating the vulnerability of Cyber-physical systems (CPSs) without relying on labeled information. Nonetheless, these generative models face challenges in identifying attack behaviors that closely resemble normal data, or deviate from the normal data distribution but are in close proximity to the manifold of the normal cluster in latent space. To tackle this problem, this article proposes a novel unsupervised dual variational generative adversarial model named MST-DVGAN, to perform anomaly detection in multivariate time series data for CPS security. The central concept is to enhance the model's discriminative capability by widening the distinction between reconstructed abnormal samples and their normal counterparts. Specifically, we propose an augmented module by imposing contrastive constraints on the reconstruction process to obtain a more compact embedding. Then, by exploiting the distribution property and modeling the normal patterns of multivariate time series, a variational autoencoder is introduced to force the generative adversarial network (GAN) to generate diverse samples. Furthermore, two augmented loss functions are designed to extract essential characteristics in a self-supervised manner through mutual guidance between the augmented samples and original samples. Finally, a specific feature center loss is introduced for the generator network to enhance its stability. Empirical experiments are conducted on three public datasets, namely SWAT, WADI and NSL_KDD. Comparing with the state-of-the-art methods, the evaluation results show that the proposed MTS-DVGAN is more stable and can achieve consistent performance improvement.

Climate-responsive urban planning through generative models: Sensitivity analysis of urban planning and design parameters for urban heat island in Singapore's residential settlements

Probabilistic programming refers to the idea of using standard programming constructs for specifying probabilistic models from machine learning and statistics, and employing generic inference algorithms for answering various queries on these models, such as posterior inference and estimation of model evidence. Although this idea itself is not new and was, in fact, explored by several programming-language and statistics researchers in the early 2000, it is only in the last few years that probabilistic programming has gained a large amount of attention among researchers in machine learning and programming languages, and that expressive and efficient probabilistic programming systems (such as Anglican, Church, Figaro, Infer.net, PSI, PyMC, Stan, and Venture) started to appear and have been taken up by a nontrivial number of users. The primary goal of my talk is to introduce probabilistic programming to the CONCUR/QUEST/FORMATS audience. At the end of my talk, I want the audience to understand basic results and techniques in probabilistic programming and to feel that these results and techniques are relevant or at least related to what she or he studies, although they typically come from foreign research areas, such as machine learning and statistics. My talk will contain both technical materials and lessons that I learnt from my machine-learning colleagues in Oxford, who are developing a highly-expressive higher-order probabilistic programming language, called Anglican. It will also include my work on the denotational semantics of higher-order probabilistic programming languages and their inference algorithms, which are jointly pursued with colleagues in Cambridge, Edinburgh, Oxford and Tubingen.

Deep probabilistic programming concatenates the strengths of deep learning to the context of probabilistic modeling for efficient and flexible computation in practice. Being an evolving field, there exist only a few expressive programming languages for uncertainty management. This paper discusses an application for analysis of ultrasound nerve segmentation-based biomedical images. Our method uses the probabilistic programming language Edward with the U-Net model and generative adversarial networks under different optimizers. The segmentation process showed the least Dice loss ("‘0.54) and the highest accuracy (0.99) with the Adam optimizer in the U-Net model with the least time consumption compared to other optimizers. The smallest amount of generative network loss in the generative adversarial network model gained was 0.69 for the Adam optimizer. The Dice loss, accuracy, time consumption and output image quality in the results show the applicability of deep probabilistic programming in the long run. Thus, we further propose a neuroscience decision support system based on the proposed approach.

Human vision, thought, and planning involve parsing and representing objects and scenes using structured representations based on part-whole hierarchies. Computer vision and machine learning researchers have recently sought to emulate this capability using neural networks, but a generative model formulation has been lacking. Generative models that leverage compositionality, recursion, and part-whole hierarchies are thought to underlie human concept learning and the ability to construct and represent flexible mental concepts. We introduce Recursive Neural Programs (RNPs), a neural generative model that addresses the part-whole hierarchy learning problem by modeling images as hierarchical trees of probabilistic sensory-motor programs. These programs recursively reuse learned sensory-motor primitives to model an image within different spatial reference frames, enabling hierarchical composition of objects from parts and implementing a grammar for images. We show that RNPs can learn part-whole hierarchies for a variety of image datasets, allowing rich compositionality and intuitive parts-based explanations of objects. Our model also suggests a cognitive framework for understanding how human brains can potentially learn and represent concepts in terms of recursively defined primitives and their relations with each other.

Pedestrian trajectory prediction in urban environments has emerged as a critical research area with extensive applications across various domains. Accurate prediction of pedestrian trajectories is essential for the safe navigation of autonomous vehicles and robots in pedestrian-populated environments. Effective prediction models must capture both the spatial interactions among pedestrians and the temporal dependencies governing their movements. Existing research primarily focuses on forecasting a single trajectory per pedestrian, limiting its applicability in real-world scenarios characterised by diverse and unpredictable pedestrian behaviours. To address these challenges, this paper introduces the Graph Convolutional Network, Spatial–Temporal Attention, and Generative Model (GSTGM) for pedestrian trajectory prediction. GSTGM employs a spatiotemporal graph convolutional network to effectively capture complex interactions between pedestrians and their environment. Additionally, it integrates a spatial–temporal attention mechanism to prioritise relevant information during the prediction process. By incorporating a time-dependent prior within the latent space and utilising a computationally efficient generative model, GSTGM facilitates the generation of diverse and realistic future trajectories. The effectiveness of GSTGM is validated through experiments on real-world scenario datasets. Compared to the state-of-the-art models on benchmark datasets such as ETH/UCY, GSTGM demonstrates superior performance in accurately predicting multiple potential trajectories for individual pedestrians. This superiority is measured using metrics such as Final Displacement Error (FDE) and Average Displacement Error (ADE). Moreover, GSTGM achieves these results with significantly faster processing speeds.

In intelligent transportation systems, effectively modeling and interpreting the complex interactions among autonomous agents, such as vehicles and pedestrians, is crucial for traffic management and safety. This paper introduces a novel generative traffic flow model that employs a generative pretrained Transformer to capture multi-agent interactions and detect anomalies in traffic patterns. We introduce a two-stage tokenization process for set-structured traffic data, efficiently encoding variable-sized agent states into fixed-length sequences suitable for generative modeling. We demonstrate that anomalies, which often indicate potential hazards or non-compliant behaviors, can be identified as deviations from learned normal interaction patterns among agents through a zero-shot detection mechanism. Our experimental results in simulated urban settings highlight the model's capability to detect various types of traffic anomalies with high accuracy. This work significantly advances agent-based traffic modeling and underscores its potential for enhancing traffic safety and efficiency in multi-agent systems.

Probabilistic programming languages have the potential to make probabilistic modeling and inference easier to use in practice, but only if inference is sufficiently fast and accurate for real applications. Thus far, this has only been possible for domain-specific languages that focus on a restricted class of models and inference algorithms. This paper proposes a design for a probabilistic programming language called Gen, embedded in Julia, that aims to be sufficiently expressive and performant for general-purpose use. The language provides constructs for automatically generating optimized implementations of custom inference tactics based on static analysis of the target probabilistic model. This paper informally describes a language design for Gen, and shows that Gen is more expressive than Stan, a widely used language for hierarchical Bayesian modeling. A first benchmark shows that a prototype implementation of Gen can be as fast as Stan, only ∼1.4x slower than a hand-coded sampler in Julia, and ∼7,500x faster than Venture, one of the only other probabilistic languages with support for custom inference.

Probabilistic programming languages (PPLs) are at the interface between statistics and the theory of programming languages. PPLs formulate statistical models as stochastic programs that enable automatic inference algorithms and optimization. Pyro [1] and its sibling NumPyro [2] are PPLs built on top of the deep learning frameworks PyTorch [3] and Jax [4], respectively. Both PPLs provide simple, highly similar interfaces for inference using efficient implementations of Hamiltonian Monte Carlo (HMC), the No-U-Turn Sampler (NUTS), and Stochastic Variational Inference (SVI). They automatically generate variational distributions from a model, automatically enumerate discrete variables, and support formulating deep probabilistic models such as variational autoencoders and deep Markov models. The Sine von Mises distribution and its skewed variant are toroidal distributions relevant to protein bioinformatics. They provide a natural way to model the dihedral angles of protein structures, which is important in protein structure prediction, simulation and analysis. We present efficient implementations of the Sine von Mises distribution and its skewing in Pyro and NumPyro, and devise a simulation method that increases efficiency with several orders of magnitude when using parallel hardware (i.e., modern CPUs, GPUs, and TPUs). We demonstrate the use of the skewed Sine von Mises distribution by modeling dihedral angles of proteins using a Bayesian mixture model inferred using NUTS, exploiting NumPyro's facilities for automatic enumeration [5].

Probabilistic programming languages aim to separate the user-specified model from the inference runtime system, allowing users to focus on model design and leave inference to the compiler and runtime systems. However, the structure of the model can affect the inference efficiency, something that is often utilized when implementing Bayesian inference solutions without probabilistic programming. In this paper, we consider a specific class of important models within phylogenetics, which are used for tree inference. Algorithms utilizing the structures of trees in such models have been used for a long time within phylogenetics, outside the scope of probabilistic programming. In particular, the use of belief propagation can significantly speed up probabilistic tree inference when combined with other approximate inferences, such as MCMC. However, there is an open problem of how to incorporate both belief-propagation (marginalization) of a subset of latent variables in typed universal probabilistic programming languages and still be able to apply approximate inference for the rest of the variables. In this paper, we propose a solution to this problem by using an approach where a user only needs to mark which latent variables in the program should be marginalized, and the rest of the belief propagation is handled automatically. We call this approach annotated automatic pruning, acknowledging the original terminology of pruning from the phylogenetic literature. We evaluate our approach using non-trivial models for tree inference, assessing performance, and testing correctness by comparing to MrBayes as the reference implementation.

Modern industry 4.0 applications are shifting towards decentralized automation of computing and cyber-physical systems (CPS), which necessitates building a robust, secure, and efficient system that performs complex interactions with other physical processes. To handle complex interactions in CPS, trust and consensus among various stakeholders is a prime concern. In a similar direction, consensus algorithms in blockchain have evolved over the years that focus on building smart, robust, and secure CPS. Thus, it is imperative to understand the key components, functional characteristics, and architecture of different consensus algorithms used in CPS. Many consensus algorithms exist in the literature with a specified set of functionalities, performance, and computing services. Motivated from these facts, in this survey, we present a comprehensive analysis of existing state-of-the-art consensus mechanisms and highlight their strength and weaknesses in decentralized CPS applications. In the first part, we present the scope of the proposed survey and identify gaps in the existing surveys. Secondly, we present the review method and objectives of the proposed survey based on research questions that address the gaps in existing studies. Then, we present a solution taxonomy of decentralized consensus mechanisms for various CPS applications. Then, open issues and challenges are also discussed in deploying various consensus mechanisms in the CPS with their merits and demerits. The proposed survey will act as a road-map for blockchain developers and researchers to evaluate and design future consensus mechanisms, which helps to build an efficient CPS for industry 4.0 stakeholders.

Probabilistic programming languages (PPLs) are an expressive means for creating and reasoning about probabilistic models. Unfortunately hybrid probabilistic programs that involve both continuous and discrete structures are not well supported by today’s PPLs. In this paper we develop a new approximate inference algorithm for hybrid probabilistic programs that first discretizes the continuous distributions and then performs discrete inference on the resulting program. The key novelty is a form of discretization that we call bit blasting , which uses a binary representation of numbers such that a domain of 2 b discretized points can be succinctly represented as a discrete probabilistic program over poly b Boolean random variables. Surprisingly, we prove that many common continuous distributions can be bit blasted in a manner that incurs no loss of accuracy over an explicit discretization and supports efficient probabilistic inference. We have built a probabilistic programming system for hybrid programs called HyBit , which employs bit blasting followed by discrete probabilistic inference. We empirically demonstrate the benefits of our approach over existing sampling-based and symbolic inference approaches

Purpose. The goal of this paper is to build a simulation environment that allows for the prediction of patient-specific tissue response by drawing samples from a generative model with a probability distribution. We propose a Gaussian Process (GP) regression approach to learn distributions over strain energy density functions including elastography, linear and hyperelastic models reported in the literature. Methods. We gather a total of 73 models characterising elastic properties of brain white matter, grey matter and abnormalities and express them as strain energy density functions. A multi-output GP is used to quantify means and confidence intervals across each anatomical region and model. We sample the GP distribution and use nonlinear optimisation to fit a Neo-Hookean meta-model to guarantee stable strain energy functions. We validate the Neo-Hookean meta-model by fitting known strain energy density functions from the literature and report optimisation cost. We also validate the ability of the GP to approximate elastic properties of tissue given a reference deformed state using simulation. Results. The GP was able to capture confidence intervals of varying strain ranges; the GP parameters and optimisation costs indicated a higher variability of hyperelastic models compared to elastography and linear models. Although one term is insufficient to fully capture hyperelastic models with higher number of terms, the resulting meta model is stable for real-time simulation within a wider range of stretches captured during mechanical characterisation of soft tissue. We demonstrated that our approach was able to approximate known elastic properties of tissue with a root-mean-squared error of 0.6 mm of node displacements when drawing six samples from a distribution of hyperelastic white matter. Conclusion. In this initial proof-of-concept, we demonstrated a GP-based approach to estimate the elastic behaviour of brain tissue through simulation by sampling a generative model comprising elastic models found in the literature.