Model-Based Oversampling for Imbalanced Sequence Classification

Drug discovery is a multi-stage process, often beginning with the identification of active molecules from a high-throughput screen or machine learning model. Once structure activity relationship trends become well established, identifying new analogs with better properties is important. Synthesizing these new compounds is a logical next step, and is key to research groups that have a synthetic chemistry team or external collaborators. Generative machine learning models have become widely adopted to generate new molecules and explore molecular space, with the goal of discovering novel compounds with desires properties. These generative models have been composed from recurrent neural networks (RNNs), Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) and are often combined with transfer learning or scoring of physicochemical properties to steer generative design. While these generative models have proven useful in generating new molecular libraries, often they are not capable of addressing a wide variety of potential problems, and often converge into similar molecular space when combined with a scoring function for desired properties. In addition, generated compounds are often not synthetically feasible, reducing their capabilities outside of virtual composition and limiting their usefulness in real-world scenarios. Here we introduce a suite of automated tools called MegaSyn representing 3 components: a new hill-climb algorithm which makes use of SMILES-based RNN generative models, analog generation software, and retrosynthetic analysis coupled with fragment analysis to score molecules for their synthetic feasibility. We now describe the development and testing of this suite of tools and propose how they might be used to optimize molecules or prioritize promising lead compounds using test case examples.

With the significant increase in the amount of solar energy sources, the variation in the production caused by the stochastic nature of these sources becomes a larger problem for the electrical grid. Having a large percentage of installed units, therefore, requires some type of both short- and long-term planning of energy production. Most planning and forecasting tools require a large amount of solar energy data for the location in question and oftentimes that data is difficult to obtain. In this paper, we propose a method for generating synthetic solar irradiance data and test its applicability on forecasting and long-term planning problems. Synthetic data are generated using various generative adversarial networks and masked autoencoder based models, which are all trained on a dataset of solar irradiance data measured on the territory of the city of Banja Luka in Bosnia and Herzegovina. The synthetic data are evaluated both on different types of short-term forecasting tools and a long-term solar plant simulation tool, and the obtained results are compared with the cases where real data is used. By comparing the results, we have concluded that by using generative models we are able to generate synthetic data which is not only realistic, but applicable for both of these uses.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

We study the problem of synthetic generation of samples of environmental features for autonomous vehicle navigation. These features are described by a spatiotemporally varying scalar field that we refer to as a threat field. The threat field is known to have some underlying dynamics subject to process noise. Some "real-world" data of observations of various threat fields are also available. The assumption is that the volume of "real-world" data is relatively small. The objective is to synthesize samples that are statistically similar to the data. The proposed solution is a generative artificial intelligence model that we refer to as a split variational recurrent neural network (S-VRNN). The S-VRNN merges the capabilities of a variational autoencoder, which is a widely used generative model, and a recurrent neural network, which is used to learn temporal dependencies in data. The main innovation in this work is that we split the latent space of the S-VRNN into two subspaces. The latent variables in one subspace are learned using the real-world data, whereas those in the other subspace are learned using the data as well as the known underlying system dynamics. Through numerical experiments we demonstrate that the proposed S-VRNN can synthesize data that are statistically similar to the training data even in the case of very small volume of real-world training data.

<p><strong>Because of its wide range of applications, generative artificial intelligence (Generative AI) has received a lot of attention in academia and industry. Despite significant advances in computer vision and natural language processing, the use of Generative AI in tabular data is still underexplored. This gap is especially significant given the prevalence of tabular data as the primary data modality. This thesis seeks to close this gap by focusing on the efficient generation, evaluation, refinement, and application of tabular data, while addressing the challenges inherent in its heterogeneous nature.</strong></p><p>The primary goal of this thesis is to develop efficient algorithms for tabular data synthesis, advancing the field of Generative AI in the tabular domain. To achieve this goal, four specific research objectives have been outlined.</p><p>First, this thesis addresses gaps in evaluation metrics by unifying a framework for the comprehensive and consistent assessment of synthetic data. The proposed framework is designed to meet diverse business requirements across various downstream tasks by incorporating a wide range of advanced metrics. These metrics cover various types of evaluations, including univariate, bivariate, multivariate, cluster, and record-level evaluations. Additionally, standardized visualizations are provided to facilitate qualitative assessments. The results indicate that the proposed evaluation framework not only enables consistent comparisons, rankings, and the selection of different data synthesis approaches but also acts as a valuable tool to promptly assess the reliability of results obtained from synthetic data.</p><p>Second, this thesis presents innovative algorithms for the generation of tabular data, aiming to significantly enhance the quality of data synthesis. One of the contributions is the development of a reversible feature engineering pipeline designed to automatically represent tabular data in an efficient format while also ensuring that the transformed data can be easily converted back to its original format. Additionally, the thesis proposes novel deep learning-based tabular data generation models that are capable of learning the joint distribution of multivariate datasets without relying on predefined distribution assumptions. Experimental results highlight the effectiveness of the proposed algorithms in handling heterogeneous data, demonstrating superior performance in most scenarios within the same training duration when compared to similar alternatives.</p><p>Third, this thesis introduces innovative synthetic data prototype selection algorithms with the goal of refining the generated samples. This approach aims to leverage the advantages of Deep Generative Models, which, once trained, can generate unlimited and diverse synthetic data. By carefully identifying and selecting high-quality samples or removing unrealistic ones, the quality of the synthetic data can be enhanced from a post-processing perspective. Building on this hypothesis, we recognize the iterative nature inherent in the data synthesis procedure, where the processes of data generation, evaluation, and refinement should operate repeatedly in a cyclical flow. During the data generation phase, the evaluation framework assists in identifying potential issues and risks based on use cases, while the refinement (post-processing) step iteratively improves synthetic data in alignment with the evaluation outcomes.</p><p>Last, this thesis applies and validates the proposed models across various domains, including business, healthcare, and government, each requiring distinct downstream tasks. Specifically, we assess our algorithms from the standpoint of data balancing using churn data, evaluate our models focusing on data augmentation with health data, and test our algorithms from the perspective of data representation considering data privacy concerns within sensitive citizen data. These applications serve as practical illustrations, demonstrating the effectiveness and utility of the proposed Generative AI models in real-world scenarios.</p><p>In summary, this thesis makes a substantial contribution to the advancement of Generative AI within the domain of tabular data. It not only presents innovative algorithms and evaluation methods but also introduces practical frameworks applicable to real-world scenarios. The findings demonstrate the considerable capacity of Generative AI to fundamentally transform tabular data in various fields, ultimately leading to improved data availability, quality, and quantity.</p>

Deep generative models (GMs) have transformed the exploration of drug-like chemical space (CS) by generating novel molecules through complex, nontransparent processes, bypassing direct structural similarity. This review examines five key architectures for CS exploration: recurrent neural networks (RNNs), variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows (NF), and Transformers. It discusses molecular representation choices, training strategies for focused CS exploration, evaluation criteria for CS coverage, and related challenges. Future directions include refining models, exploring new notations, improving benchmarks, and enhancing interpretability to better understand biologically relevant molecular properties.

In the past few years, de novo molecular design has increasingly been using generative models from the emergent field of Deep Learning, proposing novel compounds that are likely to possess desired properties or activities. De novo molecular design finds applications in different fields ranging from drug discovery and materials sciences to biotechnology. A panoply of deep generative models, including architectures as Recurrent Neural Networks, Autoencoders, and Generative Adversarial Networks, can be trained on existing data sets and provide for the generation of novel compounds. Typically, the new compounds follow the same underlying statistical distributions of properties exhibited on the training data set Additionally, different optimization strategies, including transfer learning, Bayesian optimization, reinforcement learning, and conditional generation, can direct the generation process toward desired aims, regarding their biological activities, synthesis processes or chemical features. Given the recent emergence of these technologies and their relevance, this work presents a systematic and critical review on deep generative models and related optimization methods for targeted compound design, and their applications.

Single-cell RNA-seq (scRNAseq) analyses performed at the cellular level aim to understand the cellular landscape of tissue sections, offer insights into rare cell-types, and identify marker genes for annotating distinct cell types. ScRNAseq analyses are widely applied to cancer research to understand tumor heterogeneity, disease progression, and resistance to therapy. Single-cell data processing is a challenging task due to its high-dimensionality, sparsity, and having imbalanced class(cell-type) distributions. An accurate cell-type identification is highly dependent on preprocessing and quality control steps. To address these issues, generative models have been widely used in recent years. Techniques frequently used include Variational Autoencoders (VAE), Generative Adversarial Networks (GANs), Gaussian-based methods, and, more recently, Flow-based (FB) generative models. We developed a Masked Affine Autoregressive transform-embedded FB (MAF-FB) model. Then, to improve MAF-FB further, we incorporated a mixture of experts (MOE) of attention mechanisms on top of it, resulting in our proposed MOE-FB model. We conducted a comparative analysis of fundamental generative models, aiming to serve as a preliminary guidance for developing novel automated scRNAseq data analysis systems. We performed a large-scale analysis by combiningfour datasets derived from pancreatic tissue sections and for further generalizability assessments, we employed Peripheral Blood Mononuclear Cells (PBMC68K and PBMC3K) and Human Cell Atlas Bone Marrow (HCA-BM10K) datasets. We utilized VAE, GAN, Gaussian Copula, and Automated Cell-Type-informed Introspective Variational Autoencoder (ACTIVA), and compared them against our two novel FB models, MAF-FB and MOE-FB for ScRnaseq synthesis. To evaluate the performances of generative models, we used various discrepancy metrics and performed automated cell-type classification tasks. We also identified differentially expressed genes for each cell type, and inferred cell-cell interactions based on ligand-receptor bindings across distinct cell-type pairs. Among the generative models, FB models, especially MOE-FB, consistently outperformed others across all experimental setups in both discrepancy metrics with comparison to the baseline test set and cell-type classification tasks (with an F1-score of 0.90 precision of 0.89 and recall of 0.92 for the integrated pancreatic datasets). MOE-FB produced biologically more relevant synthetic data, and ligand–receptor–based cell–cell interactions inferred from the synthetic cells closely resemble the original data, achieving an RMSE of 0.65 against the corresponding pancreatic test set. These findings highlight the potential and promising use of FB models, especially MOE-FB, in scRNAseq analyses.

The de novo drug design based on SMILES format is a typical sequence-processing problem. Previous methods based on recurrent neural network (RNN) exhibit limitation in capturing long-range dependency, resulting in a high invalid percentage in generated molecules. Recent studies have shown the potential of Transformer architecture to increase the capacity of handling sequence data. In this work, the encoder module in the Transformer is used to build a generative model. First, we train a Transformer-encoder-based generative model to learn the grammatical rules of known drug molecules and a predictive model to predict the activity of the molecules. Subsequently, transfer learning and reinforcement learning were used to fine-tune and optimize the generative model, respectively, to design new molecules with desirable activity. Compared with previous RNN-based methods, our method has improved the percentage of generating chemically valid molecules (from 95.6 to 98.2%), the structural diversity of the generated molecules, and the feasibility of molecular synthesis. The pipeline is validated by designing inhibitors against the human BRAF protein. Molecular docking and binding mode analysis showed that our method can generate small molecules with higher activity than those carrying ligands in the crystal structure and have similar interaction sites with these ligands, which can provide new ideas and suggestions for pharmaceutical chemists.

We argue that there is a need for substantially more research on the use of generative data models in the validation and evaluation of visualization techniques. For example, user studies will require the display of representative and uncon-founded visual stimuli, while algorithms will need functional coverage and assessable benchmarks. However, data is often collected in a semi-automatic fashion or entirely hand-picked, which obscures the view of generality, impairs availability, and potentially violates privacy. There are some sub-domains of visualization that use synthetic data in the sense of generative data models, whereas others work with real-world-based data sets and simulations. Depending on the visualization domain, many generative data models are side projects as part of an ad-hoc validation of a techniques paper and thus neither reusable nor general-purpose. We review existing work on popular data collections and generative data models in visualization to discuss the opportunities and consequences for technique validation, evaluation, and experiment design. We distill handling and future directions, and discuss how we can engineer generative data models and how visualization research could benefit from more and better use of generative data models.

Developmental and Epileptic Encephalopathies (DEEs) are rare, severe conditions often discussed by families on social media, offering valuable insights into their experiences. Identifying these messages amidst unrelated content is crucial but challenging due to data imbalance. This study evaluates different uses of generative large language models (LLMs) for binary classification of DEE-related experiences within social media posts. Using CamemBERT as a baseline, we compared two strategies: zero-shot prompt-based classification and synthetic data generation for minority class augmentation. While zero-shot prompting underperformed, the addition of 2% synthetic data improved all metrics (macro/positive F1, precision and recall). Higher proportions of synthetic data led to decreased precision. These findings underscore the potential of hybrid approaches combining fine-tuning and domain-specific synthetic data for addressing data imbalance in rare disease contexts. Further validation across models and datasets is needed.

This study assesses the effectiveness of deep learning models, specifically Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and Long Short-Term Memory (LSTM), in assessing synthetic grip strength data for stress ball and pencil grips. The aim is to determine the optimal model for grip strength classification and tackle issues such as temporal data and class imbalance. Synthetic data was created, preprocessed, and utilized to train the models on Google Colab employing TensorFlow and Keras. Hyperparameters, such as CNN kernel dimensions and RNN/LSTM sequence lengths, were refined. Performance was evaluated by accuracy, precision, F1 scores, and loss metrics. The RNN model surpassed both CNN and LSTM, with an accuracy of 94.99% and a precision of 0.56, particularly excelling in the classification of “Normal” grip strength instances. CNN and LSTM had difficulties with time dependencies and class imbalance, leading to reduced accuracy and increased loss. Although RNN has proven to be the most effective approach, subsequent research should concentrate on mitigating class imbalance and investigating hybrid models to improve performance.

A critical task for system operators is the precise identification of the root causes underlying an error situation. This identification is fundamental in deciding optimal maintenance actions, such as replacing a component versus calibrating it. However, the actual causes of an error are often neither measured nor unique. The measured quantities are the result of complex interactions between different error causes and system variables. Root cause identification in this context becomes a matter of inferring hidden causes from their measurable effects. This challenge is notably pronounced in cyber-physical systems comprising control loops. Control mechanisms, integral to maintaining system performance, introduce a layer of complexity in diagnostics and ultimately complicate the isolation of the underlying causes of errors. To address this challenge, we introduce a two-step approach to derive the hidden causes as a statistical inference task. First, we develop a generative model leveraging existing control software and expert-based insights into the mechanisms of errors, i.e., a simulator of synthetic data given some hidden error causes. Then, we transform the generative model into a probabilistic program on which statistical inference can be executed within a probabilistic programming language framework. This inference effectively estimates the hidden causes given some measured data from the system. Being intrinsically a statistical approach, these inferences come with a confidence interval. We applied this methodology to an industrial printer’s sheet transport belt, operating in a closed-loop configuration. Our approach successfully discerned the contributions of three distinct hidden causes to the belt’s deviation from its intended position. This paper highlights the efficacy of generative modeling followed by a probabilistic programming approach in unraveling complex interactions within cyber-physical systems for optimal maintenance.

Generative models, such as diffusion models, have made significant advancements in recent years, enabling the synthesis of high‐quality realistic data across various domains. Here, the adaptation and training of a diffusion model on super‐resolution microscopy images are explored. It is shown that the generated images resemble experimental images, and that the generation process does not exhibit a large degree of memorization from existing images in the training set. To demonstrate the usefulness of the generative model for data augmentation, the performance of a deep learning‐based single‐image super‐resolution (SISR) method trained using generated high‐resolution data is compared against training using experimental images alone, or images generated by mathematical modeling. Using a few experimental images, the reconstruction quality and the spatial resolution of the reconstructed images are improved, showcasing the potential of diffusion model image generation for overcoming the limitations accompanying the collection and annotation of microscopy images. Finally, the pipeline is made publicly available, runnable online, and user‐friendly to enable researchers to generate their own synthetic microscopy data. This work demonstrates the potential contribution of generative diffusion models for microscopy tasks and paves the way for their future application in this field.

Recurrent neural networks (RNNs) were designed for dealing with time-series data and have recently been used for creating predictive models from functional magnetic resonance imaging (fMRI) data. However, gathering large fMRI datasets for learning is a difficult task. Furthermore, network interpretability is unclear. To address these issues, we utilize multitask learning and design a novel RNN-based model that learns to discriminate between classes while simultaneously learning to generate the fMRI time-series data. Employing the long short-term memory (LSTM) structure, we develop a discriminative model based on the hidden state and a generative model based on the cell state. The addition of the generative model constrains the network to learn functional communities represented by the LSTM nodes that are both consistent with the data generation as well as useful for the classification task. We apply our approach to the classification of subjects with autism vs. healthy controls using several datasets from the Autism Brain Imaging Data Exchange. Experiments show that our jointly discriminative and generative model improves classification learning while also producing robust and meaningful functional communities for better model understanding.