Computational Study Protocol: Leveraging Synthetic Data to Validate a Benchmark Study for Differential Abundance Tests for 16S Microbiome Sequencing Data

Synthetic data's utility in benchmark studies depends on its ability to closely mimic real-world conditions and reproduce results obtained from experimental data. Building on Nearing et al.'s study (1), who assessed 14 differential abundance tests using 38 experimental 16S rRNA datasets in a case-control design, we are generating synthetic datasets that mimic the experimental data to verify their findings. We will employ statistical tests to rigorously assess the similarity between synthetic and experimental data and to validate the conclusions on the performance of these tests drawn by Nearing et al. (1). This protocol adheres to the SPIRIT guidelines, demonstrating how established reporting frameworks can support robust, transparent, and unbiased study planning. We replicate Nearing et al.'s (1) methodology, incorporating synthetic data simulated using two distinct tools, mirroring the 38 experimental datasets. Equivalence tests will be conducted on a non-redundant subset of 46 data characteristics comparing synthetic and experimental data, complemented by principal component analysis for overall similarity assessment. The 14 differential abundance tests will be applied to synthetic and experimental datasets, evaluating the consistency of significant feature identification and the number of significant features per tool. Correlation analysis and multiple regression will explore how differences between synthetic and experimental data characteristics may affect the results. Synthetic data enables the validation of findings through controlled experiments. We assess how well synthetic data replicates experimental data, try to validate previous findings with the most recent versions of the DA methods and delineate the strengths and limitations of synthetic data in benchmark studies. Moreover, to our knowledge this is the first computational benchmark study to systematically incorporate synthetic data for validating differential abundance methods while strictly adhering to a pre-specified study protocol following SPIRIT guidelines, contributing to transparency, reproducibility, and unbiased research.

To assess the validity of privacy-preserving synthetic data by comparing results from synthetic versus original EHR data analysis. A published retrospective cohort study on real-world effectiveness of COVID-19 vaccines by Maccabi Healthcare Services in Israel was replicated using synthetic data generated from the same source, and the results were compared between synthetic versus original datasets. The endpoints included COVID-19 infection, symptomatic COVID-19 infection and hospitalization due to infection and were also assessed in several demographic and clinical subgroups. In comparing synthetic versus original data estimates, several metrices were utilized: standardized mean differences (SMD), decision agreement, estimate agreement, confidence interval overlap, and Wald test. Synthetic data were generated five times to assess the stability of results. The distribution of demographic and clinical characteristics demonstrated very small difference (< 0.01 SMD). In the comparison of vaccine effectiveness assessed in relative risk reduction between synthetic versus original data, there was a 100% decision agreement, 100% estimate agreement, and a high level of confidence interval overlap (88.7%-99.7%) in all five replicates across all subgroups. Similar findings were achieved in the assessment of vaccine effectiveness against symptomatic COVID-19 Infection. In the comparison of hazard ratios for COVID 19-related hospitalization and odds ratio for symptomatic COVID-19 Infection, the Wald tests suggested no significant difference between respective effect estimates in all five replicates for all patient subgroups but there were disagreements in estimate and decision metrices in some subgroups and replicates. Overall, comparison of synthetic versus original real-world data demonstrated good validity and reliability. Transparency on the process to generate high fidelity synthetic data and assurances of patient privacy are warranted.

BackgroundThe exponential growth in patient data collection by healthcare providers, governments, and private industries is yielding large and diverse datasets that offer new insights into critical medical questions. Leveraging extensive computational resources, Machine Learning and Artificial Intelligence are increasingly utilized to address health-related issues, such as predicting outcomes from Electronic Health Records and detecting patterns in multi-omics data. Despite the proliferation of medical devices based on Artificial Intelligence, data accessibility for research is limited due to privacy concerns. Efforts to de-identify data have met challenges in maintaining effectiveness, particularly with large datasets. As an alternative, synthetic data, that replicate main statistical properties of real patient data, are proposed. However, the lack of standardized evaluation metrics complicates the selection of appropriate synthetic data generation methods. Effective evaluation of synthetic data must consider resemblance, utility and privacy, tailored to specific applications. Despite available metrics, benchmarking efforts remain limited, necessitating further research in this area.ResultsWe present SynthRO (Synthetic data Rank and Order), a user-friendly tool for benchmarking health synthetic tabular data across various contexts. SynthRO offers accessible quality evaluation metrics and automated benchmarking, helping users determine the most suitable synthetic data models for specific use cases by prioritizing metrics and providing consistent quantitative scores. Our dashboard is divided into three main sections: (1) Loading Data section, where users can locally upload real and synthetic datasets; (2) Evaluation section, in which several quality assessments are performed by computing different metrics and measures; (3) Benchmarking section, where users can globally compare synthetic datasets based on quality evaluation.ConclusionsSynthetic data mitigate concerns about privacy and data accessibility, yet lacks standardized evaluation metrics. SynthRO provides an accessible dashboard helping users select suitable synthetic data models, and it also supports various use cases in healthcare, enhancing prognostic scores and enabling federated learning. SynthRO’s accessible GUI and modular structure facilitate effective data evaluation, promoting reliability and fairness. Future developments will include temporal data evaluation, further broadening its applicability.

BackgroundPrivacy restrictions limit access to protected patient-derived health information for research purposes. Consequently, data anonymization is required to allow researchers data access for initial analysis before granting institutional review board approval. A system installed and activated at our institution enables synthetic data generation that mimics data from real electronic medical records, wherein only fictitious patients are listed.ObjectiveThis paper aimed to validate the results obtained when analyzing synthetic structured data for medical research. A comprehensive validation process concerning meaningful clinical questions and various types of data was conducted to assess the accuracy and precision of statistical estimates derived from synthetic patient data.MethodsA cross-hospital project was conducted to validate results obtained from synthetic data produced for five contemporary studies on various topics. For each study, results derived from synthetic data were compared with those based on real data. In addition, repeatedly generated synthetic datasets were used to estimate the bias and stability of results obtained from synthetic data.ResultsThis study demonstrated that results derived from synthetic data were predictive of results from real data. When the number of patients was large relative to the number of variables used, highly accurate and strongly consistent results were observed between synthetic and real data. For studies based on smaller populations that accounted for confounders and modifiers by multivariate models, predictions were of moderate accuracy, yet clear trends were correctly observed.ConclusionsThe use of synthetic structured data provides a close estimate to real data results and is thus a powerful tool in shaping research hypotheses and accessing estimated analyses, without risking patient privacy. Synthetic data enable broad access to data (eg, for out-of-organization researchers), and rapid, safe, and repeatable analysis of data in hospitals or other health organizations where patient privacy is a primary value.

This study presents a novel approach using Wasserstein Generative Adversarial Networks with Gradient Penalty (WGAN-GP) to generate synthetic electroencephalography (EEG) and electrocardiogram (ECG) waveforms. The synthetic EEG data represent concentration and relaxation mental states, while the synthetic ECG data correspond to normal and abnormal states. By addressing the challenges of limited biophysical data, including privacy concerns and restricted volunteer availability, our model generates realistic synthetic waveforms learned from real data. Combining real and synthetic datasets improved classification accuracy from 92% to 98.45%, highlighting the benefits of dataset augmentation for machine learning performance. The WGAN-GP model achieved 96.84% classification accuracy for synthetic EEG data representing relaxation states and optimal accuracy for concentration states when classified using a fusion of convolutional neural networks (CNNs). A 50% combination of synthetic and real EEG data yielded the highest accuracy of 98.48%. For EEG signals, the real dataset consisted of 60-s recordings across four channels (TP9, AF7, AF8, and TP10) from four individuals, providing approximately 15,000 data points per subject per state. For ECG signals, the dataset contained 1200 real samples, each comprising 140 data points, representing normal and abnormal states. WGAN-GP outperformed a basic generative adversarial network (GAN) in generating reliable synthetic data. For ECG data, a support vector machine (SVM) classifier achieved an accuracy of 98% with real data and 95.8% with synthetic data. Synthetic ECG data improved the random forest (RF) classifier’s accuracy from 97% with real data alone to 98.40% when combined with synthetic data. Statistical significance was assessed using the Wilcoxon signed-rank test, demonstrating the robustness of the WGAN-GP model. Techniques such as discrete wavelet transform, downsampling, and upsampling were employed to enhance data quality. This method shows significant potential in addressing biophysical data scarcity and advancing applications in assistive technologies, human-robot interaction, and mental health monitoring, among other medical applications.

e19554 Background: Adding daratumumab to standard multiple myeloma (MM) regimens improves response rates and clinical outcomes. Systematic reviews (meta-analyses) allow further evaluation of its effectiveness in MM patients with high-risk cytogenetic factors but are limited by model assumptions. his study addresses these limitations using a novel synthetic individual patient data (SynIPD) generation algorithm to enable a more robust analysis. Methods: Advances in AI have enabled the creation of synthetic data that mimics the statistical properties of real patient data, empowering aggregated analyses of clinical trials. This study focuses on newly diagnosed MM (NDMM) and relapsed/refractory MM (RRMM) patients with high-risk cytogenetic profiles. Six studies were included: ALCYONE, CASSIOPEIA, and MAIA for NDMM, and CANDOR, CASTOR, and POLLUX for RRMM. Large language models extracted clinical evidence, and SynIPD was generated, incorporating cytogenetic profiles. Random-effects meta-analyses were conducted, and results were compared with pooled analyses using synthetic data. Results: Synthetic data were generated for three studies for NDMM (ALCYONE, CASSIOPEIA, MAIA) and three studies for RRMM (CANDOR, CASTOR, POLLUX). The generated hazard ratios (HRs) for each study closely aligned with reported HRs for high-risk patients, with a maximum relative error of less than 2%. By pooling these datasets, we obtained synthetic datasets for NDMM and RRMM patients. Stratified Cox regression analyses yielded HRs (95% confidence interval) of 0.764 (0.529-1.104) for NDMM and 0.431 (0.294-0.613) for RRMM. Random-effects meta-analyses showed HRs (95% confidence interval) of 0.764 (0.529-1.104) for NDMM and 0.425 (0.287-0.621) for RRMM. Compared with traditional meta-analysis techniques, the synthetic data approach provides additional clinical insights that were not available in the original publications. For instance, the clinical results for the three NDMM studies are summarized as KM curves in Figure 1, which were not reported in the original studies. Conclusions: Our results suggest that the addition of daratumumab to standard regimens improves PFS for RRMM patients with high-risk cytogenetic factors, although its benefit for NDMM patients is not statistically significant. Pooling synthetic data enhances the robustness of conventional meta-analysis outcomes. The identification of HRs and corresponding confidence intervals through SynIPD improves the credibility of our proposed method, offering a reliable approach to evaluating the efficacy of daratumumab in high-risk patients. Since SynIPD includes individual covariate information, it provides deep clinical insights beyond merely enhancing meta-analysis. In other words, any statistical results derived from the original IPD can be effectively approximated using SynIPD.

Generation of synthetic full-scale burst test data for corroded pipelines using the tabular generative adversarial network

High-quality tabular data is a crucial requirement for developing data-driven applications, especially healthcare-related ones, because most of the data nowadays collected in this context is in tabular form. However, strict data protection laws complicates the access to medical datasets. Thus, synthetic data has become an ideal alternative for data scientists and healthcare professionals to circumvent such hurdles. Although many healthcare institutions still use the classical de-identification and anonymization techniques for generating synthetic data, deep learning-based generative models such as generative adversarial networks (GANs) have shown a remarkable performance in generating tabular datasets with complex structures. This paper examines the GANs’ potential and applicability within the healthcare industry, which often faces serious challenges with insufficient training data and patient records sensitivity. We investigate several state-of-the-art GAN-based models proposed for tabular synthetic data generation. Healthcare datasets with different sizes, numbers of variables, column data types, feature distributions, and inter-variable correlations are examined. Moreover, a comprehensive evaluation framework is defined to evaluate the quality of the synthetic records and the viability of each model in preserving the patients’ privacy. The results indicate that the proposed models can generate synthetic datasets that maintain the statistical characteristics, model compatibility and privacy of the original data. Moreover, synthetic tabular healthcare datasets can be a viable option in many data-driven applications. However, there is still room for further improvements in designing a perfect architecture for generating synthetic tabular data.

ObjectiveThis study sought to evaluate whether synthetic data derived from a national coronavirus disease 2019 (COVID-19) dataset could be used for geospatial and temporal epidemic analyses.Materials and MethodsUsing an original dataset (n = 1 854 968 severe acute respiratory syndrome coronavirus 2 tests) and its synthetic derivative, we compared key indicators of COVID-19 community spread through analysis of aggregate and zip code-level epidemic curves, patient characteristics and outcomes, distribution of tests by zip code, and indicator counts stratified by month and zip code. Similarity between the data was statistically and qualitatively evaluated.ResultsIn general, synthetic data closely matched original data for epidemic curves, patient characteristics, and outcomes. Synthetic data suppressed labels of zip codes with few total tests (mean = 2.9 ± 2.4; max = 16 tests; 66% reduction of unique zip codes). Epidemic curves and monthly indicator counts were similar between synthetic and original data in a random sample of the most tested (top 1%; n = 171) and for all unsuppressed zip codes (n = 5819), respectively. In small sample sizes, synthetic data utility was notably decreased.DiscussionAnalyses on the population-level and of densely tested zip codes (which contained most of the data) were similar between original and synthetically derived datasets. Analyses of sparsely tested populations were less similar and had more data suppression.ConclusionIn general, synthetic data were successfully used to analyze geospatial and temporal trends. Analyses using small sample sizes or populations were limited, in part due to purposeful data label suppression—an attribute disclosure countermeasure. Users should consider data fitness for use in these cases.

Realistic currents in seas and oceans are almost always changing in depth thus indicating on the presence of shear in the mean ambient flow. However, analysis methodologies interpreting directional wave data gathered by in-situ measurement devices such as: buoys, pressure gauges and Acoustic Doppler Current Profilers (ADCPs) utilize potential irrotational flow theory which cannot account for the rotational shearing currents. The effects of shearing currents on the wave direction estimations were studied on synthetic ADCP data of waves propagating in a predetermined spread. The synthetic data was generated employing the Rayleigh Boundary Value Problem (BVP) and a selected ambient current profile. The potential data processing led to significant errors in wave directional spread estimation for common shearing currents (~10&amp;#176;&amp;#160; in mean wave direction for the presented example). This finding is of great importance, as it addresses the influence of an ambient current profile on wave propagation direction. The obtained results suggest that there is an uncertainty with the confidence of any wave directional spread ever presented by in-situ wave measurement devices. A new methodology was developed for estimating directional wave spectra based on rotational flow physics by acquiring new terms emanate from wave-shearing current interaction governing equations. This included a derivation of new numerical transfer functions for the fluid&amp;#8217;s physical properties based on the Rayleigh BVP. Then, by applying classical cross- and auto-spectral analysis on time-series data sets, the directional spread function was numerically reconstructed. The newly derived data processing methodology was applied to the same synthetic ADCP data sets. It was found to be capable of reconstructing the spread with great accuracy (0.4&amp;#176; in mean wave direction for the presented example). In addition, to modeling and synthetic data, field measurement data from several campaigns were also analyzed showing the importance of accounting for the vertical shear. This makes it a prominent methodology for estimating directional wave spectra in realistic oceanic conditions.

Synthetic face datasets for training Face Recognition (FR) models have gained significant interest in recent years due to privacy concerns associated with real facial data. Recent advances in generative techniques, based mainly on Generative Adversarial Networks (GANs) and Diffusion Models, have led to the proliferation of various synthetic datasets. Although the performance of FR models trained on synthetic data is becoming similar to those of models trained on real data when evaluated on standard benchmarks such as LFW, CFP-FP, AgeDB, and CALFW, little is known about their effectiveness in more challenging scenarios involving individuals with high facial similarity, such as doppelgängers (lookalikes) and identical twins. This work addresses this gap by evaluating state-of-the-art FR models trained on real or synthetic datasets across four testing datasets that feature high facial similarity: HDA Doppelgänger, DoppelVer, 3D-TEC, and ND-Twins-2009-2010. We analyze verification performances, including subgroup analysis by ethnicity, gender, and age, to better understand demographic disparities. Our results show that the gap between synthetic and real face datasets is larger in these challenging scenarios than on typical benchmarks. We also observe similar biases in FR models trained on synthetic or real data, suggesting that current synthetic datasets may inherit biases through the use of real data at some stage in their pipeline. Reproducibility code, trained models, and instructions are available at.

This paper concerns the use of synthetic data for protecting the confidentiality of business data during statistical analysis. Synthetic datasets are constructed by replacing sensitive values in a confidential dataset with draws from statistical models estimated on the confidential dataset. Unfortunately, the process of generating effective statistical models can be a difficult and labour-intensive task. Recently, it has been proposed to use easily-implemented methods from machine learning instead of statistical model estimation in the data synthesis task. J. Drechsler and J.P. Reiter [1] have conducted an evaluation of four such methods, and have found that regression trees could give rise to synthetic datasets which provide reliable analysis results as well as low disclosure risks. Their conclusion was based on simulations using a subset of the 2002 Uganda census public use file, and it is an interesting question whether the same conclusion applies to other types of data with different characteristics. For example, business data have quite different characteristics from population census and survey data. Business data generally have few variables that are mostly categorical, and often have highly skewed distributions with outliers. In this paper we investigate the applicability of regression-tree-based methods for constructing synthetic business data. We give a detailed example comparing exploratory data analysis and linear regression results under two variants of a regression-tree-based synthetic data approach. We also include an evaluation of the analysis results with respect to the results of analysis of the original data. We further investigate the impact of different stopping criteria on performance. Our example provides evidence that synthesisers based on regression trees may not be immediately applicable in the context of business data. Further investigation, including further simulation studies with larger datasets, is certainly indicated.

Due to a significant shortage of healthcare staff, medical facilities are increasingly challenged by the need to deploy current staff more intensively, which can lead to significant complications for patients and staff. Digital surgical assistance systems that track all instruments used in procedures can make a significant contribution to relieving the load on staff, increasing efficiency, avoiding errors and improving hygiene. Due to data safety concerns, laborious data annotation and the complexity of the scenes, as well as to increase prediction accuracy, the provision of synthetic data is key to enabling the wide use of artificial intelligence for object recognition and tracking in OR settings. In this study, a synthetic data generation pipeline is introduced for the detection of eight surgical instruments during open surgery. Using 3D models of the instruments, synthetic datasets consisting of color images and annotations were created. These datasets were used to train common object detection networks (YOLOv8) and compared against networks solely trained on real data. The comparison, conducted on two real image datasets with varying complexity, revealed that networks trained on synthetic data demonstrated better generalization capabilities. A sensitivity analysis showed that synthetic data-trained networks could detect surgical instruments even at higher occlusion levels than real data-trained networks. Additionally, 1920 datasets were generated using different parameter combinations to evaluate the impact of various settings on detection performance. Key findings include the importance of object visibility, occlusion, and the inclusion of occlusion objects in improving detection accuracy. The results highlight the potential of synthetic datasets to simulate real-world conditions, enhance network generalization, and address data shortages in specialized domains like surgical instrument detection.

Traditional datasets are often limited, which can affect the accuracy of analyses. Additionally, the use of students’ real data raises privacy concerns. Generative Adversarial Networks (GANs) offer a solution by generating synthetic data that closely mirrors real-world data without compromising sensitive information. The research explored the application of GANs to enhance student performance datasets by addressing challenges related to data scarcity and privacy in educational research. In the research, GANs were utilized to generate synthetic student performance data. The accuracy of the data was assessed using Mean Absolute Percentage Error (MAPE), with values ranging from 0.004% to 19.92% across various statistical measures and means. These results demonstrated the reliability of the synthetic data, making it suitable for further analysis. The synthetic datasets were then analyzed using the Apriori algorithm, a well-known method in data mining for discovering significant patterns and relationships. A lower bound minimum support of 0.1 (10%) and a minimum confidence threshold of 0.6 (60%) were applied, ensuring the identification of meaningful associations. The analysis reveals important patterns and relationships among student attributes and behaviors. The research highlights the potential of GANs to advance data-driven educational research. By generating high-quality synthetic data, GANs allow researchers to conduct comprehensive analyses while addressing privacy concerns. The research contributes to the methodological approach to data augmentation in education, offering new opportunities for ethical and robust research.

Preserving logical and functional dependencies in synthetic tabular data