Privacy-Preserving Remote Sensing Image Generation and Classification With Differentially Private GANs

This study aims to explore the feasibility of employing generative adversarial networks (GAN) to generate synthetic contrast-enhanced ultrasound (CEUS) from grayscale ultrasound images of patients with thyroid nodules while dispensing with the need for ultrasound contrast agents injection. Patients who underwent preoperative thyroid CEUS examinations between January 2020 and July 2022 were collected retrospectively. The cycle-GAN framework integrated paired and unpaired learning modules was employed to develop the non-invasive image generation process. The syntheticCEUS imageswas generated in three phases: pre-arterial, plateau, and venous. The evaluation included quantitative similarity metrics, classification performance, and qualitative assessment by radiologists. CEUS videos of 360 thyroid nodules from 314 patients (45 years ± 12 [SD]; 272 women) in the internal dataset and 202 thyroid nodules from 183 patients (46 years ± 13 [SD]; 148 women) in the external dataset were included. In the external testing dataset, quantitative analysis revealed a significant degree of similarity between real and synthetic CEUS images (structure similarity index, 0.89 ± 0.04; peak signal-to-noise ratio, 28.17 ± 2.42). Radiologists deemed 126 of 132 [95%] synthetic CEUS images diagnostically useful. The accuracy of radiologists in distinguishing between real and synthetic images was 55.6% (95% CI: 0.49, 0.63), with an AUC of 61.0% (95% CI: 0.65, 0.68). No statistically significant difference (p > 0.05) was observed when radiologists assessed peak intensity and enhancement patterns using real CEUS and synthetic CEUS. Both quantitative analysis and radiologist evaluations exhibited that synthetic CEUS images generated by generative adversarial networks were similar to real CEUS images. QuestionIt is feasible to generate synthetic thyroid contrast-enhanced ultrasound images using generative adversarial networks without ultrasound contrast agents injection. FindingsCompared to real contrast-enhanced ultrasound images, synthetic contrast-enhanced ultrasound images exhibited high similarity and image quality. Clinical relevanceThis non-invasive and intelligent transformation may reduce the requirement for ultrasound contrast agents in certain cases, particularly in scenarios where ultrasound contrast agents administration is contraindicated, such as in patients with allergies, poor tolerance, or limited access to resources.

Generative Adversarial Networks (GANs) have received a great deal of attention due in part to recent success in generating original, high-quality samples from visual domains. However, most current methods only allow for users to guide this image generation process through limited interactions. In this work we develop a novel GAN framework that allows humans to be “in-the-loop” of the image generation process. Our technique iteratively accepts relative constraints of the form “Generate an image more like image A than image B”. After each constraint is given, the user is presented with new outputs from the GAN, informing the next round of feedback. This feedback is used to constrain the output of the GAN with respect to an underlying semantic space that can be designed to model a variety of different notions of similarity (e.g. classes, attributes, object relationships, color, etc.). In our experiments, we show that our GAN framework is able to generate images that are of comparable quality to equivalent unsupervised GANs while satisfying a large number of the constraints provided by users, effectively changing a GAN into one that allows users interactive control over image generation without sacrificing image quality.

AN ASSESSMENT OF SPATIAL VARIATION OF LAND SURFACE CHARACTERISTICS OF MINNA, NIGER STATE NIGERIA FOR SUSTAINABLE URBANIZATION USING GEOSPATIAL TECHNIQUES

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

PurposeTo develop generative adversarial networks (GANs) that synthesize realistic anterior segment optical coherence tomography (AS-OCT) images and evaluate deep learning (DL) models that are trained on real and synthetic datasets for detecting angle closure.MethodsThe GAN architecture was adopted and trained on the dataset with AS-OCT images collected from the Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, synthesizing open- and closed-angle AS-OCT images. A visual Turing test with two glaucoma specialists was performed to assess the image quality of real and synthetic images. DL models, trained on either real or synthetic datasets, were developed. Using the clinicians’ grading of the AS-OCT images as the reference standard, we compared the diagnostic performance of open-angle vs. closed-angle detection of DL models and the AS-OCT parameter, defined as a trabecular-iris space area 750 µm anterior to the scleral spur (TISA750), in a small independent validation dataset.ResultsThe GAN training included 28,643 AS-OCT anterior chamber angle (ACA) images. The real and synthetic datasets for DL model training have an equal distribution of open- and closed-angle images (all with 10,000 images each). The independent validation dataset included 238 open-angle and 243 closed-angle AS-OCT ACA images. The image quality of real versus synthetic AS-OCT images was similar, as assessed by the two glaucoma specialists, except for the scleral spur visibility. For the independent validation dataset, both DL models achieved higher areas under the curve compared with TISA750. Two DL models had areas under the curve of 0.97 (95% confidence interval, 0.96–0.99) and 0.94 (95% confidence interval, 0.92–0.96).ConclusionsThe GAN synthetic AS-OCT images appeared to be of good quality, according to the glaucoma specialists. The DL models, trained on all-synthetic AS-OCT images, can achieve high diagnostic performance.Translational RelevanceThe GANs can generate realistic AS-OCT images, which can also be used to train DL models.

Deep learning methods have made remarkable strides in surface defect detection. But, they heavily rely on large amount of training data, which can be a costly endeavor, especially for specific applications like steel strip surface defect detection, where acquiring and labeling large-scale data is impractical due to the rarity of certain defective categories in production environment. Hence, realistic defect image synthesis can greatly alleviate this issue. However, training image generation networks also demand substantial data, making image data augmentation merely an auxiliary effort. In this work, we propose a Generative Adversarial Network (GAN)-based image synthesis framework. We selectively extract the defect edges of the original image as well as the background texture information, and use them as network input through the spatially-adaptive (de)normalization (SPADE) module. This enriches the input information, thus significantly reducing the amount of training data for GAN network in image generation, and enhancing the background details as well as the defect boundaries in the generated images. Additionally, we introduce a novel generator loss term that balances the similarity and perceptual fidelity between synthetic and real images by constraining high-level features at different feature levels. This provides more valuable information for data augmentation in training object detection models using synthetic images. Our experimental results demonstrate the sophistication of the proposed image synthesis method and its effectiveness in data augmentation for steel strip surface defect detection tasks.

Large annotated datasets are required for training deep learning models, but in medical imaging data sharing is often complicated due to ethics, anonymization and data protection legislation. Generative AI models, such as generative adversarial networks (GANs) and diffusion models, can today produce very realistic synthetic images, and can potentially facilitate data sharing. However, in order to share synthetic medical images it must first be demonstrated that they can be used for training different networks with acceptable performance. Here, we therefore comprehensively evaluate four GANs (progressive GAN, StyleGAN 1–3) and a diffusion model for the task of brain tumor segmentation (using two segmentation networks, U-Net and a Swin transformer). Our results show that segmentation networks trained on synthetic images reach Dice scores that are 80%–90% of Dice scores when training with real images, but that memorization of the training images can be a problem for diffusion models if the original dataset is too small. Our conclusion is that sharing synthetic medical images is a viable option to sharing real images, but that further work is required. The trained generative models and the generated synthetic images are shared on AIDA data hub.

Chest radiography is one of the most ubiquitous imaging modalities, playing an essential role in screening, diagnosis and disease management. However, chest radiography interpretation is a time-consuming and complex task, requiring the availability of experienced radiologists. As such, automated diagnosis systems for pathology detection have been proposed aiming to reduce the burden on radiologists and reduce variability in image interpretation. While promising results have been obtained, particularly since the advent of deep learning, there are significant limitations in the developed solutions, namely the lack of representative data for less frequent pathologies and the learning of biases from the training data, such as patient position, medical devices and other markers as proxies for certain pathologies. The lack of explainability is also a challenge for the adoption of these solutions in clinical practice.Generative adversarial networks could play a significant role as a solution for these challenges as they allow to artificially create new realistic images. This way, new synthetic chest radiography images could be used to increase the prevalence of less represented pathology classes and decrease model biases as well as improving the explainability of automatic decisions by generating samples that serve as examples or counter-examples to the image being analysed, ensuring patient privacy.In this study, a few-shot generative adversarial network is used to generate synthetic chest radiography images. A minimum Fréchet Inception Distance score of 17.83 was obtained, allowing to generate convincing synthetic images. Perceptual validation was then performed by asking multiple readers to classify a mixed set of synthetic and real images. An average accuracy of 83.5% was obtained but a strong dependency on reader experience level was observed. While synthetic images showed structural irregularities, the overall image sharpness was a major factor in the decision of readers. The synthetic images were then validated using a MobileNet abnormality classifier and it was shown that over 99% of images were classified correctly, indicating that the generated images were correctly interpreted by the classifier. Finally, the use of the synthetic images during training of a YOLOv5 pathology detector showed that the addition of the synthetic images led to an improvement of mean average precision of 0.05 across 14 pathologies.In conclusion, the usage of few-shot generative adversarial networks for chest radiography image generation was shown and tested in multiple scenarios, establishing a baseline for future experiments to increase the applicability of generative models in clinical scenarios of automatic CXR screening and diagnosis tools.

Breast cancer is the leading cause of death in women worldwide; prevention of possible death from breast cancer can be decreased by early identification ultrasound image analysis by classifying ultrasound images into three classes (Normal, Benign, and Malignant), where the dataset used has imbalanced data. Imbalanced data cause the classification system only to recognize the majority class, so it is necessary to handle imbalanced data. In this study, imbalanced data can be handled by implementing the Deep Convolution Generative Adversarial Network (DCGAN) method as the addition of synthetic images to the training data. The DCGAN method generates synthetic images with feature learning on a Convolutional Neural Network (CNN), making DCGAN more stable than the basic generative adversarial network method. Synthetic and original images were further classified using the CNN GoogleNet method, which performs well in image classification and with reasonable computation cost. Synthetic ultrasound images were generated using a tuning hyperparameter in the DCGAN method to adjust the input size on GoogleNet for imbalanced data handling. From the experiment result, the implementation of DCGAN-GoogleNet has a higher accuracy in handling imbalanced data than conventional augmentation and other previous research, with an accuracy value reaching 91.61%, which is 1% to 4% higher than the accuracy value in the previous method.

In recent years, generative adversarial network (GAN) has attracted great attention due to its impressive performance and potential numerous applications, such as data augmentation, real-like image synthesis, image compression improvement, etc. The generator in GAN learns the density of the distribution from real data in order to generate high fidelity fake samples from latent space and deceive the discriminator. Despite its advantages, GAN can easily memorize training samples because of the high model complexity of deep neural networks. Thus, training a GAN with sensitive or private data samples may compromise the privacy of training data. To address this privacy issue, we propose a novel \textit{Privacy Preserving Generative Adversarial Network} (PPGAN) that perturbs the objective function of discriminator by injecting Laplace noises based on functional mechanism to guarantee the differential privacy of training data. Since generator training is considered as a post-processing step while guaranteeing differential privacy of discriminator, the trained generator should be differentially private to effectively protect data samples. Through detailed privacy analysis, we theoretically prove that PPGAN can provide such strict differential privacy guarantee. With extensive simulation study on the benchmark dataset MNIST, we show the efficacy of the proposed PPGAN under practical privacy budgets.

From real–time tumor classification to operative outcome prediction, applications of machine learning to neurosurgery are powerful. However, the translation of many of these applications are restricted by the lack of “big data” in neurosurgery. Important restrictions in patient privacy and sharing of imaging data reduce the diversity of the datasets used to train resulting models and therefore limit generalizability. Synthetic learning is a recent development in machine learning that generates synthetic data from real data and uses the synthetic data to train downstream models while preserving patient privacy. Such an approach has yet to be successfully demonstrated in the spine surgery domain. Spine radiographs were collected from the VinDR–SpineXR dataset, with 1470 labeled as abnormal and 2303 labeled as normal. A conditional generative adversarial network (GAN) was trained on the radiographs to generate a spine radiograph and normal/abnormal label. A modified conditional GAN (SpineGAN) was trained on the same task. A convolutional neural network (CNN) was trained using the real data to label abnormal radiographs. A CNN was trained to label abnormal radiographs using synthetic images from the GAN and in a separate experiment from SpineGAN. Using the real radiographs, an AUC of 0.856 was achieved in abnormality classification. Training on synthetic data generated by the standard GAN (AUC of 0.814) and synthetic data generated by our SpineGAN (AUC of 0.830) resulted in similar classifier performance. SpineGAN generated images with higher FID and lower precision scores, but with higher recall and increased performance when used for synthetic learning. The successful application of synthetic learning was demonstrated in the spine surgery domain for the classification of spine radiographs as abnormal or normal. A modified domain–relevant GAN is introduced for the generation of spine images, evidencing the importance of domain–relevant generation techniques in synthetic learning. Synthetic learning can allow neurosurgery to use larger and more diverse patient imaging sets to train more generalizable algorithms with greater patient privacy.

Differential privacy is often used in location privacy protection because of its strict reasoning and proof privacy guarantee. When users make continuous location query, it will cause noise superposition, which leads to the decline of query accuracy. At present, although differential privacy based on rule tree structure can reduce the query error, it will generate a lot of invalid zero nodes. The data structure is too large, and more improvement in the query accuracy can be further investigated. In this paper, we proposed a differential privacy location privacy protection method based on generative adversary network. Firstly, the definition of location data privacy protection under differential privacy mechanism is given, and then resume density aware network under differential privacy mechanism. Based on density aware network, the privacy protection problem of location data can be transformed into the distribution of fitting trajectory length. Finally, we use Markov chain to generate a new trajectory, and introduce the generative adversarial network to construct a set differential privacy protection method. Compared with other methods to improve the accuracy of differential privacy query, this method can effectively reduce the problem of query accuracy decline caused by noise superposition in continuous query, and can adapt to lbs location query service in different density environments.KeywordsGenerative adversarial networkLocation differencePrivacy protectionMarkov chainDensity aware network

Road networks play an important role in navigation and city planning. However, current methods mainly adopt the supervised strategy that needs paired remote sensing images and segmentation images. These data requirements are difficult to achieve. The pair segmentation images are not easy to prepare. Thus, to alleviate the burden of acquiring large quantities of training images, this study designed an improved generative adversarial network to extract road networks through a weakly supervised process named WSGAN. The proposed method is divided into two steps: generating the mapping image and post-processing the binary image. During the generation of the mapping image, unlike other road extraction methods, this method overcomes the limitations of manually annotated segmentation images and uses mapping images that can be easily obtained from public data sets. The residual network block and Wasserstein generative adversarial network with gradient penalty loss were used in the mapping network to improve the retention of high-frequency information. In the binary image post-processing, this study used the dilation and erosion method to remove salt-and-pepper noise and obtain more accurate results. By comparing the generated road network results, the Intersection over Union scores reached 0.84, the detection accuracy of this method reached 97.83%, the precision reached 92.00%, and the recall rate reached 91.67%. The experiments used a public dataset from Google Earth screenshots. Benefiting from the powerful prediction ability of GAN, the experiments show that the proposed method performs well at extracting road networks from remote sensing images, even if the roads are covered by the shadows of buildings or trees.

The Digital Imaging and Remote Sensing Image Generation (DIRSIG) tool is a first principles-based synthetic image generation model, developed at the Rochester Institute of Technology (RIT) over the past 20+ years. By calculating the sensor reaching radiance between the bandpass 0.2 to 20μm, it produces multi or hyperspectral remote sensing images. By integrating independent first principles based sub-models, such as MODTRAN, DIRSIG generates a representation of what a sensor would see with high radiometric fidelity. Currently, DIRSIG only models spatial/spectral synthetic images. In order to detect temporal changes in a process within the scene, a process model, which links the observable signatures of interest temporally, should be developed and incorporated into DIRSIG. These process models could be external time-dependent sub-models or pre-defined process models by the users to predict the state of the objects in the scene at a specific time. In this paper, a notional system of two tanks connected by a pipe is built, with hot water coming into tank A through a second pipe and with cooled water released from tank B through a third pipe. This is a simple hydrodynamic & thermodynamic model, controlled by the state of valves in the scenario. The initial temperature and height of the water in the two tanks are pre-defined by the user. Surface temperatures as a function of time are then predicted and captured as characterization maps, which are then mapped onto DIRSIG geometry using UV mapping technique. Finally, a spatial-spectral-temporal synthetic remote sensing image is produced.

Synthetic Bone Marrow Smears Are a Privacy-Preserving Substitute for Developing Accurate Leukemia Classification Models in Hematological Microscopy