SlabGAN: a method for generating efficient 3D anisotropic medical volumes using generative adversarial networks

The fusion of Federated Learning (FL) and deep generative models is transforming medical imaging by enabling privacy-preserving and data-efficient machine learning. Training large-scale deep models on radiological imaging data remains challenging due to data scarcity, heterogeneity, and strict privacy constraints that limit data sharing across institutions. FL addresses these challenges by employing collaborative model training that does not expose raw patient data, while generative models synthesize realistic medical images to alleviate scarcity and imbalance. The convergence of these two fields has given rise to federated generative models (FGMs), which extend Generative Adversarial Networks (GANs), Variational Autoencoders (VAE), and Diffusion models into distributed and privacy-preserving environments. FGMs have demonstrated significant potential in various medical imaging tasks, including data augmentation, image reconstruction, cross-modality conversion, and enhancing classification and segmentation models. Despite promising progress, the field remains in its early stages, facing open challenges in communication efficiency, scalability, data heterogeneity, and clinical reliability. This survey provides a comprehensive review of FGMs in medical imaging, analyzing their architectures, applications, and benchmark datasets, while highlighting key challenges and outlining future research directions to advance privacy-preserving generative modeling in healthcare.

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Longterm Recurrent Convolutional Network (LRCN). The 3DED- GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neural network architecture to minimize GPU memory usage and incorporates an Encoder-Decoder pair into a Long Shortterm Memory Network. By handling the 3D model as a sequence of 2D slices, LRCN transforms a coarse 3D shape into a more complete and higher resolution volume. While 3D-ED-GAN captures global contextual structure of the 3D shape, LRCN localizes the fine-grained details. Experimental results on both real-world and synthetic data show reconstructions from corrupted models result in complete and high-resolution 3D objects.

Convolutional Neural Networks (CNNs) achieve excellent computer-assisted diagnosis with sufficient annotated training data. However, most medical imaging datasets are small and fragmented. In this context, Generative Adversarial Networks (GANs) can synthesize realistic/diverse additional training images to fill the data lack in the real image distribution; researchers have improved classification by augmenting data with noise-to-image (e.g., random noise samples to diverse pathological images) or image-to-image GANs (e.g., a benign image to a malignant one). Yet, no research has reported results combining noise-to-image and image-to-image GANs for further performance boost. Therefore, to maximize the DA effect with the GAN combinations, we propose a two-step GAN-based DA that generates and refines brain Magnetic Resonance (MR) images with/without tumors separately: (i) Progressive Growing of GANs (PGGANs), multi-stage noise-to-image GAN for high-resolution MR image generation, first generates realistic/diverse 256×256 images; (ii) Multimodal UNsupervised Image-to-image Translation (MUNIT) that combines GANs/Variational AutoEncoders or SimGAN that uses a DA-focused GAN loss, further refines the texture/shape of the PGGAN-generated images similarly to the real ones. We thoroughly investigate CNN-based tumor classification results, also considering the influence of pre-training on ImageNet and discarding weird-looking GAN-generated images. The results show that, when combined with classic DA, our two-step GAN-based DA can significantly outperform the classic DA alone, in tumor detection (i.e., boosting sensitivity 93.67% to 97.48%) and also in other medical imaging tasks.

The problem of 3D reconstruction of medical images is very challenging due to the disadvantages of less texture and serious speckle noise. In this paper, we propose a medical image 3D reconstruction framework based on transfer learning and Structure-from-Motion. Unlike some existing methods, which need special hardware devices, or be effective for a particular modal of images, our unified framework does not depend on any special equipment and can be applied to diverse modalities of medical images, such as ultrasonic image, computed tomography (CT) etc. Its contribution covers three aspects: First, it uses Structure-From-Motion(SFM) to estimate the camera structure and pose of consecutive frames end-to-end, which is suitable for clinical auxiliary diagnosis. Second, strong and robust photogrammetry knowledge, including depth, the camera structure and pose, can be pre-learned from the natural images with the richer texture, and then, are transferred to the medical images learning. Besides, Med-3D is proposed, a GAN model based on self-supervised learning framework, in the generator, the spatial coplanar characteristics to constrain the cloud points reconstructed from a single captured frame or adjacent frames are considered for learning more precise features. Experimental results on the medical data of ultrasound images, and CT in the open clinical library, demonstrate that our method can reconstruct geometric structures from 2D slices and has excellent accuracy in terms of reconstruction effects.

3D object reconstruction has always been a hot topic in computer vision. Especially in recent years, many methods of learning volumetric predictions achieve robust 3D reconstruction using deep networks with 3D convolutional operations, which are direct analogies to classical 2D ones. However, the majority of methods employ strong shape priors and exist computationally waste in predicting 3D shapes. In this paper, we propose 3D-Mask-GAN, a novel framework to efficiently accomplish the task of unsupervised single-view 3D object reconstruction. We use 3D Generative Adversarial Networks (GAN) to predict the 3D shape from the single-view image and improve reconstruction accuracy by applying 2D projection masks instead of 3D priors simultaneously. The key idea is to insert a projector (a build-in camera system to approximate the true rendering pipeline) into the framework of 3D GAN to synthesize novel masks for optimization. We learn single-class and multi-class objects to evaluate our network. Experimental results show that our framework achieves impressive performance with fewer training iterations in terms of unsupervised shape predictions.

The need for medical image encryption is increasingly pronounced, for example, to safeguard the privacy of the patients' medical imaging data. In this article, a novel deep learning-based key generation network (DeepKeyGen) is proposed as a stream cipher generator to generate the private key, which can then be used for encrypting and decrypting of medical images. In DeepKeyGen, the generative adversarial network (GAN) is adopted as the learning network to generate the private key. Furthermore, the transformation domain (that represents the "style" of the private key to be generated) is designed to guide the learning network to realize the private key generation process. The goal of DeepKeyGen is to learn the mapping relationship of how to transfer the initial image to the private key. We evaluate DeepKeyGen using three data sets, namely, the Montgomery County chest X-ray data set, the Ultrasonic Brachial Plexus data set, and the BraTS18 data set. The evaluation findings and security analysis show that the proposed key generation network can achieve a high-level security in generating the private key.

Remote display and interaction of 3D reconstruction of medical images has been achieved utilizing VTK (the Visualization Toolkit) and HTML5. Aiming at the problem that the network coordination of the medical 3D reconstruction platforms are generally unsatisfactory, the feasibility of achieving multi-frames-based remote interaction of 3D reconstruction of medical images using Canvas and WebSocket in HTML5 has been discussed and verified by experiments. Furthermore, a method to display 3D images on pure Web client based on B/S (Browser/Server) framework has been put forward. The 3D reconstruction system with the application of this method has been proved to be effective in displaying and interacting on both PC and mobile platform with almost real-time effects. This feasible method has certain practical application in remote medical assistance.

Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images which could affect the clinical interpretation and diagnosis, and (3) the vanilla GAN architecture may suffer from unstable training and collapse mode that can also affect the SISR results. In this paper, we propose a novel lesion focused SR (LFSR) method, which incorporates GAN to achieve perceptually realistic SISR results for brain tumour MRI images. More importantly, we test and make comparison using recently developed GAN variations, e.g., Wasserstein GAN (WGAN) and WGAN with Gradient Penalty (WGAN-GP), and propose a novel multi-scale GAN (MS-GAN), to achieve a more stabilised and efficient training and improved perceptual quality of the super-resolved results. Based on both quantitative evaluations and our designed mean opinion score, the proposed LFSR coupled with MS-GAN has performed better in terms of both perceptual quality and efficiency.

The purpose of this study is to develop a novel computer-aided diagnosis (CAD) scheme to facilitate breast mass classification, which is based on the latest transferring generative adversarial networks (GAN) technology. Although GAN is one of the most popular techniques for image augmentation, it requires a relatively large original dataset to achieve satisfactory results, which may not be available for most of the medical imaging tasks. To address this challenge, we developed a novel transferring GAN, which was built based on the deep convolutional generative adversarial networks (DCGAN). This novel model was first pre-trained on a dataset of non-mass mammogram patches. Then the generator and the discriminator were fine-tuned on the mass dataset. A supervised loss was integrated with the discriminator, such that it can be used to directly classify the benign/malignant masses. We retrospectively assembled a total of 25,000 non-mass patches and 1024 mass images to assess this model, using classification accuracy and receiver operating characteristic (ROC) curve. The results demonstrated that our proposed approach improved the accuracy and area under the ROC curve (AUC) by 6.0% and 3.5% respectively, when compared with the classifiers trained without conventional data augmentation. This investigation may provide a new perspective for researchers to effectively train the GAN models on medical imaging tasks with limited datasets.

Medical imaging is becoming indispensable for patients' healthcare. Machine learning plays an essential role in the medical imaging field, including computer-aided diagnosis, image segmentation, image registration, image fusion, image-guided therapy, image annotation, and image database retrieval. With advances in medical imaging, new imaging modalities and methodologies such as cone-beam/multislice CT, 3D ultrasound imaging, tomosynthesis, diffusion-weighted magnetic resonance imaging (MRI), positron-emission tomography (PET)/CT, electrical impedance tomography, and diffuse optical tomography, new machine-learning algorithms/applications are demanded in the medical imaging field. Because of large variations and complexityit is generally difficult to derive analytic solutions or simple equations to represent objects such as lesions and anatomy in medical images. Therefore, tasks in medical imaging require “learning from examples” for accurate representation of data and prior knowledge. Because of its essential needs, machine learning in medical imaging is one of the most promising, growing fields. The main aim of this special issue is to help advance the scientific research within the broad field of machine learning in medical imaging. The special issue was planned in conjunction with the International Workshop on Machine Learning in Medical Imaging (MLMI 2010) [1], which was the first workshop on this topic, held at the 13th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2010) in September, 2010, in Beijing, China. This special issue is one in a series of special issues of journals on this topic [2]; it focuses on major trends and challenges in this area, and it presents work aimed at identifying new cutting-edge techniques and their use in medical imaging. The quality level of the submissions for this special issue was very high. A total of 17 papers were submitted to this issue in response to the call for papers. Based on a rigorous review process, 10 papers (59%) were accepted for publication in the special issue. The special issue starts by a review of studies on a class of machine-learning techniques, called pixel/voxel-based machine learning, in medical imaging by K. Suzuki. A series of medical imaging applications of machine-learning techniques are presented. A large variety of applications are well represented here, including organ modeling by D. Wang et al. and X. Qiao and Y.-W. Chen, brain function estimation by V. Michel et al., image reconstruction by H. Shouno et al., lesion classification by P. Wighton et al., modality classification by X.-H. Han and Y.-W. Chen, lesion segmentation by M. Zortea et al., organ segmentation by S. Alzubi et al., and visualization of molecular signals by F. Mattoli et al. Also, the issue covers various biomedical imaging modalities, including MRI by D. Wang et al., CT by X. Qiao and Y.-W. Chen and H. Shouno et al., functional MRI by V. Michel et al., dermoscopy by P. Wighton et al. and M. Zortea et al., scintigraphy by X.-H. Han and Y.-W. Chen, ultrasound imaging by X.-H. Han and Y.-W. Chen, radiography by X.-H. Han and Y.-W. Chen, MR angiography by S. Alzubi et al., and microscopy by F. Mattoli et al. as well as a variety of organs, including the kidneys by D. Wang et al., liver by X. Qiao and Y.-W. Chen, brain by V. Michel et al. and F. Mattoli et al., chest by S. Alzubi et al., skin by P. Wighton et al. and M. Zortea et al., and heart by F. Mattoli et al. Various machine-learning techniques were developed/used to solve the respective problems, including structured dictionary learning by D. Wang et al., generalized N-dimensional principal component analysis by X. Qiao et al., multiclass sparse Bayesian regression by V. Michel et al., Bayesian hyperparameter inference by H. Shouno et al., supervised learning of probabilistic models based on maximum aposteriori estimation and conditional random fields by P. Wighton et al., joint kernel equal contribution in support vector classification by X.-H. Han and Y.-W. Chen, and iterative hybrid classification by M. Zortea et al. We are grateful to all authors for their excellent contributions to this special issue and to all reviewers for their reviews and constructive suggestions. We hope that this special issue will inspire further ideas for creative research, advance the field of machine learning in medical imaging, and facilitate the translation of the research from bench to bedside. Kenji Suzuki Pingkun Yan Fei Wang Dinggang Shen

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

<p>High data collection costs and complicated data access regulations increase the demand for synthetic data. Generative Adversarial Networks (GANs) are a novel generative framework with great potential for high quality synthetic data generation. GANs formulate the true distribution of data implicitly, and the success of GANs are often measured based on the similarity of generated data to this true distribution. GANs were originally designed to work with continuous data. However, many important real-world datasets such as medical images involve discontinuous distributions. GAN training for discontinuous distributions is relatively more challenging, as the training procedure often suffers from instability and mode collapse issues. This dissertation focuses on designing novel GAN architectures to generate representative synthetic image data, and proposes new structures to alleviate GANs' mode collapse issue. As part of this thesis, novel applications of image data generation with GANs have been also investigated for important problems arising in the telecommunication industry and medical domain. Specifically, we first explore various GAN structures to generate engineered electromagnetic surfaces. We consider the continuous approximation of the data and explore the capabilities of feed-forward and convolutional GANs for synthetic data generation. Next, we introduce a novel GAN architecture to address the problem of mode collapse in GAN training. The proposed structure incorporates a third network that penalizes the generator for generating low diversity samples. Lastly, we study the challenging problem of object generation in 3D space using GANs, and we propose extensions to existing 3D GAN structures to generate connected 3D volumes. Additionally, we explore a more challenging version of this 3D volume generation problem by generating connected volumes packed with spheres. This research has applications in radiosurgery treatment planning, and the proposed 3D GAN structure can help generate rare, unseen 3D tumor volumes and information on how to treat these tumors. Accordingly, our analysis contributes to overcoming data scarcity issues (e.g., due to privacy considerations) for an important practical problem in the medical domain.</p>

3D or multi-dimensional medical image reconstruction is attracting more and more attention in modern era. Rapid volumetric image reconstruction is of paramount importance to clinicians for prompt diagnosis and analysis of complex tissue alternations. However, due to a high demand on computation for either the analytical or iterative image reconstruction algorithms, it is almost impossible to meet the requirement of real-time reconstruction. On the other hand, with high computing capability and broad banding characteristic, recently multi-core processors provide a new possibility and hopeful start for some of the most computation-demanding applications, such as medical imaging, digital media and entertainment equipment. This study aims to parallelize an analytical 3D CT image reconstruction algorithm, the so-called FDK method, and to implement it on an existing multi-core platform system, towards a real-time 3D medical image reconstruction. In this paper, the parallelization of the 3D image reconstruction algorithm, the coding optimizations and a comparative result of run time savings are described.

Medical imaging technologies have drastically changed the way healthcare professionals diagnose and treat patients. However, medical imaging datasets are limited in terms of size and diversity, which can lead to inaccurate diagnosis and treatment plans. To overcome these limitations, Generative Adversarial Networks (GANs) have been used to generate new medical images based on existing datasets. GANs are a type of deep learning artificial intelligence that can use existing datasets to create new synthetic images that look almost indistinguishable from real images. GANs can be used to generate medical images of organs, tissues, and diseases that are not present in existing datasets, providing healthcare professionals with more accurate and diverse datasets to diagnose and treat patients. GANs for medical image synthesis is a powerful tool that can revolutionize the healthcare industry. In this paper an investigation of the most used Gans architecture are presented. a state of the art of the latest works applying Gans are also discussed based on the adopted modality.

Purpose: Sharing medical images between institutions, or even inside the same institution, is restricted by various laws and regulations; research projects requiring large datasets may suffer as a result. These limitations might be addressed by an abundant supply of synthetic data that (1)are representative (i.e., the synthetic data could produce comparable research results as the original data) and (2)do not closely resemble the original images (i.e., patient privacy is protected). We introduce a framework that generates data with these requirements leveraging generative adversarial network (GAN) ensembles in a controlled fashion. Approach: To this end, an adaptive ensemble scaling strategy with the objective of representativeness is defined. A sampled Fréchet distance-based constraint was then created to eliminate poorly converged candidates. Finally, a mutual information-based validation metric was embedded into the framework to confirm there are visual differences between the original and the generated synthetic images. Results: The applicability of the solution is demonstrated with a case study for generating three-dimensional brain metastasis (BM) from T1-weighted contrast-enhanced MRI studies. A previously published BM detection system was reported to produce 9.12 false-positives at 90% detection sensitivity based on the original data. By using the synthetic data generated with the proposed framework, the system produced 9.53 false-positives at the same sensitivity level. Conclusions: Achieving comparable algorithm performance relying solely on synthetic data unveils a significant potential to eliminate/reduce patient privacy concerns when sharing data in medical imaging.