Rank3DGAN: Semantic Mesh Generation Using Relative Attributes

Over the past few years, Generative Adversarial Networks (GANs) have garnered increased interest among researchers in Computer Vision, with applications including, but not limited to, image generation, translation, imputation, and super-resolution. Nevertheless, no GAN-based method has been proposed in the literature that can successfully represent, generate or translate 3D facial shapes (meshes). This can be primarily attributed to two facts, namely that (a) publicly available 3D face databases are scarce as well as limited in terms of sample size and variability (e.g., few subjects, little diversity in race and gender), and (b) mesh convolutions for deep networks present several challenges that are not entirely tackled in the literature, leading to operator approximations and model instability, often failing to preserve high-frequency components of the distribution. As a result, linear methods such as Principal Component Analysis (PCA) have been mainly utilized towards 3D shape analysis, despite being unable to capture non-linearities and high frequency details of the 3D face—such as eyelid and lip variations. In this work, we present 3DFaceGAN, the first GAN tailored towards modeling the distribution of 3D facial surfaces, while retaining the high frequency details of 3D face shapes. We conduct an extensive series of both qualitative and quantitative experiments, where the merits of 3DFaceGAN are clearly demonstrated against other, state-of-the-art methods in tasks such as 3D shape representation, generation, and translation.

Flat design is a modern style of graphics design that minimizes the number of design attributes required to convey 3D shapes. This approach suits design contexts requiring simplicity and efficiency, such as mobile computing devices. This `less-is-more' design inspiration has posed significant challenges in practice since it selects from a restricted range of design elements (e.g., color and resolution) to represent complex shapes. In this work, we investigate a means of computationally generating a specialized 2D flat representation - image formed by black-and-white patches - from 3D shapes. We present a novel framework that automatically abstracts 3D man-made shapes into 2D binary images at multiple scales. Based on a set of identified design principles related to the inference of geometry and structure, our framework jointly analyzes the input 3D shape and its counterpart 2D representation, followed by executing a carefully devised layout optimization algorithm. The robustness and effectiveness of our method are demonstrated by testing it on a wide variety of man-made shapes and comparing the results with baseline methods via a pilot user study. We further present two practical applications that are likely to benefit from our work.

With the rapid advancement of 3D scanning devices, large and complicated 3D shapes are becoming ubiquitous, and require large amount of resources to store and transmit them efficiently. This makes shape compression a demanding technique in order for the user to reduce the data transmission latency. Existing shape compression methods could achieve very low bit‐rates by sacrificing shape quality. But none of them guarantees the preservation of salient feature lines that users care. In addition, many 3D shapes come with parametric information for texture mapping purposes. In this paper we describe a spectral method to compress the geometric shapes equipped with arbitrary valid parametric information. It guarantees to preserve user‐specified feature lines while achieving a high compression ratio. By applying the spectral shape analysis – Dirichlet Manifold Harmonics, in the 2D parametric domain, this method provides a progressive compression mechanism to trade‐off between bit‐rate and shape quality. Experiments show that this method provides very low bit‐rate with high shape‐quality and still guarantees the preservation of user‐specified feature lines.

<p>High data collection costs and complicated data access regulations increase the demand for synthetic data. Generative Adversarial Networks (GANs) are a novel generative framework with great potential for high quality synthetic data generation. GANs formulate the true distribution of data implicitly, and the success of GANs are often measured based on the similarity of generated data to this true distribution. GANs were originally designed to work with continuous data. However, many important real-world datasets such as medical images involve discontinuous distributions. GAN training for discontinuous distributions is relatively more challenging, as the training procedure often suffers from instability and mode collapse issues. This dissertation focuses on designing novel GAN architectures to generate representative synthetic image data, and proposes new structures to alleviate GANs' mode collapse issue. As part of this thesis, novel applications of image data generation with GANs have been also investigated for important problems arising in the telecommunication industry and medical domain. Specifically, we first explore various GAN structures to generate engineered electromagnetic surfaces. We consider the continuous approximation of the data and explore the capabilities of feed-forward and convolutional GANs for synthetic data generation. Next, we introduce a novel GAN architecture to address the problem of mode collapse in GAN training. The proposed structure incorporates a third network that penalizes the generator for generating low diversity samples. Lastly, we study the challenging problem of object generation in 3D space using GANs, and we propose extensions to existing 3D GAN structures to generate connected 3D volumes. Additionally, we explore a more challenging version of this 3D volume generation problem by generating connected volumes packed with spheres. This research has applications in radiosurgery treatment planning, and the proposed 3D GAN structure can help generate rare, unseen 3D tumor volumes and information on how to treat these tumors. Accordingly, our analysis contributes to overcoming data scarcity issues (e.g., due to privacy considerations) for an important practical problem in the medical domain.</p>

The 3D image construction for the 2D images is a longstanding problem, explored by the number of computer graphics, computer vision, and machine learning research organizations from decades. After the evolution of deep learning architectures, scholars have shown their interest in developing 3D images from the 2D greyscale or RGB images because this perspective shows a significant influence in the discipline of computer vision. The applications of this conversion found in various fields of medical image analysis, robotic vision, game design, lunar explorations, 3D modelling, military, geographical structuring, physics, support models, etc. Potentially, when a 2D image converted to 3D representation, the same image can be viewed from different angles and directions. The 3D structure, which in turn generated, is much more informative than 2D images as it contains information about distance from the camera to a particular object. In this paper, we discussed various exiting methods for generating 3D representations from 2D images, using 3D representational data as well as without 3D representational data and proposing a novel approach for the construction of 3D models from existing 2D images using GAN. Generative adversarial networks (GANs) have shown tremendous results in generating new fake data from existing data, where we cannot detect the false data. Various other architectures of GAN, like HOLO-GAN, IG-GAN, have also been proposed to meet the need to convert 2D to 3D representation, which produced excellent results. After analysing, we provide an extensive comparative review on methods and architectures, which can convert 2D images to 3D objects and express our thoughts on the ideas proposed. Further, the concept of GAN extended to represent 360 view images, panorama images in 3D structures, which plays a vital role in spherical view analysis and synthesis, virtual reality design, augmented reality design, 3D modelling of data, etc.

With the advent of commodity 3D capturing devices and better 3D modeling tools, 3D shape content is becoming increasingly prevalent. Therefore, the need for shape retrieval algorithms to handle large-scale shape repositories is more and more important. This track provides a benchmark to evaluate large-scale 3D shape retrieval based on the ShapeNet dataset. It is a continuation of the SHREC 2016 large-scale shape retrieval challenge with a goal of measuring progress with recent developments in deep learning methods for shape retrieval. We use ShapeNet Core55, which provides more than 50 thousands models over 55 common categories in total for training and evaluating several algorithms. Eight participating teams have submitted a variety of retrieval methods which were evaluated on several standard information retrieval performance metrics. The approaches vary in terms of the 3D representation, using multi-view projections, point sets, volumetric grids, or traditional 3D shape descriptors. Overall performance on the shape retrieval task has improved significantly compared to the iteration of this competition in SHREC 2016. We release all data, results, and evaluation code for the benefit of the community and to catalyze future research into large-scale 3D shape retrieval (website: https://www.shapenet.org/shrec17).

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Longterm Recurrent Convolutional Network (LRCN). The 3DED- GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neural network architecture to minimize GPU memory usage and incorporates an Encoder-Decoder pair into a Long Shortterm Memory Network. By handling the 3D model as a sequence of 2D slices, LRCN transforms a coarse 3D shape into a more complete and higher resolution volume. While 3D-ED-GAN captures global contextual structure of the 3D shape, LRCN localizes the fine-grained details. Experimental results on both real-world and synthetic data show reconstructions from corrupted models result in complete and high-resolution 3D objects.

RJAN: Region‐based joint attention network for 3D shape recognition

Recent advances in generative adversarial networks (GANs) have led to remarkable achievements in face image synthesis. While methods that use style-based GANs can generate strikingly photorealistic face images, it is often difficult to control the characteristics of the generated faces in a meaningful and disentangled way. Prior approaches aim to achieve such semantic control and disentanglement within the latent space of a previously trained GAN. In contrast, we propose a framework that a priori models physical attributes of the face such as 3D shape, albedo, pose, and lighting explicitly, thus providing disentanglement by design. Our method, MOST-GAN, integrates the expressive power and photorealism of style-based GANs with the physical disentanglement and flexibility of nonlinear 3D morphable models, which we couple with a state-of-the-art 2D hair manipulation network. MOST-GAN achieves photorealistic manipulation of portrait images with fully disentangled 3D control over their physical attributes, enabling extreme manipulation of lighting, facial expression, and pose variations up to full profile view.

Datasets for acquiring 3D bone shapes: a systematic review.

We present a mathematically well-founded approach for the synthetic modeling of turbulent flows using generative adversarial networks (GAN). Based on the analysis of chaotic, deterministic systems in terms of ergodicity, we outline a mathematical proof that GAN can actually learn to sample state snapshots from the invariant measure of the chaotic system. Based on this analysis, we study a hierarchy of chaotic systems starting with the Lorenz attractor and then carry on to the modeling of turbulent flows with GAN. As training data, we use fields of velocity fluctuations obtained from large-eddy simulations (LES). Two architectures are investigated in detail: we use a deep, convolutional GAN (DCGAN) to synthesize the turbulent flow around a cylinder. We furthermore simulate the flow around a low-pressure turbine stator using the pix2pixHD architecture for a conditional DCGAN being conditioned on the position of a rotating wake in front of the stator. The settings of adversarial training and the effects of using specific GAN architectures are explained. We thereby show that GAN are efficient in simulating turbulence in technically challenging flow problems on the basis of a moderate amount of training data. GAN training and inference times significantly fall short when compared with classical numerical methods, in particular, LES, while still providing turbulent flows in high resolution. We furthermore analyze the statistical properties of the synthesized and LES flow fields, which agree excellently. We also show the ability of the conditional GAN to generalize over changes of geometry by generating turbulent flow fields for positions of the wake that are not included in the training data.

PSGAN: Revisit the binary discriminator and an alternative for face frontalization

As a new unsupervised learning algorithm framework, generative adversarial networks(GAN) have been favored by more and more researchers, and it has become a research hotspot now. GAN is inspired by the two-person zero-sum game theory in game theory. Its unique adversarial training idea can generate high-quality samples and has more powerful feature learning and feature expression capabilities than traditional machine learning algorithms. At present, GAN has achieved remarkable success in the field of computer vision, especially in the field of sample generation. Every year, a large number of GAN-related research papers are produced, reflecting the fiery degree of research on GAN model. Aiming at the hot model of GAN, first introduce the research status of GAN; then introduce the theory and framework of GAN, which analyzes in detail why the gradient disappears and the mode collapses during the training of GAN; then discussed some typical GAN improvement models, and summarized their theoretical improvements, advantages, limitations, application scenarios and implementation costs; Finally, the application results of GAN in data generation, image super-resolution, and image style conversion are shown, and the current challenges and future research directions of GAN are discussed.

The direct extension of 2D image learning to three-dimensional space is 3D point cloud learning. Recently, point cloud learning has shown significant results in 3D shape estimation and semantic segmentation. Despite these advancements, fundamental problems in point cloud learning still pose significant challenges. These problems include representation learning, shape generation, shape segmentation, and shape matching. In this paper, we propose a cognitive self-attention based learning approach to aggregate global representation of 3D shapes from point cloud data. We also integrate 3D point data with a binary tree structure to build a point cloud generator. We further design a novel Generative Adversarial Network (GAN) architecture to generate point clouds resembling the ground truth that could be used for unsupervised learning of 3D shapes. Relying on a self-attention mechanism, our framework, called SAPCGAN, aggregates the global graph features to correct the structural information of 3D shapes in the latent space. Finally, we compare the performance of two gradient penalty methods used in stabilizing the training of our GAN system. We show that our framework has high training efficiency and can provide state-of-the-art results in 3D point cloud generation. The performance of our is demonstrated with both quantitative and qualitative experimental evaluations. Furthermore, the generated 3D point clouds can be segmented into their natural semantic parts, such as, for example the four legs of a chair, the wings of an air plane, or the four wheels of a car.

We consider the task of photo-realistic unconditional image generation (generate high quality, diverse samples that carry the same visual content as the image) on mobile platforms using Generative Adversarial Networks (GANs). In this paper, we propose a novel approach to trade-off image generation accuracy of a GAN for the energy consumed (compute) at run-time called Scale-Energy Trade-off GAN (SETGAN). GANs usually take a long time to train and consume a huge memory hence making it difficult to run on edge devices. The key idea behind SETGAN for an image generation task is for a given input image, we train a GAN on a remote server and use the trained model on edge devices. We use SinGAN, a single image unconditional generative model, that contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. During the training process, we determine the optimal number of scales for a given input image and the energy constraint from target edge device. Results show that with the SETGAN's unique client-server based architecture, we were able to achieve 56% gain in energy for a loss of 3% to 12% SSIM accuracy. Also, with the parallel multi-scale training, we obtain around 4x gain in training time on the server.