Generative Adversarial Networks with Bi-directional Normalization for Semantic Image Synthesis

In this paper, we focus on the task of generating realistic images given an input semantic layout, which is also called semantic image synthesis. Most of previous methods are based on conditional generative adversarial networks mechanism, which is stacks of convolution, normalization, and non-linearity layers. However, these methods easily generate blurred regions and distorted structures. There are two limits existing: their normalization layers are unable to make a good balance between keeping semantic layout information and geometric changes; and cannot effectively aggregated multi-level feature. To address the above problems, we propose a novel method which incorporates multi-level gate feature aggregation mechanism (GFA) and spatially adaptive batch-instance normalization (SPAda-BIN) for semantic image synthesis. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, in terms of both visual fidelity and quantitative metrics.

By leveraging semantic (label) information, supervised hashing has demonstrated better accuracy than unsupervised hashing in many real applications. Because the hashing-code learning problem is essentially a discrete optimization problem which is hard to solve, most existing supervised hashing methods try to solve a relaxed continuous optimization problem by dropping the discrete constraints. However, these methods typically suffer from poor performance due to the errors caused by the relaxation. Some other methods try to directly solve the discrete optimization problem. However, they are typically time-consuming and unscalable. In this paper, we propose a novel method, called column sampling based discrete supervised hashing (COSDISH), to directly learn the discrete hashing code from semantic information. COSDISH is an iterative method, in each iteration of which several columns are sampled from the semantic similarity matrix and then the hashing code is decomposed into two parts which can be alternately optimized in a discrete way. Theoretical analysis shows that the learning (optimization) algorithm of COSDISH has a constant-approximation bound in each step of the alternating optimization procedure. Empirical results on datasets with semantic labels illustrate that COSDISH can outperform the state-of-the-art methods in real applications like image retrieval.

The existing dehazing algorithms are problematic because of dense haze being unevenly distributed on the images, and the deep convolutional dehazing network relying too greatly on large-scale datasets. To solve these problems, this paper proposes a generative adversarial network based on the deep symmetric Encoder-Decoder architecture for removing dense haze. To restore the clear image, a four-layer down-sampling encoder is constructed to extract the semantic information lost due to the dense haze. At the same time, in the symmetric decoder module, an attention mechanism is introduced to adaptively assign weights to different pixels and channels, so as to deal with the uneven distribution of haze. Finally, the framework of the generative adversarial network is generated so that the model achieves a better training effect on small-scale datasets. The experimental results showed that the proposed dehazing network can not only effectively remove the unevenly distributed dense haze in the real scene image, but also achieve great performance in real-scene datasets with less training samples, and the evaluation indexes are better than other widely used contrast algorithms.

In this paper, we focus on the semantic image synthesis task that aims at transferring semantic label maps to photo-realistic images. Existing methods lack effective semantic constraints to preserve the semantic information and ignore the structural correlations in both spatial and channel dimensions, leading to unsatisfactory blurry and artifact-prone results. To address these limitations, we propose a novel Dual Attention GAN (DAGAN) to synthesize photo-realistic and semantically-consistent images with fine details from the input layouts without imposing extra training overhead or modifying the network architectures of existing methods. We also propose two novel modules, i.e., position-wise Spatial Attention Module (SAM) and scale-wise Channel Attention Module (CAM), to capture semantic structure attention in spatial and channel dimensions, respectively. Specifically, SAM selectively correlates the pixels at each position by a spatial attention map, leading to pixels with the same semantic label being related to each other regardless of their spatial distances. Meanwhile, CAM selectively emphasizes the scale-wise features at each channel by a channel attention map, which integrates associated features among all channel maps regardless of their scales. We finally sum the outputs of SAM and CAM to further improve feature representation. Extensive experiments on four challenging datasets show that DAGAN achieves remarkably better results than state-of-the-art methods, while using fewer model parameters.

The task of spatial layout estimation of monocular image is to segment an RGB image of indoor scenes with semantic surface labels (i.e., ceiling, floor, front wall, left wall, and right wall). Most recent methods have to produce layout hypotheses based on the estimated edge map or semantic labels, and then rank the layout hypotheses. In this paper, we present an end-to-end framework that can directly output the layout type and keypoint coordinates (defined in the LSUN challenge). The proposed method takes advantage of transfer learning via learning on the fake samples, i.e., plenty of artificial {type, keypoints, edge map} triplets are generated to learn the mapping from edge maps to keypoint coordinates. Generative adversarial network (GAN) is implemented in this work for domain adaptation of the edge maps. Experimental results show that the proposed method can achieve state-of-the-art layout estimation performance on benchmark datasets.

Semantic image synthesis is a challenging task with many practical applications. Albeit remarkable progress has been made in semantic image synthesis with spatially-adaptive normalization and existing methods normalize the feature activations under the coarse-level guidance (e.g., semantic class). However, different parts of a semantic object (e.g., wheel and window of car) are quite different in structures and textures, making blurry synthesis results usually inevitable due to the missing of fine-grained guidance. In this paper, we propose a novel normalization module, termed as REtrieval-based Spatially AdaptIve normaLization (RESAIL), for introducing pixel level fine-grained guidance to the normalization architecture. Specifically, we first present a retrieval paradigm by finding a content patch of the same semantic class from training set with the most similar shape to each test semantic mask. Then, RESAIL is presented to use the retrieved patch for guiding the feature normalization of corresponding region, and can provide pixel level fine-grained guidance, thereby greatly mitigating blurry synthesis results. Moreover, distorted ground-truth images are also utilized as alternatives of retrieval-based guidance for feature normalization, further benefiting model training and improving visual quality of generated images. Experiments on several challenging datasets show that our RESAIL performs favorably against state-of-the-arts in terms of quantitative metrics, visual quality, and subjective evaluation. The source code and pre-trained models will be publicly available.

Language model enhanced surface chloride concentration determination for concrete within splash environment based on limited field records

USIS: A unified semantic image synthesis model trained on a single or multiple samples

This article presents a robust and efficient system for euro and Mexican banknote recognition. A high banknote recognition and classification rate was achieved using neural networks and deep learning with real scene images taken with both sunlight and artificial light. Without extracting characteristics by hand, the convolutional neural networks was fed with raw images. Analysis and experiments were carried out on banknotes based on key features, such as; watermarks, portraits on the bills, bill value written in words and numbers, and the complete banknotes. It was concluded that both the color information and some regions of the banknotes, as well as the banknote denomination written in words and numbers and the complete banknote, is the appropriate information to achieve a high rate of banknote classification and recognition. The experimental results show that the proposed approach is promising with quite remarkable results; it performs an efficient and robust classification using real scene images taken with both sunlight and artificial light and is invariant to banknote rotation and translation. A high recognition rate was achieved for Mexican banknotes and for euros. At present, the results contained herein are an improvement over thosereported in the state of the art.

Semantic image synthesis via text description is a desirable and challenging task, which requires more protection of the text irrelevant content in the original image. Existing methods directly modify the original image, which become more difficult when encountering high resolution image, and the generated images are also blurred and lack in detail. This paper presents a novel network architecture to progressively manipulate an image starting from low-resolution, while introducing the original image of corresponding size at different stages with our proposed union module to avoid losing of detail. And the progressive design of the network allows us to modify the image from coarse into fine. Compared with the previous methods, our new method can successfully manipulate a high resolution image and generate a new image with background protection and fine details. The experimental results on CUB-200-2011 dataset show that the proposed approach outperforms existing methods in terms of image detail, background protection and high resolution generation.

Image dehazing has always been a challenging topic in image processing. The development of deep learning methods, especially the generative adversarial networks (GAN), provides a new way for image dehazing. In recent years, many deep learning methods based on GAN have been applied to image dehazing. However, GAN has two problems in image dehazing. Firstly, For haze image, haze not only reduces the quality of the image but also blurs the details of the image. For GAN network, it is difficult for the generator to restore the details of the whole image while removing the haze. Secondly, GAN model is defined as a minimax problem, which weakens the loss function. It is difficult to distinguish whether GAN is making progress in the training process. Therefore, we propose a guided generative adversarial dehazing network (GGADN). Different from other generation adversarial networks, GGADN adds a guided module on the generator. The guided module verifies the network of each layer of the generator. At the same time, the details of the map generated by each layer are strengthened. Network training is based on the pre-trained VGG feature model and L1-regularized gradient prior which is developed by new loss function parameters. From the dehazing results of synthetic images and real images, the proposed method is better than the state-of-the-art dehazing methods.

Sensory evaluation can be influenced by semantic information such as gender descriptions. Gender categories are associated with tactile information (e.g., female = soft/smooth, while male = hard/rough). Feminine scents (e.g., floral) are typically perceived as soft and smooth. Thus, semantic labels of gender (feminine/masculine qualities) may influence congruent sensory evaluation (i.e., female = soft/smooth, male = hard/rough). This study examined how semantically labeled scent-gender associations influence the evaluation of scent and texture. Specifically, we examined whether “feminine” and “masculine” labels applied to neutral scents that have not been associated with gender influence scent and haptic evaluation. Participants sniffed a feminine-labeled or masculine-labeled scent embedded on soft and rough papers. They then evaluated the scent (e.g., gender perception) and texture (e.g., hedonic evaluation). The results demonstrated that participants who sniffed a feminine-labeled (vs. masculine-labeled) scent perceived it as more feminine. However, contrary to our expectations, gender labeling of scent did not influence haptic evaluation. These findings indicate that semantic labeling of scents (i.e., feminine/masculine) may alter the gender perception of a scent but not the tactile evaluation. Practical implications for (online) sensory marketing are discussed.

Traditional video steganography methods primarily rely on modifying concealed spaces for embedding, thereby exhibiting a certain degree of security and embedding capacity. Nevertheless, these methods do not fully capitalize on the rich semantic information inherent in videos, limiting their overall effectiveness. In this paper, an AI‐generated video steganography scheme based on semantic segmentation is proposed. The mapping relationship between secret and semantic information is established by using a semantic segmentation model. The secret information can be converted into semantic labels by semantic histograms or pixels means, and semantic labels containing secret information are obtained and input into the video‐to‐video model to drive the generation of stego videos. After receiving the stego video, the receiver extracts the secret information using a pre‐defined specific embedding mode, including the methods of sub‐block partitioning and embedding capacity per frame. The experimental results show that the stego video has good visual quality, security, and robustness against various noise attacks.

Maps generated by many visual Simultaneous Localization and Mapping algorithms consist of geometric primitives such as points, lines or planes. These maps offer a topographic representation of the environment, but they contain no semantic information about the environments. Object classifiers leveraging advances in machine learning are highly accurate and reliable, capable of detecting and classifying thousands of objects. Classifiers can be incorporated into a SLAM pipeline to add semantic information to a scene. Frequently, this semantic information is conducted for each frame of the image, but semantic labeling is not persistent over time. In this work, we present a nonparametric statistical approach to perform matching/association of objects detected over consecutive image frames. These associated classified objects are then localized in the accrued map using an unsupervised clustering method. We test our approach on multiple data sets, and it shows strong performance in terms of objects correctly associated from frame to frame. We also have tested our algorithm on three data sets in our lab environment using tag markers to demonstrate the accuracy of classified object localization process.

The paper proposes a solution based on Generative Adversarial Network (GAN) for solving jigsaw puzzles. The problem assumes that an image is divided into equal square pieces, and asks to recover the image according to information provided by the pieces. Conventional jigsaw puzzle solvers often determine the relationships based on the boundaries of pieces, which ignore the important semantic information. In this paper, we propose JigsawGAN, a GAN-based auxiliary learning method for solving jigsaw puzzles with unpaired images (with no prior knowledge of the initial images). We design a multi-task pipeline that includes, (1) a classification branch to classify jigsaw permutations, and (2) a GAN branch to recover features to images in correct orders. The classification branch is constrained by the pseudo-labels generated according to the shuffled pieces. The GAN branch concentrates on the image semantic information, where the generator produces the natural images to fool the discriminator, while the discriminator distinguishes whether a given image belongs to the synthesized or the real target domain. These two branches are connected by a flow-based warp module that is applied to warp features to correct the order according to the classification results. The proposed method can solve jigsaw puzzles more efficiently by utilizing both semantic information and boundary information simultaneously. Qualitative and quantitative comparisons against several representative jigsaw puzzle solvers demonstrate the superiority of our method.