Small vehicle classification in the wild using generative adversarial network

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

Image super-resolution is of great interest in medical imaging field. However, different from natural images studied in computer vision field, the low-resolution (LR) medical imaging data is often a stack of high-resolution (HR) 2D slices with large slice thickness. Consequently, the goal of super-resolution for medical imaging data is to reconstruct the missing slice(s) between any two consecutive slices. Since some modalities (e.g., T1-weighted MR image) are often acquired with high-resolution (HR) image, it is intuitive to harness the prior self-similarity information in the HR image for guiding the super-resolution of LR image (e.g., T2-weighted MR image). The conventional way is to find the profile of patchwise self-similarity in the HR image and then use it to reconstruct the missing information at the same location of LR image. However, the local morphological patterns could vary significantly across the LR and HR images, due to the use of different imaging protocols. Therefore, such direct (un-supervised) adaption of self-similarity profile from HR image is often not effective in revealing the actual information in the LR image. To this end, we propose to employ the existing image information in the LR image to supervise the estimation of self-similarity profile by requiring it not only being optimal in representing patches in the HR image, but also producing less reconstruction errors for the existing image information in the LR image. Moreover, to make the anatomical structures spatially consistent in the reconstructed image, we simultaneously estimate the self-similarity profiles for a stack of patches across consecutive slices by solving a group sparse patch representation problem. We have evaluated our proposed super-resolution method on both simulated brain MR images and real patient images with multiple sclerosis lesion, achieving promising results with more anatomical details and sharpness.

Deep convolutional neural networks (CNNs), trained on corresponding pairs of high- and low-resolution images, achieve state-of-the-art performance in single-image super- resolution and surpass previous signal-processing based approaches. However, their performance is limited when applied to real photographs. The reason lies in their train- ing data: low-resolution (LR) images are obtained by bicu- bic interpolation of the corresponding high-resolution (HR) images. The applied convolution kernel significantly differs from real-world camera-blur. Consequently, while current CNNs well super-resolve bicubic-downsampled LR images, they often fail on camera-captured LR images. To improve generalization and robustness of deep super- resolution CNNs on real photographs, we present a ker- nel modeling super-resolution network (KMSR) that incor- porates blur-kernel modeling in the training. Our pro- posed KMSR consists of two stages: we first build a pool of realistic blur-kernels with a generative adversarial net- work (GAN) and then we train a super-resolution network with HR and corresponding LR images constructed with the generated kernels. Our extensive experimental vali- dations demonstrate the effectiveness of our single-image super-resolution approach on photographs with unknown blur-kernels.

To reduce external disturbances and achieve high vertical resolution, the scanning time for white-light interference microscopy is very short. Because capturing high-resolution (HR) images is time consuming, low-resolution (LR) images are acquired instead. However, HR images are more desirable because they contain more details. To ensure high vertical resolution and high image resolution, one feasible solution is to process the scanned LR images to HR images by single image super-resolution (SISR). In this paper, an interference image super-resolution (IISR) model based on a generative adversarial network (GAN) is proposed. The generator is based on the enhanced super-resolution generative adversarial network (ESRGAN) architecture. With the aim of acquiring more realistic images, the discriminator network is designed using a modified DenseNet architecture, in which the pooling layers are replaced with dilated convolutional layers. The perceptual loss is optimized, and the content loss is upgraded to a continuously differentiable piecewise function. Various microscopy images are tested, including images with and without interference fringes. The IISR model has been proven to restore LR images to HR images. The comparative experiments prove that the proposed model achieves better visual quality than other models, preserving more realistic details.

Image super-resolution is a research endeavour that has gained notoriety in computer vision. The research goal is to increase the spatial dimensions of an image using corresponding low-resolution and high-resolution image pairs to enhance the perceptual quality. The challenge of maintaining such perceptual quality lies in developing appropriate algorithms that learn to reconstruct higher-quality images from their lower-resolution counterparts. Recent methods employ deep learning algorithms to reconstruct textural details prevalent in low-resolution images. Since corresponding image pairs are non-trivial to collect, researchers attempt super-resolution by creating synthetic low-resolution representations of high-resolution images. Unfortunately, such methods employ ineffective downscaling operations to achieve synthetic low-resolution images. These methods fail to generalize well on real-world images that may suffer different degradations. A different angle is offered to solve the task of image super-resolution by investigating the plausibility of learning the degradational operation using generative adversarial networks. A two-stage generative adversarial network along with two architectural variations is proposed to solve the task of real-world super-resolution from low-resolution images with unknown degradations. It is demonstrated that learning to downsample images in a weakly supervised manner is an impactful and viable approach for super-resolution.

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

We propose a new high-quality up-scaling technique that extends the existing example based super-resolution (SR) framework. Our approach is based on the fundamental idea that a low-resolution (LR) image could be generated from any of the multiple possible high-resolution (HR) images. Therefore it would be more natural to use multiple predictors of HR patch from LR patch instead of single one. In this work we build a generic framework to estimate an HR image from LR one using an adaptive prior (select the predictor locally) based on the local statistics of LR images. We use natural image patch prior as the HR image statistics. We partition the natural images into documents and group them to discover the inherent topics using probabilistic Latent Semantic Analysis (pLSA) and also learn the dual dictionaries of HR and LR image patch pairs for each of the topics using sparse dictionary learning technique. Then for test image we infer locally which topic it corresponds to and then we use the corresponding learned dual dictionary to generate HR image. Experimental results show the effectiveness of our method over existing state-of-art methods.KeywordsSparse RepresentationSparse CodeSuper ResolutionProbabilistic Latent Semantic AnalysisBicubic InterpolationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we demonstrate simple algorithms that project low resolution (LR) images differing in subpixel shifts on a high resolution (HR) also called super resolution (SR) grid. The algorithms are very effective in accuracy as well as time efficiency. A number of spatial interpolation techniques using nearest neighbor, inverse-distance weighted averages, Radial Basis Functions (RBF) etc. are used in projection. For best accuracy of reconstructing SR image by a factor of two requires four LR images differing in four independent subpixel shifts. The algorithm has two steps: i) registration of low resolution images and (ii) shifting the low resolution images to align with reference image and projecting them on high resolution grid based on the shifts of each low resolution image using different interpolation techniques. Experiments are conducted by simulating low resolution images by subpixel shifts and subsampling of original high resolution image and the reconstructing the high resolution images from the simulated low resolution images. The results of accuracy of reconstruction are compared by using mean squared error measure between original high resolution image and reconstructed image. The algorithm was tested on remote sensing images and found to outperform previously proposed techniques such as Iterative Back Projection algorithm (IBP), Maximum Likelihood (ML) algorithms. The algorithms are robust and are not overly sensitive to the registration inaccuracies.

Although the majority of existing License Plate (LP) recognition techniques have significant improvements in accuracy, they are still limited to ideal situations in which training data is correctly annotated with restricted scenarios. Moreover, images or videos are frequently used in monitoring systems that have Low Resolution (LR) quality. In this work, the problem of LP detection in digital images is addressed in the images of a naturalistic environment. Single-stage character segmentation and recognition are combined with adversarial Super-Resolution (SR) approaches to improve the quality of the LP by processing the LR images into High-Resolution (HR) images. This work proposes effective changes to the SRGAN network regarding the number of layers, an activation function, and the appropriate loss regularization using Total Variation (TV) loss. The main paper contribution can be summarized into presenting an end-to-end deep learning framework based on generative adversarial networks (GAN), which is able to generate realistic super-resolution images. Also, proposed adding a TV regularization to the loss function to help the model enhance the resolution of images. The proposed SRGAN can handle tiny <inline-formula> <tex-math notation="LaTeX">$72\times 72$ </tex-math></inline-formula> images of LPs. The paper explores how SRGAN performed over different datasets from many aspects, such as visual analysis, PSNR, SSIM, and Optical Character Recognition (OCR). The experiments demonstrate that the suggested SRGAN can generate high-resolution images that improve the accuracy of the license plate recognition stage compared to other systems.

Multiframe super-resolution is the problem of reconstructing a single high-resolution (HR) image from several low-resolution (LR) versions of it. We assume that the original HR image undergoes different linear transforms, where each transform can be approximated as a set of linear shift-invariant transforms over different subregions of the HR image. The linearly transformed versions of the HR image are then downsampled, resulting in different LR images. Under the assumption of linearity, these LR images can form a basis that spans the set of the polyphase components (PPCs) of the HR image. We propose sampling rate diversity, where a secondary LR image, acquired by a secondary sensor of different (lower) sampling rate, is used as a reference to make known portions (subpolyphase components) of the PPCs of the reconstructed HR image. This setup allows for non-parametric reconstruction of the PPCs, where no knowledge of the underlying transforms is required, by solving for the expansion coefficients of the PPCs, in terms of the LR basis.

The deep learning models for the Single Image Super-Resolution (SISR) task have found success in recent years. However, one of the prime limitations of existing deep learning-based SISR approaches is that they need supervised training. Specifically, the Low-Resolution (LR) images are obtained through known degradation (for instance, bicubic downsampling) from the High-Resolution (HR) images to provide supervised data as an LR-HR pair. Such training results in a domain shift of learnt models when real-world data is provided with multiple degradation factors not present in the training set. To address this challenge, we propose an unsupervised approach for the SISR task using Generative Adversarial Network (GAN), which we refer to hereafter as DUS-GAN. The novel design of the proposed method accomplishes the SR task without degradation estimation of real-world LR data. In addition, a new human perception-based quality assessment loss, i.e., Mean Opinion Score (MOS), has also been introduced to boost the perceptual quality of SR results. The pertinence of the proposed method is validated with numerous experiments on different reference-based (i.e., NTIRE Real-world SR Challenge validation dataset) and no-reference based (i.e., NTIRE Real-world SR Challenge Track-1 and Track-2) testing datasets. The experimental analysis demonstrates committed improvement from the proposed method over the other state-of-the-art unsupervised SR approaches, both in terms of subjective and quantitative evaluations on different reference metrics (i.e., LPIPS, PI-RMSE graph) and no-reference quality measures such as NIQE, BRISQUE and PIQE. We also provide the implementation of the proposed approach (https://github.com/kalpeshjp89/DUSGAN) to support reproducible research.

Vehicle Classification Method Based on Single-Point Magnetic Sensor

Modern face recognition systems extract face representations using deep neural networks (DNNs) and give excellent identification and verification results, when tested on high resolution (HR) images. However, the performance of such an algorithm degrades significantly for low resolution (LR) images. A straight forward solution could be to train a DNN, using simultaneously, high and low resolution face images. This approach yields a definite improvement at lower resolutions but suffers a performance degradation for high resolution images. To overcome this shortcoming, we propose to train a network using both HR and LR images under the guidance of a fixed network, pretrained on HR face images. The guidance is provided by minimising the KL-divergence between the output Softmax probabilities of the pretrained (i.e., Teacher) and trainable (i.e., Student) network as well as by sharing the Softmax weights between the two networks. The resulting solution is tested on down-sampled images from FaceScrub and MegaFace datasets and shows a consistent performance improvement across various resolutions. We also tested our proposed solution on standard LR benchmarks such as TinyFace and SCFace. Our algorithm consistently outperforms the state-of-the-art methods on these datasets, confirming the effectiveness and merits of the proposed method.

This chapter concentrates the problem of recovery a high-resolution (HR) image from a single low-resolution input image. Recent research proposed to deal with the image super-resolution problem with sparse coding, which is based on the well reconstruction of any local image patch by a sparse linear combination of an appropriately chosen over-complete dictionary. Therein the chosen LR (Low-resolution) and HR (High-resolution) dictionaries have to be exactly corresponding for well reconstructing the local image patterns. However, the conventional sparse coding based image super-resolution usually achieves a global dictionary D=[D l ; D h ] by jointly training the concatenated LR and HR local image patches, and then reconstruct the LR and HR image as a linear combination of the separated D l and D h . This strategy only can achieve the global minimum reconstructing error of LR and HR local patches, and usually cannot obtain the exactly corresponding LR and HR dictionaries. In addition, the accurate coefficients for reconstructing the HR image patch using HR dictionary D h are also unable to be estimated using only the LR input and the LR dictionary D l . Therefore, this paper proposes to firstly learn the HR dictionary D h from the features of the training HR local patches, and then propagates the HR dictionary to the LR one, called as HR2LR dictionary propagation, by mathematical proving and statistical analysis. The effectiveness of the proposed HR2LR dictionary propagation in sparse coding for super-resolution is demonstrated by comparison with the conventional super-resolution approaches such as sparse coding and interpolation.