Characteristic Regularisation for Super-Resolving Face Images

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

Contemporary domain generalization (DG) and multisource unsupervised domain adaptation (UDA) methods mostly collect data from multiple domains together for joint optimization. However, this centralized training paradigm poses a threat to data privacy and is not applicable when data are non-shared across domains. In this work, we propose a new approach called Collaborative Optimization and Aggregation (COPA), which aims at optimizing a generalized target model for decentralized DG and UDA, where data from different domains are non-shared and private. Our base model consists of a domain-invariant feature extractor and an ensemble of domain-specific classifiers. In an iterative learning process, we optimize a local model for each domain, and then centrally aggregate local feature extractors and assemble domain-specific classifiers to construct a generalized global model, without sharing data from different domains. To improve generalization of feature extractors, we employ hybrid batch-instance normalization and collaboration of frozen classifiers. For better decentralized UDA, we further introduce a prediction agreement mechanism to overcome local disparities towards central model aggregation. Extensive experiments on five DG and UDA benchmark datasets show that COPA is capable of achieving comparable performance against the state-of-the-art DG and UDA methods without the need for centralized data collection in model training.

Accurate evaluation of orchard areas from remote sensing images is of great importance in economic and ecological aspects. In practice, the differences in distributions between remote sensing images and the lack of data labels make the semantic segmentation model impossible to use in new data. Unsupervised domain adaptation (UDA) methods can improve the performance of the model in the target domain by aligning the source domain and the target domain. However, due to the class mismatch problem and the interference of high-dimensional feature complexity, most UDA methods cannot achieve satisfactory results in orchard areas segmentation task. To address these issues, we propose an UDA model for orchard areas segmentation by developing a class feature aggregate discriminator. The class feature aggregate discriminator is designed to distinguish intra-domain classes and align inter-domain classes, and class feature aggregate can represent class information of different domains, which helps the model to avoid the interference of complex information. In addition, adversarial loss reweighting is introduced to the novel model, which makes the segmentation model pay more attention to the orchard areas. To verify the effectiveness of the proposed method, we conducted extensive experiments in three different remote sensing images around Yichang City. Compared to the baseline model, the proposed approach improves IoU by 27.68%, and we achieve high gains of 6.07% in IoU over other UDA methods. The larger gain indicates that our proposed method has great potential in cross-domain orchard areas segmentation.

Recently, imaging systems have exhibited remarkable image restoration performance through optimized optical systems and deep-learning-based models. Despite advancements in optical systems and models, severe performance degradation occurs when the predefined optical blur kernel differs from the actual kernel while restoring and upscaling the images. This is because super-resolution (SR) models assume that a blur kernel is predefined and known. To address this problem, various lenses could be stacked, and the SR model could be trained with all available optical blur kernels. However, infinite optical blur kernels exist in reality; thus, this task requires the complexity of the lens, substantial model training time, and hardware overhead. To resolve this issue by focusing on the SR models, we propose a kernel-attentive weight modulation memory network by adaptively modulating SR weights according to the shape of the optical blur kernel. The modulation layers are incorporated into the SR architecture and dynamically modulate the weights according to the blur level. Extensive experiments reveal that the proposed method improves peak signal-to-noise ratio performance, with an average gain of 0.83 dB for blurred and downsampled images. An experiment with a real-world blur dataset demonstrates that the proposed method can handle real-world scenarios.

Lightweight image super-resolution with sliding Proxy Attention Network

Purpose To determine whether the unsupervised domain adaptation (UDA) method with generated images improves the performance of a supervised learning (SL) model for prostate cancer (PCa) detection using multisite biparametric (bp) MRI datasets. Materials and Methods This retrospective study included data from 5150 patients (14 191 samples) collected across nine different imaging centers. A novel UDA method using a unified generative model was developed for PCa detection using multisite bpMRI datasets. This method translates diffusion-weighted imaging (DWI) acquisitions, including apparent diffusion coefficient (ADC) and individual diffusion-weighted (DW) images acquired using various b values, to align with the style of images acquired using b values recommended by Prostate Imaging Reporting and Data System (PI-RADS) guidelines. The generated ADC and DW images replace the original images for PCa detection. An independent set of 1692 test cases (2393 samples) was used for evaluation. The area under the receiver operating characteristic curve (AUC) was used as the primary metric, and statistical analysis was performed via bootstrapping. Results For all test cases, the AUC values for baseline SL and UDA methods were 0.73 and 0.79 (P < .001), respectively, for PCa lesions with PI-RADS score of 3 or greater and 0.77 and 0.80 (P < .001) for lesions with PI-RADS scores of 4 or greater. In the 361 test cases under the most unfavorable image acquisition setting, the AUC values for baseline SL and UDA were 0.49 and 0.76 (P < .001) for lesions with PI-RADS scores of 3 or greater and 0.50 and 0.77 (P < .001) for lesions with PI-RADS scores of 4 or greater. Conclusion UDA with generated images improved the performance of SL methods in PCa lesion detection across multisite datasets with various b values, especially for images acquired with significant deviations from the PI-RADS-recommended DWI protocol (eg, with an extremely high b value). Keywords: Prostate Cancer Detection, Multisite, Unsupervised Domain Adaptation, Diffusion-weighted Imaging, b Value Supplemental material is available for this article. © RSNA, 2024.

Most existing super-resolution (SR) methods are designed to restore high resolution (HR) images from certain low resolution (LR) images with a simple degradation, e.g. bicubic downsampling. Their generalization capability to real-world degradation is limited because it often couples several degradation factors such as noise and blur. To solve this problem, existing blind SR methods rely on either explicit degradation estimation or translation to bicubicly downsampled LR images, where inaccurate estimation or translation would severely deteriorate the SR performance. In this paper, we propose a plug-and-play module, which could be applied to any existing image super-resolution model for feature-level adaptation to improve the generalization ability to real-world degraded images. Specifically, a degradation encoder is proposed to compute an implicit degradation representation with a ranking loss based on the degradation level as supervision. The degradation representation then works as a kind of condition and is applied to the existing image super-resolution model pretrained on bicubicly downsampled LR images through the proposed region-aware modulation. With the proposed method, the base super-resolution model could be fine-tuned to adapt to the condition of degradation representation for further improvement. Experimental results on both synthetic and real-world datasets show that the proposed image SR method with compact model size performs favorably against state-of-the-art methods. Our source code is publicly available at https://github.com/wangyue7777/blindsr_daa.KeywordsBlind super-resolutionMultiple unknown degradationsFeature-level adaptationRanking lossRegion-aware modulation

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

Unsupervised cross-domain person re-identification by instance and distribution alignment

With the introduction of deep learning, a significant amount of research has been conducted in the field of computer vision in the past decade. In particular, research on object detection (OD) continues to progress rapidly. However, despite these advances, some limitations need to be overcome to enable real-world applications of deep learning-based OD models. One such limitation is inaccurate OD when image quality is poor or a target object is small. The performance degradation phenomenon for small objects is similar to the fundamental limitations of an OD model, such as the constraint of the receptive field, which is a difficult problem to solve using only an OD model. Therefore, OD performance can be hindered by low image quality or small target objects. To address this issue, this study investigates the compatibility of super-resolution (SR) and OD techniques to improve detection, particularly for small objects. We analyze the combination of SR and OD models, classifying them based on architectural characteristics. The experimental results show a substantial improvement when integrating OD detectors with SR models. Overall, it was demonstrated that, when the evaluation metrics (PSNR, SSIM) of the SR models are high, the performance in OD is correspondingly high as well. Especially, evaluations on the MS COCO dataset reveal that the enhancement rate for small objects is 9.4% higher compared to all objects. This work provides an analysis of SR and OD model compatibility, demonstrating the potential benefits of their synergistic combination. The experimental code can be found on our GitHub repository.

Abstract. As deep learning models advance, their use in Super Resolution (SR) tasks has become pivotal for enhancing remote sensing low-resolution (LR) satellite images. However, the decision making processes within these models remain opaque, especially in remote sensing applications where transparency is critical. This paper focuses on applying Explainable Artificial Intelligence (XAI) techniques, particularly Local Attribution Maps (LAMs), to analyze and interpret the internal behavior of both general purpose and remote sensing specific SR neural networks. General purpose models like Generative Adversarial Network for Super Resolution (SRGAN), Enhanced Deep Super-Resolution Network (EDSR), Efficient Super-Resolution Transformer (ESRT), and Hybrid-Attention Transformer (HAT), although highly effective in SR tasks, were originally designed for broader image enhancement challenges; in contrast, Hybrid-Scale Self-Similarity Exploitation Network (HSENet) and Multi-scale Enhanced Network (MEN) are tailored for the unique complexities of remote sensing, such as varied textures and intricate scene features. By leveraging LAMs, we highlight how different networks prioritize and process features such as edges, textures, and high frequency details to generate super resolved outputs. The comparative study between general purpose and remote sensing specific networks outlines each model’s strengths and weaknesses in managing the data present in remote sensing imagery. This approach addresses the previously unexplored application of LAMs in remote sensing for SR, contributing to the research in this field and providing deeper insights into the interpretability and transparency of widely used SR models. Furthermore, by drawing parallels between feature attribution in SR and classification tasks, we suggest new pathways for integrating semantic information to refine model transparency and performance in remote sensing applications.

Rethinking unsupervised domain adaptation for semantic segmentation

State-of-the-art (SOTA) super-resolution (SR) models can generate high-quality images. However, they require a large external memory bandwidth, making it impossible to implement these models on hardware. Although some work has presented different kinds of layer fusion to reduce memory traffic, they can only work on simple model architectures and only consider the feature extraction part. To solve the above issues, this article proposes structure adaptive fusion (SAF) for the feature extraction part to avoid intermediate feature map I/O. This method selects the repetitive structure as the fusion unit and fuses multiple ones to meet buffer size and memory bandwidth constraints, which can deal with different SR models. In addition, we also propose channel-aware addressing for the upscale part to avoid off-chip data transfers. The proposed methods achieve over 90% of memory traffic reduction in all tested SOTA models. Compared to the SOTA fusion method, our approach requires a 52% smaller buffer size and up to 61% lower memory bandwidth for the same number of fusion layers.

Unsupervised deep domain adaptation algorithm for video based human activity recognition via recurrent neural networks

Cardiac segmentation on late gadolinium enhancement MRI: A benchmark study from multi-sequence cardiac MR segmentation challenge.