Compressed image restoration via deep deblocker driven unified framework

Recent years have witnessed growing interest in machine learning-based models and techniques for low-dose X-ray CT (LDCT) imaging tasks. The methods can typically be categorized into supervised learning methods and unsupervised or model-based learning methods. Supervised learning methods have recently shown success in image restoration tasks. However, they often rely on large training sets. Model-based learning methods such as dictionary or transform learning do not require large or paired training sets and often have good generalization properties, since they learn general properties of CT image sets. Recent works have shown the promising reconstruction performance of methods such as PWLS-ULTRA that rely on clustering the underlying (reconstructed) image patches into a learned union of transforms. In this paper, we propose a new Supervised-UnsuPERvised (SUPER) reconstruction framework for LDCT image reconstruction that combines the benefits of supervised learning methods and (unsupervised) transform learning-based methods such as PWLS-ULTRA that involve highly image-adaptive clustering. The SUPER model consists of several layers, each of which includes a deep network learned in a supervised manner and an unsupervised iterative method that involves image-adaptive components. The SUPER reconstruction algorithms are learned in a greedy manner from training data. The proposed SUPER learning methods dramatically outperform both the constituent supervised learning-based networks and iterative algorithms for LDCT, and use much fewer iterations in the iterative reconstruction modules.

When a facial image is blurred, it significantly affects high-level vision tasks such as face recognition. The purpose of facial image deblurring is to recover a clear image from a blurry input image, which can improve the recognition accuracy, etc. However, general deblurring methods do not perform well on facial images. Therefore, some face deblurring methods have been proposed to improve performance by adding semantic or structural information as specific priors according to the characteristics of the facial images. In this paper, we survey and summarize recently published methods for facial image deblurring, most of which are based on deep learning. First, we provide a brief introduction to the modeling of image blurring. Next, we summarize face deblurring methods into two categories: model-based methods and deep learning-based methods. Furthermore, we summarize the datasets, loss functions, and performance evaluation metrics commonly used in the neural network training process. We show the performance of classical methods on these datasets and metrics and provide a brief discussion on the differences between model-based and learning-based methods. Finally, we discuss the current challenges and possible future research directions.

Learning-based methods have been proved to perform well in a variety of areas in the biomedical field, such as biomedical image segmentation, and histopathological image analysis. Deep learning, as the most recently presented approach of learning-based methods, has attracted more and more attention. For instance, massive researches of deep learning methods for image reconstructions of computed tomography (CT) and magnetic resonance imaging (MRI) have been reported, indicating the great potential of deep learning for inverse problems. Optical technology-related medical imaging modalities including diffuse optical tomography (DOT), fluorescence molecular tomography (FMT), bioluminescence tomography (BLT), and photoacoustic tomography (PAT) are also dramatically innovated by introducing learning-based methods, in particular deep learning methods, to obtain better reconstruction results. This review depicts the latest researches on learning-based optical tomography of DOT, FMT, BLT, and PAT. According to the most recent studies, learning-based methods applied in the field of optical tomography are categorized as kernel-based methods and deep learning methods. In this review, the former are regarded as a sort of conventional learning-based methods and the latter are subdivided into model-based methods, post-processing methods, and end-to-end methods. Algorithm as well as data acquisition strategy are discussed in this review. The evaluations of these methods are summarized to illustrate the performance of deep learning-based reconstruction.

Image restoration (IR) methods based on neural network algorithms have shown great success. However, the hardware circuits that can perform real-time IR task with high-effective analog computation are few in the literature. To address such problem, we propose a memristor-based continuous Hopfield neural network (HNN) circuit for processing the IR task in this work. In our circuit, a single memristor crossbar array is used to represent synaptic weights and perform matrix operations. Current feedback operation amplifiers are utilized to achieve integral operation and output function. Given these designs, the proposed circuit can perform continuous recursive operations in parallel and process different optimization problems with the programmability of the memristor array. On the basis of the proposed circuit, binary and greyscale image restorations are conducted through self-organizing network operations, providing a hardware implementation platform for IR tasks. Comparative simulations show the designed HNN circuit provides effective improvements in terms of speed and accuracy compared with software simulation. Moreover, the hardware circuit shows good robustness to memristive variation and input noise.

Focusing on features with higher importance degree: A significance-wise mechanism for image restoration

As a long-standing task, image restoration aims to recover the latent sharp image from its degraded counterpart. In recent years, owing to the strong ability of self-attention in capturing long-range dependencies, Transformer based methods have achieved promising performance on multifarious image restoration tasks. However, the canonical self-attention leads to quadratic complexity with respect to input size, hindering its further applications in image restoration. In this paper, we propose a Strip Attention Network (SANet) for image restoration to integrate information in a more efficient and effective manner. Specifically, a strip attention unit is proposed to harvest the contextual information for each pixel from its adjacent pixels in the same row or column. By employing this operation in different directions, each location can perceive information from an expanded region. Furthermore, we apply various receptive fields in different feature groups to enhance representation learning. Incorporating these designs into a U-shaped backbone, our SANet performs favorably against state-of-the-art algorithms on several image restoration tasks. The code is available at https://github.com/c-yn/SANet.

Recent works that utilized deep models have achieved superior results in various image restoration (IR) applications. Such approach is typically supervised, which requires a corpus of training images with distributions similar to the images to be recovered. On the other hand, the shallow methods, which are usually unsupervised remain promising performance in many inverse problems, e.g., image deblurring and image compressive sensing (CS), as they can effectively leverage nonlocal self-similarity priors of natural images. However, most of such methods are patch-based leading to the restored images with various artifacts due to naive patch aggregation in addition to the slow speed. Using either approach alone usually limits performance and generalizability in IR tasks. In this paper, we propose a joint low-rank and deep (LRD) image model, which contains a pair of triply complementary priors, namely, internal and external, shallow and deep, and non-local and local priors. We then propose a novel hybrid plug-and-play (H-PnP) framework based on the LRD model for IR. Following this, a simple yet effective algorithm is developed to solve the proposed H-PnP based IR problems. Extensive experimental results on several representative IR tasks, including image deblurring, image CS and image deblocking, demonstrate that the proposed H-PnP algorithm achieves favorable performance compared to many popular or state-of-the-art IR methods in terms of both objective and visual perception.

Image restoration aims to reconstruct a high-quality image from a degraded low-quality observation. Recently, Transformer models have achieved promising performance on image restoration tasks due to their powerful ability to model long-range dependencies. However, the quadratically growing complexity with respect to the input size makes them inapplicable to practical applications. In this paper, we develop an efficient convolutional network for image restoration by enhancing multi-scale representation learning. To this end, we propose an omni-kernel module that consists of three branches, i.e., global, large, and local branches, to learn global-to-local feature representations efficiently. Specifically, the global branch achieves a global perceptive field via the dual-domain channel attention and frequency-gated mechanism. Furthermore, to provide multi-grained receptive fields, the large branch is formulated via different shapes of depth-wise convolutions with unusually large kernel sizes. Moreover, we complement local information using a point-wise depth-wise convolution. Finally, the proposed network, dubbed OKNet, is established by inserting the omni-kernel module into the bottleneck position for efficiency. Extensive experiments demonstrate that our network achieves state-of-the-art performance on 11 benchmark datasets for three representative image restoration tasks, including image dehazing, image desnowing, and image defocus deblurring. The code is available at https://github.com/c-yn/OKNet.

Head pose estimation methods can be broadly classified into learning-based methods and model-based methods. The learning based methods use machine learning techniques to directly predict the pose from image appearance, while the model-based methods link the 2D observation (e.g. facial landmarks) and 3D model through the projection model for pose estimation. However, both methods may have difficulty on images with very low quality (e.g. low resolution, occlusion, and noisy images). For example, there would be limited appearance information to generate accurate landmark detection on lowquality images for reliable face pose estimation. To tackle pose estimation on low-quality images, we propose to combine the learning and model based methods. Specifically, we first build the relationship between facial landmark locations and image appearance using the Restricted Boltzmann Machine (RBM) model. Then, we link the landmark locations and 3D model analytically using the projection model. By combining the RBM model with the projection model, without explicit landmark detection, we predict the head pose with a KL-divergence based method and a gradient-based method. Experimental results demonstrate the effectiveness of the proposed method.

In image restoration tasks, like denoising and super resolution, continual modulation of restoration levels is of great importance for real-world applications, but has failed most of existing deep learning based image restoration methods. Learning from discrete and fixed restoration levels, deep models cannot be easily generalized to data of continuous and unseen levels. This topic is rarely touched in literature, due to the difficulty of modulating well-trained models with certain hyper-parameters. We make a step forward by proposing a unified CNN framework that consists of few additional parameters than a single-level model yet could handle arbitrary restoration levels between a start and an end level. The additional module, namely AdaFM layer, performs channel-wise feature modification, and can adapt a model to another restoration level with high accuracy. By simply tweaking an interpolation coefficient, the intermediate model - AdaFM-Net could generate smooth and continuous restoration effects without artifacts. Extensive experiments on three image restoration tasks demonstrate the effectiveness of both model training and modulation testing. Besides, we carefully investigate the properties of AdaFM layers, providing a detailed guidance on the usage of the proposed method.

Image restoration tasks such as deraining, deblurring, and dehazing are crucial for enhancing the environmental perception of autonomous vehicles and traffic systems, particularly for tasks like vehicle detection, pedestrian detection and lane line identification. While transformer-based models excel in these tasks, their prohibitive computational complexity hinders real-world deployment on resource-constrained platforms. To bridge this gap, this paper introduces a novel Soft Knowledge Distillation (SKD) framework, designed specifically for creating highly efficient yet powerful image restoration models. Our core innovation is twofold: first, we propose a Multi-dimensional Cross-Net Attention(MCA) mechanism that allows a compact student model to learn comprehensive attention relationships from a large teacher model across both spatial and channel dimensions, capturing fine-grained details essential for high-quality restoration. Second, we pioneer the use of a contrastive learning loss at the reconstruction level, treating the teacher’s outputs as positives and the degraded inputs as negatives, which significantly elevates the student’s reconstruction quality. Extensive experiments demonstrate that our method achieves a superior trade-off between performance and efficiency, notably enhancing downstream tasks like object detection. The primary contributions of this work lie in delivering a practical and compelling solution for real-time perceptual enhancement in autonomous systems, pushing the boundaries of efficient model design.

Multiple isotope simultaneous acquisition SPECT imaging can provide spatially and temporally registered images with just one image acquisition. We have previously demonstrated that, with model-based crosstalk compensation, simultaneously acquired <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">99m</sup> Tc/ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">201</sup> TI dual-isotope cardiac imaging can provide images with quality and quantitative accuracy that is comparable with that from separate acquisition. Given the capability of modern SPECT systems to acquire projections in multiple energy windows, it is potentially possible to simultaneously assess rest and stress perfusion status with <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">201</sup> TI and a <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">99m</sup> Tc labeled agent, and innervation status with <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">123</sup> I-MIBG. However, because these three isotopes have very close emission energies and energy windows, crosstalk contamination can severely degrade the quality of simultaneously acquired images. In this work we developed a model-based method for estimating and compensating for the crosstalk and downscatter contamination in simultaneously acquired triple-isotope images. The crosstalk model includes the effects of the photon interactions inside the object and collimator-detector system. This crosstalk model was incorporated into the forward projection step of an iterative reconstruction algorithm to provide crosstalk compensation. We evaluated the proposed model-based method using Monte Carlo simulated simultaneous triple-isotope projection data. Results showed that there was very good agreement between the model-based estimates and the MC simulated projection data in each of the isotopes' energy windows. After compensation, the crosstalk in the simultaneously acquired images was effectively removed. In conclusion, we showed that it is possible to simultaneously acquire the cardiac SPECT imaging with <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">99m</sup> Tc/ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">201</sup> Tl/ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">123</sup> I labeled multiple tracers. Using the proposed model-based crosstalk compensation method, the qualities of the simultaneously acquired triple-isotope images were comparable with the single acquisitions.

Underwater laser imaging is of great significance in underwater search and marine science, <i>etc</i>. However, traditional underwater laser imaging is often of poor quality with noises and blurs, moreover, the resolution of the image is also low. In order to obtain clear underwater images with high resolution and quality, here, we have designed a range gated imaging underwater imaging system and realized an image restoration approach. In this paper, based on the introduction to the imaging system and image restoration algorithm, the experiment is established by setting the imaging system under water in the lake to capture the underwater targets. With the proposed underwater image restoration approach, images of high quality could be retrieved which proves that the method is able to identify the target ~10 meters away underwater.

In this paper, we explore the role of Instance Normalization in low-level vision tasks. Specifically, we present a novel block: Half Instance Normalization Block (HIN Block), to boost the performance of image restoration networks. Based on HIN Block, we design a simple and powerful multi-stage network named HINet, which consists of two subnetworks. With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8× and 2.9× speedup respectively. For image deblurring, we get comparable performance with 22.5% of its MACs and 3.3× speedup on REDS and GoPro datasets. For image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4× speedup. With HINet, we won the 1st place on the NTIRE 2021 Image Deblurring Challenge - Track2. JPEG Artifacts, with a PSNR of 29.70.

This paper introduces a matrix-free variant of the Davidon-Fletcher-Powell (DFP) method for unconstrained optimization problems with applications in compressive sensing and image restoration. The main contribution lies in the new search direction incorporating a scaling parameter that ensures the satisfaction of the sufficient descent condition, independent of the line search conditions. A rigorous convergence analysis guarantees the boundedness and theoretical validity of the proposed method. Comprehensive numerical experiments on benchmark unconstrained optimization test problems and compressive sensing problems demonstrate the efficiency and robustness of the algorithm. Specifically, in image restoration tasks, our method outperforms CG-DESCENT, MDL, and NSMA, achieving a 100% success rate compared to 95.8%, 84.5%, and 53.5%, respectively. Additionally, results on computational time, relative error, and PSNR confirm the superior performance of the proposed approach. These findings establish the proposed method as a competitive alternative for large-scale optimization problems.