Multi-Scale Detail Enhancement Network for Image Super-Resolution

In this study, we investigated how shapes are classified based on local and global features by four representative convolutional neural networks (CNNs), i.e., AlexNet, VGG, ResNet and Inception. While the local features are based on simple components, such as orientation of line segment, the global features are based on the whole object, such as whether an object has a hole. For example, solid triangles and solid squares are differentiated by local features, solid circles and rings are differentiated by a global feature. Two sets of experiments were performed in this research. In the first experiment, we examined how the four CNNs pre-trained on ImageNet (with transfer learning) learned to differentiate the regular shapes (equilateral-triangles, squares, circles and rings). Our results showed that the pre-trained CNNs exhibited faster learning rates in the tasks discriminating the local features than in the tasks discriminating the global feature. However, the transfer learning of discriminating the global feature in regular shapes were better generalized to irregular shapes than the transfer learning of discriminating local features. In the second experiment, the CNNs were trained from scratch (with random weights initialization) to discriminate local and global features in regular and irregular shapes. Different from the transfer learning, the CNNs exhibited faster learning rates in discriminating the global feature than the local features. Similar to transfer learning, the CNNs exhibited excellent learning generalization to discriminating the global feature of irregular shapes, but poor learning generalization to discriminating the local features in the irregular shapes. The overarching goal of this research is to create a paradigm and benchmark to directly compare how the CNNs and primate visual systems process geometrical invariants. In contrast to the ImageNet approach which employs natural images to train CNNs, we employed the “ShapeNet” approach which features geometrical shapes with well-defined properties. The ShapeNet approach will not only help elucidate the strengths and limitations of CNN computation, but also provide insights into visual information processing in the primates.

Single image super-resolution using multi-scale feature enhancement attention residual network

Recent studies have shown that the use of deep convolutional neural networks (CNNs) can improve the performance of single image super-resolution reconstruction (SISR) methods. However, the existing CNN-based SISR model ignores the multi-scale features and shallow and deep features of the image, resulting in relatively low image reconstruction performance. To address these issues, this paper proposes a new multi-scale fractal residual network (MSFRN) for image super-resolution. On the basis of residual learning, a multi-scale fractal residual block (MSFRB) is designed. This block uses convolution kernels of different sizes to extract image multi-scale features and uses multiple paths to extract and fuse image features of different depths. Then, the shallow features extracted at the shallow feature extraction stage and the local features output by all MSFRBs are used to perform global hierarchical feature fusion. Finally, through sub-pixel convolution, the fused global features are used to reconstruct high-resolution images from low-resolution images. The experimental results on the five standard benchmark datasets show that MSFRN improved subjective visual effects and objective image quality evaluation indicators, and is superior to other state-of-the-art SISR methods.

Single image super-resolution (SISR) with deep convolutional neural networks has recently attracted increasing attention due to its potentials to generate rich details. To obtain better fidelity and visual quality, most of existing methods are of heavy design with the depth of network. However, the lack of high-frequency information in the deep network and the limit of non-local context relations of pixels remain the core issue. In addition, there are many arbitrary-scale image super-resolution tasks, so it is a focus that provides an effective and efficient model for super resolution with arbitrary-scale. To this end, we propose a Feature Preserving and Enhancing Network (FPEN) based on implicit representation, which is aims to preserving high-frequency information in deep network and enhancing non-local contextual features of pixels to output more realistic images at arbitrary scales. In particular, our proposed High Frequency Preserving Block (HFPB) can divide the features into high-frequency components and low-frequency components, and allocate more operations to high-frequency components to ensure that high-frequency components can be preserved in the network. Since high-frequency information contains the details and texture of the image, it can restore the finer details of the image. Moreover, Pixel Continuity Attention (PiCA) module our proposed utilizes visual cues observed from pixels to adaptively recalibrate the pixel range that needs attention in the image to better achieve feature enhancement and generate smoother images. Extensive experiments conducted on benchmark SISR models and datasets show that Feature Preserving and Enhancing Network can be employed for various SISR tasks with arbitrary-scale to obtain the better visual quality than other state-of-the-art SR algorithms.

After continuous development, many researchers are exploring how to better utilize global and local information in single image super-resolution (SISR). Various methods based on convolutional neural network (CNN) and Transformer structures have emerged, but few studies have mentioned how to combine these two parts of information. We study the use of self-attention mechanism to integrate local and global information, aiming to make the model better balance the weights of the two parts of information. At the same time, in order to avoid the huge amount of computation brought by Transformer, we use the selective state space model VMamba to extract global information to achieve the effect of reducing computational complexity and lightweight network. Based on the above situation, we propose a High-frequency Mamba Network (HFMN) for SISR, which includes the local high-frequency extraction module Local High-Frequency Feature Block (LHFB), the global feature extraction module Mamba-Based Attention Block (MAB) based on VMamba, and the dual attention fusion module Dual-information Interactive Attention Block (DIAB). It can better incorporate local and global information and has linear complexity in the global feature extraction branch. Experiments on multiple benchmark datasets demonstrate that the network outforms recent SOTA methods in SISR while using fewer parameters. All codes are available at https://github.com/taoWuuu/HFMN .

Recently deep convolutional neural network (CNN) has made significant achievement in Single Image Super-Resolution (SISR). Most CNN-based SISR methods used the default L2 norm of the error. However, for Hyperspectral Image (HSI), this loss function may bring spectral inconsistencies. The main reason is that most methods did not pay much attention to spectral loss. To HSI, the loss function should capture not only spatial information but also spectral consistency. In this paper, a Multi-Losses Function Network (MLFN) simultaneously considering spatial and spectral information is proposed, and is composed of two parts: one is Concatenate Dense Residual Network (CDRN), and the other is Loss Network (LN). CDRN is an image reconstruction network which can utilize the hierarchical features extracted from the low-resolution image. LN includes pixel-wise spatial loss and spectral loss which drive the learning of the entire reconstruction model. The experimental results prove that the proposed MLFN can enhance spatial resolution with the consistency of the spectrum of HSI preserved.

Deep convolutional neural networks (CNNs), trained on corresponding pairs of high- and low-resolution images, achieve state-of-the-art performance in single-image super- resolution and surpass previous signal-processing based approaches. However, their performance is limited when applied to real photographs. The reason lies in their train- ing data: low-resolution (LR) images are obtained by bicu- bic interpolation of the corresponding high-resolution (HR) images. The applied convolution kernel significantly differs from real-world camera-blur. Consequently, while current CNNs well super-resolve bicubic-downsampled LR images, they often fail on camera-captured LR images. To improve generalization and robustness of deep super- resolution CNNs on real photographs, we present a ker- nel modeling super-resolution network (KMSR) that incor- porates blur-kernel modeling in the training. Our pro- posed KMSR consists of two stages: we first build a pool of realistic blur-kernels with a generative adversarial net- work (GAN) and then we train a super-resolution network with HR and corresponding LR images constructed with the generated kernels. Our extensive experimental vali- dations demonstrate the effectiveness of our single-image super-resolution approach on photographs with unknown blur-kernels.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

Recently, the deep convolutional neural network (CNN) has made remarkable progress in single image super resolution(SISR). However, blindly using the residual structure and dense structure to extract features from LR images, can cause the network to be bloated and difficult to train. To address these problems, we propose a simple and efficient distilling with residual network(DRN) for SISR. In detail, we propose residual distilling block(RDB) containing two branches, while one branch performs a residual operation and the other branch distills effective information. To further improve efficiency, we design residual distilling group(RDG) by stacking some RDBs and one long skip connection, which can effectively extract local features and fuse them with global features. These efficient features beneficially contribute to image reconstruction. Experiments on benchmark datasets demonstrate that our DRN is superior to the state-of-the-art methods, specifically has a better trade-off between performance and model size.

Ultra-high-definition display technology is widely used in broadcasting, but there is a huge contradiction between its ultra-high-resolution content and short storage. Super-Resolution (SR) can effectively alleviate this contradiction. Recently, State-of-the-art image SR approaches leveraging Deep Convolutional Neural Networks (DCNNs) have demonstrated high-quality reconstruction performance. However, most of them suffer from large model parameters, which restricts their practical application. Besides, image SR for large scaling factors (e.g., ×8) is a tricky issue when the parameters diminish. To remedy these issues, we propose the Lightweight Multi-scale Aggregation Network (LMAN) for the image SR, which works well for both small and large scaling factors with limited parameters. Specifically, we propose a Group-wise Multi-scale Block (GMB) in which a group convolution is exploited for extracting and fusing multi-scale features before a channel attention layer to obtain discriminative features. Additionally, we present a novel Hierarchical Spatial Attention (HSA) mechanism to jointly and adaptively fuse local and global hierarchical features for high-resolution image reconstruction. Extensive experiments illustrate that our LMAN achieves superior performance against state-of-the-art methods with similar parameters and in particular for large scaling factors such as 4× and 8×.

Recent studies on video super-resolution (SR) has shown that convolutional neural network (CNN) combined with motion compensation (MC) is able to merge information from multiple low-resolution (LR) frames to generate high quality images. However, Most SR and MC modules based on deep CNN simply increase the depth of the network, which cannot make full use of hierarchical and multi-scale features, thereby achieving relatively low performance. To address the above problem, a novel multi-scale residual dense Network (MSRDN) is proposed in this paper. A multi-scale residual density block (MSRDB) is first designed to extract abundant local features through dense convolution layer, which helps to adaptively detect image features of different scales with convolution kernels of different scales. Then, we redesign SR module and MC module with MSRDB, which adaptively learns more effective features from local features and uses global feature fusion to jointly and adaptively learn global hierarchical features. Comparative results on Vid4 dataset demonstrate that MSRDB can make more full use of feature information, which helps to effectively improve the reconstruction performance of video SR.

MSAR-Net: Multi-scale attention based light-weight image super-resolution

An efficient lightweight network for single image super-resolution

In the harsh underground environment, the transmission gear in the rocker arm of the shearer is susceptible to impact load, frequently leading to failures that result in economic losses or casualties. Therefore, real-time and accurate gear fault diagnosis is crucial. Existing diagnostic models struggle with insufficient feature extraction and low diagnostic accuracy when addressing nonlinear, nonsmooth signals with strong noise in this environment. Therefore, an innovative parallel integration model of Swin Transformer and multiscale convolution is proposed to extract both global and local features under different fault modes and noise conditions. A spatial interaction block (SIB) is introduced into the Swin Transformer branch to enhance global information acquisition. In multiscale convolutional neural network (CNN) branches, depthwise separable convolutions with different expansion coefficients are utilized for multiscale extraction of local features to reduce computational complexity. Considering the differences between global and local features, a feature interaction module (FIM) is added between the two branches to realize the interaction between local and global features, considerably improving the model diagnosis performance. Furthermore, to meet the high real-time diagnosis requirements, a cloud-edge collaboration (CEC) framework is designed to realize the timely alarm of gear faults through the efficient collaboration between edge nodes and the industrial cloud platform. Using the vibration data collected from the rocker arm loading test bed of Taizhong Coal Machinery and the gear data provided by Southeast University for validation, the results show that the diagnostic precision of the proposed method can reach up to 99.55%. Even when the signal-to-noise ratio (SNR) is −2 db, the accuracy is still 93.67%, which is superior to other advanced comparison models, and significantly reduces the completion time of fault diagnosis tasks, and has strong anti-noise and real-time diagnosis capabilities.

With the availability of commercial Light Field (LF) cameras, LF imaging has emerged as an up-and-coming technology in computational photography. However, the spatial resolution is significantly constrained in commercial micro-lens-based LF cameras because of the inherent multiplexing of spatial and angular information. Therefore, it becomes the main bottleneck for other applications of light field cameras. This paper proposes an adaptation module in a pre-trained Single Image Super-Resolution (SISR) network to leverage the powerful SISR model instead of using highly engineered light field imaging domain-specific Super Resolution models. The adaption module consists of a Sub-aperture Shift block and a fusion block. It is an adaptation in the SISR network to further exploit the spatial and angular information in LF images to improve the super-resolution performance. Experimental validation shows that the proposed method outperforms existing light field super-resolution algorithms. It also achieves PSNR gains of more than 1 dB across all the datasets as compared to the same pre-trained SISR models for scale factor 2, and PSNR gains 0.6 − 1 dB for scale factor 4.