Depth image super-resolution based on joint sparse coding

A sparse dictionary model for image superresolution is presented, which unifies the feature patches of high-resolution (HR) and low-resolution images using sparse dictionary coding. This method builds a sparse association between middle-frequency and high-frequency image components and realizes simultaneously match searching and optimization methods. Comparison with sparse coding method shows sparse dictionary is more compact and effective. Sparse K-SVD algorithm is applied for optimization to speed up sparse coding. Some experiments with real images show that our method outperforms other learning-based super-resolution algorithms.

In this paper, we propose the convolutional sparse coding based model in the wavelet domain for the task of single image super-resolution (SISR). The conventional sparse coding based approaches work on overlapping image patches and use the dictionary atoms to sparse code an image patch. Further, at the final stage, an overlap-add mechanism is used to get the final high-resolution image estimate. However, these algorithms fail to take into account the consistency present in the overlapping patches which limits their performance. We propose the use of wavelet integrated convolutional sparse coding approach where instead of dictionary atoms we utilize the convolution summations between the learned filters and their mappings for sparse representation based SISR. The use of wavelets is proposed owing to their unique directional and compact features. A pair of filters are learned along with a mapping function for each wavelet sub-band to exploit the consistency among patches. The proposed wavelet integrated convolutional sparse coding model helps capture useful contextual information. The proposed model is evaluated on publicly available datasets for different scale-up parameters. To show the efficacy of the proposed model we compare it with recent state-of-the-art algorithms. The visual results along with the quantitative ones indicate that the proposed model performs well for the tasks of super-resolution.

We frame the task of predicting a semantic labeling as a sparse reconstruction procedure that applies a target-specific learned transfer function to a generic deep sparse code representation of an image. This strategy partitions training into two distinct stages. First, in an unsupervised manner, we learn a set of dictionaries optimized for sparse coding of image patches. These generic dictionaries minimize error with respect to representing image appearance and are independent of any particular target task. We train a multilayer representation via recursive sparse dictionary learning on pooled codes output by earlier layers. Second, we encode all training images with the generic dictionaries and learn a transfer function that optimizes reconstruction of patches extracted from annotated ground-truth given the sparse codes of their corresponding image patches. At test time, we encode a novel image using the generic dictionaries and then reconstruct using the transfer function. The output reconstruction is a semantic labeling of the test image. Applying this strategy to the task of contour detection, we demonstrate performance competitive with state-of-the-art systems. Unlike almost all prior work, our approach obviates the need for any form of hand-designed features or filters. Our model is entirely learned from image and ground-truth patches, with only patch sizes, dictionary sizes and sparsity levels, and depth of the network as chosen parameters. To illustrate the general applicability of our approach, we also show initial results on the task of semantic part labeling of human faces. The effectiveness of our data-driven approach opens new avenues for research on deep sparse representations. Our classifiers utilize this representation in a novel manner. Rather than acting on nodes in the deepest layer, they attach to nodes along a slice through multiple layers of the network in order to make predictions about local patches. Our flexible combination of a generatively learned sparse representation with discriminatively trained transfer classifiers extends the notion of sparse reconstruction to encompass arbitrary semantic labeling tasks.

Image representation using local features and sparse coding (SC) plays a very important role in image classification when the dataset is fairly large. Despite of its worldwide popularity, there are still some improving space in classification efficiency and computational investment in training and coding phrase of SC. In this paper, we put forward a novel object categories recognition method from two aspects. First, the contextual relevance between image patches are fully utilized by merging local feature of every sub-patch with its neighboring ones into strong context features to generate the multiple sparse representations, which are received by the SC and multi-scale max pooling SPM(Spatial Pyramid Matching), respectively. Second, while calculating the sparse coefficients of SC, we need to solve L1-regularized least square problem. Screening out the zero coefficients and discarding the corresponding inactive codewords before solving Lasso problem can remarkably speed up the optimization. The proposed method outperforms state-of-the-art performance in a large number of image categorization experiments on several benchmarks: the ground truth dataset (21 Land-Use database), the event dataset (UIUC-Sport dataset), and the object recognition dataset (Caltech101 dataset).

Image representation using local features and sparse coding (SC) plays a very important role in image classification when the dataset is fairly large. Despite of its worldwide popularity, there are still some improving space in classification efficiency and computational investment in training and coding phrase of SC. In this paper, we put forward a novel object categories recognition method from two aspects. First, the contextual relevance between image patches are fully utilized by merging local feature of every sub-patch with its neighboring ones into strong context features to generate the multiple sparse representations, which are received by the SC and multi-scale max pooling SPM(Spatial Pyramid Matching), respectively. Second, while calculating the sparse coefficients of SC, we need to solve L1-regularized least square problem. Screening out the zero coefficients and discarding the corresponding inactive codewords before solving Lasso problem can remarkably speed up the optimization. The proposed method outperforms state-of-the-art performancein a large number of image categorization experiments on several benchmarks: the ground truth dataset (21 Land-Use database), the event dataset (UIUC-Sport dataset), and the object recognition dataset (Caltech101 dataset).

Depth image super-resolution is a challenging problem, since normally high upscaling factors are required (e.g., 16×), and depth images are often noisy. In order to achieve large upscaling factors and resilience to noise, we propose a Robust Algorithm for Depth imAge super Resolution (RADAR) that combines the power of finite rate of innovation (FRI) theory with multimodal dictionary learning. Given a low-resolution (LR) depth image, we first model its rows and columns as piece-wise polynomials and propose an FRI-based depth upscaling (FDU) algorithm to super-resolve the image. Then, the upscaled moderate quality (MQ) depth image is further enhanced with the guidance of a registered high-resolution (HR) intensity image. This is achieved by learning multimodal mappings from the joint MQ depth and HR intensity pairs to the HR depth, through a recently proposed triple dictionary learning (TDL) algorithm. Moreover, to speed up the super-resolution process, we introduce a new projection-based rapid upscaling (PRU) technique that pre-calculates the projections from the joint MQ depth and HR intensity pairs to the HR depth. Compared with the state-of-the-art deep learning-based methods, our approach has two distinct advantages: we need a fraction of training data but can achieve the best performance, and we are resilient to mismatches between training and testing datasets. The extensive numerical results show that the proposed method outperforms other state-of-the-art methods on either noise-free or noisy datasets with large upscaling factors up to 16× and can handle unknown blurring kernels well.

Depth map super-resolution using stereo-vision-assisted model

With the rapid development of various satellite sensors, automatic and advanced scene classification technique is urgently needed to process a huge amount of satellite image data. Recently, a few of research works start to implant the sparse coding for feature learning in aerial scene classification. However, these previous research works use the single-layer sparse coding in their system and their performances are highly related with multiple low-level features, such as scale-invariant feature transform (SIFT) and saliency. Motivated by the importance of feature learning through multiple layers, we propose a new unsupervised feature learning approach for scene classification on very high resolution satellite imagery. The proposed unsupervised feature learning utilizes multipath sparse coding architecture in order to capture multiple aspects of discriminative structures within complex satellite scene images. In addition, the dense low-level features are extracted from the raw satellite data by using different image patches with varying size at different layers, and this approach is not limited to a particularly designed feature descriptors compared with the other related works. The proposed technique has been evaluated on two challenging high-resolution datasets, including the UC Merced dataset containing 21 different aerial scene categories with a 1 foot resolution and the Singapore dataset containing 5 land-use categories with a 0.5m spatial resolution. Experimental results show that it outperforms the state-of-the-art that uses the single-layer sparse coding. The major contributions of this proposed technique include (1) a new unsupervised feature learning approach to generate feature representation for very high-resolution satellite imagery, (2) the first multipath sparse coding that is used for scene classification in very high-resolution satellite imagery, (3) a simple low-level feature descriptor instead of many particularly designed low-level descriptor, such as SIFT descriptors and saliency, (4) evaluation on two satellite image datasets that come from different sensor sources.

This chapter concentrates the problem of recovery a high-resolution (HR) image from a single low-resolution input image. Recent research proposed to deal with the image super-resolution problem with sparse coding, which is based on the well reconstruction of any local image patch by a sparse linear combination of an appropriately chosen over-complete dictionary. Therein the chosen LR (Low-resolution) and HR (High-resolution) dictionaries have to be exactly corresponding for well reconstructing the local image patterns. However, the conventional sparse coding based image super-resolution usually achieves a global dictionary D=[D l ; D h ] by jointly training the concatenated LR and HR local image patches, and then reconstruct the LR and HR image as a linear combination of the separated D l and D h . This strategy only can achieve the global minimum reconstructing error of LR and HR local patches, and usually cannot obtain the exactly corresponding LR and HR dictionaries. In addition, the accurate coefficients for reconstructing the HR image patch using HR dictionary D h are also unable to be estimated using only the LR input and the LR dictionary D l . Therefore, this paper proposes to firstly learn the HR dictionary D h from the features of the training HR local patches, and then propagates the HR dictionary to the LR one, called as HR2LR dictionary propagation, by mathematical proving and statistical analysis. The effectiveness of the proposed HR2LR dictionary propagation in sparse coding for super-resolution is demonstrated by comparison with the conventional super-resolution approaches such as sparse coding and interpolation.

Many image super-resolution algorithms can be formulated in a mapping framework based on the natural image prior. Generally, the mapping function with free parameters is learned by minimizing the reconstruction mapping error. In this paper, we obtain a considerable image super-resolution algorithm which gains in accuracy and speed by combining joint mapping learning with fast approximations of sparse coding. A novel “dictionary” training method for single image super-resolution based on feed-forward neural network is proposed. The training algorithm alternates between solving sparse coding problem and learning joint mapping relation problem. The learning process enforces that the sparse coding of a low-resolution image patch can be regarded as the shared latent coordinates for reconstructing its underlying high-resolution image patch with the image high-resolution dictionary. Experiments validate that our learning method shows excellent results both quantitatively and qualitatively.

Effective representation methods and proper signal priors are crucial in most signal processing applications. In this thesis we focus on different structured models and we design appropriate schemes that allow the discovery of low dimensional latent structures that characterise and identify the signals. Motivated by the highly non-linear structure of most datasets, we firstly investigate the geometry of manifolds. Manifolds are low dimensional, non-linear structures that are naturally employed to describe sets of strongly related signals such as the images of a 3-D object captured from different viewpoints. However, the use of manifolds in applications is not straightforward due to their usually non-analytic and non-linear form. We propose here a way to `disassemble' a manifold into simpler components by approximating it with affine subspaces. Our objective is to discover a set of low dimensional affine subspaces that can represent manifold data accurately while preserving the manifold's structure. To this end, we employ a greedy technique that iteratively merges manifold samples into groups based on the difference of local tangents. We use our algorithm to approximate synthetic and real manifolds and to demonstrate that it is competitive to state-of-the-art techniques. Then, we consider structured sparse representations of signals and we propose a new sparsity model, where signals are essentially composed of a small number of structured {\it molecules }. We define the molecules to be linear combinations of a small number of atoms in a redundant dictionary. Our multi-level model takes into account the energy distribution of the significant signal components in addition to their support. It permits to define typical visual patterns and recognise them in prototypical or deformed form. We define a new structural difference measure between molecules and their deformed versions, which is based on their sparse codes and we create an algorithm for decomposing signals into molecules that can account for different deviations in the internal molecule structure. Our experiments verify the benefits of the new image model in various restoration tasks and they confirm that the development of proper models that extend the mere notion of sparsity can be very useful for various inverse problems in imaging. Finally, we investigate the problem of learning molecule representations directly in the sparse code domain. We constrain sparse codes to be linear combinations of a few, possibly deformed, molecules and we design an algorithm that can learn the structure from the codes without transforming them back into the signal domain. To this end, we take advantage of our structural difference which is based on the sparse codes and we devise a scheme for representing the codes with molecules and learn the molecules at the same time. To illustrate the effectiveness of our proposed algorithm we apply it to various synthetic and real datasets and we compare the results with traditional sparse coding and dictionary learning techniques. From the experiments, we verify the superior performance of our scheme in interpreting and recognising correctly the underlying structure. In short, in this thesis we are interested in low-dimensional, structured models. Among the various choices, we focus on manifolds and sparse representations and we propose schemes that enhance their structural properties and highlight their effectiveness in signal representations.

A depth map upsampling method using depth local features is proposed. Depth discontinuity and depth variance are extracted from a low-resolution depth map and a colour image as the depth local features. They are incorporated into an energy function for the Markov random field (MRF)-based depth upsampling. The high-resolution depth map is obtained by optimising the energy function using belief propagation. The experimental results show that the proposed method outperforms other depth upsampling approaches in terms of the bad pixel rate.

Resolution enhancement based on learning the sparse association of image patches

In order to solve the issues of inadequate feature description and inefficient feature learning model existing in current classification methods, this article proposes a multi-channel joint sparse learning model for three-dimensional (3D) non-rigid object classification. First, the authors adopt a multi-level measurement of intrinsic properties to create complementary shape descriptors. Second, they build independent and informative bag of features (BoF) by embedding these shape descriptors into the visual vocabulary space. Third, a max-dependency and min-redundancy criterion is applied for optimal feature filtering on each BoF dictionary based on mutual information; meanwhile, each dictionary is learned and weighted according to its contribution to the classification task, and then a compact multi-channel joint sparse learning model is constructed. Finally, the authors train the joint sparse learning model followed by a Softmax classifier to implement efficient shape classification. The experimental results show that the proposed method has stronger feature representation ability and promotes greatly the discrimination of sparse coding coefficients. Thus, the promising classification performance and the powerful robustness can be obtained compared to the state-of-the-art methods.

The depth maps captured by RGB-D cameras usually are of low resolution, entailing recent efforts to develop depth super-resolution (DSR) methods. However, several problems remain in existing DSR methods. First, conventional DSR methods often suffer from unexpected artifacts. Secondly, high-resolution (HR) RGB features and low-resolution (LR) depth features are often fused in shallow layers only. Thirdly, only the last layer of features is used for reconstruction. To address the above problems, we propose Coupled U-Net (CU-Net), a new color image guided DSR method built on two U-Net branches for HR color images and LR depth maps, respectively. The CU-Net embeds a dual skip connection structure to leverage the feature interaction of the two branches, and a multi-scale fusion to fuse the deeper and multi-scale features of two branch decoders for more effective feature reconstruction. Moreover, a channel attention module is proposed to eliminate artifacts. Extensive experiments show that the proposed CU-Net outperforms state-of-the-art methods.