Color-Guided Depth Map Super-Resolution via Joint Graph Laplacian and Gradient Consistency Regularization

The edge information plays a key role in the restoration of a depth map. Most conventional methods assume that the color image and depth map are consistent in edge areas. However, complex texture regions in the color image do not match exactly with edges in the depth map. In this paper, firstly, we point out that in most cases the consistency between normal map and depth map is much higher than that between RGB-D pairs. Then we propose a dual normal-depth regularization term to guide the restoration of depth map, which constrains the edge consistency between normal map and depth map back and forth. Moreover, considering the bimodal characteristic of weight distribution that exists in depth discontinuous areas, a reweighted graph Laplacian regularizer is proposed to promote this bimodal characteristic. And this regularization is incorporated into a unified optimization framework to effectively protect the piece-wise smoothness(PWS) characteristics of depth map. By treating depth image as graph signal, the weight between two nodes is adapted according to its content. The proposed method is tested for both noise-free and noisy cases, and is compared against the state-of-the-art methods on both synthesis and real captured datasets. Extensive experimental results demonstrate the superior performance of our method compared with most state-of-the-art works in terms of both objective and subjective quality evaluations. Specifically, our method is more effective on edge areas and more robust to noises.

Depth information is being widely used in many real-world tasks, such as 3DTV, 3D scene reconstruction, multi-view rendering, etc. However, the captured depth maps in practice usually suffer from quality degradations, including low-resolution and noise corruption, which limit their further applications. Noise-aware super-resolution of depth maps is a challenging task and has received increasingly more attention in recent years. In this paper, we propose a novel method based on the plug-and-play scheme, which casts two powerful graph-based tools-the graph Laplacian regularizer and 3D graph Fourier transform-into a unified ADMM optimization framework. It can be performed in an iterative manner with easily treatable convex optimization sub-problems. Experiments results demonstrate that our method achieves superior performance compared with the state-of-the-art works with respect to both objective and subjective quality evaluations.

Although RGBD cameras can provide depth information in real scenes, the captured depth map is often of low resolution and insufficient quality compared to the color image. Typically, most of the existing methods work by assuming that the edges in depth map and its corresponding color image are more likely to occur simultaneously. However, when the color image is rich in detail, the high-frequency information which is non-existent in the depth map will be introduced into the depth map. In this paper, we propose a CNN-based method to detect the co-occurrent structural edge for color-guided depth map super-resolution. Firstly, we design an edge detection convolutional neural network (CNN) to obtain the co-occurrent structural edge in depth map and its corresponding color image. Then we pack the obtained co-occurrent structural edges and the interpolated low-resolution depth maps into another customized CNN for depth map super-resolution. The presented scheme can effectively interpret and exploit the structural correlation between the depth map and the color image. Additionally, recursive learning is adopted to reduce the parameters of the customized CNN for depth map super-resolution and avoid overfitting. Experimental results demonstrate the effectiveness and reliability of our proposed approach by comparing with the state-of-the-art methods.

This paper presents a new algorithm for the joint restoration of depth and intensity (DI) images constructed using a gated SPAD-array imaging system. The three dimensional (3D) data consists of two spatial dimensions and one temporal dimension, and contains photon counts (i.e., histograms). The algorithm is based on two steps: (i) construction of a graph connecting patches of pixels with similar temporal responses, and (ii) estimation of the DI values for pixels belonging to homogeneous spatial classes. The first step is achieved by building a graph representation of the 3D data, while giving a special attention to the computational complexity of the algorithm. The second step is achieved using a Fisher scoring gradient descent algorithm while accounting for the data statistics and the Laplacian regularization term. Results on laboratory data show the benefit of the proposed strategy that improves the quality of the estimated DI images.

Sparse unmixing aims at finding the optimal subset of endmembers in a spectral library to approximate the observed data, and has received increasing attention as it can circumvent the estimation of the endmember. In this paper, a graph Laplacian regularized spectral-spatial-sparse unmixing algorithm is proposed, namely, gLapS3U, incorporating the graph Laplacian regularization to consider the similarity between pixels of the whole image, and enforcing the spectral-spatial-sparse constraints to enhance the local spatial information as well as the sparsity of the abundance solution jointly. Experimental results on simulated and real data show the superiority of the proposed algorithm compared with state-of-the-art existing methods.

Graph-based semi-supervised learning (SSL) algorithms have been widely studied in the last few years. Most of these algorithms were designed from unconstrained optimization problems using a Laplacian regularizer term as smoothness functional in an attempt to reflect the intrinsic geometric structure of the data's marginal distribution. Although a number of recent research papers are still focusing on unconstrained methods for graph-based SSL, a recent statistical analysis showed that many of these algorithms may be unstable on transductive regression. Therefore, we focus on providing new constrained methods for graph-based SSL. We begin by analyzing the regularization framework of existing unconstrained methods. Then, we incorporate two normalization constraints into the optimization problem of three of these methods. We show that the proposed optimization problems have closed-form solution. By generalizing one of these constraints to any distribution, we provide generalized methods for constrained graph-based SSL. The proposed methods have a more flexible regularization framework than the corresponding unconstrained methods. More precisely, our methods can deal with any graph Laplacian and use higher order regularization, which is effective on general SSL taks. In order to show the effectiveness of the proposed methods, we provide comprehensive experimental analyses. Specifically, our experiments are subdivided into two parts. In the first part, we evaluate existing graph-based SSL algorithms on time series data to find their weaknesses. In the second part, we evaluate the proposed constrained methods against six state-of-the-art graph-based SSL algorithms on benchmark data sets. Since the widely used best case analysis may hide useful information concerning the SSL algorithms' performance with respect to parameter selection, we used recently proposed empirical evaluation models to evaluate our results. Our results show that our methods outperforms the competing methods on most parameter settings and graph construction methods. However, we found a few experimental settings in which our methods showed poor performance. In order to facilitate the reproduction of our results, the source codes, data sets, and experimental results are freely available.

The edge information plays key role in the restoration of depth map. Most conventional methods assume that the RGB-D pairs are consistent in edge areas. In this paper, firstly, we point out that in most cases the consistency between normal map and depth map(N-D pairs) are much higher than that be-tween RGB-D pairs. Then we propose a dual regularization term to guide the restoration of depth map, which constrains the consistency between N-D pairs back and forth. Moreover, a reweighted graph Laplacian prior is incorporated into a unified optimization framework to effectively protect piece-wise smoothness(PWS) characteristics of depth map. By treating depth maps as graph signals, the weight between two nodes is adapted according to its content. Extensive experimental results demonstrate the superior performance of our method compared with other state-of-the-art works in terms of objective and subjective quality evaluations.

Non-negativity and dependence constrained sparse coding for image classification

When it comes to evaluating perceptual quality of digital media for overall quality of experience assessment in immersive video applications, typically two main approaches stand out: Subjective and objective quality evaluation. On one hand, subjective quality evaluation offers the best representation of perceived video quality assessed by the real viewers. On the other hand, it consumes a significant amount of time and effort, due to the involvement of real users with lengthy and laborious assessment procedures. Thus, it is essential that an objective quality evaluation model is developed. The speed-up advantage offered by an objective quality evaluation model, which can predict the quality of rendered virtual views based on the depth maps used in the rendering process, allows for faster quality assessments for immersive video applications. This is particularly important given the lack of a suitable reference or ground truth for comparing the available depth maps, especially when live content services are offered in those applications. This paper presents a no-reference depth map quality evaluation model based on a proposed depth map edge confidence measurement technique to assist with accurately estimating the quality of rendered (virtual) views in immersive multi-view video content. The model is applied for depth image-based rendering in multi-view video format, providing comparable evaluation results to those existing in the literature, and often exceeding their performance.

Consumer depth sensors such as time-of-flight camera or Kinect have gained significant popularity in recently. However, the captured depth maps suffer from limited spatial resolution and a variety of noise, making such depth maps difficult to be directly applied in related applications. In this paper, we present a novel single depth map super-resolution method, aiming to reconstruct high-resolution depth map from its associated low-resolution depth map without any auxiliary information. Particularly, we exploit the depth local self-similarity to assist in constructing patch pairs in terms of high-resolution and low-resolution depth edge patches, and then deduce a high-resolution depth edge map via Markov model. Finally, we implement a joint bilateral filter to reconstruct the high-resolution depth map. Experimental results show that our method overcomes existing methods on the benchmark database as well as Kinect captured depth maps.

As an advanced spectral imaging technology, hyperspectral has critical applications in remote sensing. Unfortunately, hyperspectral images (HSIs) are frequently contaminated by diverse noise interference during capture. It is desirable to remove these mixed noises and recover clean HSIs accurately. Current approaches struggle to deliver great performance because they fail to effectively utilize the spectral correlations in hyperspectral data. This paper introduces an innovative hyperspectral image denoising algorithm based on the tensorial weighted Schatten-p norm and graph Laplacian regularization named TWSPGLR. Firstly, to improve the accuracy of low-rank tensor recovery, the tensorial weighted Schatten- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p$</tex> norm is introduced to recover clean hyperspectral data. Secondly, we introduce a spectral constraint to enhance restoration accuracy by efficiently exploiting the spectral correlations of hyperspectral data. Finally, experimental results demonstrate the superiority of TWSPGLR compared with the state-of-the-art methods for HSI denoising.

Previous chapter Next chapter Full AccessProceedings Proceedings of the 2016 SIAM International Conference on Data Mining (SDM)GSpartan: a Geospatio-Temporal Multi-task Learning Framework for Multi-location PredictionJianpeng Xu, Pang-Ning Tan, Lifeng Luo, and Jiayu ZhouJianpeng Xu, Pang-Ning Tan, Lifeng Luo, and Jiayu Zhoupp.657 - 665Chapter DOI:https://doi.org/10.1137/1.9781611974348.74PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutAbstract This paper presents a novel geospatio-temporal prediction framework called GSpartan to simultaneously build local regression models at multiple locations. The framework assumes that the local models share a common, low-rank representation, which makes them amenable to multi-task learning. GSpartan learns a set of base models to capture the spatio-temporal variabilities of the data and represents each local model as a linear combination of the base models. A graph Laplacian regularization is used to enforce constraints on the local models based on their spatial autocorrelation. We also introduce sparsity-inducing norms to perform feature selection for the base models and model selection for the local models. Experimental results using historical climate data from 37 weather stations showed that, on average, GSpartan outperforms single-task learning and other existing multi-task learning methods in more than 65% of the stations, which increases to 81% when there are fewer training examples. Previous chapter Next chapter RelatedDetails Published:2016eISBN:978-1-61197-434-8 https://doi.org/10.1137/1.9781611974348Book Series Name:ProceedingsBook Code:PRDT16Book Pages:1-867

BackgroundSelecting genes and pathways indicative of disease is a central problem incomputational biology. This problem is especially challenging when parsingmulti-dimensional genomic data. A number of tools, such asL1-norm based regularization and its extensions elastic netand fused lasso, have been introduced to deal with this challenge. However, theseapproaches tend to ignore the vast amount of a priori biological networkinformation curated in the literature.ResultsWe propose the use of graph Laplacian regularized logistic regression to integratebiological networks into disease classification and pathway association problems.Simulation studies demonstrate that the performance of the proposed algorithm issuperior to elastic net and lasso analyses. Utility of this algorithm is alsovalidated by its ability to reliably differentiate breast cancer subtypes using alarge breast cancer dataset recently generated by the Cancer Genome Atlas (TCGA)consortium. Many of the protein-protein interaction modules identified by ourapproach are further supported by evidence published in the literature. Sourcecode of the proposed algorithm is freely available athttp://www.github.com/zhandong/Logit-Lapnet.ConclusionLogistic regression with graph Laplacian regularization is an effective algorithmfor identifying key pathways and modules associated with disease subtypes. Withthe rapid expansion of our knowledge of biological regulatory networks, thisapproach will become more accurate and increasingly useful for miningtranscriptomic, epi-genomic, and other types of genome wide associationstudies.

The K Nearest Neighbor (kNN) method has widely been used in the applications of data mining and machine learning due to its simple implementation and distinguished performance. However, setting all test data with the same k value in the previous kNN methods has been proven to make these methods impractical in real applications. This article proposes to learn a correlation matrix to reconstruct test data points by training data to assign different k values to different test data points, referred to as the Correlation Matrix kNN (CM-kNN for short) classification. Specifically, the least-squares loss function is employed to minimize the reconstruction error to reconstruct each test data point by all training data points. Then, a graph Laplacian regularizer is advocated to preserve the local structure of the data in the reconstruction process. Moreover, an ℓ 1 -norm regularizer and an ℓ 2, 1 -norm regularizer are applied to learn different k values for different test data and to result in low sparsity to remove the redundant/noisy feature from the reconstruction process, respectively. Besides for classification tasks, the kNN methods (including our proposed CM-kNN method) are further utilized to regression and missing data imputation. We conducted sets of experiments for illustrating the efficiency, and experimental results showed that the proposed method was more accurate and efficient than existing kNN methods in data-mining applications, such as classification, regression, and missing data imputation.

Numerous studies have shown that microRNAs are associated with the occurrence and development of human diseases. Thus, studying disease-associated miRNAs is significantly valuable to the prevention, diagnosis and treatment of diseases. In this paper, we proposed a novel method based on matrix completion and non-negative matrix factorization (MCNMF)for predicting disease-associated miRNAs. Due to the information inadequacy on miRNA similarities and disease similarities, we calculated the latter via two models, and introduced the Gaussian interaction profile kernel similarity. In addition, the matrix completion (MC)was employed to further replenish the miRNA and disease similarities to improve the prediction performance. And to reduce the sparsity of miRNA-disease association matrix, the method of weighted K nearest neighbor (WKNKN)was used, which is a pre-processing step. We also utilized non-negative matrix factorization (NMF)using dual L2,1-norm, graph Laplacian regularization, and Tikhonov regularization to effectively avoid the overfitting during the prediction. Finally, several experiments and a case study were implemented to evaluate the effectiveness and performance of the proposed MCNMF model. The results indicated that our method could reliably and effectively predict disease-associated miRNAs.