Signal reconstruction using sparse tree representations

The discrete cosine transform (DCT) offers superior energy compaction properties for a large class of functions and has been employed as a standard tool in many signal and image processing applications. However, it suffers from spurious behavior in the vicinity of edge discontinuities in piecewise smooth signals. To leverage the sparse representation provided by the DCT, in this article, we derive a framework for the inverse polynomial reconstruction in the DCT expansion. It yields the expansion of a piecewise smooth signal in terms of polynomial coefficients, obtained from the DCT representation of the same signal. Taking advantage of this framework, we show that it is feasible to recover piecewise smooth signals from a relatively small number of DCT coefficients with high accuracy. Furthermore, automatic methods based on minimum description length principle and cross-validation are devised to select the polynomial orders, as a requirement of the inverse polynomial reconstruction method in practical applications. The developed framework can considerably enhance the performance of the DCT in sparse representation of piecewise smooth signals. Numerical results show that denoising and image approximation algorithms based on the proposed framework indicate significant improvements over wavelet counterparts for this class of signals.

Significant advances in compressive sensing and sparse signal encoding have provided a rich set of mathematical tools for signal analysis and representation. In addition to novel formulations for enabling sparse solutions to underdetermined systems, exciting progress has taken place in efficiently solving these problems from an optimization theoretic viewpoint. The focus of the wide body of literature in compressive sensing/sparse signal representations has however been on the problem of signal recovery from a small number of measurements (equivalently a sparse coefficient vector). This tutorial will discuss the design of sparse signal representations explicitly for the purposes of signal classification. The tutorial will focus on and build upon two significant recent advances. First, the work by Wright et al. which advocates the use of a dictionary (or basis) matrix comprising of class-specific training sub-dictionaries. In this framework, a test signal is modeled as a sparse linear combination of training vectors in the dictionary, sparsity being enforced by the assertion that only coefficients corresponding to one class (from which the test signal is drawn) ought to be active. The second set of ideas we leverage are recent key contributions in model-based compressive sensing where prior information or constraints on sparse coefficients are used to enhance signal recovery. These ideas will be combined towards the exposition of current trends: namely the development of class-specific priors or constraints to capture structure on sparse coefficients that helps explicitly distinguish between signal classes. In the second part of the tutorial, applications will be discussed including: 1.) structured sparsity for classification of medical imagery for diagnostics, 2.) low-rank approximation and sparse recovery for visual data reconstruction, and 3.) sparse representations for target detection and classification in hyperspectral imagery (guest speaker from the US Army Research Lab).

The Bayesian inference is widely used in many scientific and engineering problems, especially in the linear inverse problems in infinite-dimensional setting where the unknowns are functions. In such problems, choosing an appropriate prior distribution is an important task. Especially when the function to infer has much detail information, such as many sharp jumps, corners, and the discontinuous and nonsmooth oscillation, the so-called total variation-Gaussian (TG) prior is proposed in function space to address it. However, the TG prior is easy to lead the blocky (staircase) effect in numerical results. In this work, we present a fractional order-TG (FTG) hybrid prior to deal with such problems, where the fractional order total variation (FTV) term is used to capture the detail information of the unknowns and simultaneously uses the Gaussian measure to ensure that it results in a well-defined posterior measure. For the numerical implementations of linear inverse problems in function spaces, we also propose an efficient independence sampler based on a transport map, which uses a proposal distribution derived from a diagonal map, and the acceptance probability associated to the proposal is independent of discretization dimensionality. And in order to take full advantage of the transport map, the hierarchical Bayesian framework is applied to flexibly determine the regularization parameter. Finally we provide some numerical examples to demonstrate the performance of the FTG prior and the efficiency and robustness of the proposed independence sampler method.

With the flood of information available today the question how to deal with high dimensional data/signals, which are cumbersome to handle, to calculate with and to store, is highly important. One approach to reducing this flood is to find sparse signal representations, as a signal that is the linear combination of a few elements from a pool of building blocks, can be reduced to the few coefficients of this representation. If these building blocks form a basis, finding the sparse representation poses no problem but unfortunately not many signal classes are sparse in a basis. Taking more building blocks, i.e. a redundant dictionary, increases the chances of having sparse representations, but actually finding them becomes very hard. This led to the development of numerous strategies and algorithms for finding sparse representations, with varying complexity and success rate. The first part of the thesis deals with two of those algorithms, Thresholding and Matching Pursuit, from a more theoretical point of view. It is shown that both those greedy algorithms can be improved with a little trick, that does not increase their complexity, and that when considering their average instead of their worst case performance they perform quite well in comparison to more complex methods. The second part of thesis treats questions concerning the whole dictionary and its properties. First it gives more evidence that sparsity is useful by extending the concept of compressed sensing to signals that are sparse not in a basis but in a redundant dictionary. Thus to record a sparse signal it is not necessary to make as many measurements as the dimension of the signal but only a multiple of the number of dictionary elements used to represent it. Next we show that dictionaries cannot only provide sparse representations but that their geometric properties can also be exploited to model data structures. Here we explain how to model different subclasses of a class of signals by incoherent subspaces, present an algorithm to learn a dictionary made out of these subspaces and then use it for classification of faces. Finally we turn back to the sparse representation problem and study the fundamental question how to find a dictionary providing sparse representations. We pick up the idea to learn a dictionary via minimisation of a continuous cost function and provide conditions, guaranteeing that the decomposition of a collection of training signals into a dictionary and a coefficient matrix constitutes a local minimum. We also analyse statistically when these conditions are fulfilled with high probability.

This paper presents a new approach to single-image super-resolution, based on sparse signal representation. Research on image statistics suggests that image patches can be well-represented as a sparse linear combination of elements from an appropriately chosen over-complete dictionary. Inspired by this observation, we seek a sparse representation for each patch of the low-resolution input, and then use the coefficients of this representation to generate the high-resolution output. Theoretical results from compressed sensing suggest that under mild conditions, the sparse representation can be correctly recovered from the downsampled signals. By jointly training two dictionaries for the low- and high-resolution image patches, we can enforce the similarity of sparse representations between the low resolution and high resolution image patch pair with respect to their own dictionaries. Therefore, the sparse representation of a low resolution image patch can be applied with the high resolution image patch dictionary to generate a high resolution image patch. The learned dictionary pair is a more compact representation of the patch pairs, compared to previous approaches, which simply sample a large amount of image patch pairs, reducing the computational cost substantially. The effectiveness of such a sparsity prior is demonstrated for both general image super-resolution and the special case of face hallucination. In both cases, our algorithm generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods. In addition, the local sparse modeling of our approach is naturally robust to noise, and therefore the proposed algorithm can handle super-resolution with noisy inputs in a more unified framework.

Geophysical Inverse Theory and Regularization Problems

Algorithms for data-driven learning of domain-specific over complete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur- concave negative log-priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen dictionary. The elements of the dictionary can be interpreted as 'concepts,' features or 'words' capable of succinct expression of events encountered in the environment. This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries, but not necessarily as succinct as one entry. To learn an environmentally-adapted dictionary capable of concise expression of signals generated by the environment, we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS, an affine scaling transformation (ACT)-like sparse signal representation algorithm recently developed at UCSD, and an update of the dictionary using these sparse representations.

Millions of digital images are captured by imaging devices on a daily basis. The way imaging devices operate follows an integral process from which the information of the original scene needs to be estimated. The estimation is done by inverting the integral process of the imaging device with the use of optimization techniques. This linear inverse problem, the inversion of the integral acquisition process, is at the heart of several image processing applications such as denoising, deblurring, inpainting, and super-resolution. We describe in detail the use of linear inverse problems in these applications. We review and compare several state-of-the-art optimization algorithms that invert this integral process. Linear inverse problems are usually very difficult to solve. Therefore, additional prior assumptions need to be introduced to successfully estimate the output signal. Several priors have been suggested in the research literature, with the Total Variation (TV) being one of the most prominent. In this thesis, we review another prior, the l0 pseudo-norm over the gradient domain. This prior allows full control over how many non-zero gradients are retained to approximate prominent structures of the image. We show the superiority of the l0 gradient prior over the TV prior in recovering genuinely piece-wise constant signals. The l0 gradient prior has shown to produce state-of-the-art results in edge-preserving image smoothing. Moreover, this general prior can be applied to several other applications, such as edge extraction, clip-art JPEG artifact removal, non-photorealistic image rendering, detail magnification, and tone mapping. We review and evaluate several state-of-the-art algorithms that solve the optimization problem based on the l0 gradient prior. Subsequently we apply the l0 gradient prior to two applications where we show superior results as compared to the current state-of-the-art. The first application is that of single-image reflection removal. Existing solutions to this problem have shown limited success because of the highly ill-posed nature of the problem. We show that the standard l0 gradient prior with a modified data-fidelity term based on the Laplacian operator is able to sufficiently remove unwanted reflections from images in many realistic scenarios. We conduct extensive experiments and show that our method outperforms the state-of-the-art. In the second application of haze removal from visible-NIR image pairs we propose a novel optimization framework, where the prior term penalizes the number of non-zero gradients of the difference between the output and the NIR image. Due to the longer wavelengths of NIR, an image taken in the NIR spectrum suffers significantly less from haze artifacts. Using this prior term, we are able to transfer details from the haze-free NIR image to the final result. We show that our formulation provides state-of-the-art results compared to haze removal methods that use a single image and also to those that are based on visible-NIR image pairs.

We consider the classification problem as a linear regression problem, and find that sparse signal representation offers the key to address this problem. Therefore, a new method, which is based on sparse representation, is proposed for classification. This new method provides insights into two critical issues in classification: sparse representation and classification. For sparse representation, we use the lasso, the elastic net and nonnegative garrote as the initial estimate of a new test sample. In the classification stage, we classify the test sample to the correct class via a simple l <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -distance measurement. Finally, we propose an efficient algorithm for computing the whole solution path of this method, and conduct extensive experiments on the number ball recognition. From the experiment results, we conclude that this method achieves high recognition rate.

Various signal processing techniques have been used for the enhancement of defect detection and defect characterisation. Cross-correlation, filtering, autoregressive analysis, deconvolution, neural network, wavelet transform and sparse signal representations have all been applied in attempts to analyse ultrasonic signals. In ultrasonic nondestructive evaluation (NDE) applications, a large number of materials have multilayered structures. NDE of multilayered structures leads to some specific problems, such as penetration, echo overlap, high attenuation and low signal-to-noise ratio. The signals recorded from a multilayered structure are a class of very special signals comprised of limited echoes. Such signals can be assumed to have a sparse representation in a proper signal dictionary. Recently, a number of digital signal processing techniques have been developed by exploiting the sparse constraint. This paper presents a review of research to date, showing the up-to-date developments of signal processing techniques made in ultrasonic NDE. A few typical ultrasonic signal processing techniques used for NDE of multilayered structures are elaborated. The practical applications and limitations of different signal processing methods in ultrasonic NDE of multilayered structures are analysed.

Efficient, real-time and automated data analysis is one of the key elements for achieving scientific success in complex engineering and physical systems, two examples of which include the JET and ITER tokamaks.One problem which is common to these fields is the determination of the pulsation modes from an irregularly sampled time series. To this end, there are a wealth of signal processing techniques that are being applied to post-pulse and real-time data analysis in such complex systems. Here, we wish to present a review of the applications of a method based on the sparse representation of signals, using examples of the synergies that can be exploited when combining ideas and methods from very different fields, such as astronomy, astrophysics and thermonuclear fusion plasmas.Examples of this work in astronomy and astrophysics are the analysis of pulsation modes in various classes of stars and the orbit determination software of the Pioneer spacecraft. Two examples of this work in thermonuclear fusion plasmas include the detection of magneto-hydrodynamic instabilities, which is now performed routinely in JET in real-time on a sub-millisecond time scale, and the studies leading to the optimization of the magnetic diagnostic system in ITER and TCV.These questions have been solved by formulating them as inverse problems, despite the fact that these applicative frameworks are extremely different from the classical use of sparse representations, from both the theoretical and computational point of view. The requirements, prospects and ideas for the signal processing and real-time data analysis applications of this method to the routine operation of ITER will also be discussed.Finally, a very recent development has been the attempt to apply this method to the deconvolution of the measurement of electric potential performed during a ground-based survey of a proto-Villanovian necropolis in central Italy.

A MATLAB implementation of the minimum relative entropy method for linear inverse problems

Algorithms for data-driven learning of domain-specific overcomplete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur-concave (CSC) negative log priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen (environmentally matched) dictionary. The elements of the dictionary can be interpreted as concepts, features, or words capable of succinct expression of events encountered in the environment (the source of the measured signals). This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries (the proverbial "25 words or less"), but not necessarily as succinct as one entry. To learn an environmentally adapted dictionary capable of concise expression of signals generated by the environment, we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS and an update of the dictionary using these sparse representations. Experiments were performed using synthetic data and natural images. For complete dictionaries, we demonstrate that our algorithms have improved performance over other independent component analysis (ICA) methods, measured in terms of signal-to-noise ratios of separated sources. In the overcomplete case, we show that the true underlying dictionary and sparse sources can be accurately recovered. In tests with natural images, learned overcomplete dictionaries are shown to have higher coding efficiency than complete dictionaries; that is, images encoded with an overcomplete dictionary have both higher compression (fewer bits per pixel) and higher accuracy (lower mean square error).

Representing signals with sparse vectors has a wide range of applications that range from image and video coding to shape representation and health monitoring. In many applications with real-time requirements, or that deal with high-dimensional signals, the computational complexity of the encoder that finds the sparse representation plays an important role. Quantum computing has recently shown promising speed-ups in many representation learning tasks. In this work, we propose a quantum version of the well-known matching pursuit algorithm. Assuming the availability of a fault-tolerant quantum random access memory, our quantum matching pursuit lowers the complexity of its classical counterpart of a polynomial factor, at the cost of some error in the computation of the inner products, enabling the computation of sparse representation of high-dimensional signals. Besides proving the computational complexity of our new algorithm, we provide numerical experiments that show that its error is negligible in practice. This work opens the path to further research on quantum algorithms for finding sparse representations, showing suitable quantum computing applications in signal processing.

Fourier phase retrieval problem is equivalent to the recovery of a two-dimensional image from its autocorrelation measurements. This problem is generally nonlinear and nonconvex. Good initialization and prior information about the support or sparsity of the target image are often critical for a robust recovery. In this paper, we show that the presence of a known reference image can help us solve the nonlinear phase retrieval problem as a sequence of small linear inverse problems. Instead of recovering the entire image at once, our sequential method recovers a small number of rows or columns by solving a linear deconvolution problem at every step. Existing methods for the reference-based (holographic) phase retrieval either assume that the reference and target images are sufficiently separated so that the recovery problem is linear or recover the image via nonlinear optimization. In contrast, our proposed method does not require the separation condition. We performed an extensive set of simulations to demonstrate that our proposed method can successfully recover images from autocorrelation data under different settings of reference placement and noise.