Topology Inference and Signal Representation Using Dictionary Learning

Effective representation methods and proper signal priors are crucial in most signal processing applications. In this thesis we focus on different structured models and we design appropriate schemes that allow the discovery of low dimensional latent structures that characterise and identify the signals. Motivated by the highly non-linear structure of most datasets, we firstly investigate the geometry of manifolds. Manifolds are low dimensional, non-linear structures that are naturally employed to describe sets of strongly related signals such as the images of a 3-D object captured from different viewpoints. However, the use of manifolds in applications is not straightforward due to their usually non-analytic and non-linear form. We propose here a way to `disassemble' a manifold into simpler components by approximating it with affine subspaces. Our objective is to discover a set of low dimensional affine subspaces that can represent manifold data accurately while preserving the manifold's structure. To this end, we employ a greedy technique that iteratively merges manifold samples into groups based on the difference of local tangents. We use our algorithm to approximate synthetic and real manifolds and to demonstrate that it is competitive to state-of-the-art techniques. Then, we consider structured sparse representations of signals and we propose a new sparsity model, where signals are essentially composed of a small number of structured {\it molecules }. We define the molecules to be linear combinations of a small number of atoms in a redundant dictionary. Our multi-level model takes into account the energy distribution of the significant signal components in addition to their support. It permits to define typical visual patterns and recognise them in prototypical or deformed form. We define a new structural difference measure between molecules and their deformed versions, which is based on their sparse codes and we create an algorithm for decomposing signals into molecules that can account for different deviations in the internal molecule structure. Our experiments verify the benefits of the new image model in various restoration tasks and they confirm that the development of proper models that extend the mere notion of sparsity can be very useful for various inverse problems in imaging. Finally, we investigate the problem of learning molecule representations directly in the sparse code domain. We constrain sparse codes to be linear combinations of a few, possibly deformed, molecules and we design an algorithm that can learn the structure from the codes without transforming them back into the signal domain. To this end, we take advantage of our structural difference which is based on the sparse codes and we devise a scheme for representing the codes with molecules and learn the molecules at the same time. To illustrate the effectiveness of our proposed algorithm we apply it to various synthetic and real datasets and we compare the results with traditional sparse coding and dictionary learning techniques. From the experiments, we verify the superior performance of our scheme in interpreting and recognising correctly the underlying structure. In short, in this thesis we are interested in low-dimensional, structured models. Among the various choices, we focus on manifolds and sparse representations and we propose schemes that enhance their structural properties and highlight their effectiveness in signal representations.

In many applications signals reside on the vertices of weighted graphs. Thus, there is the need to learn low dimensional representations for graph signals that will allow for data analysis and interpretation. Existing unsupervised dimensionality reduction methods for graph signals have focused on dictionary learning. In these works the graph is taken into consideration by imposing a structure or a parametrization on the dictionary and the signals are represented as linear combinations of the atoms in the dictionary. However, the assumption that graph signals can be represented using linear combinations of atoms is not always appropriate. In this paper we propose a novel representation framework based on non-linear and geometry-aware combinations of graph signals by leveraging the mathematical theory of Optimal Transport. We represent graph signals as Wasserstein barycenters and demonstrate through our experiments the potential of our proposed framework for low-dimensional graph signal representation.

This paper seeks to combine dictionary learning and hierarchical image representation in a principled way. To make dictionary atoms capturing additional information from extended receptive fields and attain improved descriptive capacity, we present a two-pass multi-resolution cascade framework for dictionary learning and sparse coding. The cascade allows collaborative reconstructions at different resolutions using the same dimensional dictionary atoms. Our jointly learned dictionary comprises atoms that adapt to the information available at the coarsest layer where the support of atoms reaches their maximum range and the residual images where the supplementary details progressively refine the reconstruction objective. The residual at a layer is computed by the difference between the aggregated reconstructions of the previous layers and the downsampled original image at that layer. Our method generates more flexible and accurate representations using much less number of coefficients. Its computational efficiency stems from encoding at the coarsest resolution, which is minuscule, and encoding the residuals, which are relatively much sparse. Our extensive experiments on multiple datasets demonstrate that this new method is powerful in image coding, denoising, inpainting and artifact removal tasks outperforming the state-of-the-art techniques.

In this paper, an adaptive Kalman filter algorithm is proposed for simultaneous graph topology learning and graph signal recovery from noisy time series. Each time series corresponds to one node of the graph and underlying graph edges express the causality among nodes. We assume that graph signals are generated via a multivariate auto-regressive processes (MAR), generated by an innovation noise and graph weight matrices. Then we relate the state transition matrix of Kalman filter to the graph weight matrices since both of them can play the role of signal propagation and transition. Our proposed Kalman filter for MAR processes, called KF-MAR, runs three main steps; prediction, update, and learn. In prediction and update steps, we fix the previously learned graph weight matrices and follow a regular Kalman algorithm for graph signal recovery. Then in the learning step, we use the last update of graph signal estimates and keep track of topology changes. Simulation results show that our proposed graph Kalman filter outperforms the available online algorithms for graph topology inference and also it can achieve the same performance of the batch method, when the number of observations increase.

Sparse representation based classification has led to interesting image recognition results, while the dictionary used for sparse coding plays a key role in it. This paper presents a novel dictionary learning (DL) method to improve the pattern classification performance. Based on the Fisher discrimination criterion, a structured dictionary, whose dictionary atoms have correspondence to the class labels, is learned so that the reconstruction error after sparse coding can be used for pattern classification. Meanwhile, the Fisher discrimination criterion is imposed on the coding coefficients so that they have small within-class scatter but big between-class scatter. A new classification scheme associated with the proposed Fisher discrimination DL (FDDL) method is then presented by using both the discriminative information in the reconstruction error and sparse coding coefficients. The proposed FDDL is extensively evaluated on benchmark image databases in comparison with existing sparse representation and DL based classification methods.

Dictionary learning with the cosparse analysis model based on summation of blocked determinants as the sparseness measure

Dictionary learning is still a challenging problem in signal and image processing. In this paper, we propose an efficient block-structured incoherent dictionary learning algorithm for sparse representations of image signals. The constrained minimization of dictionary learning is achieved by iteratively alternating between sparse coding and dictionary update. Without relying on any prior knowledge of the group structure for the input data, we develop a two-stage clustering method that identifies the underlying block structure of the dictionary under certain restricted constraints. The two-stage clustering method mainly consists of affinity propagation and agglomerative hierarchical clustering. To meet the conditions of both the upper bound and the lower bound of the mutual coherence of dictionary atoms, we introduce a regularization term for the objective function to adjust the block coherence of the overcomplete dictionary. The experiments on synthetic data and real images demonstrate that the proposed dictionary learning algorithm has lower representation error, higher visual quality and better reconstructed results than most of the state-of-the-art methods.

Algorithms for data-driven learning of domain-specific overcomplete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur-concave (CSC) negative log priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen (environmentally matched) dictionary. The elements of the dictionary can be interpreted as concepts, features, or words capable of succinct expression of events encountered in the environment (the source of the measured signals). This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries (the proverbial "25 words or less"), but not necessarily as succinct as one entry. To learn an environmentally adapted dictionary capable of concise expression of signals generated by the environment, we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS and an update of the dictionary using these sparse representations. Experiments were performed using synthetic data and natural images. For complete dictionaries, we demonstrate that our algorithms have improved performance over other independent component analysis (ICA) methods, measured in terms of signal-to-noise ratios of separated sources. In the overcomplete case, we show that the true underlying dictionary and sparse sources can be accurately recovered. In tests with natural images, learned overcomplete dictionaries are shown to have higher coding efficiency than complete dictionaries; that is, images encoded with an overcomplete dictionary have both higher compression (fewer bits per pixel) and higher accuracy (lower mean square error).

Sparse coding allows the representation of complex data as a linear combination of basis sparse vectors (alternatively called atoms or codewords), a collection of which constitutes a dictionary. Dictionary learning is a learning process aimed at finding a small number of optimal basis vectors for a more accurate representation of the original data. The existing dictionary learning methods do not address the inherent uncertainty of the input data in their learning processes. To compensate for the uncertainty, and to obtain a flexible and effective learning system, we introduce a new adaptive fuzzy dictionary learning (AFDL) method for image classification purposes. The new method iteratively alternates between sparse coding based on a given dictionary and an adaptive fuzzy dictionary learning approach to learn (improve) dictionary atoms. The adjustability of the dictionary and coefficients vectors, in this method, provide us a more accurate and straight representation of input data. AFDL was applied on magnetic resonance images from the cancer image archive datasets, for medical image classification of cancer tumors. Finally, the overall experimental results clearly show that our approach outperforms its rival techniques in terms of accuracy, sensitivity, and specificity. Convergence speed in the experimental results shows that AFDL can achieve its acceptable precision in a reasonable time.

Graph signals arise in various applications, ranging from sensor networks to social media data. The high-dimensional nature of these signals implies that they often need to be compressed in order to be stored and transmitted. The common framework for graph signal compression is based on sampling, resulting in a set of continuous-amplitude samples, which in turn have to be quantized into a finite bit representation. In this work, we study the joint design of graph signal sampling along with quantization, for graph signal compression. We focus on bandlimited graph signals, and show that the compression problem can be represented as a task-based quantization setup, in which the task is to recover the spectrum of the signal. Based on this equivalence, we propose a joint design of the sampling and recovery mechanisms for a fixed quantization mapping, and present an iterative algorithm for dividing the available bit budget among the discretized samples. Furthermore, we show how the proposed approach can be realized using graph filters combining elements corresponding the neighbouring nodes of the graph, thus facilitating distributed implementation at reduced complexity. Our numerical evaluations on both synthetic and real world data shows that the joint sampling and quantization method yields a compact finite bit representation of high-dimensional graph signals, which allows reconstruction of the original signal with accuracy within a small gap of that achievable with infinite resolution quantizers.

This paper introduces a design method for densergraph-frequency graph Fourier frames (DGFFs) to enhance graph signal processing and analysis. The graph Fourier transform (GFT) enables us to analyze graph signals in the graph spectral domain and facilitates various graph signal processing tasks, such as filtering, sampling and reconstruction, denoising, and so on. However, the conventional GFT faces two significant limitations. First, unlike the discrete Fourier transform and its variants (such as discrete cosine transforms), the graph frequencies of the derived graph Fourier basis (GFB) from a given graph tend to be unevenly distributed or localized, which leads to biased spectral analysis. Second, the GFB used in GFT does not provide an efficient sparse representation of graph signals compared to overcomplete systems like frames. To overcome these challenges, we propose adding oscillating vectors with intermediate graph frequencies between the original vectors in the GFB for both undirected and directed graphs, constructing GFFs with densergraph frequencies. The resulting DGFFs are expected to enable more accurate graph signal analysis. Furthermore, we propose a graph filtering method based on the DGFFs. In experiments, we apply the DGFFs to practical applications such as graph signal recovery, demonstrating superior performance compared to existing GFBs.

Dictionary learning (DL) techniques aim to find sparse signal representations that capture prominent characteristics in a given data. Such methods operate on a data matrix $Y\in \mathbb {R}^{N\times M}$ , where each of its columns $y_i\in \mathbb {R}^N$ constitutes a training sample, and these columns together represent a sampling from the data manifold. For signals $y\in \mathbb {R}^N$ residing on weighted graphs, an additional challenge is incorporating the underlying geometric structure of the data domain into the learning process. In such cases, the topological graph structure may provide a crucial interpretation for the columns, while the data manifold itself may also possess a low-dimensional intrinsic structure that should be taken into account. In this work, we propose a novel dictionary learning algorithm for graph signals that simultaneously takes into account the underlying structure in both the signal and the manifold domains. Specifically, we require that the dictionary atoms are smooth with respect to the graph topology, as encapsulated by the graph Laplacian matrix. Furthermore, we propose to learn this graph Laplacian within the dictionary learning process, adapting it to promote the desired smoothness. Utilizing the manifold structure, we propose to encourage the smoothness of the sparse representations on the data manifold in a similar manner. Both these smoothness forces implicitly enhance the learned dictionary. The efficiency of the proposed approach is demonstrated on synthetic examples as well as on real data, showing that it outperforms other dictionary learning methods in typical problems such as resistance to noise and data completion.

Sparse representation-based classifiers have shown outstanding accuracy and robustness in image classification tasks even with the presence of intense noise and occlusion. However, it has been discovered that the performance degrades significantly either when test image is not aligned with the dictionary atoms or the dictionary atoms themselves are not aligned with each other, in which cases the sparse linear representation assumption fails. In this paper, having both training and test images misaligned, we introduce a novel sparse coding framework that is able to efficiently adapt the dictionary atoms to the test image via large displacement optical flow. In the proposed algorithm, every dictionary atom is automatically aligned with the input image and the sparse code is then recovered using the adapted dictionary atoms. A corresponding supervised dictionary learning algorithm is also developed for the proposed framework. Experimental results on digit datasets recognition verify the efficacy and robustness of the proposed algorithm.

For face recognition, conventional dictionary learning (DL) methods have disadvantages. In the paper, we propose a novel robust, discriminative and comprehensive DL (RDCDL) model. The proposed model uses sample diversities of the same face image to make the dictionary robust. The model includes class-specific dictionary atoms and disturbance dictionary atoms, which can well represent the data from different classes. Both the dictionary and the representation coefficients of data on the dictionary introduce discriminative information, which improves effectively the discrimination capability of the dictionary. The proposed RDCDL is extensively evaluated on benchmark face image databases, and it shows superior performance to many state-of-the-art sparse representation and dictionary learning methods for face recognition.

In this paper, we present an automated system for robust biometric recognition based upon sparse representation and dictionary learning. In sparse representation, extracted features from the training data are used to develop a dictionary. Training data of real world applications are likely to be exposed to geometric transformations, which is a big challenge for designing of discriminative dictionaries. Classification is achieved by representing the extracted features of the test data as a linear combination of entries in the dictionary. We propose joint weighted dictionary learning and classifier training (JWDL-CT) approach which simultaneously learns from a set of training samples along with weight vectors that correspond to the atoms in the learnt dictionary. The components of the weight vector associated with an atom represent the relationship between the atom and each of the classes. The weight vectors and atoms are jointly obtained during the dictionary learning. In the proposed method, a constraint is imposed on the correlation between the atoms to decrease the similarity between these atoms. The proposed dictionary learning objective function enhances the class-discrimination capabilities of individual atoms that renders the designed dictionaries especially suitable for classification of query images with very sparse representation. Experiments conducted on the West Virginia University (WVU) and the University of Notre Dame (UND) datasets for ear recognition show that the proposed method outperforms other state-of-the-art classifiers.