Supervoxel-based segmentation of 3D imagery with optical flow integration for spatiotemporal processing

Segmentation is a low-level processing aimed at the partition of an image in disjoint regions, each one homogeneous with respect to some properties like intensity, texture, shape, etc. Such a task is needed in many high-level processing and applications in such diverse fields as remote-sensing, medical imaging, video coding, and industrial automation, just to name a few. In this work of thesis, we first deal with the problem of color based image segmentation, that is based on some form of homogeneity in the color/spectral properties of the image, and then move to the more complex task of texture based segmentation, where the aim is to recognize complex structures in the image which are typically non homogeneous in terms of spectral properties. Concerning color based segmentation, we focus on a specific class of Markov Random Field (MRF) probability models, namely the Tree-Structured Markov Random Fields (TS-MRF), recently proposed to provide a hierarchical multiscale description of images. Based on such a model, the unsupervised image segmentation is carried out by means of a sequence of nested class splits, where each class is modeled as a local binary MRF. As a first contribution, we first extend the model to generic tree structures removing the unnecessary binary constraint, and then devise a new TS-MRF unsupervised segmentation technique which improves upon the original algorithm by selecting a better tree structure and eliminating spurious classes. Such results are obtained by using the Mean-Shift procedure to estimate the number of pdf modes at each node, and to obtain a more reliable initial clustering for subsequent MRF optimization. To this end, we devise a new reliable and fast clustering algorithm based on the Mean-Shift technique, which makes use of a variable-bandwidth strategy based on the k-Nearest Neighbors (k-NN) technique, and is implemented with a speed-up strategy that cuts significantly the computational complexity. Experimental results on synthetic data and real life images from the remote sensing domain prove the potential of the proposed method. Turning to texture based segmentation, we present a novel multiscale texture model, and a related algorithm for the unsupervised segmentation of color images. Elementary textures are characterized by their spatial interactions with neighboring regions along selected directions. Such interactions are modeled in turn by means of a set of Markov chains, one for each direction, whose parameters are collected in a feature vector that synthetically describes the texture. Based on the feature vectors, the texture are then recursively merged, giving rise to larger and more complex textures, which appear at different scales of observation: accordingly, the model is named Hierarchical Multiple Markov Chain (H-MMC). The Texture Fragmentation and Reconstruction (TFR) algorithm, addresses the unsupervised segmentation problem based on the H-MMC model. The ``fragmentation'' step allows one to find the elementary textures of the model, while the ``reconstruction'' step defines the hierarchical image segmentation based on a probabilistic measure (texture score) which takes into account both region scale and inter-region interactions. The performance of the proposed method was assessed through the Prague segmentation benchmark, based on mosaics of real natural textures, and also tested on real-world natural images from the Berkeley dataset. Finally, an application of the TFR algorithm for the unsupervised segmentation of multiresolution remotely sensed images is proposed, based on the significant presence of textural content in last generation high-resolution satellite images. The obtained results further assess the potential of the proposed method for such real life applications.

This paper presents a new two-step pseudo maximum a posteriori (MAP) segmentation method for the Markov random field (MRF)-modeled image because the exact MAP estimation is hard to implement due to intractable complexity. The expectation maximization (EM) and Markov Chain Monte Carlo (MCMC) methods are adopted to estimate the parameters for the MRF model due to their comparatively good performance. Although the image segmentation algorithms via graph cuts have become very popular nowadays, our proposed algorithm still performs significantly better in automatic identification and segmentation of fuzzy images than them, which is shown by the quantitative results on synthesized images. In practical applications, the proposed two-step pseudo MAP method is superior in segmenting the fuzzy laser images reflected from the weld pool surfaces during the P-GMAW welding process.

Recent work on early vision such as image segmentation, image restoration, stereo matching, and optical flow models these problems using Markov Random Fields. Although this formulation yields an NP-hard energy minimization problem, good heuristics have been developed based on graph cuts and belief propagation. Nevertheless both approaches still require tens of seconds to solve stereo problems on recent PCs. Such running times are impractical for optical flow and many image segmentation and restoration problems. We show how to reduce the computational complexity of belief propagation by applying the Four Color Theorem to limit the maximum number of labels in the underlying image segmentation to at most four. We show that this provides substantial speed improvements for large inputs to a variety of vision problems, while maintaining competitive result quality.KeywordsImage SegmentationBelief PropagationMarkov Random FieldsImage RestorationStereo MatchThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Interactive Image Segmentation has attracted much attention in the vision and graphics community recently. A typical application for interactive image segmentation is foreground/background segmentation based on user specified brush labellings. The problem can be formulated within the binary Markov Random Field (MRF) framework which can be solved efficiently via graph cut [1]. However, no attempt has yet been made to handle segmentation of multiple regions using graph cuts. In this paper, we propose a multiclass interactive image segmentation algorithm based on the Potts MRF model. Following [2], this can be converted to a multiway cut problem first proposed in [2] and solved by expansion-move algorithms for approximate inference [2]. A faster algorithm is proposed in this paper for efficient solution of the multiway cut problem based on partial optimal labeling. To achieve this, we combine the one-vs-all classifier fusion framework with the expansion-move algorithm for label inference over large images. We justify our approach with both theoretical analysis and experimental validation.

Unsupervised segmentation of weather images into features that correspond to physical storms is a fundamental and difficult problem. Treating an infrared satellite image as a Markov random field, the Kolmogorov-Smirnov distance between the local distribution of spatial statistics and the global statistics of classified regions is used to segment the image using a relaxation algorithm. An outlier class is utilized to capture as yet unclassified pixels. We demonstrate the results of different initialization methods on the final segmentation and point out where the method is deficient.

Combining multiple expert annotations using semi-supervised learning and graph cuts for medical image segmentation

In order to overcome the poor anti-noise performance of traditional fuzzy C-Means (FCM) algorithm in image segmentation, a novel improved FCM algorithm was proposed in this paper based on Particle Swarm Optimization (PSO) algorithm and Markov Random Field (MRF) model, which can make full use of the global searching ability of PSO and the spatial information integrating ability of MRF for image segmentation. In this algorithm, the image segmentation is converted to a PSO optimization problem, in which the fitness function is set up to containing the spatial information based on the spectral value and the neighboring pixels modeled by MRFs. And segmentation results can be iteratively obtained during the PSO iterations according to the newly designed membership function of FCM in which the spatial information is integrated. The experiments herein reported in this paper illustrate the better performance of this algorithm than the traditional FCM algorithm and the PSO algorithm for image segmentation.

Many vision applications have been formulated as Markov Random Field (MRF) problems. Although many of them are discrete labeling problems, continuous formulation often achieves great improvement on the qualities of the solutions in some applications such as stereo matching and optical flow. In continuous formulation, however, it is much more difficult to optimize the target functions. In this paper, we propose a new method called fusion move driven Markov Chain Monte Carlo method (MCMC-F) that combines the Markov Chain Monte Carlo method and the fusion move to solve continuous MRF problems effectively. This algorithm exploits powerful fusion move while it fully explore the whole solution space. We evaluate it using the stereo matching problem. We empirically demonstrate that the proposed algorithm is more stable and always finds lower energy states than the state-of-the art optimization techniques.

Chapter 10 - Graph cut-based segmentation

In this paper, we present an unsupervised change detection approach in temporal sets of SAR images. The change detection is represented as a task of energy minimization and the energy function is minimized using graph cuts. Neighboring pixels are taken into account in a priority sequence according to their distance from the center pixel, and the energy function is formed based on Markov Random Field (MRF) model. Graph cuts algorithm is employed for computing maximum a-posteriori (MAP) estimates of the MRF. Experiments results obtained on a SAR data set confirm the effectiveness of the proposed approach. The comparisons between graph cuts algorithm and iterated conditional modes (ICM) algorithm about the quality of change map and running time of energy minimization illustrate that graph cuts algorithm is a huge improvement over ICM.

The application of a Markov random field (MRF) state model in an expectation-maximization (EM)-based approach to unsupervised image segmentation is investigated. In the calculation of the marginal distribution of the state field, it is shown that the use of the expected state values for interacting pixel sites in the computation of the MRF energy function may be interpreted as a mean-field approximation. The implications of calculating a self-consistent expectation of the state field are considered. EM convergence criteria are considered, and a criterion based upon divergence is proposed. Experimental results based on synthetic data illustrate the performance advantage of the mean-field approximation and the computational advantage of using self-consistent expectations. >

The exemplar-based data clustering problem can be formulated as minimizing an energy function defined on a Markov random field (MRF). However, most algorithms for optimizing the MRF energy function cannot be directly applied to the task of clustering, as the problem has a high-order energy function. In this paper, we first show that the high-order energy function for the clustering problem can be simplified as a pairwise energy function with the metric property, and consequently it can be optimized by the α-expansion move algorithm based on graph cut. Then, the original expansion move algorithm is improved in the following two aspects: 1) Instead of solving a minimal s-t graph cut problem, we show that there is an explicit and interpretable solution for minimizing the energy function in the clustering problem. Based on this interpretation, a fast α-expansion move algorithm is proposed, which is much more efficient than the graph-cut-based algorithm. 2) The fast α-expansion move algorithm is further improved by extending its move space so that a larger energy value reduction can be achieved in each iteration. Experiments on benchmark data sets show that the enhanced expansion move algorithm has a better performance, compared to other state-of-the-art exemplar-based clustering algorithms.

Image segmentation is the process of partitioning a digital image into a set of homogeneous regions (according to some homogeneity criterion) to facilitate a subsequent higher-level analysis. In this context, the present paper proposes an unsupervised and graph-based method of image segmentation, which is driven by an application goal, namely, the generation of image segments associated with a user-defined and application-specific goal. A graph, together with a random grid of source elements, is defined on top of the input image. From each source satisfying a goal-driven predicate, called seed, a propagation algorithm assigns a cost to each pixel on the basis of similarity and topological connectivity, measuring the degree of association with the reference seed. Then, the set of most significant regions is automatically extracted and used to estimate a statistical model for each region. Finally, the segmentation problem is expressed in a Bayesian framework in terms of probabilistic Markov random field (MRF) graphical modeling. An ad hoc energy function is defined based on parametric models, a seed-specific spatial feature, a background-specific potential, and local-contextual information. This energy function is minimized through graph cuts and, more specifically, the alpha-beta swap algorithm, yielding the final goal-driven segmentation based on the maximum a posteriori (MAP) decision rule. The proposed method does not require deep a priori knowledge (e.g., labelled datasets), as it only requires the choice of a goal-driven predicate and a suited parametric model for the data. In the experimental validation with both magnetic resonance (MR) and synthetic aperture radar (SAR) images, the method demonstrates robustness, versatility, and applicability to different domains, thus allowing for further analyses guided by the generated products.

We address the problem of robust normal reconstruction by dense photometric stereo, in the presence of complex geometry, shadows, highlight, transparencies, variable attenuation in light intensities, and inaccurate estimation in light directions. The input is a dense set of noisy photometric images, conveniently captured by using a very simple set-up consisting of a digital video camera, a reflective mirror sphere, and a handheld spotlight. We formulate the dense photometric stereo problem as a Markov network and investigate two important inference algorithms for Markov Random Fields (MRFs)--graph cuts and belief propagation--to optimize for the most likely setting for each node in the network. In the graph cut algorithm, the MRF formulation is translated into one of energy minimization. A discontinuity-preserving metric is introduced as the compatibility function, which allows alpha-expansion to efficiently perform the maximum a posteriori (MAP) estimation. Using the identical dense input and the same MRF formulation, our tensor belief propagation algorithm recovers faithful normal directions, preserves underlying discontinuities, improves the normal estimation from one of discrete to continuous, and drastically reduces the storage requirement and running time. Both algorithms produce comparable and very faithful normals for complex scenes. Although the discontinuity-preserving metric in graph cuts permits efficient inference of optimal discrete labels with a theoretical guarantee, our estimation algorithm using tensor belief propagation converges to comparable results, but runs faster because very compact messages are passed and combined. We present very encouraging results on normal reconstruction. A simple algorithm is proposed to reconstruct a surface from a normal map recovered by our method. With the reconstructed surface, an inverse process, known as relighting in computer graphics, is proposed to synthesize novel images of the given scene under user-specified light source and direction. The synthesis is made to run in real time by exploiting the state-of-the-art graphics processing unit (GPU). Our method offers many unique advantages over previous relighting methods and can handle a wide range of novel light sources and directions.

This letter presents a precise and rapid clustering method for synthetic aperture radar (SAR) images by embedding a Markov random field (MRF) model in the clustering space and using graph cuts (GCs) to search the optimal clusters for the data. The proposed method is optimal in the sense of maximum a posteriori (MAP). It automatically works in a two-loop way: an outer loop and an inner loop. The outer loop determines the cluster number using a pseudolikelihood information criterion based on MRF modeling, and the inner loop is designed in a ldquohardrdquo membership expectation-maximization (EM) style: in the E step, with fixed parameters, the optimal data clusters are rapidly searched under the criterion of MAP by the GC; and in the M step, the parameters are estimated using current data clusters as ldquohardrdquo membership obtained in the E step. The two steps are iterated until the inner loop converges. Experiments on both simulated and real SAR images test the performance of the algorithm.