Learning Coupled Prior Shape and Appearance Models for Segmentation

Active Appearance Models (AAM), which have been recently introduced by Cootes et al., describe the shape of objects and gray level appearance from a set of example images. An AAM is created from user-placed contours defining the shape of objects of interest in each training image. The information about shape changes observed in the training set is used to model the shape variation. Principle component analysis (PCA) is utilized to model gray level variation observed in the training set. The resulting model describes objects as a linear combination of eigen vectors both in shape and gray levels applied to the mean image. The main purpose of this work is to investigate the clinical potential of AAMs for segmentation of cardiovascular MR images acquired in routine clinical practice. An AAM was constructed using 102 end- diastolic short-axis cardiac MR images at the papillary muscle level from normals and patients with varying pathologies. The resulting AAM is a compact representation consisting of a mean image and a limited number of coefficients of eigen vectors, representing 97% of shape and gray level variation observed in the training set. The segmentation performance is tested in 60 end-diastolic short-axis cardiac MR images from different patients.

Automatic prostate segmentation from MR images is an important task in various clinical applications such as prostate cancer staging and MR-guided radiotherapy planning. However, the large appearance and shape variations of the prostate in MR images make the segmentation problem difficult to solve. Traditional Active Shape/Appearance Model (ASM/AAM) has limited accuracy on this problem, since its basic assumption, i.e., both shape and appearance of the targeted organ follow Gaussian distributions, is invalid in prostate MR images. To this end, the authors propose a sparse dictionary learning method to model the image appearance in a nonparametric fashion and further integrate the appearance model into a deformable segmentation framework for prostate MR segmentation. To drive the deformable model for prostate segmentation, the authors propose nonparametric appearance and shape models. The nonparametric appearance model is based on a novel dictionary learning method, namely distributed discriminative dictionary (DDD) learning, which is able to capture fine distinctions in image appearance. To increase the differential power of traditional dictionary-based classification methods, the authors' DDD learning approach takes three strategies. First, two dictionaries for prostate and nonprostate tissues are built, respectively, using the discriminative features obtained from minimum redundancy maximum relevance feature selection. Second, linear discriminant analysis is employed as a linear classifier to boost the optimal separation between prostate and nonprostate tissues, based on the representation residuals from sparse representation. Third, to enhance the robustness of the authors' classification method, multiple local dictionaries are learned for local regions along the prostate boundary (each with small appearance variations), instead of learning one global classifier for the entire prostate. These discriminative dictionaries are located on different patches of the prostate surface and trained to adaptively capture the appearance in different prostate zones, thus achieving better local tissue differentiation. For each local region, multiple classifiers are trained based on the randomly selected samples and finally assembled by a specific fusion method. In addition to this nonparametric appearance model, a prostate shape model is learned from the shape statistics using a novel approach, sparse shape composition, which can model nonGaussian distributions of shape variation and regularize the 3D mesh deformation by constraining it within the observed shape subspace. The proposed method has been evaluated on two datasets consisting of T2-weighted MR prostate images. For the first (internal) dataset, the classification effectiveness of the authors' improved dictionary learning has been validated by comparing it with three other variants of traditional dictionary learning methods. The experimental results show that the authors' method yields a Dice Ratio of 89.1% compared to the manual segmentation, which is more accurate than the three state-of-the-art MR prostate segmentation methods under comparison. For the second dataset, the MICCAI 2012 challenge dataset, the authors' proposed method yields a Dice Ratio of 87.4%, which also achieves better segmentation accuracy than other methods under comparison. A new magnetic resonance image prostate segmentation method is proposed based on the combination of deformable model and dictionary learning methods, which achieves more accurate segmentation performance on prostate T2 MR images.

The use of appearance and shape priors in image segmentation is known to improve accuracy; however, existing techniques have several drawbacks. For instance, most active shape and appearance models require landmark points and assume unimodal shape and appearance distributions, and the level set representation does not support construction of local priors. In this paper, we present novel appearance and shape models for image segmentation based on a differentiable implicit parametric shape representation called a disjunctive normal shape model (DNSM). The DNSM is formed by the disjunction of polytopes, which themselves are formed by the conjunctions of half-spaces. The DNSM's parametric nature allows the use of powerful local prior statistics, and its implicit nature removes the need to use landmarks and easily handles topological changes. In a Bayesian inference framework, we model arbitrary shape and appearance distributions using nonparametric density estimations, at any local scale. The proposed local shape prior results in accurate segmentation even when very few training shapes are available, because the method generates a rich set of shape variations by locally combining training samples. We demonstrate the performance of the framework by applying it to both 2-D and 3-D data sets with emphasis on biomedical image segmentation applications.

In this paper, we develop and validate an open source, fully automatic algorithm to localize the left ventricular (LV) blood pool centroid in short axis cardiac cine MR images, enabling follow-on automated LV segmentation algorithms. The algorithm comprises four steps: (i) quantify motion to determine an initial region of interest surrounding the heart, (ii) identify potential 2D objects of interest using an intensity-based segmentation, (iii) assess contraction/expansion, circularity, and proximity to lung tissue to score all objects of interest in terms of their likelihood of constituting part of the LV, and (iv) aggregate the objects into connected groups and construct the final LV blood pool volume and centroid. This algorithm was tested against 1140 datasets from the Kaggle Second Annual Data Science Bowl, as well as 45 datasets from the STACOM 2009 Cardiac MR Left Ventricle Segmentation Challenge. Correct LV localization was confirmed in 97.3% of the datasets. The mean absolute error between the gold standard and localization centroids was 2.8 to 4.7mm, or 12 to 22% of the average endocardial radius. Graphical abstract Fully automated localization of the left ventricular blood pool in short axis cardiac cine MR images.

This paper presents a new deformable modeling strategy aimed at integrating shape and appearance in a unified space. If we think traditional deformable models as active contours or evolving curve fronts, the new deformable shape and appearance models we propose are deforming disks or volumes. Each model has not only boundary shape but also interior appearance. The model shape is implicitly embedded in a higher dimensional space of distance transforms, thus represented by a distance map image. In this way, both shape and appearance of the model are defined in the pixel space. A common deformation scheme, the Free Form Deformations (FFD), parameterizes warping deformations of the volumetric space in which the model is embedded in, hence deforming both model boundary and interior simultaneously.

Vessel segmentation is a challenging task due to the complexity of vascular networks and limitations of imaging modalities to accurately capture thin structures. Analytical models based on geometric appearance and/or edge-based assumptions have been shown to be sub-optimal in segmenting vessels. In this paper, a novel approach for learning vessel appearance models from localized-vessel image patches is presented. This approach uses subspace clustering methods based on sparse representation of signals to identify local vessel appearance models from a training dataset. This paper also presents a hierarchical subspace clustering framework, which improves clustering speeds in presence of large number of subspaces. The preliminary results obtained for segmenting retinal vessel images using learned appearance models on the publicly available DRIVE database, yields an accuracy of 0.9268 at 0.1344 false detections and reduces the search space up to 80%.

Robust and accurate segmentation methods are important for the computerized evaluation of medical images. For treatment of rheumatoid arthritis, joint damage assessment in radiographs of hands is frequently used for monitoring disease progression. Current clinical scoring methods are based on visual measurements that are time-consuming and subject to intra and inter-reader variance. A solution may be found in the development of partially automated assessment procedures. This requires reliable segmentation algorithms. Our work demonstrates a segmentation method based on multiple connected active appearance models (AAM) with multiple search steps using different quality levels. The quality level can be regulated by setting the image resolution and the number of landmarks in the AAMs. We performed experiments using two models of different quality levels for shape and texture information. Both models included AAMs for the carpal region, the metacarpals, and all phalanges. By starting an iterative search with the faster, low-quality model, we were able to determine the initial parameters of the second, high-quality model. After the second search, the results showed successful segmentation for 22 of 30 test images. For these images, 70% of the landmarks were found within 1.3 mm difference from manual placement by an expert. The multi-level search approach resulted in a reduction of 50% in calculation time compared to a search using a single model. Results are expected to improve when the model is refined by increasing the number of training examples and the resolution of the models.

In this work we automatically segment the left ventricle (LV) in cardiac MR images in the end-diastole (ED) and end-systole (ES) phases using a novel approach that combines statistical and deterministic deformable models. A 3D Active Appearance Model (AAM) is used to segment the ED phase. The AAM texture model is trained on radial samples from gradient magnitude images to make the fitting process faster and more discriminative. A trained ED-to-ES shape correspondence model is used to map a given ED shape to an ES shape. Once the AAM model converges to a shape in ED, the correspondence model is used to get an approximate ES shape. We segment the LV in the ES phase by first fitting a deformable superquadric to the AAM converged shape (in ED) using data range forces and then tracking the LV using image and data range forces (for the ES shape obtained from correspondence model). We test our approach by performing leave-one-out training on a 35 patient datasets. The data comprises 19 normal patients and 16 patients having heart abnormalities (cardiomyopathy and myocardial infarction). The composition makes it a challenging data collection with significant shape variation. The performance of our method is evaluated by measuring the mismatch between automatically segmented and expert delineated contours using the Mean Perpendicular Distance (MPD) and Dice metrics. The average MPD is 2.6mm for ED and 3.7mm for ES (error mostly towards the apex and base). The average Dice is 0.9 for ED and 0.8 for ES. These results show good potential for clinical use.

Statistical region-based segmentation methods such as the Active Appearance Model (AAM) are used for establishing dense correspondences in images based on learning the variation in shape and pixel intensities in a training set. For low resolution 2D images correspondences can be recovered reliably in real-time. However, as resolution increases this becomes infeasible due to excessive storage and computational requirements. In this paper we propose to reduce the textural components by modelling the coefficients of a wedgelet based regression tree instead of the original pixel intensities. The wedgelet regression trees employed are based on triangular domains and estimated using cross validation. The wedgelet regression trees are functional descriptions of the intensity information and serve to 1) reduce noise and 2) produce a compact textural description. The wedgelet enhanced appearance model is applied to a case study of human faces. Compression ratios of the texture information of 1:40 is obtained without sacrificing segmentation accuracy notably, even at compression ratios of 1:150 fair segmentation is achieved.

We present an improved bundle adjustment method based on the online learned appearance subspaces of 3D points. Our method incorporates the additional information from the learned appearance models into bundle adjustment. Through the online learning of the appearance models, we are able to include more plausible observations of 2D features across diverse viewpoints. Bundle adjustment can benefit from such an increase in the number of observations. Our formulation uses the appearance information to impose additional constraints on the optimization. The detailed experiments with ground-truth data show that the proposed method is able to enhance the reliability of 2D correspondences, and more important, can improve the accuracy of camera motion estimation and the overall quality of 3D reconstruction.

Recently, much work has been devoted to multiple object tracking on the one hand and to appearance model adaptation for a single object tracker on the other side. In this paper, we do both tracking of multiple objects (faces of people) in a meeting scenario and on-line learning to incrementally update the models of the tracked objects to account for appearance changes during tracking. Additionally, we automatically initialize and terminate tracking of individual objects based on low-level features, i.e. face color, face size, and object movement. For tracking a particle filter is incorporated to propagate sample distributions over time. Numerous experiments on meeting data demonstrate the capabilities of our tracking approach. Additionally, we provide an empirical verification of appearance model learning during tracking of an outdoor scene which supports a more robust tracking.

This paper describes a Multi-View Active Appearance Model (AAM) for coherent segmentation of multiple cardiac views. Cootes' AAM framework was adapted by considering shapes and intensities from multiple views, while eliminating trivial difference in object pose in different views. This way, the coherence in organ shape and intensities between different views is modeled, and utilized during image search. The method is validated in two substantially different and novel applications: segmentation of combined end-diastolic and end-systolic left ventricular X-ray angiograms, and simultaneous segmentation of a combination of four chamber, two chamber and short-axis cardiac MR views.

Building a robust and fully automatic framework for human motion tracking in 2D images and videos remains a challenging task in computer vision due to cluttered backgrounds, self-occlusions, variations of body shape and complexities of human postures. In this paper we propose a robust framework for human motion tracking without motion priors. The proposed framework builds an accurate/uncontaminated specific appearance model and then tracks the target’s postures with this specific appearance model. The main contribution of this work is a novel process to build an accurate appearance model by identifying non-target pixels and removing them. In addition, for the goal of tracking in multiple scales, a novel strategy for scale evaluation and adjustment is proposed to adaptively change the scale values during the tracking process. Experiments show that the accurate specific appearance model outperforms existing work, and the proposed tracking system is able to successfully track challenging sequences with different appearances, motions, scales and angles of view.

UTOMATED video surveillance addresses real-time observation of people and vehicles within a busy environment, leading to a description of their actions and interactions. The technical issues include moving object detection and tracking, object classification, human motion analysis, and activity understanding, touching on many of the core topics of computer vision, pattern analysis, and aritificial intelligence. Video surveillance has spawned large research projects in the United States, Europe, and Japan, and has been the topic of several international conferences and workshops in recent years. There are immediate needs for automated surveillance systems in commercial, law enforcement, and military applications. Mounting video cameras is cheap, but finding available human resources to observe the output is expensive. Although surveillance cameras are already prevalent in banks, stores, and parking lots, video data currently is used only “after the fact” as a forensic tool, thus losing its primary benefit as an active, real-time medium. What is needed is continuous 24-hour monitoring of surveillance video to alert security officers to a burglary in progress or to a suspicious individual loitering in the parking lot, while there is still time to prevent the crime. In addition to the obvious security applications, video surveillance technology has been proposed to measure traffic flow, detect accidents on highways, monitor pedestrian congestion in public spaces, compile consumer demographics in shopping malls and amusement parks, log routine maintainence tasks at nuclear facilities, and count endangered species. The numerous military applications include patrolling national borders, measuring the flow of refugees in troubled areas, monitoring peace treaties, and providing secure perimeters around bases and embassies. The 11 papers in this special section illustrate topics and techniques at the forefront of video surveillance research. These papers can be loosely organized into three categories. Detection and tracking involves real-time extraction of moving objects from video and continuous tracking over time to form persistent object trajectories. C. Stauffer and W.E.L. Grimson introduce unsupervised statistical learning techniques to cluster object trajectories produced by adaptive background subtraction into descriptions of normal scene activity. Viewpoint-specific trajectory descriptions from multiple cameras are combined into a common scene coordinate system using a calibration technique described by L. Lee, R. Romano, and G. Stein, who automatically determine the relative exterior orientation of overlapping camera views by observing a sparse set of moving objects on flat terrain. Two papers address the accumulation of noisy motion evidence over time. R. Pless, T. Brodský, and Y. Aloimonos detect and track small objects in aerial video sequences by first compensating for the self-motion of the aircraft, then accumulating residual normal flow to acquire evidence of independent object motion. L. Wixson notes that motion in the image does not always signify purposeful travel by an independently moving object (examples of such “motion clutter” are wind-blown tree branches and sun reflections off rippling water) and devises a flow-based salience measure to highlight objects that tend to move in a consistent direction over time. Human motion analysis is concerned with detecting periodic motion signifying a human gait and acquiring descriptions of human body pose over time. R. Cutler and L.S. Davis plot an object’s self-similarity across all pairs of frames to form distinctive patterns that classify bipedal, quadripedal, and rigid object motion. Y. Ricquebourg and P. Bouthemy track apparent contours in XT slices of an XYT sequence volume to robustly delineate and track articulated human body structure. I. Haritaoglu, D. Harwood, and L.S. Davis present W4, a surveillance system specialized to the task of looking at people. The W4 system can locate people and segment their body parts, build simple appearance models for tracking, disambiguate between and separately track multiple individuals in a group, and detect carried objects such as boxes and backpacks. Activity analysis deals with parsing temporal sequences of object observations to produce high-level descriptions of agent actions and multiagent interactions. In our opinion, this will be the most important area of future research in video surveillance. N.M. Oliver, B. Rosario, and A.P. Pentland introduce Coupled Hidden Markov Models (CHMMs) to detect and classify interactions consisting of two interleaved agent action streams and present a training method based on synthetic agents to address the problem of parameter estimation from limited real-world training examples. M. Brand and V. Kettnaker present an entropyminimization approach to estimating HMM topology and

Video object segmentation (VOS) is a highly challenging problem since the initial mask, defining the target object, is only given at test-time. The main difficulty is to effectively handle appearance changes and similar background objects, while maintaining accurate segmentation. Most previous approaches fine-tune segmentation networks on the first frame, resulting in impractical frame-rates and risk of overfitting. More recent methods integrate generative target appearance models, but either achieve limited robustness or require large amounts of training data. We propose a novel VOS architecture consisting of two network components. The target appearance model consists of a light-weight module, which is learned during the inference stage using fast optimization techniques to predict a coarse but robust target segmentation. The segmentation model is exclusively trained offline, designed to process the coarse scores into high quality segmentation masks. Our method is fast, easily trainable and remains highly effective in cases of limited training data. We perform extensive experiments on the challenging YouTube-VOS and DAVIS datasets. Our network achieves favorable performance, while operating at higher frame-rates compared to state-of-the-art. Code and trained models are available at https://github.com/andr345/frtm-vos.