Recursive structure and motion from image sequences using shape and depth spaces

Over the last decade, it has become increasingly affordable to digitize 2D and 3D shape information using multiple modalities, such as (video) cameras, image-based reconstruction systems, laser-range scanners, or depth cameras. If these dense models can be processed and described in an efficient and informative way, they can be used in applications, such as ergonomic design, virtual shopping, scientific and medical visualization, realistic simulation, photo-realistic rendering, the design of natural user interfaces, and semantic scene understanding. Traditionally, the notion of shape has been studied either by analyzing projections of shapes in images or by analyzing a sparse set of marker positions on 3D shapes. Typical tasks in 2D shape analysis include segmenting objects in images and tracking objects across a sequence of images, and typical tasks in 3D shape analysis include reconstructing the three-dimensional object depth from input images and identifying corresponding points on different 3D models. The analysis and processing of shape data becomes especially challenging because of the increasing amount of data captured by sensors used to acquire shapes, and because modern applications such as natural user interfaces require real-time processing of the input shapes. Meeting these challenges requires models of shape analysis that are compact and informative, thereby allowing the development of algorithms that can process large datasets efficiently. To achieve these goals, interdisciplinary approaches are needed that use concepts from a variety of research areas, including numerical computing, differential geometry, deformable shape modeling, sparse data representation, and machine learning. On the algorithmic side, many shape analysis tasks are modeled using partial differential equations, which can be solved using tools from the field of numerical computing. The fields of differential geometry and deformable shape modeling have recently begun to influence shape analysis methods. Furthermore, tools from the field of sparse representations, which aim to describe input data using a compressible representation with respect to a set of carefully selected basis elements, have the potential to significantly reduce the amount of data that needs to be processed in shape analysis tasks. The related field of machine learning offers similar potential. This seminar brought together 28 researchers from North America and Europe engaged in recent and upcoming developments in shape analysis who view these challenges from different perspectives and who together discussed the pressing open problems and novel solutions to them.

The quality and information content of biological images can be significantly enhanced by postacquisition processing using deconvolution and denoising. However, when imaging complex biological samples, such as neurons, stained with fluorescence labels, the signal level of different structures can differ by several orders of magnitude. This poses a challenge as current image reconstruction algorithms are focused on recovering low signals and generally have sample-dependent performance, requiring tedious manual tuning. This is one of the main hurdles for their wide adoption by nonspecialists. In this work, we modify the general constrained reconstruction method (in our case utilizing a total variation constraint) so that both bright and dim structures can drive the deconvolution with equal force. In this way, we can simultaneously obtain high-quality reconstruction across a wide range of signals within a single image or image sequence. The algorithm is tested on both simulated and experimental data. When compared with current state-of-art algorithms, our algorithm outperforms others in terms of maintaining the resolution in the high-signal areas and reducing artefacts in the low-signal areas. The algorithm was also tested on image sequences where one set of parameters are used to reconstruct all images, with blind evaluation by a group of biologists demonstrating a marked preference for the images produced by our method. This means that our method is suitable for batch processing of image sequences obtained from either spatial or temporalscanning. LAY DESCRIPTION: Fluorescence microscopy images of complex biological samples contain a wide range of signal levels. This signal variation leads current reconstruction algorithms, which aim to enhance the quality of the raw images, to have sample-dependent performance. In this work, we design a new optimization that allows the reconstruction to "pay equal eqattention to" both bright and dim structures. In this way, we can simultaneously recover both bright and dim structures within a single image or image sequence, as validated when the algorithm was quantitatively tested on both simulated and experimental data. When our method was evaluated alongside current state of art algorithms by a group of biologists, our algorithm was considered qualitativelysuperior.

The discriminant function of a subspace method is provided by using correlation matrices that reflect the averaged feature of a category. As a result, it will not work well on unknown input patterns that are far from the average. To address this problem, we propose two kinds of weighted correlation matrices for subspace methods. The globally weighted correlation matrix (GWCM) attaches importance to training patterns that are far from the average. Then, it can reflect the distribution of patterns around the category boundary more precisely. The computational cost of a subspace method using GWCMs is almost the same as that using ordinary correlation matrices. The locally weighted correlation matrix (LWCM) attaches importance to training patterns that are near to an input pattern to be classified. Then, it can reflect the distribution of training patterns around the input pattern in more detail. The computational cost of a subspace method with LWCM at the recognition stage does not depend on the number of training patterns, while those of the conventional adaptive local and the nonlinear subspace methods do. We show the advantages of the proposed methods by experiments made on the MNIST database of handwritten digits.

This paper presents a shape analysis on orthogonal feature curves of 3D bald head scans with the intention of predicting scalp shape under the hair. While there are currently a number of large scale 3D head data collections available around the world, they unfortunately all suffer from hair obstruction preventing an accurate description of true scalp shape. This study is aimed at exploring the relationship between a small set of head anthropometric measurements and the feature curves of the scalp shape based on a small set of 3D bald head scans. The feature curves include scalp profile along the sagittal plane, coronal plane and a cross-sectional curve at the level of glabella.Introduction: The ever increasing availability of 3D scans of the human head has been a valuable source of information for improving head shape related equipment design and engineering with the goal of improving overall fit, sizing and comfort. However the existence of the hair prevents the actual recording and analysis of true scalp shape, and thus the analysis of scalp shape remains an elusive work. For this reason, much of the data analysis based on 3D head scans is mostly limited on facial shape and some facial landmarks. To overcome this limitation, it is desirable to be able to reconstruct the cranium shape from a few easy to obtain anthropometric measurements. In order to achieve this goal we obtained three orthogonal feature curves from 83 bald head scans and build regression equation between principal component scores of these curves and a number of head measurements such as head length, head breadth and tragion-to-top of head distance. The prediction error of these equations are evaluated. The approach: A total of 83 bald head scans taken in the 2012 anthropometric survey of US Army personnel (ANSUR II) from male soldiers are available for the analysis. From these bald head scans we identified three feature curves that principally define the scalp shape. These curves are a profile along the sagittal plane, a profile curve along the coronal plane and a cross-sectional curve at glabella level. These curves are then sampled in an equal angular space and grouped by their respective plane. The collection of each feature curves forms a shape space. We applied Principal Component Analysis (PCA) to those shape curves and decomposed the shape variation of each curve group into its respective principal axes. After conducting the PCA each curve group has a reduced dimension of 4~5 principal components that account for 95% of total variance. It shows major shape variations within each feature curve group. Then we apply multiple linear regression to the above head measurements and PCA scores of each curve group (Principal component regression or PCR). The regression equations for each curve and each PC were evaluated for their predicting power. From these regression equations the constituent feature curves can be selected from respective shape space based on head anthropometric measurements. Then PCR was also applied to the shape space of combined three curves as compared to prediction power in individual curve spaces.The results and discussion: PCR results from all orthogonal curve spaces and combined three curve space have similar predicting power as their R-squared value falling between 0.6 ~ 0.7. With a set of feature curves it is possible to further derive a 3D shape of the scalp. Although the three head measurements used in this study are effectively correlated to the first and second principal component in each group, other principal components contribute to subtle shape variations. This is an area need further analysis. We will also investigate methods to predicting 3D scalp shape from three feature curves and accuracy of the reconstruction of 3D scalp from those curves.

More and more analyses of biological shapes are using the techniques of geometric morphometrics based on configurations of landmarks in two or three dimensions. A fundamental concept at the core of these analyses is Kendall’s shape space and local approximations to it by shape tangent spaces. Kendall’s shape space is complex because it is a curved surface and, for configurations with more than three landmarks, multidimensional. This paper uses the shape space for triangles, which is the surface of a sphere, to explore and visualize some properties of shape spaces and the respective tangent spaces. Considerations about the dimensionality of shape spaces are an important step in understanding them, and can offer a coordinate system that can translate between positions in the shape space and the corresponding landmark configurations and vice versa. By simulation studies “walking” along that are great circles around the shape space, each of them corresponding to the repeated application of a particular shape change, it is possible to grasp intuitively why shape spaces are curved and closed surfaces. From these considerations and the available information on shape spaces for configurations with more than three landmarks, the conclusion emerges that the approach using a tangent space approximation in general is valid for biological datasets. The quality of approximation depends on the scale of variation in the data, but existing analyses suggest this should be satisfactory to excellent in most empirical datasets.

Precise modeling of the program heap is fundamental for understanding the behavior of a program, and is thus of significant interest for many optimization applications. One of the fundamental properties of the heap that can be used in a range of optimization techniques is the sharing relationships between the elements in an array or collection. If an analysis can determine that the memory locations pointed to by different entries of an array (or collection) are disjoint, then in many cases loops that traverse the array can be vectorized or transformed into a thread-parallel version. This paper introduces several novel sharing properties over the concrete heap and corresponding abstractions to represent them. In conjunction with an existing shape analysis technique, these abstractions allow us to precisely resolve the sharing relations in a wide range of heap structures (arrays, collections, recursive data structures, composite heap structures) in a computationally efficient manner. The effectiveness of the approach is evaluated on a set of challenge problems from the JOlden and SPECjvm98 suites. Sharing information obtained from the analysis is used to achieve substantial thread-level parallel speedups.

Many software model checkers are based on predicate abstraction. If the verification goal depends on pointer structures, the approach does not work well, because it is difficult to find adequate predicate abstractions for the heap. In contrast, shape analysis, which uses graph-based heap abstractions, can provide a compact representation of recursive data structures. We integrate shape analysis into the software model checker Blast. Because shape analysis is expensive, we do not apply it globally. Instead, we ensure that, like predicates, shape graphs are computed and stored locally, only where necessary for proving the verification goal. To achieve this, we extend lazy abstraction refinement, which so far has been used only for predicate abstractions, to three-valued logical structures. This approach does not only increase the precision of model checking, but it also increases the efficiency of shape analysis. We implemented the technique by extending Blast with calls to Tvla.

The goal of this work is to track human joints in image sequences or video fragments. In this work, we attempt to develop an algorithm to analog the perceiving process of human vision system. Based on the knowledge of human body model and shape analysis techniques used in pattern recognition, after the user selecting out edges correspond to tracked objects interactively in the first frame, the algorithm tracks human joints by using the outline and other important inner edges of a moving human figure. It does not need special markers and avoids feature searching, thus largely improves the tracking speed. Moreover, the algorithm is applied to real image sequences with very encouraging results. Experimental results indicate that, our algorithm is very robust to object moving and rotating, varying with colors, illumination and intervals between frames.

In this paper, we propose an algorithm to extract heart rate frequency from video camera using the Multiple SIgnal Classification (MUSIC) algorithm. This leads to improved accuracy of the estimated heart rate frequency in cases the performance is limited by the number of samples and frame rate. Monitoring vital signs remotely can be exploited for both non-contact physiological and psychological diagnosis. The color variation recorded by ordinary cameras is used for heart rate monitoring. The orthogonality between signal space and noise space is used to find more accurate heart rate frequency in comparison with traditional methods. It is shown via experimental results that the limitation of previous methods can be overcome by using subspace methods.

The fact that the physical shapes of man-made objects are subject to overlapping influences—such as technological, economic, geographic, and stylistic progressions—holds great information potential. On the other hand, it is also a major analytical challenge to uncover these overlapping trends and to disentagle them in an unbiased way. This article explores a novel mathematical approach to extract archaeological insights from ensembles of similar artifact shapes. We show that by considering all shape information in a find collection , it is possible to identify shape patterns that would be difficult to discern by considering the artifacts individually or by classifying shapes into predefined archaeological types and analyzing the associated distinguishing characteristics. Recently, a series of high-resolution digital representations of artifacts have become available. Such datasets enable the application of extremely sensitive and flexible methods of shape analysis. We explore this potential on a set of 3D models of ancient Greek and Roman sundials, with the aim of providing alternatives to the traditional archaeological method of “trend extraction by ordination” (typology). In the proposed approach, each 3D shape is represented as a point in a shape space —a high-dimensional, curved, non-Euclidean space. Proper consideration of its mathematical properties reduces bias in data analysis and thus improves analytical power. By performing regression in shape space, we find that for Roman sundials, the bend of the shadow-receiving surface of the sundials changes with the latitude of the location. This suggests that, apart from the inscribed hour lines, also a sundial’s shape was adjusted to the place of installation. As an example of more advanced inference, we use the identified trend to infer the latitude at which a sundial, whose location of installation is unknown, was placed. We also derive a novel method for differentiated morphological trend assertion, building upon and extending the theory of geometric statistics and shape analysis. Specifically, we present a regression-based method for statistical normalization of shapes that serves as a means of disentangling parameter-dependent effects (trends) and unexplained variability . In addition, we show that this approach is robust to noise in the digital reconstructions of the artifact shapes.

A scene-based algorithm is developed to compensate for bias nonuniformity in focal-plane arrays. Nonuniformity can be extremely problematic, especially for mid- to far-infrared imaging systems. The technique is based on use of estimates of interframe subpixel shifts in an image sequence, in conjunction with a linear-interpolation model for the motion, to extract information on the bias nonuniformity algebraically. The performance of the proposed algorithm is analyzed by using real infrared and simulated data. One advantage of this technique is its simplicity; it requires relatively few frames to generate an effective correction matrix, thereby permitting the execution of frequent on-the-fly nonuniformity correction as drift occurs. Additionally, the performance is shown to exhibit considerable robustness with respect to lack of the common types of temporal and spatial irradiance diversity that are typically required by statistical scene-based nonuniformity correction techniques.

We consider the problem of removing and replacing clouds in satellite image sequences, which has a wide range of applications in remote sensing. Our approach first detects and removes the cloud-contaminated part of the image sequences. It then recovers the missing scenes from the clean parts using the proposed "TECROMAC" (TEmporally Contiguous RObust MAtrix Completion) objective. The objective function balances temporal smoothness with a low rank solution while staying close to the original observations. The matrix whose the rows are pixels and columns are days corresponding to the image, has low-rank because the pixels reflect land-types such as vegetation, roads and lakes and there are relatively few variations as a result. We provide efficient optimization algorithms for TECROMAC, so we can exploit images containing millions of pixels. Empirical results on real satellite image sequences, as well as simulated data, demonstrate that our approach is able to recover underlying images from heavily cloud-contaminated observations.

This paper develops a shape space framework for collision-aware geometric modeling, where basic geometric operations automatically avoid inter-penetration. Shape spaces are a powerful tool for surface modeling, shape analysis, nonrigid motion planning, and animation, but past formulations permit nonphysical intersections. Our framework augments an existing shape space using a repulsive energy such that collision avoidance becomes a first-class property, encoded in the Riemannian metric itself. In turn, tasks like intersection-free shape interpolation or motion extrapolation amount to simply computing geodesic paths via standard numerical algorithms. To make optimization practical, we develop an adaptive collision penalty that prevents mesh self-intersection, and converges to a meaningful limit energy under refinement. The final algorithms apply to any category of shape, and do not require a dataset of examples, training, rigging, nor any other prior information. For instance, to interpolate between two shapes we need only a single pair of meshes with the same connectivity. We evaluate our method on a variety of challenging examples from modeling and animation.

In this study,the Euler subtle motion amplification (MA) algorithm is applied to bridge vibration analysis. First,a series of digital images of bridge structures is continuously captured using an ordinary digital camera. Then,the micro-vibration of the bridge is amplified using the related algorithm of Euler MA. The time history image of the bridge specimen under conventional video sampling frequency is obtained by decomposing the amplified digital image. Lastly,the holographic dynamic displacement measurement results of the test bridge are obtained by analyzing the edge features of the marking points in the image sequences. The proposed analysis method is feasible compared with the displacement sensor and dial indicator. It can meet the application requirements of bridge dynamic displacement measurement. It can also perform preliminary quantitative analysis of several vibration characteristics of bridges and identify the micro-vibration displacements of a bridge structure.

Hand is an important part of human in communicating with other persons and interacting with objects in the world. In this thesis, we will explore hand analysis, which includes hand detection, hand posture and hand shape analysis.
\n
\nFirst I will focus on hand detection. We start by exploring pixel-level hand detection. It is a challenging problem because the appearance of hands can vary greatly in images. We use a random forest framework in order to predict hand mask and hand part labels in an efficient and robust manner. Our approach exploits the inherent contextual information from structured hand labelling for pixel-level hand detection and hand part labelling. Having obtained a pixel-level hand probability map, we present an end-to-end system to detect hands in a bounding box fashion. Our model can not only detect and localize the hands but also provide detailed information within the bounding box by using a multi-task convolutional neural network. Experimental results show that hand detection can be improved by utilizing more information such as hand shape and hand landmark points than only using bounding boxes.
\n
\nIn the second part of my thesis, we will concentrate on two specific problems based on the good hand detection results. The first problem is to recognize the hand postures efficiently. We present a flexible method for hand posture recognition by fusing information from color and depth images using kernel descriptors. Existing methods usually focus on designing intuitive features for color and depth images. On the contrary, our method extracts common patch-level features, and combines them by means of kernel descriptors. Linear SVM is used to predict the class label efficiently.
\nThe second problem is related to hand shape. We present an interesting application of shape analysis to train a hand shape recognizer automatically from simple sketches, such as a “stick-figure” of a hand shape. We introduce the Hand Boltzmann Machine, a generative model built upon unsupervised learning, to represent the hand shape space of a binary image, and formulate the user provided sketches as an initial guidance for sampling to generate realistic hand shape samples. A hand shape recognizer is trained from these samples and used to classify the hand shapes intuitively.