Fine-Grained Semantic Segmentation of Motion Capture Data using Dilated Temporal Fully-Convolutional Networks

We present a novel approach to learn motion controllers for real-time character animation based on motion capture data. We employ a tree-based regression algorithm for reinforcement learning, which enables us to generate motions that require planning. This approach is more flexible and more robust than previous strategies. We also extend the learning framework to include parameterized motions and interpolation. This enables us to control the character more precisely with a small amount of motion data. Finally, we present results of our algorithm for three different types of controllers.

This paper aims to propose a novel approach to generate new generic human walking patterns using motion-captured data, leading to a real-time engine intended for virtual humans animation. The method applies the PCA (principal component analysis) technique on motion data acquired by an optical system to yield a reduced dimension space where not only interpolation, but also extrapolation are possible, controlled by quantitative speed parameter values. Moreover, with proper normalization and time warping methods, the generic presented engine can produce walking motions with continuously varying human height and speed with real-time reactivity

In this work, we focus on fast and efficient recognition of motions in multi-attribute continuous motion sequences. 3D motion capture data, animation motion data, and sensor data from gesture sensing devices are examples of multi-attribute continuous motion sequences. These sequences have multiple attributes rather than only one attribute as time series data has. Motions can have different rates and durations, and the resulting data can thus have different lengths. Also, motion data can have noises due to transitions between successive motions. Hence, traditional distance measuring approaches used for time series data (such as Euclidean distances or dynamic time-warped distances) are not suitable for recognition in multi-attribute motion sequences. Hence, we have defined a similarity measure based on the analysis of singular value decomposition (SVD) properties of similar multi-attribute motions. A five-phase algorithm has then been proposed that gives good pruning power by exploiting the proximity of continuous motion data. We experimented this algorithm with data from different sources: 3D motion capture devices, animation motions, and CyberGlove gesture sensing device. These experiments show that our algorithm can segment and recognize long motion streams with high accuracy and in real time without knowing beforehand the number of motions in a stream.

Physical simulation is often proposed as a way to generate motion for interactive characters. A simulated character has the potential to adapt to changing terrain and disturbances in a realistic and robust manner. In this paper, we present a balancing control algorithm based on a simplified dynamic model, an inverted pendulum on a cart. The simplified model lacks the degrees of freedom found in a full human model, so we analyze a captured reference motion in a preprocessing step and use that information about human running patterns to supplement the balance algorithms provided by the inverted pendulum controller. At run-time, the controller plans a desired motion at every frame based on the current estimate of the pendulum state and a predicted pendulum trajectory. By tracking this time-varying trajectory, our controller creates a running character that dynamically balances, changes speed and makes turns. The initial controller can be optimized to further improve the motion quality with an objective function that minimizes the difference between a planned desired motion and a simulated motion. We demonstrate the power of this approach by generating running motions at a variety of speeds (3 m/s to 5m/s), following a curved path, and in the presence of disturbance forces and a skipping motion.

The presented work illustrates the potential of an IK algorithm enforcing priorities among constraints for motion editing. Motion capture is a very efficient technique to deliver believable motions but usually most of the captured motions need to be edited before they match the end-user's needs. Motion editing can be achieved by applying constraints to parts of the animated character while trying to retain most of the original motion. However all constraints are not equal: some have more importance than others for the animator. The sequences presented here were obtained with an IK solver that allows the user to associate a priority level to a constraint (without limitation in the number of priority levels). We first provide a short overview of the edited sequence prior to describe the general framework of the test application and the potential of the IK algorithm with priorities. We focus especially on the first case where three priority levels are exploited to provide a fine control over the arm animation. The two next cases focus on the specification of goals attached to mobile frames. A general discussion reviews the advantages and limitations of the recent state of the IK algorithm.

The film and video-game industries' rapidly increasing demand for realistic virtual characters is pushing for the development of fast and efficient character animation techniques. The use of motion capture has become an industry standard. While motion capture systems allow the production of high fidelity full-body motions, in most cases, the complex finger movements are later animated manually. Several methods have been suggested to automatically synthesize finger motions. An overview can be found in Wheatland et al. [WWS*15]. Most of these methods either do not consider interactions and collisions or are computationally intensive.

Human motion is a complex activity of the central nervous system (CNS) and muscles. Performance of a human motion can be decomposed into three components: estimation of trajectory; calculation of required signal for muscles; and performance of movement. The CNS conducts the first two tasks and the muscles perform the third task. This paper presents the development of a mathematical model and a Matlab Simulink plant for human gait movement. An internal model predictive control (MPC) is setup and plays as the human CNS to estimate the trajectory and to calculate the required signal for muscles to perform the movement. MPC calculates the required torques for each joint and generate optimal trajectories subject to human physical constraints for muscles. Results of simulation are analyzed and compared to the real human gait motions captured by a real motion capture system (Vicon). Finally, conclusions and recommendations from this research are withdrawn.

This paper presents Camera Motion Graphs, a technique to easily and efficiently generate cinematographic sequences in real-time dynamic 3D environments. A camera motion graph consists of (i) pieces of original camera trajectories attached to one or multiple targets, (ii) generated continuous transitions between camera trajectories and (iii) transitions representing cuts between camera trajectories. Pieces of original camera trajectories are built by extracting camera motions from real movies using vision-based techniques, or relying on motion capture techniques using a virtual camera system. A transformation is proposed to recompute all the camera trajectories in a normalized representation, making camera paths easily adaptable to new 3D environments through a specific retargeting technique. The camera motion graph is then constructed by sampling all pairs of camera trajectories and evaluating the possibility and quality of continuous or cut transitions. Results illustrate the simplicity of the technique, its adaptability to different 3D environments and its efficiency.

Motion segmentation is one of the key techniques in the context of motion analysis and generation. The basic idea is to split motion capture data into continuous segments that can be used to generate new motion sequences. For most applications, this segmentation is done manually leading to inaccurate and inconsistent results. This makes it difficult to conceive general methods for subsequent reassembly. This paper proposes an automatic segmentation of motion capture data that results in deterministic segmentation points. The method can be considered as an advanced zero crossing segmentation technique. As zero crossing performs poorly on weak motion, a threshold is defined detecting phases of week motion. We distinguish two states: One for the resting phase and one for phases of movement. Splitting this second phase again, the presented approach leads to a symbolic level allowing later steps to be carried out without the need of considering spatio-temporal dependencies.

We present a data-driven method for completion of corrupted marker-based motion capture data. Our novel approach is especially suitable for challenging cases, e.g. if complete marker sets of multiple body parts are missing over a long period of time. Without the need for extensive preprocessing we are able to fix missing markers across different actors and motion styles. Our approach can be used for incrementally increasing prior-databases, as the underlying search technique for similar motions scales well to huge databases.

Character animations produced with motion capture data have many of the stylistic details seen in human motion while those generated with simulation are physically realistic for the dynamic parameters of the character. We combine these two approaches by tracking and modifying human motion capture data using dynamic simulation and constraints. The tracking system generates motion that is appropriate for the graphical character while maintaining characteristics of the original human motion. The system imposes contact and task constraints to add dynamic impacts for interactions with the environment and to modify motions at the behavior level. The system is able to edit motion data to account for changes in the character and the environment as well as create smooth transitions between motion capture sequences. We demonstrate the power of combining these two approaches by tracking data for a variety of upper-body motions and by animating models with differing kinematic and dynamic parameters.

There are many applications for which it is necessary to illustrate motion in a static image using visual cues which do not represent a physical entity in the scene, yet are widely understood to convey motion. For example, consider the task of illustrating the desired movements for exercising, dancing, or a given sport technique. Traditional artists have developed techniques to specify desired movements precisely (technical illustrators) and suggest motion (cartoonists) in an image.In this paper, we present an interactive system to synthesize a 2D image of an animated character by generating artist-inspired motion cues derived from 3D skeletal motion capture data. The primary cues include directed arrows, noise waves, and stroboscopic motion. First, the user decomposes the animation into short sequences containing individual motions which can be represented by visual cues. The system then allows the user to determine a suitable viewpoint for illustrating the movement, to select the proper level in the joint hierarchy, as well as to fine-tune various controls for the depiction of the cues themselves. While the system does provide adapted default values for each control, extracted from the motion capture data, it allows fine-tuning for greater expressiveness. Moreover, these cues are drawn in real time, and maintain a coherent display with changing viewpoints.We demonstrate the benefit of our interactive system on various motion capture sequences.

In the framework of virtual prototyping, a virtual human is a useful tool to evaluate what one can reach or see. However because engineers and industrial designers are not necessarily skilled animators, it is important to provide intuitive or high-level interfaces to control virtual characters. With those considerations in mind, in this thesis we investigate three interfaces to control virtual humans for reaching tasks. First we propose an online full-body control using motion capture technology. The posture is reconstructed with an inverse kinematics solver. Coupled with a collision avoidance module, it is a powerful interface to control interactively virtual humans in cluttered environment. Secondly, we propose a novel interface for the full-body online control of differing-height characters. Instead of scaling the markers to the size of the virtual human, we suggest inversely scaling the virtual world. We conduct a user-study to evaluate if such interface is susceptible to help the user experiences the efforts involved by the differing-height character to complete a reaching task. Finally, we investigate motion graph technology for navigation coupled with reaching motions.

Using motion capture systems to animate humanlike figures still remains difficult when the movements are complex or need to be adapted to geometric constraints. We propose a new method to blend several captured movements while adapting the trajectories to new skeletons and to unknown environments. For each body part (considered as resources), a priority is defined for each movement (considered as consumers). The trajectories applied to the skeleton consist of a weighted sum of the motions trajectories. A new technique to compute the weights is proposed. Finally, the system adapts the resulting trajectories to the synthetic skeleton and to the environment. The results enabled to animate up to one hundred actors in interactive environments

The advancement of motion-sensing technology and depth cameras has led to a vast opportunity in motion analysis and monitoring applications, kinesiology analysis, and safeguarding intangible cultural heritage (ICH). A technique that allows a computer to understand human behavior is necessary to analyze and identify the motion using the motion capture data. The integration of motion sensing technology such as markerless motion capture devices, inertial sensors, and deep learning techniques gives an innovative approach to recording, analyzing, and visually recognizing human choreographic motion. Convolutional Neural Network (CNN) is one of the best-known techniques in learning patterns from images and videos and is most widely used among deep learning architectures for vision applications. This study explored different CNN architecture to determine the best prediction classifier based on its performances, such as VGG19, InceptionV3, and MobileNetV2. This study aims to perform an image classification approach of one of the Philippines' cultural dances, Aeta dances, utilizing skeleton-based motion capture data using CNN. The test results were assessed based on the generated training accuracy and evaluation of the loss function to assess the models' overall efficiency. VGG19 produced the highest model cultural dance classification accuracy among the three architectures, which resulted in 98.68% compared to InceptionV3 and MobileNetV2. Thus, the VGG19 model illustrates the optimal transfer learning result implies the best fit model than InceptionV3 and MobileNetV2.