Probabilistic Articulated Real-Time Tracking for Robot Manipulation

Visual object tracking based on discriminant DCT features

We present DVTrack, a visual object tracking method that leverages a Deformable Attention Transformer (DAT) to jointly improve efficiency and accuracy. Recent Transformer-based trackers typically rely on dense global attention over all image tokens, which incurs high computational cost due to redundant processing of background regions. To address these issues, DVTrack explicitly focuses the attention sampling on target-related regions: the deformable attention backbone adaptively selects a small set of informative locations around the object, thereby suppressing redundant background tokens while preserving discriminative target cues. In our Siamese tracking framework, the template and search images are first processed independently by the DAT backbone. The proposed feature aggregation module then fuses these backbone features and effectively models intra- and inter-feature dependencies, where a fusion encoder and a focus decoder enable template features to guide search features toward the target. A lightweight Center and Scale Offset Predictor (CSOP) head then converts these focused features into precise bounding boxes. Extensive experiments on benchmark datasets demonstrate that DVTrack achieves state-of-the-art in LaSOT, TrackingNet, and competitive performance in GOT-10k, UAV123 in terms of success rate and precision. Our model also exhibits robustness against challenges such as occlusions, rapid motion, and background clutter. Furthermore, DVTrack reduces computational complexity, highlighting its potential for real-time applications in visual object tracking. Code is available at this https URL.

Most common methods for proper 3D capture of human movement require a laboratory environment and setting of markers, sensors accessories body segments. However, the high cost of these equipments is a limiting factor in our work environment. Motion capture systems like Microsoft's Kinect TM camera present alternative new approaches to motion capture technology. This paper proposes the design and implementation of a marker-less motion capture system of low cost, using the Microsofts Kinect camera. The system allows the capture of 20 joint points at 30 frames per second (fps) and exports the Cartesian coordinates (x, y, z) of each joint point for further analysis. Furthermore, the system plots in 3D the human skeleton tracking. The motion capture system was tested to imitate human movement with a planar robot of 9 degrees of freedom (dof), another application was to generate trajectories for a planar biped robot and the last application described is the control of final effector of virtual Puma robot from motion capture. Its data presents less accuracy and a lower rate than market equipment. On the other hand, the low-cost of Kinect sensor and easy installation to use in comparison with traditional methods, show it as an excellent alternative for robotic applications that do not require much accuracy.

Generation of human depth images with body part labels for complex human pose recognition

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.11https://github.com/baeckgoo/ir-hand

Kinect has been utilized as a cost-effective, easy-to-use motion capture sensor using the Kinect skeleton algorithm. However, a limited number of landmarks and inaccuracies in tracking the landmarks’ positions restrict Kinect’s capability. In order to increase the accuracy of motion capturing using Kinect, joint use of the Kinect skeleton algorithm and Kinect-based marker tracking was applied to track the 3D coordinates of multiple landmarks on human. The motion’s kinematic parameters were calculated using the landmarks’ positions by applying the joint constraints and inverse kinematics techniques. The accuracy of the proposed method and OptiTrack (NaturalPoint, Inc., USA) was evaluated in capturing the joint angles of a humanoid (as ground truth) in a walking test. In order to evaluate the accuracy of the proposed method in capturing the kinematic parameters of a human, lower body joint angles of five healthy subjects were extracted using a Kinect, and the results were compared to Perception Neuron (Noitom Ltd., China) and OptiTrack data during ten gait trials. The absolute agreement and consistency between each optical system and the robot data in the robot test and between each motion capture system and OptiTrack data in the human gait test were determined using intraclass correlations coefficients (ICC3). The reproducibility between systems was evaluated using Lin’s concordance correlation coefficient (CCC). The correlation coefficients with 95% confidence intervals (95%CI) were interpreted substantial for both OptiTrack and proposed method (ICC > 0.75 and CCC > 0.95) in humanoid test. The results of the human gait experiments demonstrated the advantage of the proposed method (ICC > 0.75 and RMSE = 1.1460°) over the Kinect skeleton model (ICC < 0.4 and RMSE = 6.5843°).

In this paper we present a method for fusing optical flow and inertial measurements. To this end, we derive a novel visual error term which is better suited than the standard continuous epipolar constraint for extracting the information contained in the optical flow measurements. By means of an unscented Kalman filter (UKF), this information is then tightly coupled with inertial measurements in order to estimate the egomotion of the sensor setup. The individual visual landmark positions are not part of the filter state anymore. Thus, the dimensionality of the state space is significantly reduced, allowing for a fast online implementation. A nonlinear observability analysis is provided and supports the proposed method from a theoretical side. The filter is evaluated on real data together with ground truth from a motion capture system.

Semantic simultaneous localization and mapping (SLAM) is susceptible to several sources of noise that hinder the accuracy of its trajectory and map estimates. Such sources include inaccurate landmark pose estimation and sensor limitations. In this paper, a novel deep learning based approach is proposed to improve the accuracy of semantic SLAM by reducing the trajectory estimation error. A deep neural network consisting of various non-linear activation functions is structured and pre-trained by means of an unsupervised greedy layer-wise pre-training technique. The network is then fine-tuned using the adaptive moment estimation (Adam) optimizer. The training datasets were collected using several simulated and realtime experiments and are composed of two parts, the estimated trajectory and the corresponding ground truth. Ground truth trajectories were obtained using a motion capture system in realtime experiments. The effectiveness of the proposed approach was shown through simulated experiments, real-time experiments, and a sequence from the Technical University of Munich (TUM) RGB-D dataset. The performance of the deep neural network (DNN) was tested with different pre-training techniques and the proposed unsupervised greedy layer-wise pre-training technique proved to perform the best across training, validation, and testing datasets in terms of reducing the mean absolute trajectory error (ATE).

The extraction of graph structures in Euclidean vector space is a topic of interest with applications in many fields, such as the analysis of vascular networks in the biomedical domain. While a number of approaches have been proposed to tackle the problem of graph extraction, a quantitative evaluation of those algorithms remains a challenging task: In many cases, manual generation of ground truth for real-world data is time-consuming, error-prone, and thus not feasible. While tools for generating synthetic datasets with corresponding ground truth exist, the resulting data often does not reflect the complexity that real-world scenarios show in morphology and topology. As a complementary or even alternative approach, we propose GERoMe, the graph extraction robustness measure, which provides a means of quantifying the stability of algorithms that extract (multi-)graphs with associated node positions from non-graph structures. Our method takes edge-associated properties into consideration and does not necessarily require ground truth data, although available ground truth information can be incorporated to additionally evaluate the correctness of the graph extraction algorithm. We evaluate the behavior of the proposed graph similarity measure and demonstrate the usefulness and applicability of our method in an exemplary study on both synthetic and real-world data.

This paper addresses issues on moving object tracking from videos. We propose a novel tracking scheme that jointly exploits local object features using consensus point correspondences, and global object appearance and shape models using adaptive particle filter-based eigen-tracking. The paper include the following main novelties: (a) employ consensus feature point correspondences to estimate the motion vector of shape model; (b) employ adaptive particle filters and motioncorrected state vector for joint appearance- and shape-based eigen-tracking. An adaptive number of particles is chosen automatically based on an updated estimation of covariancematrix. Further, online learning is made adaptive to avoid learning using partially-occluded objects. The proposed scheme is realized by integrating SURF and RANSAC [8, 9] for estimating consensus point correspondences, and modify an existing particle filter-based eigen-tracking [4]. Experiment results on tracking moving objects in videos have shown that the proposed scheme provides more accurate tracking, especially for objects with fast motion or long-term partial occlusions. The average number of particles is significantly reduced. Comparisons have been made with an existing method, results have shown that the proposed scheme has provided an improved tracking accuracy at the cost of more computations.

Nowadays, most industrial robots have been designed to be mechanically stiff with rigid link. When a robot with heavy payload is running in fast motions, the residual vibrations of the end-effector are primary caused by joint flexibility. Therefore, the flexibility of joint cannot be neglected. This paper presents a systematic approach to dynamic modeling and residual vibration mode analysis for an industrial robot with rigid links and flexible joints (RLFJ). First, the flexibility of the joint is modeled as a torsional spring and the dynamic equations for this robot are derived by using Lagrange's method. Then, to examine residual vibration properties of the system, numerical simulation is carried out. The following important conclusions are drawn from simulation results: (1) the proposed dynamic model can represent joint flexibility of industrial robot and the joint flexibility causes the residual vibration of the end-effector; (2) for a given RLFJ model, the residual vibration is mainly affected by the payload mass, path of the task and joint stiffness.

Autonomous navigation technologies are continuously improving in terms of performance and safety. A key subfield of autonomous navigation for mobile robots is localization or precise positioning, involving the robot understanding its position relative to key points of interest in the environment. Traditionally, localization is purely dependent on the quality of input received from onboard sensors and therefore localization quality degrades in the presence of high sensor error or other faults. Sensor fusion is useful to combat this. Sensor fusion is used to combine various data types from different sensors to gain a lower overall uncertainty or error in the resulting data, when compared to the separate data from each individual sensor. In this study a four-wheel mobile robot with front-wheel driving and steering is localized during navigation. The investigation is performed in simulation and with a physical prototype. The mobile robot has several sensors including a wheel encoder on each wheel, an inertial measurement unit as well as an indoor GPS system. A designed Kalman Filter called The Combined Kalman Filter has been designed to fuse the sensor input data in order to output the position data required to localize the robot in (X, Y) cartesian coordinates. This is completed in the presence of sensor bias error and noise with the output path compared against a pre-selected ground truth trajectory. The proposed filter is based on the Extended Kalman Filter to contend with the non-linear model arising from the four-wheel steerable robot with the steering system following the Ackermann steering condition. The robot model and sensors are simulated in MATLAB for a chosen ground truth trajectory with results highlighting any detected faults and final accurate position information. It was seen that the estimated filter output had high accuracy when compared with the ground truth. Compared with other investigations the proposed filter has reduced mean error and minimal deviations compared with the ground truth path. In addition, this filtering model can be used in general for the localization of other actuated steering vehicles and aims to widen the research in this field as the majority of research in this area leverages differential drive robots which require less complex modeling.

Event cameras are bioinspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, due to the fundamentally different structure of the sensor's output, new algorithms that exploit the high temporal resolution and the asynchronous nature of the sensor are required. Recent work has shown that a continuous-time representation of the event camera pose can deal with the high temporal resolution and asynchronous nature of this sensor in a principled way. In this paper, we leverage such a continuous-time representation to perform visual-inertial odometry with an event camera. This representation allows direct integration of the asynchronous events with microsecond accuracy and the inertial measurements at high frequency. The event camera trajectory is approximated by a smooth curve in the space of rigid-body motions using cubic splines. This formulation significantly reduces the number of variables in trajectory estimation problems. We evaluate our method on real data from several scenes and compare the results against ground truth from a motion-capture system. We show that our method provides improved accuracy over the result of a state-of-the-art visual odometry method for event cameras. We also show that both the map orientation and scale can be recovered accurately by fusing events and inertial data. To the best of our knowledge, this is the first work on visual-inertial fusion with event cameras using a continuous-time framework.

In this paper, a unified control framework is proposed to realize a robotic ball catching task with only a moving single-camera (eye-in-hand) system able to catch flying, rolling, and bouncing balls in the same formalism. The thrown ball is visually tracked through a circle detection algorithm. Once the ball is recognized, the camera is forced to follow a baseline in the space so as to acquire an initial dataset of visual measurements. A first estimate of the catching point is initially provided through a linear algorithm. Then, additional visual measurements are acquired to constantly refine the current estimate by exploiting a nonlinear optimization algorithm and a more accurate ballistic model. A classic partitioned visual servoing approach is employed to control the translational and rotational components of the camera differently. Experimental results performed on an industrial robotic system prove the effectiveness of the presented solution. A motion-capture system is employed to validate the proposed estimation process via ground truth.

Despite remarkable progress, visual object tracking is still a challenging task as objects usually suffer from significant appearance changes, fast motion, and serious occlusion. In this paper, we propose an anti-occlusion correlation filter-based tracking method (AO-CF) for robust visual tracking. We first propose an occlusion criterion based on continuous response values. Based on the criterion, objects are divided into four categories to adaptively identify the occlusion of objects. Then we propose a new detection condition for detecting proposals. When the occlusion criterion is triggered, the re-detection mechanism is executed and the tracker is commanded to stop, and then the re-detector selects the most reliable proposal to reinitialize the tracker. Experimental results show that our method outperforms other state-of-the-art trackers in terms of both precision rate and success rate on the widely used object tracking benchmark dataset. In addition, AO-CF is able to achieve real-time tracking speed.