Spacecraft Pose Estimation Using a Monocular Camera

We propose and demonstrate an efficient pose estimation method, specifically small tilt angle variation, for visual servo control system design using a monocular camera, eliminating the need for iterative computations. This paper addresses the challenges associated with accurately estimating the relative pose between a chaser, equipped with a monocular camera, and a target that provides visual cues, particularly in scenarios where the captured image shows only minimal variations in the target's tilt angle. Traditional approaches often rely on direct feedback methods from image data or designs for pose estimation and control separately, necessitating computationally expensive iterations for pose estimation or complex implementation of visual servoing. To address these challenges, we adopt an extended Kalman filter, known as a robust nonlinear observer, and propose a non‐iterative approach that integrates motion dynamics models and control inputs into pose estimation from image data. This approach enables efficient integration of pose estimation and control solely relying on a monocular camera, reducing computational costs and improving robustness. Experimental validation using the ball‐and‐plate system demonstrates the effectiveness of our proposed method. In our setup, the camera is deliberately positioned directly above the pose estimation target, creating a scenario with minimal image variation in the tilt angle. Despite this condition, our approach achieves efficient and accurate pose estimation, demonstrating its feasibility for high‐speed pose estimation and control without iterative computations.

Spacecraft pose estimation is an important technology to maintain or change the spacecraft orientation in space. For spacecraft pose estimation, when two spacecraft are relatively distant, the depth information of the space point is less than that of the measuring distance, so the camera model can be seen as a weak perspective projection model. In this paper, a spacecraft pose estimation algorithm based on four symmetrical points of the spacecraft outline is proposed. The analytical solution of the spacecraft pose is obtained by solving the weak perspective projection model, which can satisfy the requirements of the measurement model when the measurement distance is long. The optimal solution is obtained from the weak perspective projection model to the perspective projection model, which can meet the measurement requirements when the measuring distance is small. The simulation results show that the proposed algorithm can obtain better results, even though the noise is large.

In the field of close-proximity space operations, RGB-D cameras have been used widely. However, publicly available spacecraft datasets for pose estimation are primarily based solely on image data. There is no publicly available dataset based on RGB-D cameras that can be directly used. Therefore, this paper introduces a novel RGB-D camera based spacecraft (RDCS) dataset. Due to high acquisition costs and confidentiality issues, it is difficult to obtain real spacecraft data and their associated pose information. To address this, the space environment and camera trajectories are constructed in Blender to capture pixel-aligned RGB images and depth maps of spacecraft from various observation angles. In addition, the RDCS dataset also includes pose annotations, bounding box labels, and corresponding mask images. The process focuses on addressing the challenges of acquiring the RGB image and the depth map of the spacecraft at the same keyframe while ensuring their pixel alignment, as well as determining the corresponding pose annotations. Finally, a spacecraft pose estimation network based on the pixel-level fused feature is applied to verify the availability of the RDCS dataset.

The small satellite revolution has opened the door to a new era in satellite development. As spacecraft hardware continues to advance, traditional monolithic satellites can be replaced with cooperative swarms of smaller spacecraft. To enable an effective autonomous mission architecture, swarms of spacecraft will need mechanisms to perform relative navigation for each member. This research presents a solution for the spacecraft identification problem using a neural network architecture to classify space objects and then applies traditional image processing techniques to perform pose estimation using monocular camera images. The trained neural network classifies multiple CubeSats detected in an image into a predefined number of classes using object pixel count and centroid information. The pose estimator then uses the classification to load a predefined wireframe object model and employs an iterative least squares approach to solve the inverse perspective-n-point problem. Preliminary results using a data set with 51,840 points show neural network classification accuracy of ~97% with network training times of less than ten minutes. The position error for spacecraft with a relative range of ~50 meters is typically less than one meter in the X and Y directions and around ~3 meters in the Z direction.

In this paper, a robust pose (i.e., position and orientation) estimation algorithm using two-views captured by a calibrated monocular camera is presented. A collection of pose hypotheses is obtained when more than the minimum number of feature points required to uniquely identify a pose are available in both the images. The pose hypotheses - unit quaternion and unit translation vectors - lie on the S3 and S2 manifolds in the Euclidean 4-space and 3-space, respectively. Probability density function (pdf) of the rotation and translation pose hypotheses is evaluated by gridding the unit spheres where a robust, coarse pose estimate is identified at the mode of the pdf. Further, a "refining" pdf of the geodesic distance from the coarse pose estimate is constructed for the hypotheses within a grid containing the coarse estimate. A refined pose estimate is obtained by averaging the low-noise hypotheses in the neighbourhood of the mode of refining pdf. Pose estimation results of the proposed method are compared with RANSAC and nonlinear mean-shift (NMS) algorithms using the Oxford Corridor sequence and the robustness to feature outliers, image noise rejection, and scalability to number of features is analyzed using the synthetic data experiments. Processing time comparison with the RANSAC and NMS algorithms indicate that the deterministic time requirement of the proposed and NMS algorithms is amenable to a variety of visual servo control applications.

Pose estimation of human–machine interactions such as bicycling plays an important role to understand and study human motor skills. In this paper, we report the development of a human whole-body pose estimation scheme with application to rider–bicycle interactions. The pose estimation scheme is built on the fusion of measurements of a monocular camera on the bicycle and a set of small wearable gyroscopes attached to the rider's upper- and lower-limbs and the trunk. A single feature point is collocated with each wearable gyroscope and also on the body segment link where the gyroscope is not attached. An extended Kalman filter (EKF) is designed to fuse the visual-inertial measurements to obtain the drift-free whole-body poses. The pose estimation design also incorporates a set of constraints from human anatomy and the physical rider–bicycle interactions. The performance of the estimation design is validated through ten subject riding experiments. The results illustrate that the maximum errors for all joint angle estimations by the proposed scheme are within 3 degs. The pose estimation scheme can be further extended and used in other types of physical human–machine interactions.

Light detection and ranging (LiDAR) sensors provide accurate 3-D point clouds for noncooperative spacecraft pose estimation. Several robust methods, such as iterative closest point, exist to perform a local refinement of the pose starting from an initial estimate. However, finding the initial pose of the spacecraft is a global optimization problem, which is challenging to solve in real time. This is especially true on space hardware with limited computing power. In addition, many spacecrafts have a shape with multiple symmetries, making an unambiguous initial pose estimation impossible. This work introduces a convolutional-neural-network-based pose estimation method, accounting for potential symmetries of the target satellite. The point clouds are projected to a 2-D depth image before being processed by the network. To generate a sufficient amount of training data, a LiDAR simulator integrating multiple effects such as reflections or laser beam divergence is developed. While being trained solely on synthetic point clouds, the pose estimation method shows to be precise, efficient, and reliable when evaluated on real point clouds taken at a hardware-in-the-loop rendezvous test facility. A runtime evaluation on potential space computing hardware is also performed to demonstrate the applicability of the method to real-time onboard pose estimation.

Multi-robot clusters are increasingly deployed in indoor environments, where effective communication and 3D perception are critical for coordinated operations. Monocular cameras, known for their lightweight design, cost-effectiveness, and versatility, present a promising solution for these tasks. However, relying solely on monocular cameras for comprehensive perception and communication presents significant challenges. To address this, we introduce MonoLDP, a novel system that leverages monocular cameras for depth estimation, mutual pose estimation, and visible light communication in indoor environments, providing an integrated framework to overcome these limitations. MonoLDP features a two-stage network: (1) a depth estimation module that infers depth from monocular images, and (2) a depth-guided 3D object recognition network for agent-relative localization and pose estimation. We created a custom dataset to validate the accuracy of MonoLDP. On our indoor dataset, MonoLDP outperforms the baseline by 43.39% in 3D detection and 42.39% in bird's-eye view detection, with an average localization error of 0.104 m and an orientation error of 1.66 degrees. Moreover, the depth estimation network demonstrates excellent performance on the NYU v2 dataset. Additionally, the system achieves a communication rate of 1.2 Kbps with a bit error rate below 10<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> at a distance of up to 4 m using LED arrays. Our code will be released at https://github.com/RavenLiang1005/MonoLDP.git.

In this paper, we present an approach to enable a humanoid robot to autonomously climb up spiral staircases. This task is substantially more challenging than climbing straight stairs since careful repositioning is needed. Our system globally estimates the pose of the robot, which is subsequently refined by integrating visual observations. In this way, the robot can accurately determine its relative position with respect to the next step. We use a 3D model of the environment to project edges corresponding to stair contours into monocular camera images. By detecting edges in the images and associating them to projected model edges, the robot is able to accurately locate itself towards the stairs and to climb them. We present experiments carried out with a Nao humanoid equipped with a 2D laser range finder for global localization and a low-cost monocular camera for short-range sensing. As we show in the experiments, the robot reliably climbs up the steps of a spiral staircase.

This paper presents a comprehensive solution for distance estimation of the following vehicle solely based on visual data from a low-resolution monocular camera. To this end, a pair of vehicles were instrumented with real-time kinematic (RTK) GPS, and the lead vehicle was equipped with custom devices that recorded video of the following vehicle. Forty trials were recorded with a sedan as the following vehicle, and then the procedure was repeated with a pickup truck in the following position. Vehicle detection was then conducted by employing a deep-learning-based framework on the video footage. Finally, the outputs of the detection were used for following distance estimation. In this study, three main methods for distance estimation were considered and compared: linear regression model, pinhole model, and artificial neural network (ANN). RTK GPS was used as the ground truth for distance estimation. The output of this study can contribute to the methodological base for further understanding of driver following behavior with a long-term goal of reducing rear-end collisions.

A novel method for monocular-based pose estimation of uncooperative spacecraft using keypoints specialized for a given target is presented. A set of robust keypoints are created by examining the effectiveness of existing localization algorithms by simulating and testing different perspectives. The feature extraction and matching is used to build a model of the spacecraft before the flight mission using the same feature extraction algorithms that can be used during the mission. Further, a visibility map is determined for each keypoint to aid in outlier filtering, matching, and measurement covariance estimation. For initialization and matching, a Convolutional Neural Network (CNN) is trained to generate descriptors robust to illumination, scale, and affine changes for the pre-computed keypoints. In the second part of the paper, we focus on pose determination and filtering after keypoint-to-model matching. While several approaches for pose acquisition have been formulated, we propose a novel method for tracking that makes use of a nonlinear filter, based on the spacecraft translational and rotational relative dynamics which estimates the covariance of the vision-based observations using the keypoint preprocessing information. Further, the estimated propagated covariance for each extracted feature is used for aiding the feature matching.

This paper presents a metric global localization in the urban environment only with a monocular camera and the Google Street View database. We fully leverage the abundant sources from the Street View and benefits from its topo-metric structure to build a coarse-to-fine positioning, namely a topological place recognition process and then a metric pose estimation by local bundle adjustment. Our method is tested on a 3 km urban environment and demonstrates both sub-meter accuracy and robustness to viewpoint changes, illumination and occlusion. To our knowledge, this is the first work that studies the global urban localization simply with a single camera and Street View.

Monocular Visual Odometry (MVO) provides a cost-effective, real-time positioning solution for autonomous vehicles. However, MVO systems face the common issue of lacking inherent scale information from monocular cameras. Traditional methods have good interpretability but can only obtain relative scale and suffer from severe scale drift in long-distance tasks. Learning-based methods under perspective view leverage large amounts of training data to acquire prior knowledge and estimate absolute scale by predicting depth values. However, their generalization ability is limited due to the need to accurately estimate the depth of each point. In contrast, we propose a novel MVO system called BEV-DWPVO. Our approach leverages the common assumption of a ground plane, using Bird's-Eye View (BEV) feature maps to represent the environment in a grid-based structure with a unified scale. This enables us to reduce the complexity of pose estimation from 6 Degrees of Freedom (DoF) to 3-DoF. Keypoints are extracted and matched within the BEV space, followed by pose estimation through a differentiable weighted Procrustes solver. The entire system is fully differentiable, supporting end-to-end training with only pose supervision and no auxiliary tasks. We validate BEV-DWPVO on the challenging long-sequence datasets NCLT, Oxford, and KITTI, achieving superior results over existing MVO methods on most evaluation metrics.

In this paper, a fully autonomous quadrotor in a heterogeneous air–ground multi-robot system is established only using minimal on-board sensors: a monocular camera and inertial measurement units (IMUs). Efficient pose and motion estimation is proposed and optimized. A continuous-discrete extended Kalman filter is applied, in which the high-frequency IMU data drive the prediction, while the estimates are corrected by the accurate and steady vision data. A high-frequency fusion at 100 Hz is achieved. Moreover, time delay analysis and data synchronizations are conducted to further improve the pose/motion estimation of the quadrotor. The complete on-board implementation of sensor data processing and control algorithms reduces the influence of data transfer time delay, enables autonomous task accomplishment and extends the work space. Higher pose estimation accuracy and smaller control errors compared to the standard works are achieved in real-time hovering and tracking experiments.

Depth estimation is usually ill-posed and ambiguous for monocular camera-based 3D multi-person pose estimation. Since LiDAR can capture accurate depth information in long-range scenes, it can benefit both the global localization of individuals and the 3D pose estimation by providing rich geometry features. Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light. Specifically, we design an effective fusion strategy to take advantage of multi-modal input data, including images and point cloud, and make full use of temporal information to guide the network to learn natural and coherent human motions. Without relying on any 3D pose annotations, our method exploits the inherent geometry constraints of point cloud for self-supervision and utilizes 2D keypoints on images for weak supervision. Extensive experiments on public datasets and our newly collected dataset demonstrate the superiority and generalization capability of our proposed method. Project homepage is at \url{https://github.com/4DVLab/FusionPose.git}.