Autonomous Plug-in Charging for Wheeled Mobile Manipulator in Unstructured Environments

Human-robot collaboration (HRC) significantly extends robotic systems’ applications when working in spaces like houses, hospitals, or laboratories. However, new challenges appear during a close collaboration between humans and robots and imitating the movement of humans by robots. Learning from demonstration (LfD), or kinesthetic teaching, is a popular approach to help teach a robot human behavior by demonstrations without the need to explicitly reprogram the robot for different procedures. In this paper, we propose a method for object manipulation, including lifting, carrying, and lowering the object through a collaboration of a human with a wheeled mobile manipulator (WMM). The WMM is first trained with the help of a human demonstrator to collaborate with the user to execute the task. Then, the WMM will independently cooperate with the user by reproducing the learned skills to perform the same task. The redundancy of the WMM will also be employed to enhance its force exertion capability in the vertical direction to offset the object’s weight. The advantages and effectiveness of the proposed method are investigated through experiments.

Research on autonomous driving in unstructured outdoor environments is less advanced than in structured urban settings due to challenges like environmental diversities and scene complexity. These environments–such as rural areas and rugged terrains–pose unique obstacles that are not common in structured urban areas. Despite these difficulties, autonomous driving in unstructured outdoor environments is crucial for applications in agriculture, mining, and military operations. Our survey reviews over 250 papers for autonomous driving in unstructured outdoor environments, covering offline mapping, pose estimation, environmental perception, path planning, end‐to‐end autonomous driving, datasets, and relevant challenges. We also discuss emerging trends and future research directions. This review aims to consolidate knowledge and encourage further research for autonomous driving in unstructured environments. To support ongoing work, we maintain an active repository of up‐to‐date literature and open‐source projects at: https://github.com/chaytonmin/Survey‐Autonomous‐Driving‐in‐Unstructured‐Environments .

This paper presents an environmental representation, called T-Net (Target Network), for robot navigation. This representation navigates a mobile robot to move in real time in an unstructured environment. The robot approximates local areas with straight paths and memorizes the paths with pairs of feature points. The T-Net consisting of the paths represents rigid environmental structure and keeps relations between the local areas. By tracking the feature points, the robot can move along the T-Net and investigate the environment in detail. The authors have applied this idea to their mobile robot which has multiple vision agents, and verified it through simulations and experimentations in a real world.

The autonomous vehicle is an automated system equipped with features like environment perception, decision-making, motion planning, and control and execution technology. Navigating in an unstructured and complex environment is a huge challenge for autonomous vehicles, due to the irregular shape of road, the requirement of real-time planning, and the nonholonomic constraints of vehicle. This paper presents a motion planning method, based on the Radial Basis Function (RBF) neural network, to guide the autonomous vehicle in unstructured environments. The proposed algorithm extracts the drivable region from the perception grid map based on the global path, which is available in the road network. The sample points are randomly selected in the drivable region, and a gradient descent method is used to train the RBF network. The parameters of the motion-planning algorithm are verified through the simulation and experiment. It is observed that the proposed approach produces a flexible, smooth, and safe path that can fit any road shape. The method is implemented on autonomous vehicle and verified against many outdoor scenes; furthermore, a comparison of proposed method with the existing well-known Rapidly-exploring Random Tree (RRT) method is presented. The experimental results show that the proposed method is highly effective in planning the vehicle path and offers better motion quality.

In this paper, we tackle the problem of category-level 9D pose estimation in the wild, given a single RGB-D frame. Using supervised data of real-world 9D poses is tedious and erroneous, and also fails to generalize to unseen scenarios. Besides, category-level pose estimation requires a method to be able to generalize to unseen objects at test time, which is also challenging. Drawing inspirations from traditional point pair features (PPFs), in this paper, we design a novel Category-level PPF (CPPF) voting method to achieve accurate, robust and generalizable 9D pose estimation in the wild. To obtain robust pose estimation, we sample numerous point pairs on an object, and for each pair our model predicts necessary SE(3)-invariant voting statistics on object centers, orientations and scales. A novel coarse-to-fine voting algorithm is proposed to eliminate noisy point pair samples and generate final predictions from the population. To get rid of false positives in the orientation voting process, an auxiliary binary disambiguating classification task is introduced for each sampled point pair. In order to detect objects in the wild, we carefully design our sim-to-real pipeline by training on synthetic point clouds only, unless objects have ambiguous poses in geometry. Under this cir-cumstance, color information is leveraged to disambiguate these poses. Results on standard benchmarks show that our method is on par with current state of the arts with real-world training data. Extensive experiments further show that our method is robust to noise and gives promising results under extremely challenging scenarios. Our code is available on https://github.com/qq456cvb/CPPF.

This paper presents a method to potentate the multi-segmented inspection robot Kairo-II to navigate in unstructured and dynamic environment. Previous methods for motion planning for such robots come from driving scenarios in highly structured areas. The virtual tube algorithm is introduced which enables a multi-segmented robot to range in such complex environment. Precise force feedback is required. Therefore, we present a sensor system which is based on strain-gauges technology. Information extracted by this sensor enables the trajectory planning algorithm to adapt its curve. Thus, the proposed system provides and evaluates key functions for compliant motion of a multi-segmented robot within unstructured environment.

This paper investigates a new method of modeling and simulation of a two wheeled Mobile Manipulators (MMs) system each equipped with a 6 DOF arm. Such system modeled in Matlab/Simulink (Simmechanic) environment is used to transport objects without explicit communication between the MMs. MMs cooperation control has received big interest in the last few years and has been suggested for various applications such as tasks involving hazardous environments, explosive handling, and space operations. However most research in MMs cooperation has been performed within 2D flat terrain environment. In this paper MMs are considered working in 3D space and we use the nonholonomic characteristics of the robotic manipulators which has been neglected in previous work. This paper describes a control algorithm for two MMs cooperation executing cooperation’s tasks. The proposed approach uses common MM sensors. The paper includes the description of the proposed control mechanism that enables ground mobile manipulators to execute complex tasks in cooperation’s.

We present a revised pipe-line of the existing 3D object detection and pose estimation framework based on point pair feature matching. This framework proposed to represent 3D target object using self-similar point pairs, and then matching such model to 3D scene using efficient Hough-like voting scheme operating on the reduced pose parameter space. Even though this work produces great results and motivated a large number of extensions, it had some general shortcoming like relatively high dimensionality of the search space, sensitivity in establishing 3D correspondences, having performance drops in presence of many outliers and low density surfaces. In this paper, we explain and address these drawbacks and propose new solutions within the existing framework. In particular, we propose to couple the object detection with a coarse-to-fine segmentation, where each segment is subject to disjoint pose estimation. During matching, we apply a weighted Hough voting and an interpolated recovery of pose parameters. Finally, all the generated hypothesis are tested via an occlusion-aware ranking and sorted. We argue that such a combined pipeline simultaneously boosts the detection rate and reduces the complexity, while improving the accuracy of the resulting pose. Thanks to such enhanced pose retrieval, our verification doesn't necessitate ICP and thus achieves better compromise of speed vs. accuracy. We demonstrate our method on existing datasets as well as on our scenes. We conclude that via the new pipe-line, point pair features can now be used in more challenging scenarios.

Bin picking refers to picking the randomly-piled objects from a bin for industrial production purposes, and robotic bin picking is always used in automated assembly lines. In order to achieve a higher productivity, a fast and robust pose estimation algorithm is necessary to recognize and localize the randomly-piled parts. This paper proposes a pose estimation algorithm for bin picking tasks using point cloud data. A novel descriptor Curve Set Feature (CSF) is proposed to describe a point by the surface fluctuation around this point and is also capable of evaluating poses. The Rotation Match Feature (RMF) is proposed to match CSF efficiently. The matching process combines the idea of the matching in 2D space of origin Point Pair Feature (PPF) algorithm with nearest neighbor search. A voxel-based pose verification method is introduced to evaluate the poses and proved to be more than 30-times faster than the kd-tree-based verification method. Our algorithm is evaluated against a large number of synthetic and real scenes and proven to be robust to noise, able to detect metal parts, more accurately and more than 10-times faster than PPF and Oriented, Unique and Repeatable (OUR)-Clustered Viewpoint Feature Histogram (CVFH).

This paper presents a learning-based model predictive trajectory planning controller for automated driving in unstructured, dynamic environments with obstacle avoidance. We first address the problem of lacking prior knowledge in unstructured environments by introducing a risk map that maps the density and motion of obstacles and the road to an occupancy risk. Model predictive control is then used to integrate trajectory planning and tracking control into one framework to bridge the gap between planning and control. Meanwhile, we use Gaussian Process (GP) regression to learn the residual model uncertainty for improving the model accuracy. An objective function considering both risks within the feasible region and vehicle dynamics is carefully formulated to obtain collision-free and kinematically-feasible local trajectories. Field experiments are performed on real unstructured environments with our automated vehicle. Experimental results demonstrate the effectiveness of the proposed algorithm for successful obstacle avoidance in various complex unstructured scenarios.

In this paper we present an artificial vision algorithm for real-time obstacle detection in unstructured environments. The images have been taken using a stereoscopical vision system. The system uses a new approach, of low computational load, to calculate a V-disparity image between left and right corresponding images, in order to estimate the cameras pitch oscillation caused by the vehicle movement. Then, the obstacles are localized by stereo matching and mapped in real world coordinates. Experimental results on sequences taken from a moving vehicle (which partecipated to the DARPA Grand Challenge 2004) in different unstructured scenarios are then presented, to demonstrate the validity of the approach.

Pose estimation is a particularly important link in the task of robotic bin-picking. Its purpose is to obtain the 6D pose (3D position and 3D posture) of the target object. In real bin-picking scenarios, noise, overlap, and occlusion affect accuracy of pose estimation and lead to failure in robot grasping. In this paper, a new point-pair feature (PPF) descriptor is proposed, in which curvature information of point-pairs is introduced to strengthen feature description, and improves the point cloud matching rate. The proposed method also introduces an effective point cloud preprocessing, which extracts candidate targets in complex scenarios, and, thus, improves the overall computational efficiency. By combining with the curvature distribution, a weighted voting scheme is presented to further improve the accuracy of pose estimation. The experimental results performed on public data set and real scenarios show that the accuracy of the proposed method is much higher than that of the existing PPF method, and it is more efficient than the PPF method. The proposed method can be used for robotic bin-picking in real industrial scenarios.

Image-based endoscopy pose estimation has been shown to significantly improve the visualization and accuracy of minimally invasive surgery (MIS). This paper proposes a method for pose estimation based on structure-depth information from a monocular endoscopy image sequence. Firstly, the initial frame location is constrained using the image structure difference (ISD) network. Secondly, endoscopy image depth information is used to estimate the pose of sequence frames. Finally, adaptive boundary constraints are used to optimize continuous frame endoscopy pose estimation, resulting in more accurate intraoperative endoscopy pose estimation. Evaluations were conducted on publicly available datasets, with the pose estimation error in bronchoscopy and colonoscopy datasets reaching 1.43 mm and 3.64 mm, respectively. These results meet the real-time requirements of various scenarios, demonstrating the capability of this method to generate reliable pose estimation results for endoscopy images and its meaningful applications in clinical practice. This method enables accurate localization of endoscopy images during surgery, assisting physicians in performing safer and more effective procedures.

This study focuses on the design of intelligent equipment for efficient operation in unstructured environments, concentrating on the construction and performance validation of a “mechanical execution-intelligent control” collaborative system. It emphasizes embedded control, sensor integration, and multi-scenario data-driven performance optimization solutions. First, an intelligent control system centered on embedded control is designed, comprising: - A long-range remote control module (utilizing MC6C/MC7RB communication units based on the 2.4GHz ISM band, enabling interference-resistant command transmission and status feedback over 800 meters) - An identification spotlight module (integrating the K210 visual recognition module, performing typical target detection via CNN Accelerator, and featuring high-efficiency LED lighting for auxiliary perception) and a scanning ranging module (utilizing the N10 LiDAR for 360° two-dimensional scanning based on TOF principles, with a 20,000 measurements/second frequency ensuring ±3cm accuracy in environmental data acquisition). All modules form a “perception-decision-execution” data closed-loop via standardized communication protocols. Second, hardware-intelligent control adaptation design was completed. This involved optimizing power unit hardware interfaces (drive adaptation between brushless DC motors and duct structures), modular partitioning of the control cabin (embedded layout integrating SLAM LiDAR and vision modules), and quick-release connection structures. These provide stable physical carriers for intelligent modules, ensuring efficient command execution. Finally, multi-scenario physical testing and data validation were conducted. Data from various operating conditions—collected via infrared speed measurement, GPS positioning, and attitude sensors—was used to calibrate metrics including water resistance, speed, and steering performance. This validated the intelligent modules' data transmission stability, environmental prediction safety, and system coordination adaptability, proving the equipment meets rapid deployment requirements in complex environments.

An increasingly important trend is to simulate a human arm using a 7-degrees-of-freedom (DOF) redundant manipulator and accomplish complex tasks in an unstructured environment by leveraging human experience. Human arm motion information acquired by a biomimetic motion capture system is mapped to planning information of the manipulator to leverage human experience in the control process and quickly improve the operational capability of the manipulator in complex environments. In this paper, extensive tests are performed for verification, and the test results prove the effectiveness of the proposed algorithm, which improve the operational performance of a redundant manipulator in a complex environment to accomplish humanoid tasks and is critical to robot-aided operations or the complete substitution of humans by robots in the future.