An integrated perception pipeline for robot mission execution in unstructured environments

The Army Research Laboratory (ARL) Robotics Collaborative Technology Alliance (CTA) conducted an assessment and evaluation of multiple algorithms for real-time detection of pedestrians in Laser Detection and Ranging (LADAR) and video sensor data taken from a moving platform. The algorithms were developed by Robotics CTA members and then assessed in field experiments jointly conducted by the National Institute of Standards and Technology (NIST) and ARL. A robust, accurate and independent pedestrian tracking system was developed to provide ground truth. The ground truth was used to evaluate the CTA member algorithms for uncertainty and error in their results. A real-time display system was used to provide early detection of errors in data collection.

Cognitive sharing of objects is fundamental in a heterogeneous robot system composed of a Unmanned Aerial Vehicle and a ground robot. Since the viewpoint of a UAV is greatly different from a ground robot, they may have different perceptions about the same objects. That makes it difficult to realize cognitive sharing. In this paper, we proposed a cognitive sharing method which is based on Geometric Relation-based Triangle Representations. The method is able to make a UAV and a ground robot identify the same object from similar objects without sharing appearance information in unstructured environment. To copy with the problem of increasing computational cost for the recognition of objects in the Region of Interest, entropy evaluation is employed to evaluate and select unique representations. We illustrated the proposed method with robots in real world.

Mobile manipulation targets applications in dynamic and unstructured environments. Motion generation methods suitable for these applications must account for end-effector task constraints, must reason about environment uncertainty, i.e. the fact that the exact state of the dynamic environment cannot be known to the robot, and should do so only using their on-board sensors. We present the Expected-Shortest-Path Elastic Roadmap (ESPER) planner as a motion generation method suitable for mobile manipulation. It integrates task-constrained, whole-body, reactive motion generation in high-dimensional configuration space with reasoning about uncertainty. In our experiments, we generate task-consistent motion in uncertain environments on a real-world mobile manipulator only relying on on-board sensors.

Many robotic tasks, such as mobile manipulation, often require interaction with unstructured environments and are subject to imperfect sensing and actuation. This brings substantial uncertainty into the problems. Reasoningunder this uncertainty can provide higher level of robustness but is computationally significantly more challenging. More specifically, sequential decision making under motion and sensing uncertainty can be formulated in a principled form as a Partially Observable Markov Decision Process (POMDP). Solving POMDPs exactly is computationally intractable due to their exponential complexity with the number of states and the depth of the planning horizon, so called, the curse of dimensionality and the curse ofhistory, respectively. In this work, we propose a novel search-based robust planning framework that sample-efficiently finds a solution with theoretical suboptimalitybounds by leveraging multiple heuristics that are designed using domain knowledge. The main contributions of this work can be summarized as follows: 1) it works with generative models, in the absence of mathematical models, similarly to the reinforcement learning paradigm, but 2) it still exhibits high sample-efficiency by bootstrapping with the domain knowledge in the form of heuristics, and 3) it is empowered with effective guidance of the search toward the goal through the systematic employment of multipleheuristics. Its solution can be returned in an anytime fashion, and converges to the bounded suboptimality with theoretical guarantees. We validate the proposed frameworks through simulation and robot experiments,which includes 2D rover navigation, PR2 parts assembly, and full size mobile manipulator truck unloading tasks. The truck unloading task is especially interesting, where the custom-built robot is to unload up to several thousands of boxes from a truck. It is a challenging real-world manipulation problem in continuous space with excessively high uncertainty, and we demonstrate the efficacy of our approach in such a domain.

LADAR is among the pre-eminent sensor modalities for autonomous vehicle navigation. Size, weight, power and cost constraints impose significant practical limitations on perception systems intended for small ground robots. In recent years, the Army Research Laboratory (ARL) developed a LADAR architecture based on a MEMS mirror scanner that fundamentally improves the trade-offs between these limitations and sensor capability. We describe how the characteristics of a highly developed prototype correspond to and satisfy the requirements of autonomous navigation and the experimental scenarios of the ARL Robotics Collaborative Technology Alliance (RCTA) program. In particular, the long maximum and short minimum range capability of the ARL MEMS LADAR makes it remarkably suitable for a wide variety of scenarios from building mapping to the manipulation of objects at close range, including dexterous manipulation with robotic arms. A prototype system was applied to a small (approximately 50 kg) unmanned robotic vehicle as the primary mobility perception sensor. We present the results of a field test where the perception information supplied by the LADAR system successfully accomplished the experimental objectives of an Integrated Research Assessment (IRA).

There is significant motivation to provide robotic systems with improved autonomy as a means to significantly accelerate deactivation and decommissioning (D&D) operations while also reducing the associated costs, removing human operators from hazardous environments, and reducing the required burden and skill of human operators. To achieve improved autonomy, this project focused on the basic science challenges leading to the development of visual servo controllers. The challenge in developing these controllers is that a camera provides 2-dimensional image information about the 3-dimensional Euclidean-space through a perspective (range dependent) projection that can be corrupted by uncertainty in the camera calibration matrix and by disturbances such as nonlinear radial distortion. Disturbances in this relationship (i.e., corruption in the sensor information) propagate erroneous information to the feedback controller of the robot, leading to potentially unpredictable task execution. This research project focused on the development of a visual servo control methodology that targets compensating for disturbances in the camera model (i.e., camera calibration and the recovery of range information) as a means to achieve predictable response by the robotic system operating in unstructured environments. The fundamental idea is to use nonlinear Lyapunov-based techniques along with photogrammetry methods to overcome the complex control issues and alleviate many of the restrictive assumptions that impact current robotic applications. The outcome of this control methodology is a plug-and-play visual servoing control module that can be utilized in conjunction with current technology such as feature recognition and extraction to enable robotic systems with the capabilities of increased accuracy, autonomy, and robustness, with a larger field of view (and hence a larger workspace). The developed methodology has been reported in numerous peer-reviewed publications and the performance and enabling capabilities of the resulting visual servo control modules have been demonstrated on mobile robot and robot manipulator platforms.

Navigating autonomous ground robots through dense and unstructured environments, such as forests, remains a major challenge due to the complexity of natural terrains and the computational burden of traditional navigation pipelines. Conventional approaches often rely on detailed prior maps, rigid rule-based systems, and computationally heavy sensor fusion techniques, which tend to lack generalization across varying environments.In this work, we present a reinforcement learning (RL) strategy for mapless navigation of ground robots in dense forested areas, relying solely on a 3D LiDAR sensor and the position of the target and the agent. The agent is trained using Proximal Policy Optimization (PPO) within a photorealistic simulation framework tailored for outdoor navigation tasks. We adopt a curriculum learning scheme that incrementally increases obstacle density during training.Experimental results in simulation show that the trained agent is capable of navigating challenging forest scenarios effectively, consistently reaching target locations even under high obstacle density. Evaluation based on Success Rate and Success Path Length (SPL) metrics highlights the robustness and adaptability of the learned policy, underscoring its potential for real-world deployment on resource-constrained robotic platforms.

Enabling humanoid robots to perform long-horizon mobile manipulation planning in real-world environments based on embodied perception and comprehension abilities has been a longstanding challenge. With the recent rise of large language models (LLMs), there has been a notable increase in the development of LLM-based planners. These approaches either utilize human-provided textual representations of the real world or heavily depend on prompt engineering to extract such representations, lacking the capability to quantitatively understand the environment, such as determining the feasibility of manipulating objects. To address these limitations, we present the Instruction-Augmented Long-Horizon Planning (IALP) system, a novel framework that employs LLMs to generate feasible and optimal actions based on real-time sensor feedback, including grounded knowledge of the environment, in a closed-loop interaction. Distinct from prior works, our approach augments user instructions into PDDL problems by leveraging both the abstract reasoning capabilities of LLMs and grounding mechanisms. By conducting various real-world long-horizon tasks, each consisting of seven distinct manipulatory skills, our results demonstrate that the IALP system can efficiently solve these tasks with an average success rate exceeding 80%. Our proposed method can operate as a high-level planner, equipping robots with substantial autonomy in unstructured environments through the utilization of multi-modal sensor inputs.

This paper is a summary of an effort to develop a powerful motion planning algorithm for the mobile manipulator. The mobile manipulator is expected to work in partially defined or unstructured environments. In our global/local approach to path planning, joint trajectories are generated for a desired Cartesian space path, designed by the global path planner. For local path planner, inverse kinematics for a redundant system is used. Obstacle avoidance and joint displacement limits for the manipulator links are considered in the motion planner. In an event of failure to obtain feasible trajectories, the task can not be accomplished. In the case of the joint constraint violation, use of Jacobian matrix element as gradient is proposed. At the point of failure, a derivation in the Cartesian space path is obtained and the re-planner gives a new path that would achieve the goal position. To calculate the deviation, a non-linear optimization problem is formulated and solved by standard sequential quadratic programming (SQP) method.

Earthworms have entirely soft bodies mainly composed of circular and longitudinal muscle bundles but can handle the complexity of unstructured environments with exceptional multifunctionality. Soft robots are naturally appropriate for mimicking soft animal structures thanks to their inherent compliance. Here, we explore the new possibility of using this compliance to coordinate the actuation movements of single-type soft actuators for not only high adaptability but the simultaneous multifunctionality of soft robots. A cross-linked actuator coordination mechanism is proposed and explained with a novel conceptual design of a cross-linked network, characterization of modular coordinated kinematics, and a modular control strategy for multiple functions. We model and analyze the motion patterns for these functions, including grabbing, manipulation, and locomotion. This further enables the combination of simultaneous multi-functions with this very simple actuator network structure. In this way, a soft modular robot is developed with demonstrations of a novel continuous-transportation mode, for which multiple objects could be simultaneously transported in unstructured environments with either mobile manipulation or pick-and-place operation. A comprehensive workflow is presented to elaborate the cross-linked actuator coordination concept, analytical modeling, modular control strategy, experimental validation, and multi-functional applications. Our understanding of actuator coordination inspires new soft robotic designs for wider robotic applications.

Vision-Language-Action (VLA) systems are crucial for autonomous decision-making in embodied intelligence. While current systems have advanced the instruction-following capabilities, their limited spatial perception often leads to suboptimal performance for mobile manipulation tasks in unstructured environments. To address this challenge, we propose Uni-Sight, an end-to-end VLA system for robust mobile manipulation. Uni-Sight unifies decision-making, perception, and control through joint training, enabling synchronized cross-component optimization. Within the system, we introduce Latent Feature Aligner (LFA) that ensures accurate target localization by aligning multi-view data. Specifically, we develop Domain Transfer Policy (DTP), a hierarchical policy constrained by LiDAR-guided spatial priors, which ensures 3D spatial understanding with limited visual coverage. Extensive experiments on 20 real-world mobile manipulation tasks demonstrate the high task success rate and robust execution performance of Uni-Sight. Our Uni-Sight achieves a 3.04× the success rate of existing methods, and exhibits superior generalization in both long-horizon and zero-shot scenes. Code and dataset are publicly available at https://github.com/trantor2nd/Uni-Sight.

Results of an experimental study of a motion control algorithm for mobile manipulators are presented. The dynamic interactions between a mobile manipulator and its vehicle are shown to lead to poor performance when a conventional fixed-base controller, which neglects these interactions, is used. The mobile manipulator control algorithm, which accounts for dynamic vehicle motions caused by manipulator motions, is shown to be stable and to perform well, while using only limited sensory data, such as would be practically available in highly unstructured field environments. >

Humanoid robots that have to operate in cluttered and unstructured environments, such as man-made and natural disaster scenarios, require sophisticated sensorimotor capabilities. A crucial prerequisite for the successful execution of whole-body locomotion and manipulation tasks in such environments is the perception of the environment and the extraction of associated environmental affordances, i.e., the action possibilities of the robot in the environment. We believe that such a coupling between perception and action could be a key to substantially increase the flexibility of humanoid robots. In this paper, we approach the affordance-based generation of whole-body actions for stable locomotion and manipulation. We incorporate a rule-based system to assign affordance hypotheses to visually perceived environmental primitives in the scene. These hypotheses are then filtered using extended reachability maps that carry stability information, for identifying reachable affordance hypotheses. We then formulate the hypotheses in terms of a constrained inverse kinematics problem in order to find whole-body configurations that utilize a chosen set of hypotheses. The proposed methods are implemented and tested in simulated environments based on RGB-D scans as well as on a real robotic platform.

To address the low efficiency and poor robustness of robot plug-in charging in unstructured environments, this paper proposes a method for plug-in autonomous charging via a wheeled mobile manipulator in unstructured environments. First, a two-stage paradigm target recognition and positioning algorithm named You Only Look Once version 8 (YOLOv8) and Point Pair Features (PPF) pose estimation is constructed based on YOLOv8 and PPF. By integrating 2D image detection and 3D point cloud processing, it can quickly and accurately recognize and locate charging guns in complex environments. Then, according to the application characteristics of unstructured scenarios, an integrated system of the wheeled mobile manipulator is designed and verified through simulation. Finally, based on the custom-developed wheeled mobile manipulator platform, plug-in autonomous charging experiments are carried out in unstructured scenarios. The results show that in three typical experimental scenarios, namely normal illumination without occlusion, dim light with slight occlusion, and dark with large-area occlusion, the method proposed in this paper can achieve rapid recognition and positioning of the charging gun, meeting the precision requirements that the angle error of the charging gun pose estimation does not exceed $$10^{\circ }$$ and that the distance error does not exceed 15mm; this demonstrates the effectiveness and robustness of the designed plug-in autonomous charging method for the wheeled mobile manipulator.

This paper presents a sensor fusion framework that incorporates a monocular vision system, laser range finder and an inertial navigation sensor (INS) for localization of mobile robots and manipulators. The proposed method is particularly useful for applications in which there is a featured wall or floor in proximity to the robots. Examples include unmanned ground vehicles (UGV), unmanned aerial vehicles (UAV), and field inspection robots. In essence, the proposed method fuses image mosaicing and dead reckoning where the sequential images of a digital camera are stitched together with the help of an inertial navigation system (INS). In the proposed model, the processing load of image mosaicing is reduced significantly, and at the same time accumulation of error of the INS is prevented. The localization method was developed as a standalone module and tested using a hardware-in-the-loop simulator explained in this paper. This module will eventually be used for autonomous navigation of a pipe inspection robot capable of carrying nondestructive testing (NDT) sensors and visual inspection instruments into large water mains.