A three-finger gripper for manipulation in unstructured environments

Humanoid robotic manipulation in unstructured environments is a challenging problem. Limited perception, communications and environmental constraints present challenges that prevent fully autonomous or purely teleoperated robots from reliably interacting with their environment. In order to achieve higher reliability in manipulation we present an approach involving remote human supervision. Strengths from both human operator and humanoid robot are leveraged through a user interface that allows the operator to perceive the remote environment through an aggregated worldmodel based on onboard sensing, while the robot can efficiently receive perceptual and semantic information from the operator. A template based manipulation approach has been successfully applied to the Atlas humanoid robot; we show real world footage of the results obtained in the DARPA Robotics Challenge Trials 2013.

This paper presents the design, modeling and control of a compact wrist, which can work in active mode with position or torque control, or passive mode with interactive force compensation. This characteristic makes it suitable for dexterous manipulation in unstructured environments, such as door opening. Mechanism design is presented briefly, and forward, inverse as well as differential kinematics are derived to map the motions and velocities between the task space and joint space. A distributed robust adaptive controller is developed for tracking control of the wrist in active mode; and a new interactive force compensation technique is proposed on the basis of force sensor measurement, which enables passive working mode of the compact wrist. A DSP based control system is developed to test the developed control methods. A prototype has been fabricated and some preliminary experiments have been conducted to validate the proposed design and to verify the developed algorithms.

Conventional suction cups lack sensing capabilities for contact-aware manipulation in unstructured environments. This paper presents FlexiCup, a multimodal suction cup with wireless electronics that integrate dual-zone vision-tactile sensing. The central zone dynamically switches between vision and tactile modalities via illumination control, while the peripheral zone provides continuous spatial awareness. The modular mechanical design supports both vacuum (sustained-contact adhesion) and Bernoulli (contactless lifting) actuation while maintaining the identical dual-zone sensing architecture, demonstrating sensing-actuation decoupling where sensing and actuation principles are orthogonally separable. We validate hardware versatility through dual control paradigms. Modular perception-driven grasping achieves comparable success rates across vacuum (90.0%) and Bernoulli (86.7%) modes using identical sensing and control pipelines, validating the sensing architecture's effectiveness across fundamentally different pneumatic principles. Diffusion-based end-to-end learning achieves 73.3% and 66.7% success on contact-aware manipulation tasks, with ablation studies confirming 13% improvements from multi-head attention coordinating dual-zone observations. Hardware designs, firmware, and experimental videos are available at the companion website: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://flexicup.junhaogong.top</uri>.

Controlled grasping using soft robotic grippers is a significant challenge that hinders their deployment in unstructured environments. This work presents a directly three-dimensional (3D) printed soft pneumatic finger that involves separate chambers for soft pneumatic-based control, actuation, and sensing (SoPCAS) simultaneously. The actuation chamber of the finger is designed as a pneumatic network that expands upon pressurization, and its sensing chamber is designed as a thin wall soft pneumatic chamber embedded with a solid air pressure sensor that measures the volume change ( i.e ., pressure change) when the finger is in contact with its environment. The soft finger is 3D printed using a commercially available thermoplastic polyurethane (TPU) and a fused deposition modeling (FDM) 3D printer. The SoPCAS finger is characterized in terms of repeatability, hysteresis, force–pressure relationship, and stability ( i.e ., airtightness). Two SoPCAS fingers are used to develop a gripper that highlights the usage of the actuation and sensing components in real-time closed-loop feedback grasping applications using a simple and manually tuned proportional-integral (PI) controller. The gripper demonstrates successful grasping of various objects with different weights, shapes, textures, and stiffnesses, making SoPCAS finger a promising candidate for applications requiring real-time force/pressure control in highly dynamic and unstructured environments.

Robotic fingers made of soft material and compliant structures usually lead to superior adaptation when interacting with the unstructured physical environment. In this paper, we present an embedded sensing solution using optical fibers for an omni-adaptive soft robotic finger with exceptional adaptation in all directions. In particular, we managed to insert a pair of optical fibers inside the finger's structural cavity without interfering with its adaptive performance. The resultant integration is scalable as a versatile, low-cost, and moisture-proof solution for physically safe human-robot interaction. In addition, we experimented with our finger design for an object sorting task and identified sectional diameters of 94\% objects within the $\pm$6mm error and measured 80\% of the structural strains within $\pm$0.1mm/mm error. The proposed sensor design opens many doors in future applications of soft robotics for scalable and adaptive physical interactions in the unstructured environment.

The practical application of machine perception to support physical manipulation in unstructured environments remains a barrier to the development of intelligent robotic systems. Recently, great progress has been made by the large-scale machine perception community, but these methods have made few contributions to the applied robotic perception. This is in part because such large-scale systems are designed to recognize category labels of large numbers of objects from a single image, rather than highly accurate, efficient, and robust pose estimation in environments for which a robot has reliable prior knowledge. In this paper, we illustrate the potential for synergistic integration of modern computer vision methods into robotics by augmenting a RANSAC-based registration method with a state-of-the art semantic segmentation algorithm. We detail a convolutional architecture for semantic labeling of the scene, modified to operate efficiently using integral images. We combine this labeling with two novel scene parsing variants of RANSAC, and show, on a new RGB-D dataset that contains complex configurations of textureless and highly specular objects, that our method demonstrates improved performance of pose estimation over the unaugmented algorithms.

In unstructured environments, robots face challenges in efficiently and accurately grasping irregular, fragile objects. To address this, this paper introduces a soft robotic hand tailored for such settings and enhances You Only Look Once v5s (YOLOv5s), a lightweight detection algorithm, to achieve efficient grasping. A rapid pneumatic network-based soft finger structure, broadly applicable to various irregularly placed objects, is designed, with a mathematical model linking the bending angle of the fingers to input gas pressure, validated through simulations. The YOLOv5s model is improved by integrating the Coordinate Attention (CA) mechanism in the backbone layer, refining the Spatial Pyramid Pooling (SPP) module for faster detection, and adjusting the loss function to prevent misalignment between predicted and actual bounding boxes, thereby enhancing computational efficiency. Experimental comparative analysis indicates that the refined model exhibits improvements in both mean average precision and recognition speed. A soft robotic grasping experimental platform was established, with precision grasping and power grasping experiments conducted using the pose and object type data generated by the enhanced YOLOv5s-CA-SPP model network. The results show that the success rate of grabbing reaches 82% with a proper grabbing posture.

The evolution of robotics technology continues to facilitate exploration and scientific study in remote environments, enabling research in areas that were previously impossible to reach. Robots operating in space and marine environments encounter similar operational challenges, as both face high operational costs, bandwidth-limited conditions, and natural, unstructured environments where dynamic obstacles might be present. Within the oceanographic domain, conventional deep-sea sampling operations involve remotely operated vehicles (ROVs) equipped with robotic manipulator arms to complete dexterous tasks at depth. While effective, deep-sea ROV operations require specialized instrumentation, highly trained shipboard personnel, and large oceanographic vessels, which make deep-sea samples inaccessible to most. This thesis presents the SHared Autonomy for Remote Collaboration (SHARC) framework, and evaluates its utility within an oceanographic context. By leveraging shared autonomy, SHARC enables shore-side operators to collaboratively carry out underwater sampling and manipulation tasks, regardless of their prior manipulator operations experience. With SHARC, operators can conduct manipulation tasks using natural language and hand gestures through a virtual reality (VR) interface. The interface provides remote operators with a contextual 3D scene understanding that is updated according to bandwidth availability. Evaluation of the SHARC framework through controlled lab experiments indicates that SHARC’s VR interface enables novice operators to complete manipulation tasks in framerate-limited conditions (i.e., &lt;0.5 frames per second) faster than expert pilots using the conventional topside controller. For both novice and expert users, the VR interface also increased the task completion rate and improved sampling precision. During sea trials, SHARC enabled collection of an underwater in-situ X-ray fluorescence (XRF) measurement at more than 1000 meters water depth in the Eastern Pacific with centimeter-level precision by remote scientists with no prior piloting experience. This demonstration provides compelling evidence of SHARC’s utility for conducting delicate operations in unstructured environments across bandwidthlimited communications, which holds relevance for improving operations in other sensitive domains where dexterity is required. SHARC’s ability to relax infrastructure requirements and engage novice shore-side users provides a promising avenue for democratizing access to deep-sea research.

Visual perception has become core technology in autonomous robotics to identify and localize objects of interest to ensure successful and safe task execution. As part of the recently concluded Robotics Collaborative Technology Alliance (RCTA) program, a collaborative research effort among government, academic, and industry partners, a vision acquisition and processing pipeline was developed and demonstrated to support manned-unmanned team ing for Army relevant applications. The perception pipeline provided accurate and cohesive situational awareness to support autonomous robot capabilities for maneuver in dynamic and unstructured environments, collaborative human-robot mission planning and execution, and mobile manipulation. Development of the pipeline involved a) collecting domain specific data, b) curating ground truth annotations, e.g., bounding boxes, keypoints, c) retraining deep networks to obtain updated object detection and pose estimation models, and d) deploying and testing the trained models on ground robots. We discuss the process of delivering this perception pipeline under limited time and resource constraints due to lack of a priori knowledge of the operational environment. We focus on experiments conducted to optimize the models despite using data that was noisy and exhibited sparse examples for some object classes. Additionally, we discuss our augmentation techniques used to enhance the data set given skewed class distributions. These efforts highlight some initial work that directly relates to learning and updating visual perception systems quickly in the field under sudden environment or mission changes.

The consideration of data set design, collection, and distribution methodology is becoming increasingly important as robots move out of fully controlled settings, such as assembly lines, into unstructured environments. Extensive knowledge bases and data sets will potentially offer a means of coping with the variability inherent in the real world. In this study, we introduce three new data sets related to mobile manipulation in human environments.

Robotics has been a paradigm for science in the last few decades. At first, scientists’ efforts were devoted to the solution of the problem of planning and control of themotion of robot manipulators. However, the motion control of robot manipulators in unstructured environments is today an attractive scientific problem. An interesting solution for motion control is the use of sensor information, such as computer vision, in the system’s feedback. Several works and tools have been developed in recent years in this field (Corke; 2005), (Chaumette and Hutchinson; 2006). The more typical approaches consider visual perception for servoing and for the so called look and move (Hutchinson et al.; 1996). Visual servoing (Kelly et al.; 2000) can be classified into two approaches: camera-in-hand or camera-to-hand (Flandin et al.; 2000). In camera-to-hand robotic systems, multiple cameras or a single camera fixed in the worldcoordinate frame capture images of both, the robot and its environment. The tracking of the object with visual feedback can be made in 2D or 3D. An interesting solution for the visual servoing of camera-in hand robot manipulators in 2D can be found in e. g. (Bonfe et al.; 2002) and (Hernandez et al.; 2008) where stability demonstration of a decoupled controller have been presented. For 3D tracking some solutions reported are look and move controller, with one camera (Sim et al.; 2002) or more than one camera (Xie et al.; 2005). A 3D visual servoing with stability analysis in continuous time is presented by (Hernandez et al.; 2008a) and (Kelly et al.; 2006) present a direct visual servoing with transpose Jacobian control technique for regulation of robot manipulators in the 3D Cartesian space. In a similar vein (Enescu et al.; 2006) present mobile robot navigation for person tracking using a stereo head-camera. In this chapter, we consider the control problem of camera-in-hand robot manipulators in 2D and 3D. In both cases only one camera is mounted on the robot’s arm, which supplies visual information of the environment, with the aim of moving the manipulator by maintaining the image of the tracked object (a sphere) in the centre of the image plane, despite the possible movements of the object. In the 3D control the constant radius is used as a feature too. In this work, the proposed control system considers two loops in cascade, an internal loop solving the robots’ joint control, and an external loop implementing a dynamic look and move visual controller. A stability analysis in discrete time is developed under the conditions that it is possible to approximate the dynamic effect of the internal loop as an external loop time delay (Corke; 1996) and (Bonfe et al.; 2002). The more classic presentations of servovisual control are velocity controllers, based in the term feature Jacobian,(Chaumette and Hutchinson; 2006) 2

Arm-manipulator coordination is a new and attractive use of robotic manipulators in unstructured environments. In this paper, we propose a reflexive motion control as a coordination mechanism for the manipulator to share the load with the arm. Reflexive motion is an inspiration of biological systems. The motion is formulated in a manner which assists the manipulator to comprehend the intention of the human arm. It reduces the strain on the arm which is due to the weight of the object. In addition, the speed of manipulation on the object is improved significantly. The effects of the reflexive behavior on stability and load sharing of the system are investigated. Experimental results are presented to prove the effectiveness of the proposed scheme.

The tactile sensor design and data measurements have been playing an important role in surface material recognition. In inevitable unstructured environments, the performance of the material classification is bottlenecked by multimodal deficiency or data collection intermittency. This article investigates the influence of surface tactile data quantity on material classification in unstructured environments. We extracted tactile data features according to the material-surface-texture model and proposed an approach to determine the scope of the time window of tactile data. We utilized the machine learning classifiers to verify the effectiveness of the hand-designed features and the determined time window. Based on the combination of the Mel frequency cepstrum coefficient and statistics, the majority of the algorithms can classify the tactile data of the upper limit (60 ms in this article) with an accuracy of at least 85%. Exploiting tactile data of the lower limit (50 ms in this article), linear discriminant analysis, quadratic discriminant analysis, and support vector machine can achieve about 93% classification accuracy, and area under curve is about 0.99. The accuracy is not significantly improved with the time window beyond the upper limit, while the performance is degraded and unstable with it beneath the lower limit.

Navigating autonomous ground robots through dense and unstructured environments, such as forests, remains a major challenge due to the complexity of natural terrains and the computational burden of traditional navigation pipelines. Conventional approaches often rely on detailed prior maps, rigid rule-based systems, and computationally heavy sensor fusion techniques, which tend to lack generalization across varying environments.In this work, we present a reinforcement learning (RL) strategy for mapless navigation of ground robots in dense forested areas, relying solely on a 3D LiDAR sensor and the position of the target and the agent. The agent is trained using Proximal Policy Optimization (PPO) within a photorealistic simulation framework tailored for outdoor navigation tasks. We adopt a curriculum learning scheme that incrementally increases obstacle density during training.Experimental results in simulation show that the trained agent is capable of navigating challenging forest scenarios effectively, consistently reaching target locations even under high obstacle density. Evaluation based on Success Rate and Success Path Length (SPL) metrics highlights the robustness and adaptability of the learned policy, underscoring its potential for real-world deployment on resource-constrained robotic platforms.

We present a three-dimensional deep convolutional neural network (3D CNN) approach for grasping unknown objects with soft hands. Soft hands are compliant and capable of handling uncertainty in sensing and actuation, but come at the cost of unpredictable deformation of the soft fingers. Traditional model-driven grasping approaches, which assume known models for objects, robot hands, and stable grasps with expected contacts, are inapplicable to such soft hands, since predicting contact points between objects and soft hands is not straightforward. Our solution adopts a deep CNN approach to find good caging grasps for previously unseen objects by learning effective features and a classifier from point cloud data. Unlike recent CNN models applied to robotic grasping which have been trained on 2D or 2.5D images and limited to a fixed top grasping direction, we exploit the power of a 3D CNN model to estimate suitable grasp poses from multiple grasping directions (top and side directions) and wrist orientations, which has great potential for geometry-related robotic tasks. Our soft hands guided by the 3D CNN algorithm show 87% successful grasping on previously unseen objects. A set of comparative evaluations shows the robustness of our approach with respect to noise and occlusions.