Arm-manipulator coordination for load sharing using reflexive motion control

The evolution of robotics technology continues to facilitate exploration and scientific study in remote environments, enabling research in areas that were previously impossible to reach. Robots operating in space and marine environments encounter similar operational challenges, as both face high operational costs, bandwidth-limited conditions, and natural, unstructured environments where dynamic obstacles might be present. Within the oceanographic domain, conventional deep-sea sampling operations involve remotely operated vehicles (ROVs) equipped with robotic manipulator arms to complete dexterous tasks at depth. While effective, deep-sea ROV operations require specialized instrumentation, highly trained shipboard personnel, and large oceanographic vessels, which make deep-sea samples inaccessible to most. This thesis presents the SHared Autonomy for Remote Collaboration (SHARC) framework, and evaluates its utility within an oceanographic context. By leveraging shared autonomy, SHARC enables shore-side operators to collaboratively carry out underwater sampling and manipulation tasks, regardless of their prior manipulator operations experience. With SHARC, operators can conduct manipulation tasks using natural language and hand gestures through a virtual reality (VR) interface. The interface provides remote operators with a contextual 3D scene understanding that is updated according to bandwidth availability. Evaluation of the SHARC framework through controlled lab experiments indicates that SHARC’s VR interface enables novice operators to complete manipulation tasks in framerate-limited conditions (i.e., <0.5 frames per second) faster than expert pilots using the conventional topside controller. For both novice and expert users, the VR interface also increased the task completion rate and improved sampling precision. During sea trials, SHARC enabled collection of an underwater in-situ X-ray fluorescence (XRF) measurement at more than 1000 meters water depth in the Eastern Pacific with centimeter-level precision by remote scientists with no prior piloting experience. This demonstration provides compelling evidence of SHARC’s utility for conducting delicate operations in unstructured environments across bandwidthlimited communications, which holds relevance for improving operations in other sensitive domains where dexterity is required. SHARC’s ability to relax infrastructure requirements and engage novice shore-side users provides a promising avenue for democratizing access to deep-sea research.

Robotics has been a paradigm for science in the last few decades. At first, scientists’ efforts were devoted to the solution of the problem of planning and control of themotion of robot manipulators. However, the motion control of robot manipulators in unstructured environments is today an attractive scientific problem. An interesting solution for motion control is the use of sensor information, such as computer vision, in the system’s feedback. Several works and tools have been developed in recent years in this field (Corke; 2005), (Chaumette and Hutchinson; 2006). The more typical approaches consider visual perception for servoing and for the so called look and move (Hutchinson et al.; 1996). Visual servoing (Kelly et al.; 2000) can be classified into two approaches: camera-in-hand or camera-to-hand (Flandin et al.; 2000). In camera-to-hand robotic systems, multiple cameras or a single camera fixed in the worldcoordinate frame capture images of both, the robot and its environment. The tracking of the object with visual feedback can be made in 2D or 3D. An interesting solution for the visual servoing of camera-in hand robot manipulators in 2D can be found in e. g. (Bonfe et al.; 2002) and (Hernandez et al.; 2008) where stability demonstration of a decoupled controller have been presented. For 3D tracking some solutions reported are look and move controller, with one camera (Sim et al.; 2002) or more than one camera (Xie et al.; 2005). A 3D visual servoing with stability analysis in continuous time is presented by (Hernandez et al.; 2008a) and (Kelly et al.; 2006) present a direct visual servoing with transpose Jacobian control technique for regulation of robot manipulators in the 3D Cartesian space. In a similar vein (Enescu et al.; 2006) present mobile robot navigation for person tracking using a stereo head-camera. In this chapter, we consider the control problem of camera-in-hand robot manipulators in 2D and 3D. In both cases only one camera is mounted on the robot’s arm, which supplies visual information of the environment, with the aim of moving the manipulator by maintaining the image of the tracked object (a sphere) in the centre of the image plane, despite the possible movements of the object. In the 3D control the constant radius is used as a feature too. In this work, the proposed control system considers two loops in cascade, an internal loop solving the robots’ joint control, and an external loop implementing a dynamic look and move visual controller. A stability analysis in discrete time is developed under the conditions that it is possible to approximate the dynamic effect of the internal loop as an external loop time delay (Corke; 1996) and (Bonfe et al.; 2002). The more classic presentations of servovisual control are velocity controllers, based in the term feature Jacobian,(Chaumette and Hutchinson; 2006) 2

In unstructured environments, robots face challenges in efficiently and accurately grasping irregular, fragile objects. To address this, this paper introduces a soft robotic hand tailored for such settings and enhances You Only Look Once v5s (YOLOv5s), a lightweight detection algorithm, to achieve efficient grasping. A rapid pneumatic network-based soft finger structure, broadly applicable to various irregularly placed objects, is designed, with a mathematical model linking the bending angle of the fingers to input gas pressure, validated through simulations. The YOLOv5s model is improved by integrating the Coordinate Attention (CA) mechanism in the backbone layer, refining the Spatial Pyramid Pooling (SPP) module for faster detection, and adjusting the loss function to prevent misalignment between predicted and actual bounding boxes, thereby enhancing computational efficiency. Experimental comparative analysis indicates that the refined model exhibits improvements in both mean average precision and recognition speed. A soft robotic grasping experimental platform was established, with precision grasping and power grasping experiments conducted using the pose and object type data generated by the enhanced YOLOv5s-CA-SPP model network. The results show that the success rate of grabbing reaches 82% with a proper grabbing posture.

Humanoid robotic manipulation in unstructured environments is a challenging problem. Limited perception, communications and environmental constraints present challenges that prevent fully autonomous or purely teleoperated robots from reliably interacting with their environment. In order to achieve higher reliability in manipulation we present an approach involving remote human supervision. Strengths from both human operator and humanoid robot are leveraged through a user interface that allows the operator to perceive the remote environment through an aggregated worldmodel based on onboard sensing, while the robot can efficiently receive perceptual and semantic information from the operator. A template based manipulation approach has been successfully applied to the Atlas humanoid robot; we show real world footage of the results obtained in the DARPA Robotics Challenge Trials 2013.

A gripper is described for manipulation in natural, unstructured environments. The specific manipulation task is to pick up surface material such as pebbles or small rocks in a natural terrain. The application is to give autonomous sampling capabilities to an autonomous vehicle for planetary exploration. The authors describe the task analysis process that led to the selection of a configuration with three soft fingers. They carry out a complete analysis of the stability of a grasp for this gripper including an analysis of the deformation of the fingers at the points of contact. The implementation of a grasp selection algorithm is described, and results on three-dimensional representations of objects computed from range data are presented. >

Automatically finding paths is a crucial and challenging task in autonomous navigation systems. The task becomes more difficult in unstructured environments such as indoor or outdoor scenes with unmarked pedestrian lanes under severe illumination conditions, complex lane surface structures, and occlusion. This paper proposes a robust method for pedestrian lane detection in such unstructured environments. The proposed method detects the walking lane in a probabilistic framework integrating both appearance of the lane region and characteristics of the lane borders. The vanishing point is employed to identify the lane borders. We propose an improved vanishing point estimation method based on orientation of color edges, and use pedestrian detection for occlusion handling. The proposed pedestrian lane detection method is evaluated on a new data set of 2000 images collected from various indoor and outdoor scenes with different types of unmarked lanes. Experimental results and comparisons with other existing methods on the new data set have shown the efficiency and robustness of the proposed method.

The consideration of data set design, collection, and distribution methodology is becoming increasingly important as robots move out of fully controlled settings, such as assembly lines, into unstructured environments. Extensive knowledge bases and data sets will potentially offer a means of coping with the variability inherent in the real world. In this study, we introduce three new data sets related to mobile manipulation in human environments.

This paper presents the design, modeling and control of a compact wrist, which can work in active mode with position or torque control, or passive mode with interactive force compensation. This characteristic makes it suitable for dexterous manipulation in unstructured environments, such as door opening. Mechanism design is presented briefly, and forward, inverse as well as differential kinematics are derived to map the motions and velocities between the task space and joint space. A distributed robust adaptive controller is developed for tracking control of the wrist in active mode; and a new interactive force compensation technique is proposed on the basis of force sensor measurement, which enables passive working mode of the compact wrist. A DSP based control system is developed to test the developed control methods. A prototype has been fabricated and some preliminary experiments have been conducted to validate the proposed design and to verify the developed algorithms.

Conventional suction cups lack sensing capabilities for contact-aware manipulation in unstructured environments. This paper presents FlexiCup, a multimodal suction cup with wireless electronics that integrate dual-zone vision-tactile sensing. The central zone dynamically switches between vision and tactile modalities via illumination control, while the peripheral zone provides continuous spatial awareness. The modular mechanical design supports both vacuum (sustained-contact adhesion) and Bernoulli (contactless lifting) actuation while maintaining the identical dual-zone sensing architecture, demonstrating sensing-actuation decoupling where sensing and actuation principles are orthogonally separable. We validate hardware versatility through dual control paradigms. Modular perception-driven grasping achieves comparable success rates across vacuum (90.0%) and Bernoulli (86.7%) modes using identical sensing and control pipelines, validating the sensing architecture's effectiveness across fundamentally different pneumatic principles. Diffusion-based end-to-end learning achieves 73.3% and 66.7% success on contact-aware manipulation tasks, with ablation studies confirming 13% improvements from multi-head attention coordinating dual-zone observations. Hardware designs, firmware, and experimental videos are available at the companion website: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://flexicup.junhaogong.top</uri>.

When a robot manipulator performs tasks in a dynamic unstructured environment, it not only should account for where and how it should move, but also ensure that the robot can effectively avoid collisions with static and moving obstacles. Planning in absolutely unstructured and dynamic environment is much more challenging, up-to-date there is no universal solution to this problem. In this paper, we believed the concept that for motion planning in real dynamic environments, usually there is only a part of a path invalidated by obstacles, and only this part needs replanning. Thus, in this paper a practical strategy was presented for motion planning in dynamic environments. First, a variant dynamic Rapidly-exploring Random Tree (RRT) planner was designed to combine global planning with local replanning. Then, the proposed dynamic RRT algorithm was implemented under Robotic Operation System (ROS) framework and applied to a 7-degree-of-freedom manipulator. Experimental results have shown that the proposed dynamic RRT planner can plan and execute movement of robot manipulator between a starting and target position while preserving task constraints and reacting to environment changes in real time.

Robot grasping under unstructured environment is a research hotspot in the realization of human-computer interaction and automation. To solve the problem of grasping efficiency and grasping of multiple object scales under unstructured environment, we propose modified GG-CNN model, a fast and efficient robotic grasping method. The proposed modified GG-CNN model uses RGB-D image as input to obtain features by convolutional layers and then reshape them back to the original size by transposed convolutional layers to estimate grasp quality and angle. The resulting grasp pose is chosen to have largest grasp quality. To achieve better robustness, background plane is estimated by the Least Square Method and used as workspace frame so that the height value of background plane remains zero. The modified model is trained and evaluated based on the Cornell Grasp Dataset and the accuracy rate is 96.63%. The proposed method is used to grasp household objects and fruits, including some challenging small objects, and the grasp success rates are 94.5% and 98.0% respectively. Experimental results show that our method can perform real-time robotic grasping under unstructured environment.

Visual recognition is the most critical function of a harvesting robot, and the accuracy of the harvesting action is based on the performance of visual recognition. However, unstructured environment, such as severe occlusion, fruits overlap, illumination changes, complex backgrounds, and even heavy fog weather, pose series of serious challenges to the detection accuracy of the recognition algorithm. Hence, this paper proposes an improved YOLO v4 model, called YOLO v4+, to cope with the challenges brought by unstructured environment. The output of each Resblock_body in the backbone is processed using a simple, parameterless attention mechanism for full dimensional refinement of extracted features. Further, in order to alleviate the problem of feature information loss, a multi scale feature fusion module with fusion weight and jump connection structure was pro-posed. In addition, the focal loss function is adopted and the hyperparameters α, γ are adjusted to 0.75 and 2. The experimental results show that the average precision of the YOLO v4+ model is 94.25% and the F1 score is 93%, which is 3.35% and 3% higher than the original YOLO v4 respectively. Compared with several state-of-the-art detection models, YOLO v4+ not only has the highest comprehensive ability, but also has better generalization ability. Selecting the corresponding augmentation method for specific working condition can greatly improve the model detection accuracy. Applying the proposed method to harvesting robots may enhance the applicability and robustness of the robotic system.

An increasingly important trend is to simulate a human arm using a 7-degrees-of-freedom (DOF) redundant manipulator and accomplish complex tasks in an unstructured environment by leveraging human experience. Human arm motion information acquired by a biomimetic motion capture system is mapped to planning information of the manipulator to leverage human experience in the control process and quickly improve the operational capability of the manipulator in complex environments. In this paper, extensive tests are performed for verification, and the test results prove the effectiveness of the proposed algorithm, which improve the operational performance of a redundant manipulator in a complex environment to accomplish humanoid tasks and is critical to robot-aided operations or the complete substitution of humans by robots in the future.

A hybrid (position and force) control algorithm of a robot manipulator with internal uncertainty and working in an unstructured environment is proposed. The internal uncertainty can be due to imperfect modelling, friction, payload or external disturbances. The unstructured environment can be due to the lack of knowledge of the designer. The problem has been cast into the framework of control of dynamical systems whose mathematical model contain uncertainties. Based solely on the bound of the uncertainties, controllers can be constructed which guarantee certain desirable performance.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The understanding of human arm stiffness have brought several significant advances to robotics. For the most part, the end-point stiffness of human arm serves as an important role in guiding and shaping the Cartesian stiffness of robot arm in the execution of complicated interaction tasks because of the convenience of using the common space where both stiffnesses function. However, investigation of the joint stiffness of human arm, on the other hand, will provide a more comprehensive perspective on the human arm stiffness and enable other appealing robotic applications, for instance, whole-arm interaction with unstructured environment. As a fundamental research for these applications, the feasibility of an online joint stiffness transfer approach from human to anthropomorphic arms is discussed in this paper. This is realized by a proposed concept of physiological joint stiffness, which is shared by the human and anthropomorphic arms. The desired joint stiffness of robot arm is transformed from the estimated joint stiffness of human arm by requiring both arms to have the same apparent physiological joint stiffness. To make the calculated joint stiffness achievable in a robot controller, the stiffness matrix is subsequently optimized to be symmetric and positive definite. Proof-of-concept experiment is performed on a fully integrated robotic teleoperation setup to validate the efficacy of the proposed method.