Object sorting using a distributed manipulator array

In recent years, the computer vision applications in the robotics have been improved to approach human-like visual perception and scene/context understanding. Following this aspiration, in this study, we explored the possibility of better object manipulation performance by connecting the visual recognition of objects to their physical attributes, such as weight and center of gravity (CoG). To develop and test this idea, an object manipulation platform is built comprising a robotic arm, a depth camera fixed at the top center of the workspace, embedded encoders in the robotic arm mechanism, and microcontrollers for position and force control. Since both the visual recognition and force estimation algorithms use deep learning principles, the test set-up was named as Deep-Table. The objects in the manipulation tests are selected from everyday life and are common to be seen on modern office desktops. The visual object localization and recognition processes are performed from two distinct branches by deep convolutional neural network architectures. We present five of the possible cases, having different levels of information availability on the object weight and CoG in the experiments. The results confirm that using our algorithm, the robotic arm can move different types of objects successfully varying from several grams (empty bottle) to around 250 g (ceramic cup) without failure or tipping. The proposed method also shows that connecting the object recognition with load estimation and contact point further improves the performance characterized by a smoother motion.

This thesis presents the design and fabrication of an autonomous mobile robot with dual six-axis robotic arms, enhanced by augmented reality (AR) and powered by artificial intelligence (AI). The innovation aims to advance object manipulation tasks in various complex and dynamic environments, potentially for industrial applications. The system's Artificial Intelligence feature enhances the robot's ability to navigate and interact with its surroundings more accurately. The AR interface also allows human operators to intuitively control and guide the robot's actions in real time when needed, bridging the gap between autonomous operation and human oversight. The dual six-axis robotic arms enable dexterous and precise object manipulation. These arms mimic the versatility and adaptability of human arms, opening new possibilities for tasks that were previously challenging for traditional robotic systems. In developing this advanced mobile robot, this work embodies a pioneering step in autonomous robotics, showcasing a blend of AI, AR, and mechanical design. The robot's capabilities in object manipulation present a leap forward in robotics, promising a future where robots can take on more complex roles in our societies, while also offering a practical solution to labor-intensive or high-risk tasks. The research concludes with extensive testing and validation of the robot, demonstrating its effectiveness and efficiency in performing complex object manipulation tasks autonomously in various controlled and real-world scenarios. The results suggest promising implications for the future of robotics and automation, reinforcing the importance of interdisciplinary approaches in technological advancement.

Stochastic control for optical manipulation of multiple microscopic objects

There are many difficulties in dexterous object manipulation by multi-fingered hands, due to redundant degree-of-freedom of the entire system and uncertainties in interaction with the object. In this paper, however, 3D object manipulation without any external sensors are attempted. Under the assumption of point contact without rolling, the object position and orientation are computed in relative sense and used as the feedback for object manipulation. Overall system dynamics including two cone-shaped finger robots and an arbitrary object is modeled, and the closed-loop system stability is analyzed based with the proposed controllers for stable grasping and object position/orientation control. In order to validate the proposed method, dynamic simulation is conducted, and showed complacent results.

In existing control techniques for optical manipulation, the effect of the Brownian motion is usually disregarded for ease of analyzing the control systems. However, due to the prevalence of the Brownian motion and its effect on micro-manipulation of tiny objects, it is important for the Brownian effect to be properly taken into account so as to ensure the performance and stability of the control systems. This paper proposes a robotic control technique to achieve automated manipulation of multiple microscopic objects with the consideration of the Brownian perturbations. In this control technique, a desired dynamic region is first initiated to enclose all the trapped micro-objects. The trapped objects are then manipulated as a whole while being kept inside the desired region, and at the same time, avoiding collisions between each other during the course of manipulation. Rigorous mathematical formulation is developed to analyze the control system for automated manipulation of multiple micro-objects with stochastic perturbations. Experimental results are presented to illustrate the performance of the proposed control technique.

Embodied intelligent agents, which represent the future of robotics, demand precise perception and real-time decision-making capabilities to achieve natural environmental interactions. However current systems face inherent limitations in unimodal sensing and cross-modal coordination, which hinder their performance in dynamic contact-rich operations. Herein, we present a fabric-based event-driven tactile interface that features an innovative woven structure with cross-fiber electrodes. It achieves breakthroughs in sensitivity (246.3kPa-1), pressure detection (>450kPa), and waterproof robustness. This interface enables millisecond-level pressure/slip dual-mode feedback for self-adaptive grasping, thereby improving the dexterous manipulation of fragile or slippery objects. For underwater scenarios, a bio-inspired visual-tactile fusion (VTF) architecture leverages tactile perception to compensate for visual limitations, demonstrating a high accuracy of 97.7% in complex tasks, including underwater transparent object manipulation and recognition of similar objects. Event-driven tactile feedback is merged with visual semantics for decision-level optimization, thereby enhancing the autonomy and adaptation of humanoid machine intelligence. It creates an innovative closed-loop cross-modal perception-decision system that builds a direct link between environmental interaction and autonomous decision-making for intelligent agent development in open-world scenarios. The superior performance of the VTF architecture dynamic interaction tasks represents a crucial step toward robotic systems with advanced intelligence.

Previous research investigating handle-response compatibility effects with graspable objects used different categories of objects as stimuli, regardless of their specific, intrinsic characteristics. The current study explores whether different types of objects’ characteristics may elicit different types of spatial compatibility, that is, handle-response and response-effect compatibility as well as their potential interaction. In Experiment 1, objects having a graspable handle opposite to either a visible functional component (i.e., handle-function objects: a teapot) or a latent functional component (handle-only objects: a pitcher lacking the spout) were presented separately in different blocks. Both the handle and the goal-directed functional components of these objects were located on the horizontal axis. In Experiment 2, handle-only objects had a handle located on the horizontal axis and a latent functional component located on the vertical axis (e.g., a cup). In both experiments, participants were required to judge the material (plastic and metal) the object was made of. Results showed that the handle-response compatibility effect was sensitive to whether the actions consequences of object manipulation took place on the horizontal rather than on the vertical axis.

Adaptive neural controller for cooperative multiple robot manipulator system manipulating a single rigid object

This paper discusses the manipulation of multiple objects by pushing. When manipulating multiple objects, the relative motion between two objects such as sliding and rotating may occur. We propose a velocity based pushing method where the region of the pusher's velocity causing the desired relative motion at each contact point is obtained as a common space of regions causing the relative motion at one of the contact points. Experimental results are also included to show the effectiveness of our idea.

Nowadays, people usually depend on augmented reality (AR) systems to obtain an augmented view in a real-world environment. With the help of advanced AR technology (e.g. object recognition), users can effectively distinguish multiple objects of different types. However, these techniques can only offer limited degrees of distinctions among different objects and cannot provide more inherent information about these objects. In this paper, we leverage RFID technology to further label different objects with RFID tags. We deploy additional RFID antennas to the COTS depth camera and propose a continuous scanning-based scheme to scan the objects, i.e., the system continuously rotates and samples the depth of field and RF-signals from these tagged objects. In this way, by pairing the tags with the objects according to the correlations between the depth of field and RF-signals, we can accurately identify and distinguish multiple tagged objects to realize the vision of tell me what I see from the augmented reality system. For example, in front of multiple unknown people wearing RFID tagged badges in public events, our system can identify these people and further show their inherent information from the RFID tags, such as their names, jobs, titles, etc. We have implemented a prototype system to evaluate the actual performance. The experiment results show that our solution achieves an average match ratio of 91% in distinguishing up to dozens of tagged objects with a high deployment density.

Ensemble summary statistics represent multiple objects on the high level of abstraction-that is, without representing individual features and ignoring spatial organization. This makes them especially useful for the rapid visual categorization of multiple objects of different types that are intermixed in space. Rapid categorization implies our ability to judge at one brief glance whether all visible objects represent different types or just variants of one type. A framework presented here states that processes resembling statistical tests can underlie that categorization. At an early stage (primary categorization), when independent ensemble properties are distributed along a single sensory dimension, the shape of that distribution is tested in order to establish whether all features can be represented by a single or multiple peaks. When primary categories are separated, the visual system either reiterates the shape test to recognize subcategories (in-depth processing) or implements mean comparison tests to match several primary categories along a new dimension. Rapid categorization is not free from processing limitations; the role of selective attention in categorization is discussed in light of these limitations.

Summary Monitoring carbon dioxide (CO2) saturation plume movement and pressure buildup is critical for ensuring the environmental safety of geological carbon storage (GCS) projects. High-fidelity numerical simulations provide accurate modeling of CO2 plume dynamics, but they are often computationally intensive. Recent advancements in data-driven models have enabled the rapid prediction of CO2 plume movement. By leveraging available simulation data sets, these models offer a more efficient alternative without compromising accuracy. In this study, we adopt the U-Net enhanced graph convolutional neural network (UGCN) to predict the spatial and temporal evolution of CO2 plume saturation and pressure buildup in saline aquifers. Utilizing the U-Net architecture, which incorporates skip connections, enables UGCN to capture high-level features and fine-grained details concurrently. We have created physics-based numerical simulation models that account for both CO2 injection and post-injection periods. By employing the Latin-hypercube sampling method, we generated a diverse range of reservoir and decision parameters, resulting in a comprehensive simulation database. We trained and tested the UGCN model on two different data sets, including a 2D radial reservoir model (Scenario 1) and a realistic Society of Petroleum Engineers Comparative Solution Project b (SPE 11b, Scenario 2) model to train and validate the UGCN model. The performance of the UGCN model was compared with other standard graph neural networks (GNNs), such as graph convolution network (GCN) and graph attention network (GAT). Notably, the UGCN model demonstrated robust performances on the blind testing dataset, achieving an R2 score of 0.993 and 0.989 for saturation predictions in Scenarios 1 and 2, respectively. Similarly, for pressure buildup the new model achieved an R2 of 0.989 and 0.999 for Scenarios 1 and 2, respectively. These prediction results indicate the effectiveness of the trained models in predicting the temporal and spatial evolution of CO2 saturation and pressure buildup predictions. Moreover, the prediction central processing unit (CPU) time for the deep learning (DL) models is significantly lower (0.02 seconds per case) than the physics-based reservoir simulator (on average, 10–15 minutes per case). This underscores the capability of the proposed method to provide predictions as accurate as physics-based simulations while reducing substantial computational costs.

We propose a direct object manipulation system that a user can manipulate objects constrained to each other by his or her hands. Because there is no haptic feedback to a user, manipulation of constrained objects is much harder than that of a simple rigid body. In addition, many physics simulator usually suffers from instability in handling the constrained objects in a virtual system. We solve the problem by adding another micro simulator which enforce the constraint between the objects, and guarantees that the rendered image of an object and its physics counterpart always satisfy the given constraint. The experiment shows that the proposed method makes a user manipulate constrained objects with no difficulties in the way that he or she manipulates such objects in the real world.

This paper discusses grasp and manipulation of multiple objects initially placed on a table. We show the kinematically necessary and sufficient condition for lifting up two cylindrical objects from a table by finger links in contact with them. It is guaranteed that if pushing points between a finger link and object is lower than the contact point between objects, two objects are detached from the table. For two grasped objects, we further show a sufficient condition for producing a small relative displacement between objects. This sufficient condition given as the function of joint torques enables us to achieve a primitive manipulation of two objects within the hand. The basic idea is also experimentally verified by using a three-fingered robot hand.

This paper proposes a modular gripping mechanism for the manipulation of multiple objects. The proposed micro gripper combines traditional machining techniques with MEMS technologies to produce a modular mechanism consisting of a sturdy, compliant aluminium base and replaceable end-effectors. This creates an easily-customisable solution for micro manipulation with an array of different micro tips for different applications. We have provided the kinematic analysis for the gripper to predict the output and have also optimised design parameters based on FEA (finite element analysis) simulation and the effects of altering flexure beam lengths. The gripper is operated by a piezo actuator capable of 18 m displacement at 150 V of applied DC voltage. This is then amplified by a factor of 8.1 to a maximum tip displacement of 154 m. This is achieved by incorporating bridge and lever amplifying techniques into the design. An initial experimental analysis of the micro gripper is provided to investigate the performance of the micro gripper and to gauge the accuracy of the theory and simulation through comparison with experimental results.