Autonomous Object Detection and Grasping Using Deep Learning for Design of an Intelligent Assistive Robot Manipulation System

Nose Tracking Assistive Robot Control for the People with Motor Dysfunctions

The paper discusses the potential of assistive service robots to support disabled and elderly people. It shows that they have considerable untapped potential in this area, but also that inappropriate implementations could increase isolation, reduce independence and lead to users feeling as though they are under surveillance. The main body of the paper presents an overview of existing applications and discusses their benefits and potential problems. This is organized by an extension of the common classification into socially and physically assistive robots by the two categories of sensory assistive and mixed assistance robots. Another more detailed classification is also presented. This discussion is introduced by an overview of many of the technological components of smart mobile robots. It is followed by a discussion of user acceptance. The problems of existing models based on either solely positive or solely negative factors are noted and a model containing both types of factors is proposed. The need for continuing research is noted and various proposals are made.

Camera Holding Robotic Devices in Urology

Sensor devices are regularly used on unmanned aerial vehicles (UAVs) as reconnaissance and intelligence gathering systems and as support for front line troops on operations. This platform provides a wealth of sensor data and has limited computational power available for processing. The objective of this work is to detect and identify objects in real-time, with a low power footprint so that it can operate on a UAV. An appraisal of current computer vision methods is presented, with reference to their performance and applicability to the objectives. Experimentation with real-time methods of background subtraction and motion estimation was carried out and limitations of each method described. A new, assumption free, data driven method for object detection and identification was developed. The core ideas of the development were based on models that propose that the human vision system analyses edges of objects to detect and separate them and perceives motion separately, a function which has been modelled here by optical flow. The initial development in the temporal domain combined object and motion detection in the analysis process. This approach was found to have limitations. The second iteration used a detection component in the spatial domain that extracts texture patches based on edge contours, their profile, and internal texture structure. Motion perception was performed separately on the texture patches using optical flow. The motion and spatial location of texture patches was used to define physical objects. A clustering method is used on the rich feature set extracted by the detection method to characterise the objects. The results show that the method carries out detection and identification of both moving and static objects, in real-time, irrespective of camera motion.

This paper presents the development of Robot Technology Components (RTCs) using RT-Middleware (RTM) for designing assistive robotic arms and human-machine interfaces (HMIs) that can be used by the disabled. RTM is an open software platform for a component-oriented robot system. An RTC is the basic functional unit for constructing a robot system using RTM. The interface definition for the basic functionality of RTCs is standardized by the Object Management Group (OMG). Therefore, software modules written as RTCs can be shared and reused in other robot systems. One feature of an assistive robotic arm is that the arm is controlled by manual operation. Various HMIs for manually operating the robotic arms have been developed according to the abilities of each user and the requirements regarding the actions that the robot can perform. These HMIs can be interchangeably connected to the assistive robotic arms by implementing the software modules for the robotic arm and the HMI as RTCs with a common software interface.

Considering the real employment situation of the disabled, it was concluded that there is demand for a robot system that can be used in a real manufacturing environment. To derive our mission statement, we conducted a survey with three criteria: `A. to assist the disabled as much as possible'; `B. to consider the real situation of employment for the disabled'; and `C. to provide assistance for the most necessary tasks for the disabled in the work space'. Based on the survey results, our goal can be stated as follows: To develop work assistive mobile robots to help persons with limb disabilities obtain jobs in manufacturing environments where they can perform simple manual labor. Here, simple labor includes such tasks as circuit inspection and moving boxes. For our goal, we developed two types of work assistive mobile robots, which are small enough to work in the confined environments of real factories. Work assistive mobile robot type I is equipped with a forklift to move boxes and a robotic arm to help with jobs requiring more dexterity. Robot type II is compact in size, adopting an omni-wheel driving mechanism, and is equipped with a lift having a foldable fork. We gathered feedback from empirical work done on the two robots.

Due to the vulnerability of deep neural networks to adversarial examples, numerous works on adversarial attacks and defenses have been burgeoning over the past several years. However, there seem to be some conventional views regarding adversarial attacks and object detection approaches that most researchers take for granted. In this work, we bring a fresh perspective on those procedures by evaluating the impact of universal perturbations on object detection at a class-level. We apply it to a carefully curated data set related to autonomous driving. We use Faster-RCNN object detector on images of five different categories: person, car, truck, stop sign and traffic light from the COCO data set, while carefully perturbing the images using Universal Dense Object Suppression algorithm. Our results indicate that person, car, traffic light, truck and stop sign are resilient in that order (most to least) to universal perturbations. To the best of our knowledge, this is the first time such a ranking has been established which is significant for the security of the data sets pertaining to autonomous vehicles and object detection in general.

In this paper, an image based visual servo (IBVS) scheme is developed for a hexacopter, equipped with a robotic soft grasper to perform autonomous object detection and grasping. The structural design of the hexacopter-soft grasper system is analyzed to study the soft grasper’s influence on the multirotor’s aerodynamics. The object detection, tracking and trajectory planning are implemented on a high-level computer which sends position and velocity setpoints to the flight controller. A soft robotic grasper is mounted on the UAV to enable the collection of various contaminants. The use of soft robotics removes excess weight associated with traditional rigid graspers, as well as simplifies the controls of the grasping mechanics. Collected experimental results demonstrate autonomous object detection, tracking and grasping. This pipeline would enable the system to autonomously collect solid and liquid contaminants in water canal based on GPS and multi-camera system. It can also be used for more complex aerial manipulation including in-flight grasping.

This paper proposes a page-turning strategy using an assistive robot that has a low-degree-of-freedom robotic hand. The robotic hand is based on human object handling characteristics, which significantly reduces the number of fingers and joints required to handle various objects. The robotic hand has right and left planar fingers that can transform their shape to handle various objects. To turn a page, the robot uses the planar fingers to push the surface of the page and then rotates the fingers. The design concept, mechanism, sensor system, strategy for page turning, and control system of the robotic hand are presented. The experimental results show that the robot can turn pages using the proposed method; however, it sometimes failed to turn the page when the robotic hand height was too low and too close to the book because the rotation of the fingers was stopped by the book. When the hand detects excessive force during page turning, the control system changes the shape of the fingers and releases the force from the book. The experimental results show the effectiveness of the control system.

In the rapid development of the world, robots have become involved in different aspects of human lives. A robotic arm is a type of machine that is able to manipulate its end effector in two or more axes, thus more mobile than a human arm would be. The most common restraint of a robotic arm is how it needs to be connected to the user through wires, which limits the overall capabilities of the robotic arm. Thus, the study aims to design, fabricate, and test a prototype of a robotic arm which can be controlled wirelessly through joystick module. The robotic arm controlled by a wireless joystick can change the position of its end effector within the three axes; X,Y, and Z axis. The prototype was then tested on its accuracy and on the maximum distance for safe performance when it is unobstructed and when it is obstructed by a glass window. The controller for the robotic arm was able to transmit data wirelessly through a radio frequency module while the robotic arm was able to receive the data using the same module. The robotic arm can be controlled at a maximum distance of 25ft with a glass obstruction, and 29 ft when unobstructed. The accuracy of the robotic arm is then tested wherein the researchers were operating the robotic arm. It was proven that it was accurate enough to be able to move its end effector to the desired point while being controlled remotely using a joystick controller. The ability of the robotic arm to carry load depends on the type of servo motor used in the robotic arm, however, the researchers opted to focus more on the accuracy of the robotic arm and not on the load carrying capacity.

In this paper, a design and development of an assistive robot are proposed that assists to the elderly and handicapped people, such as providing information and care with daily routine supports with the ability to perform various tasks, including the contact safety. Firstly a clear function requirement for such assistive robots was made. Secondly different sub-parts have been mounted to synthesize/unify the functions such as high power, dexterity/competence, mobility, safety and the cost-effectiveness. Additionally, a set of different applications, like the robot moves to other rooms/locations according to the commands given by the user, and the arm performs the task of grabbing and pouring/serving. Based on the application, the assistive prototype robot ROSWITHA (RObot System WITH Autonomy) was developed successfully, that operates using three major systems- the vision system, the arm system and the navigation system. The mechanisms are the camera, LIDAR, ultrasonic sensors, dual arm with gripper and two wheeled mobile vehicle including two castor wheels. Some experiments are performed focusing on assistive tasks using ROSWITHA's sub-systems, indicating that the robot software can be integrated and become extremely useful to improve the living style of the elderly and handicap people in future.

Bidirectional Human-Robot Learning: Imitation and Skill Improvement

This paper reports the design and development of a real-time IMU-Vision-based hybrid control algorithm to interact with a 6-DOF Kinova virtual robotic arm. Human Robot Interaction (HRI) control scheme proposed in this paper utilizes an embedded gyroscopic sensor from a Myo Gesture Control Armband's inertial measurement unit and an 800*600-pixel resolution from a Microsoft HD camera. The algorithm exploits the mathematical features of a numerical discrete time integrator and a mean filter to process the raw angular velocity data from the gyroscope. The processed data further provides the angular displacements to the end-effector of the robotic arm during clockwise or counterclockwise actions along x, y, and z axes from the user. This also facilitates the end effector (gripper) motion which was also controlled simultaneously by the roll action through threshold comparison in the algorithm. A vision based feedback system was designed using a computer vision toolbox and blob analysis technique in order to make the system more reliable and to control the distance of the end-effector while reaching to desired objects. The results demonstrated a significant control of the 6-DOF virtual robotic limb using the gyroscopic information and user inputs. The virtual robotic arm stopped the movement after reaching 320 mm from the desired object as expected. For three different objects, the maximum error between the real and the measured distance was calculated as 15.3 cm for cylindrical object. Due to its smooth control and arm gesture controller, this technology has the potential to assist people with either physical impairments or neurological disorders to perform activities of daily living using assistive robotic arm in the near future.

Recent advancements in sensor technologies, including camera-based systems integrated with computer vision and deep learning, have significantly transformed Advanced Driving Assistance Systems (ADAS) and autonomous vehicles, driving innovations in safety, efficiency, and environmental perception. As these systems become increasingly essential for autonomous navigation, object detection, semantic segmentation, and tracking technologies are now the foundation for achieving reliable, real-time decision-making in dynamic driving environments. In this study, we propose Vision-based Integrated Traffic Analysis (VITA), a unified multitask learning model that integrates object detection and semantic segmentation with the cutting-edge Cross-View Tracking (CVT) method. VITA’s key innovation lies in its ability to seamlessly integrate camera sensor-based tracking functionalities, enhancing system robustness and accuracy across diverse perspectives. We present a encoder-decoder architecture that enables unprecedented levels of recognition speed and accuracy, outperforming conventional methods in challenging real-world scenarios. Our results demonstrate VITA’s transformative potential, particularly through multi-camera sensor configurations that enable seamless object detection associations across diverse viewpoints. Evaluations on multiple datasets, including Cityscapes, VeRi-776, and a self-collected Taiwan traffic dataset, show superior performance: Road IoU 91.0%, mAP 41.0% (Cityscapes), mAP 87%, CMC 95.9% (VeRi-776), and MOTA 87%, mAP 86.5%, CMC 93.4% (self-collected). These results underline the robustness and real-world applicability of our model, offering a significant leap forward in sensor-based perception for ADAS.

This video demonstrates the use of a robotic surgical assist arm, specifically the Zimmer Biomet WalterLorenz arm, in a complex head and neck reconstructive surgery involving a scapular tip/latissimus dorsi free flap for a 67-year-old patient with advanced squamous cell carcinoma of the oral cavity. The key objective is to showcase how the robotic arm enhances surgical precision and efficiency by providing stable retraction, reducing the need for additional assistants, and minimizing fatigue-related challenges during prolonged procedures. The video highlights the setup, positioning, and intraoperative use of the robotic arm, including its role in maintaining optimal visualization and facilitating delicate dissections and osteotomies. The clinical significance of this technology lies in its ability to improve surgical ergonomics and outcomes by enabling the surgical team to focus more effectively on the intricate aspects of the procedure. This video serves as a valuable resource for surgeons interested in integrating robotic assist technology into head and neck reconstructive surgeries.