Integrating Tactile and Force Feedback with Finite Element Models

This work focuses on the design and testing of teleoperation controllers which are required to discriminate changes in compliance, addressing the question of which controller architecture performs the best in the high fidelity application of telesurgery. Three teleoperator controller architectures are compared for their ability to detect objects in compliant environments. These architectures are: position error based force feedback (PERR), kinesthetic force feedback (KFF), and position and force feedback (P+FF). The gains for each controller are chosen based on stability, tracking performance, and fidelity of the system. Stability is determined by a robust stability criterion. A sensitivity function is used to determine a tracking criterion. A new fidelity measure is introduced which looks at the sensitivity of the transmitted impedance to changes in the environmental compliance. Experiments are conducted to determine which control architecture allows the operator to most easily determine a change in compliance. This experimental task is designed to mimic palpation of soft tissue performed in medical procedures. The results suggest that the hybrid controller (P+FF) outperforms both PERR and KFF.

The rapid advancements in robotics and automation have highlighted the need for robotic systems capable of adapting to the diverse physical properties of objects. Traditional grippers often lack the versatility to handle both hard and soft objects without extensive reprogramming or hardware adjustments. Previous studies have explored various approaches to this challenge, including tactile sensors and force feedback mechanisms to distinguish object properties. For instance, research by Calandra et al. (2018) utilized deep learning with tactile data to enable robotic hands to identify objects and adjust grip accordingly. Similarly, the paper by Li et al. (2020) “Design and performance characterization of a soft robot hand with fingertip haptic feedback for teleoperation”, focuses on designing and characterizing a soft robotic hand with fingertip haptic feedback for teleoperation emphasizing real-time tactile sensing and feedback mechanisms. However, these studies primarily focus on tactile feedback or specialized hardware, limiting their applicability in scenarios where such systems are not available or practical.This study introduces a novel machine learning-based approach, focusing on the use of visual data alone to classify and adapt to the hardness or softness of objects. By leveraging the CIFAR-100 dataset, we trained a deep learning model based on the ResNet50 architecture, achieving significant results in binary classification of hard and soft objects. The CIFAR-100 dataset, consisting of 100 diverse object categories/classes, was reorganized into two classes: hard (39 categories) and soft (61 categories). The ResNet50 model, pre-trained on ImageNet, was fine-tuned specifically for this task, with modifications to the last 210 layers to enhance its adaptability.Data augmentation techniques, including rotations, translations, shearing, zooming, and horizontal flipping, were applied to simulate real-world variations, ensuring robust learning. The model was further refined with additional fully connected layers, dropout, and batch normalization to prevent overfitting. Optimized using the AdamW optimizer, the model achieved a training accuracy of 83.31% and a validation accuracy of 80.25%, with a test accuracy of 80%. The precision, recall, and F1-scores were 0.82, 0.86, and 0.84 for the soft object class, and 0.76, 0.71, and 0.73 for the hard object class, demonstrating the model’s effectiveness in distinguishing between hard and soft objects without the need for specialized sensors. It is also observed that the accuracy of the model in relation to hard objects is significantly lesser as compared to soft objects. It is understood that the CIFAR-100 dataset (comprising of 100 classes) is inadequate for model training, so we are exploring the ILSVRC (ImageNet subset) dataset for model training in future. The research is useful in a variety of fields where the focus lies on object handling. In everyday life, we encounter a wide array of objects with varying degrees of hardness, requiring different levels of care and precision during handling. By relying on visual data and machine learning algorithms, as demonstrated in this research, robotic systems can become more autonomous and versatile, reducing the burden on human operators and improving overall efficiency. This approach can lead to cost savings by reducing the need for specialized hardware, such as tactile sensors, which are often expensive and difficult to integrate.

Virtual environments are often used in pre-wearing training and assessment of prosthetic control abilities. Here, we developed a virtual prosthetic hand training platform for evaluation of closed-loop control of grasp force. Biorealistic controllers emulated a pair of antagonistic muscles that actuated the thumb and index fingers of the hand. Surface electromyographic (sEMG) signals from a pair of antagonistic residual muscles drove the biorealistic controllers. Tactile forces from fingertip sensors were conveyed to amputees through evoked tactile sensations (ETS) elicited at the projected finger map (PFM) areas of the stump. A forearm amputee subject participated in force tracking or holding tasks using the virtual hand with residual muscle EMGs, or the contralateral intact hand. Root-mean-square error (RMSE) was used as outcome measure of motor performance. Results in this subject showed that the biorealistic controller enabled the virtual hand to track and maintain grasping forces. The best performance in both tasks was achieved by the contralateral intact hand with visual feedback. The roles of visual or tactile feedback in force tracking or maintaining were also assessed with the virtual hand. For force holding task, hybrid tactile and visual feedback with biorealistic control had a better performance than single visual or tactile feedback in terms of RMSE, success rate, and force variability. While in the force pursuing task, tactile feedback did not seem to add visual feedback in following the target force. The study suggests that training may be required for a novel virtual hand user to perceive and integrate multiple modalities of feedback information, so as to optimize the closed-loop control ability.

This paper presents the development of a compact tactile display and its integration in teleoperation. The system's operation is based on the display of surface shape to an area of the fingertip through a 4 × 4 array of tactors moving perpendicularly to the skin surface. The tactors are spring loaded and are actuated remotely by dc motors through a flexible tendon transmission. This novel implementation of conventional actuation principles achieves a compact design with superior performance compared to devices of a similar footprint, demonstrating an excellent combination of tactor spatiotemporal resolution, force, and amplitude. The display's ergonomic design and high performance make it suitable for integration on haptic devices for tactile feedback in VR and in Teleoperation. This paper presents the design, control, and performance of the tactile display and of the transmission system. It also demonstrates its integration on an Omega7 force feedback device for the teleoperation of an LWR KUKA manipulator. An experiment is presented where users teleoperated the stylus of the robot in a 3D contour following task with and without tactile feedback. In this experiment, force feedback from the slave is fused with model-based local tactile feedback. Subjects' performances indicate an improvement in teleoperation when both tactile and force feedback are present.

Haptic technologies and applications have received enormous attention in the last decade. The incorporation of haptic modality into multimedia applications adds excitement and enjoyment to an application. It also adds a more natural feel to multimedia applications, that otherwise would be limited to vision and audition, by engaging as well the user's sense of touch, giving a more intrinsic feel essential for ambient intelligent applications. However, the improvement of an application's Quality of Experience (QoE) by the addition of haptic feedback is still not completely understood. The research presented in this paper focuses on the effect of haptic feedback and what it potentially adds to the experience of the user as opposed to the traditional visual and auditory feedback. In essence, it investigates certain issues regarding stylus-based haptic education applications and haptic-enhanced entertainment videos. To this end, we used two haptic applications: the haptic handwriting learning tool to experiment with force feedback haptic interaction and the tactile YouTube application for tactile haptic feedback. In both applications, our analysis shows that the addition of haptic feedback will increase the QoE in the absence of fatigue or discomfort for this category of applications. This implies that the incorporation of haptic modality (both force feedback as well as tactile feedback) has positively contributed to the overall QoE for the users.

Integrating tactile and force feedback for highly dynamic tasks: Technological, experimental and epistemological aspects

The use of haptic technologies has recently become immensely essential in Human-Computer Interaction to improve user experience and performance. With the introduction of tactile feedback on a touchscreen device, commonly known as surface haptics, several applications and interaction paradigms have become a reality. However, the effects of tactile feedback on the preference of 2D images in visuo-tactile exploration task on touchscreen devices remain largely unknown. In this study, we investigated differences of preference score (the tendency of participants to like/dislike a 2D image based on its visual and tactile properties), reach time, interaction time, and response time under four conditions of feedback: no tactile feedback, high-quality of tactile information (sharp tactile texture), low-quality of tactile information (blurred tactile texture), and incorrect tactile information (mismatch tactile texture). The tactile feedback is rendered in the form of roughness that is simulated by modulating the friction between the finger and the surface and is derived from the 2D image. Thirty-six participants completed visuo-tactile exploration tasks for a total of 36 trials (3 2D images × 4 tactile textures × 3 repetitions). Results showed that the presence of tactile feedback enhanced users’ preference (tactile feedback conditions were rated significantly higher than the no tactile feedback condition for preference regardless of the quality/correctness of tactile feedback). This finding is also supported through results from self-reporting where 88.89% of participants preferred to experience the 2D image with tactile feedback. Additionally, the presence of tactile feedback resulted in significantly larger interaction time and response time compared to the no tactile feedback condition. Furthermore, the quality and correctness of tactile information significantly impacted the preference rating (sharp tactile textures were rated statistically higher than blurred tactile and mismatched tactile textures). All of these findings demonstrate that tactile feedback plays a crucial role in users’ preference and thus motivates further the development of surface haptic technologies.

Robot-assisted thoracoscopic surgery has been applied for general thoracic operations. Its advantages include not only those of minimally invasive surgery but also those of magnified three-dimensional vision and angulation of the robotic arm. However, there are no direct tactile sensation and force feedback, which can cause unwanted organ damage. We therefore used laparoscopy simultaneously to avoid a blind intraperitoneal area during robotic surgery for diaphragmatic eventration via transthoracic approach and describe the technique herein.

When we use a tool to explore or manipulate an object, friction between the surface of the tool and the fingerpads generates skin stretch cues that are related to the interaction forces between the tool and the object. In this study, we emulate these naturally occurring skin stretch cues in order to convey force direction and magnitude information to users during teleoperation. We hypothesize that skin stretch feedback is a useful substitute for kinesthetic force feedback in force-sensitive teleoperated tasks. In this study, ten participants performed teleoperated palpation to determine the orientation of a stiff region in a surrounding artificial tissue using five feedback conditions: skin stretch, force, reduced gain force, graphic, and vibration. When participants received skin stretch feedback, they localized the stiff region as well as with force feedback, with no increase in task completion time. Additionally, participants receiving skin-stretch feedback localized the stiff region statistically significantly more accurately than those using vibration feedback. Although participants using skin stretch exhibited higher interaction forces than when using force, vibration, and graphical feedback, skin stretch statistically significantly decreased interaction forces compared with reduced gain force feedback. Thus, skin-stretch feedback is a compelling substitute for force feedback and may be useful in scenarios where force feedback is reduced or infeasible.

Telemanipulation (master-slave operation) was developed in order to replace human being in hazardous environment, with using a slave device (e.g. robot). In this paper many kind of master devices are studied, for the precise control. Remote control needs much information about remote environment for good and safe human-machine interaction. Existing master devices with vision system or audio feedback lacks most of such information, therefor force feedback, called haptics become significant recently. In this paper we propose a new concept of force feedback. This system can overcome the bottlenecks of other feedback system in a user friendly way. Force sensor and laser distance sensor communicates the information from the gripper's status to the teleoperator by using force feedback module on a glove. Pneumatic pressure gives the operator distance information, while a Magnetorheological Fluid (MR-Fluid) based actuator presents the gripper's force. The experiment result shows the possibility of usage of such force feedback glove in combination with a robotic arm.

1. This research aims to study the edge computing devices with a faster processing and signal connecting function based on development of the physical layer and magnetic sensing devices. It is important matter for smart robot in multi-axis motor control. Based on the variable robot properties, it includes humanoid robots, wheeled machines, and fixed robots, respectively, are contented. This research is constructed as follows: to establishing an edge computing device system, the image vision system with neural-network learning theory to control robot arm path, multiple magnetic sensors in physical layer to analyze the signals of the motor, voltage, current, temperature, etc., are transmitted signal to the edge computing device and IoTs computing server system.2. IntroductionThe Industry 4.0 concept represents the trend of data exchange and automation in the manufacturing industry. It is defined by cloud computing, cyber-physical systems and the IoTs. In Industry 4.0, computers and automation technologies are integrated with robots and their components [1,2]. These robots and other components are remotely connected to a computer system, and the computer system is equipped with algorithms that can control and learning robots with minimal human intervention. For the motor device, the robot composes the control of the DC motor and relate to the accessories vital role. However, there are many factors that can misguide the controller's work, to make an unable to accurately control and/or have the desired dynamic behavior. For example, the resistance (Ohmic) of the coil winding of the motor is changed due to the temperature is different. If the vibration level is high and the wear of the rotor ball bearing becomes larger, the friction coefficient will also be affected. Therefore, the cloud computing environment of industrial factories, a continuous processing is required to re-evaluate the precise values of motor parameters and update the corresponding controller settings. In addition, it should be noted that the torque and speed sensors of the motor must have the correct selection to assist for achieving the correctness of the program.3. Theory and methods: multi-axis motor sensing and visual images of robotic armsCollaborative robots focus on intelligent motor control. Based on this construction, multiple sensor component signals are added. The physical information such as voltage, current, vibration, electric heating, magnetic flux sensing, and motor position is diagnosed through integrated control. This study, the artificial intelligent motor with the sensing signals, is used to control motor speed, positioning, constant horsepower, constant torque, force feedback, vibration resistance, overheating, overload, and instantaneous short circuit, as shown in Figure 1. The electromagnetic control of this paper is as follows:A. Single robotic arm function automation: Directly control the robot motor, at the same time, it has higher precision and speed to perform specific tasks repeatedly, but most of them cannot clearly actuate without an electromagnetic sensor.B. Part of the arm automation: It assists decision-making through environmental sensing input signals. In this study, robots use vision and sensors to determine the decision-making process, which is not easy to deal with accidental arm interference.C. Conditional autonomous robot: Control the arm according to the environmental monitoring behavior, but the external server can operate in real time.D. Highly autonomous robot: an environment that needs to be defined in an area or any situation without human operation.4. Experimental results and discussionBased on the motor control of intelligent robots, this research proposes a better and faster algorithm structure, which is a neural network-like combined with particle swarm method, which is used to calculate the convergence result of a lower loss function, which can accurately control the motor parameters and combine the IoTs, as shown in Figure 2. First, the robotic arm establishes multi-input sensing values, using the above algorithm to evaluate the robotic arm motor parameters, the prevention of interference from peripheral physical sensing, and the signal processing of external visual images. The issues raised based on past research experience include:(1) Establish edge and fog calculations to control the voltage and current of the motor, (2) establish neural network algorithm to calculate motor parameter, (3) construct the image vision system, (4) establish a smart robot platform, including wheel-shaped and human-shaped robots as experiments, which uses the above-mentioned deep learning theory to construct the test of the robot's motor motion and signal transmission response, (5) to use the multiple sensors, motor electromagnetic signals, piezoelectric sensors, and load signals, as the force feedback of the robot to calculate and judge. Finally, (6) to integrate the edge computing of the sensing layer, the fog computing of the signal processing layer, and faster computing layer through the IoTs cloud server. The IoTs chip is to use the dual-core Linkit-7697 (Taiwan’s MediaTek Inc). The chip can perform as a cloud computing and remote device, which indeed meet the requires of providing multi-sensing force feedback for cooperative robot handling objects. **

The control and operation of mechanical manipulators by a human and the use of sensory tactile and force feedback is reviewed. The terms telepresence, teleproprioception, and teletouch are defined and relevant technologies that have or could have been applied to teleoperation are discussed. An ideal method of tactile sensory feedback for teleoperators that is based upon reproduction of the object's contour is discussed, and its practicality considered. Previously developed components that could be used to build a system incorporating sensory tactile and force feedback are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In this paper, a mouse-shaped haptic device is proposed. The device solves conventional problems of previously developed devices such as structure and control complexity and operating difficulties. We adopted multiple finger inputs in order to carry out complicated tasks and in order to highly adapt to the environment. To develop the multi-fingered haptic device, we focused on anatomical knowledge and neurophysiology. Since there is a primacy of the visual information over somatic sensation, we believe the movable range of human fingers does not necessarily have to be completely satisfied, i.e. visual information can compensate for the difference between the displacement of the haptic device and that of the slave device in a master-slave system. We applied this feature to miniaturize and simplify the device. To confirm the usefulness of our developed device for a virtual reality system, we carried out three experiments: position control, familiarization and force feedback. The results show the effectiveness of the developed haptic device.

This paper reports the results of three experiments on tactile perception, spatial representation, and tactile display design for assistive mobility devices for blind, partially sighted, and deaf blind people. The results indicate the potentials of tactile displays for supporting environmental exploration and mobility. Voluntary test subjects showed reasonably good ability to determine the direction of motion of an arrow, with best recognition rates in the up and right directions. They showed reasonably good ability to use a tactile display to detect and avoid obstacles after a very short learning period and more limited ability to learn and remember an environmental representation using information from a tactile display and walking through the environment without specific instructions.

Vitreo-retinal surgery encompasses the surgical procedures performed on the vitreous humor and the retina. A procedure typically consists of the removal of the vitreous humor, the peeling of a membrane and/or the repair of a retinal detachment. Operations are performed with needle shaped instruments which enter the eye through surgeon made scleral openings. An instrument is moved by hand in four degrees of freedom (three rotations and one translation) through this opening. Two rotations (? and ? ) are for a lateral instrument tip movement. The other two DoFs (z and ?) are the translation and rotation along the instrument axis. Actuation of for example a forceps can be considered as a fifth DoF. Characteristically, the manipulation of delicate, micrometer range thick intraocular tissue is required. Today, eye surgery is performed with a maximum of two instruments simultaneously. The surgeon relies on visual feedback only, since instrument forces are below the human detection limit. A microscope provides the visual feedback. It forces the surgeon to work in a static and non ergonomic body posture. Although the surgeon’s proficiency improves throughout his career, hand tremor may become a problem around his mid-fifties. Robotically assisted surgery with a master-slave system enhances dexterity. The slave with instrument manipulators is placed over the eye. The surgeon controls the instrument manipulators via haptic interfaces at the master. The master and slave are connected by electronic hardware and control software. Implementation of tremor filtering in the control software and downscaling of the hand motion allow prolongation of the surgeon’s career. Furthermore, it becomes possible to do tasks like intraocular cannulation which can not be done by manually performed surgery. This thesis focusses on the master console. Eye surgery procedures are observed in the operating room of different hospitals to gain insight in the requirements for the master. The master console as designed has an adjustable frame, a 3D display and two haptic interfaces with a coarse adjustment arm each. The console is mounted at the head of the operating table and is combined with the slave. It is compact, easy to place and allows the surgeon to have a direct view on and a physical contact with the patient. Furthermore, it fits in today’s manual surgery arrangement. Each haptic interface has the same five degrees of freedom as the instrument inside the eye. Through these interfaces, the surgeon can feel the augmented instrument forces. Downscaling of the hand motion results in a more accurate instrument movement compared to manually performed surgery. Together with the visual feedback, it is like the surgeon grasps the instrument near the tip inside the eye. The similarity between hand motion and motion of the instrument tip as seen on the display results in an intuitive manipulation. Pre-adjustment of the interface is done via the coarse adjustment arm. Mode switching enables to control three or more instruments manipulators with only two interfaces. Two one degree of freedom master-slave systems with force feedback are built to derive the requirements for the haptic interface. Hardware in the loop testing provides valuable insights and shows the possibility of force feedback without the use of force sensors. Two five DoF haptic interfaces are realized for bimanual operation. Each DoF has a position encoder and a force feedback motor. A correct representation of the upscaled instrument forces is only possible if the disturbance forces are low. Actuators are therefore mounted to the fixed world or in the neighborhood of the pivoting point for a low contribution to the inertia. The use of direct drive for ' and and low geared, backdriveable transmissions for the other three DoFs gives a minimum of friction. Disturbance forces are further minimized by a proper cable layout and actuator-amplifier combinations without torque ripple. The similarity in DoFs between vitreo-retinal eye surgery and minimally invasive surgery (MIS) enables the system to be used for MIS as well. Experiments in combination with a slave robot for laparoscopic and thoracoscopic surgery show that an instrument can be manipulated in a comfortable and intuitive way. User experience of surgeons and others is utilized to improve the haptic interface further. A parallel instead of a serial actuation concept for the ' and DoFs reduces the inertia, eliminates the flexible cable connection between frame and motor and allows that the heat of the motor is transferred directly to the frame. A newly designed z-?? module combines the actuation and suspension of the hand held part of the interface and has a three times larger z range than in the first design of the haptic interface.