A High Performance Tactile Feedback Display and Its Integration in Teleoperation.

Haptic feedback should integrate kinaesthetic and tactile feedback. However current haptic displays do not satisfy the stringent performance and design requirements for integration in teleoperation and VR. This work presents the development of a compact, high performance tactile display for the fingertip. The compact design, high performance, reliability, and simple connectivity of this display make it suitable for immediate integration in current VR and master-slave haptic systems. In terms of performance this display achieves an excellent combination of force, amplitude and spatiotemporal resolution at the tactors, surpassing the performance of devices of a similar footprint. Its operation is based on the display of surface shape to an area of the fingertip through a 4×4 array of vertically moving tactors. The tactors are spring loaded and are actuated remotely by dc motors through a flexible tendon transmission. This work presents the overall design, control and performance of the device. A preliminary analysis of the transmission system is presented and is used to compensate for output errors induced by component elasticity.

Haptic technologies and applications have received enormous attention in the last decade. The incorporation of haptic modality into multimedia applications adds excitement and enjoyment to an application. It also adds a more natural feel to multimedia applications, that otherwise would be limited to vision and audition, by engaging as well the user's sense of touch, giving a more intrinsic feel essential for ambient intelligent applications. However, the improvement of an application's Quality of Experience (QoE) by the addition of haptic feedback is still not completely understood. The research presented in this paper focuses on the effect of haptic feedback and what it potentially adds to the experience of the user as opposed to the traditional visual and auditory feedback. In essence, it investigates certain issues regarding stylus-based haptic education applications and haptic-enhanced entertainment videos. To this end, we used two haptic applications: the haptic handwriting learning tool to experiment with force feedback haptic interaction and the tactile YouTube application for tactile haptic feedback. In both applications, our analysis shows that the addition of haptic feedback will increase the QoE in the absence of fatigue or discomfort for this category of applications. This implies that the incorporation of haptic modality (both force feedback as well as tactile feedback) has positively contributed to the overall QoE for the users.

The use of haptic technologies has recently become immensely essential in Human-Computer Interaction to improve user experience and performance. With the introduction of tactile feedback on a touchscreen device, commonly known as surface haptics, several applications and interaction paradigms have become a reality. However, the effects of tactile feedback on the preference of 2D images in visuo-tactile exploration task on touchscreen devices remain largely unknown. In this study, we investigated differences of preference score (the tendency of participants to like/dislike a 2D image based on its visual and tactile properties), reach time, interaction time, and response time under four conditions of feedback: no tactile feedback, high-quality of tactile information (sharp tactile texture), low-quality of tactile information (blurred tactile texture), and incorrect tactile information (mismatch tactile texture). The tactile feedback is rendered in the form of roughness that is simulated by modulating the friction between the finger and the surface and is derived from the 2D image. Thirty-six participants completed visuo-tactile exploration tasks for a total of 36 trials (3 2D images × 4 tactile textures × 3 repetitions). Results showed that the presence of tactile feedback enhanced users’ preference (tactile feedback conditions were rated significantly higher than the no tactile feedback condition for preference regardless of the quality/correctness of tactile feedback). This finding is also supported through results from self-reporting where 88.89% of participants preferred to experience the 2D image with tactile feedback. Additionally, the presence of tactile feedback resulted in significantly larger interaction time and response time compared to the no tactile feedback condition. Furthermore, the quality and correctness of tactile information significantly impacted the preference rating (sharp tactile textures were rated statistically higher than blurred tactile and mismatched tactile textures). All of these findings demonstrate that tactile feedback plays a crucial role in users’ preference and thus motivates further the development of surface haptic technologies.

Integration of the correct tactile and kinesthetic force feedback response with an accurate computational model of a compliant environment is a formidable challenge. We examine several design issues that arise in the construction of a compliance renderer, specifically the interaction between impedances of tactile displays, impedances of robot arms, and the computational model. We also describe an implementation of a compliance rendering system combining a low-impedance robot arm for large workspace kinesthetic force feedback, a high-impedance shape display for distributed tactile feedback to the finger pad, and a real-time finite element modeler. To determine the efficacy of the integration of tactile and kinesthetic force feedback components, we conducted a study examining the user’s ability to discriminate stiffness. Subjects were able to reliably detect a 20% difference in rendered material stiffness using our compliance rendering system.

This paper reports the results of three experiments on tactile perception, spatial representation, and tactile display design for assistive mobility devices for blind, partially sighted, and deaf blind people. The results indicate the potentials of tactile displays for supporting environmental exploration and mobility. Voluntary test subjects showed reasonably good ability to determine the direction of motion of an arrow, with best recognition rates in the up and right directions. They showed reasonably good ability to use a tactile display to detect and avoid obstacles after a very short learning period and more limited ability to learn and remember an environmental representation using information from a tactile display and walking through the environment without specific instructions.

The control and operation of mechanical manipulators by a human and the use of sensory tactile and force feedback is reviewed. The terms telepresence, teleproprioception, and teletouch are defined and relevant technologies that have or could have been applied to teleoperation are discussed. An ideal method of tactile sensory feedback for teleoperators that is based upon reproduction of the object's contour is discussed, and its practicality considered. Previously developed components that could be used to build a system incorporating sensory tactile and force feedback are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In this paper, a mouse-shaped haptic device is proposed. The device solves conventional problems of previously developed devices such as structure and control complexity and operating difficulties. We adopted multiple finger inputs in order to carry out complicated tasks and in order to highly adapt to the environment. To develop the multi-fingered haptic device, we focused on anatomical knowledge and neurophysiology. Since there is a primacy of the visual information over somatic sensation, we believe the movable range of human fingers does not necessarily have to be completely satisfied, i.e. visual information can compensate for the difference between the displacement of the haptic device and that of the slave device in a master-slave system. We applied this feature to miniaturize and simplify the device. To confirm the usefulness of our developed device for a virtual reality system, we carried out three experiments: position control, familiarization and force feedback. The results show the effectiveness of the developed haptic device.

Vitreo-retinal surgery encompasses the surgical procedures performed on the vitreous humor and the retina. A procedure typically consists of the removal of the vitreous humor, the peeling of a membrane and/or the repair of a retinal detachment. Operations are performed with needle shaped instruments which enter the eye through surgeon made scleral openings. An instrument is moved by hand in four degrees of freedom (three rotations and one translation) through this opening. Two rotations (? and ? ) are for a lateral instrument tip movement. The other two DoFs (z and ?) are the translation and rotation along the instrument axis. Actuation of for example a forceps can be considered as a fifth DoF. Characteristically, the manipulation of delicate, micrometer range thick intraocular tissue is required. Today, eye surgery is performed with a maximum of two instruments simultaneously. The surgeon relies on visual feedback only, since instrument forces are below the human detection limit. A microscope provides the visual feedback. It forces the surgeon to work in a static and non ergonomic body posture. Although the surgeon’s proficiency improves throughout his career, hand tremor may become a problem around his mid-fifties. Robotically assisted surgery with a master-slave system enhances dexterity. The slave with instrument manipulators is placed over the eye. The surgeon controls the instrument manipulators via haptic interfaces at the master. The master and slave are connected by electronic hardware and control software. Implementation of tremor filtering in the control software and downscaling of the hand motion allow prolongation of the surgeon’s career. Furthermore, it becomes possible to do tasks like intraocular cannulation which can not be done by manually performed surgery. This thesis focusses on the master console. Eye surgery procedures are observed in the operating room of different hospitals to gain insight in the requirements for the master. The master console as designed has an adjustable frame, a 3D display and two haptic interfaces with a coarse adjustment arm each. The console is mounted at the head of the operating table and is combined with the slave. It is compact, easy to place and allows the surgeon to have a direct view on and a physical contact with the patient. Furthermore, it fits in today’s manual surgery arrangement. Each haptic interface has the same five degrees of freedom as the instrument inside the eye. Through these interfaces, the surgeon can feel the augmented instrument forces. Downscaling of the hand motion results in a more accurate instrument movement compared to manually performed surgery. Together with the visual feedback, it is like the surgeon grasps the instrument near the tip inside the eye. The similarity between hand motion and motion of the instrument tip as seen on the display results in an intuitive manipulation. Pre-adjustment of the interface is done via the coarse adjustment arm. Mode switching enables to control three or more instruments manipulators with only two interfaces. Two one degree of freedom master-slave systems with force feedback are built to derive the requirements for the haptic interface. Hardware in the loop testing provides valuable insights and shows the possibility of force feedback without the use of force sensors. Two five DoF haptic interfaces are realized for bimanual operation. Each DoF has a position encoder and a force feedback motor. A correct representation of the upscaled instrument forces is only possible if the disturbance forces are low. Actuators are therefore mounted to the fixed world or in the neighborhood of the pivoting point for a low contribution to the inertia. The use of direct drive for ' and and low geared, backdriveable transmissions for the other three DoFs gives a minimum of friction. Disturbance forces are further minimized by a proper cable layout and actuator-amplifier combinations without torque ripple. The similarity in DoFs between vitreo-retinal eye surgery and minimally invasive surgery (MIS) enables the system to be used for MIS as well. Experiments in combination with a slave robot for laparoscopic and thoracoscopic surgery show that an instrument can be manipulated in a comfortable and intuitive way. User experience of surgeons and others is utilized to improve the haptic interface further. A parallel instead of a serial actuation concept for the ' and DoFs reduces the inertia, eliminates the flexible cable connection between frame and motor and allows that the heat of the motor is transferred directly to the frame. A newly designed z-?? module combines the actuation and suspension of the hand held part of the interface and has a three times larger z range than in the first design of the haptic interface.

Whereas vision and force feedback—either at the wrist or at the joint level—for robotic manipulation purposes has received considerable attention in the literature, the benefits that tactile sensors can provide when combined with vision and force have been rarely explored. In fact, there are some situations in which vision and force feedback cannot guarantee robust manipulation. Vision is frequently subject to calibration errors, occlusions and outliers, whereas force feedback can only provide useful information on those directions that are constrained by the environment. In tasks where the visual feedback contains errors, and the contact configuration does not constrain all the Cartesian degrees of freedom, vision and force sensors are not sufficient to guarantee a successful execution. Many of the tasks performed in our daily life that do not require a firm grasp belong to this category. Therefore, it is important to develop strategies for robustly dealing with these situations. In this article, a new framework for combining tactile information with vision and force feedback is proposed and validated with the task of opening a sliding door. Results show how the vision-tactile-force approach outperforms vision-force and force-alone, in the sense that it allows to correct the vision errors at the same time that a suitable contact configuration is guaranteed.

The rapid advancements in robotics and automation have highlighted the need for robotic systems capable of adapting to the diverse physical properties of objects. Traditional grippers often lack the versatility to handle both hard and soft objects without extensive reprogramming or hardware adjustments. Previous studies have explored various approaches to this challenge, including tactile sensors and force feedback mechanisms to distinguish object properties. For instance, research by Calandra et al. (2018) utilized deep learning with tactile data to enable robotic hands to identify objects and adjust grip accordingly. Similarly, the paper by Li et al. (2020) “Design and performance characterization of a soft robot hand with fingertip haptic feedback for teleoperation”, focuses on designing and characterizing a soft robotic hand with fingertip haptic feedback for teleoperation emphasizing real-time tactile sensing and feedback mechanisms. However, these studies primarily focus on tactile feedback or specialized hardware, limiting their applicability in scenarios where such systems are not available or practical.This study introduces a novel machine learning-based approach, focusing on the use of visual data alone to classify and adapt to the hardness or softness of objects. By leveraging the CIFAR-100 dataset, we trained a deep learning model based on the ResNet50 architecture, achieving significant results in binary classification of hard and soft objects. The CIFAR-100 dataset, consisting of 100 diverse object categories/classes, was reorganized into two classes: hard (39 categories) and soft (61 categories). The ResNet50 model, pre-trained on ImageNet, was fine-tuned specifically for this task, with modifications to the last 210 layers to enhance its adaptability.Data augmentation techniques, including rotations, translations, shearing, zooming, and horizontal flipping, were applied to simulate real-world variations, ensuring robust learning. The model was further refined with additional fully connected layers, dropout, and batch normalization to prevent overfitting. Optimized using the AdamW optimizer, the model achieved a training accuracy of 83.31% and a validation accuracy of 80.25%, with a test accuracy of 80%. The precision, recall, and F1-scores were 0.82, 0.86, and 0.84 for the soft object class, and 0.76, 0.71, and 0.73 for the hard object class, demonstrating the model’s effectiveness in distinguishing between hard and soft objects without the need for specialized sensors. It is also observed that the accuracy of the model in relation to hard objects is significantly lesser as compared to soft objects. It is understood that the CIFAR-100 dataset (comprising of 100 classes) is inadequate for model training, so we are exploring the ILSVRC (ImageNet subset) dataset for model training in future. The research is useful in a variety of fields where the focus lies on object handling. In everyday life, we encounter a wide array of objects with varying degrees of hardness, requiring different levels of care and precision during handling. By relying on visual data and machine learning algorithms, as demonstrated in this research, robotic systems can become more autonomous and versatile, reducing the burden on human operators and improving overall efficiency. This approach can lead to cost savings by reducing the need for specialized hardware, such as tactile sensors, which are often expensive and difficult to integrate.

During interaction with remote environments, the operator may benefit from the addition of force feedback to the ubiquitous visual feedback. However, the apparatus required for reactive force feedback (feedback which imposes the remote environment's motion-constraints on the user by applying joint torques) is cumbersome and expensive, especially when implemented in conjunction with high degree-offreedom precision joint motion sensing. Non-reactive, tactile feedback can provide similar information, and can be implemented at much lower cost. The purposes of this research were (1)todesign and demonstrate an inexpensive tactile feedback system, and (2) to determine the extent to which such a system could aid in the performance of a simple teleoperation task. After some experimentation with some different display technologies, and preliminary design, a vibrotactile display was chosen because of its low weight, size, and low cost. The final design consisted of two voice-coils, one each for the thumb and the index finger, which were driven by a 250 Hz variableamplitude signal produced by an analog electronics unit which was controlled by a PC. Experimental results are provided to show that the addition of the tactile display provides a small but significant improvement in manual tracking performance over the use of the visual display alone, and that the tracking task may be performed with only the tactile display. In further experiments the tactile display is compared with reactive force-feedback and is shown to confer most of the reactive display's performance improvement over tracking with only a visual display.

The development of successful tactile displays can be assisted by the capacity to assess their performance in integrated virtual reality applications. The design of single tactor or desktop tactile displays has been useful for scientific analysis of the mechanisms of touch. Nevertheless, it does not provide useful information about the quality of immersion in VR environments integrated with tactile feedback where user mobility and comfort are paramount. The present work introduces a new wireless portable tactile display for the finger. This new device demonstrates excellent levels of force output combined with good bandwidth in a configuration of a wearable 4times4 array of vertically moving tactors. The focus of this design is small size, comfort, mobility and ultimately integration into a complete Virtual Reality system for tactile surface texture exploration with 6DOF in the finger's work envelope

This study examined the impact of adding vibrotactile feedback to existing force feedback in a VR-Haptic needle insertion simulation. A new programmable tactile feedback system using direct digital synthesis (DDS) to drive LRA and piezo actuators was designed and integrated into the VR needle simulation. A user study measured needle insertion performance, usability, and user preferences for realistic vein insertion. Results showed that vibrotactile feedback (via piezo and LRA) combined with force feedback improved needle insertion performance over LRA and realism over Force Feedback alone. Participants preferred the combination of force and vibrotactile feedback. While vibrotactile feedback cannot replace force feedback, it serves as a valuable supplemental cue, enhancing overall realism. This suggests that dual-modal haptic feedback, combining force (kinesthetic) and tactile (cutaneous) cues, can address the limitations of single-modal force feedback.

During traditional surgery, the surgeons’ hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn’t include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator’s side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.

Skin deformation has been previously shown as vital for lump detection during the direct manipulation of an object. A deformation-based tactile feedback system is developed and presented that senses and displays this tactile information for palpation in tele-surgical applications. A biologically-inspired tactile sensor modelled from the human fingertip is used to obtain the skin deformation during interaction. It does this by measuring the displacement of the sensor’s artificial intermediate epidermal ridges using a simple, computationally-efficient algorithm. The design of a previously-published tactile shape display is then recreated and improved for relaying this sensed information on to a human user’s fingertip. This tactile display uses remote actuation to reduce the mass of the display, avoiding the issue of adding a large mass to a tele-operation interface. The tactors within the developed display exhibit 2.5 mm displacement, with a 2.5 mm spacing, 12 Hz bandwidth and a stiffness of 5.0 N/mm. A linear relationship is found between sensor deformation and tactor displacement and the spatial performance of the system is proven by successfully detecting lumps within artificial muscle tissue. This new deformation-based tactile system offers an intuitive sense of touch with minimal processing.KeywordsTactile SensorTactile FeedbackSensor MembraneTactile DisplayHuman FingerThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.