A tactile sensing and feedback system for tumor localization

This paper provides a solution for fast haptic information gain during soft tissue palpation using a Variable Lever Mechanism (VLM) probe. More specifically, we investigate the impact of stiffness variation of the probe to condition likelihood functions of the kinesthetic force and tactile sensors measurements during a palpation task for two sweeping directions. Using knowledge obtained from past probing trials or Finite Element (FE) simulations, we implemented this likelihood conditioning in an autonomous palpation control strategy. Based on a recursive Bayesian inferencing framework, this new control strategy adapts the sweeping direction and the stiffness of the probe to detect abnormal stiff inclusions in soft tissues. This original control strategy for compliant palpation probes shows a sub-millimeter accuracy for the 3D localization of the nodules in a soft tissue phantom as well as a 100% reliability detecting the existence of nodules in a soft phantom.

Haptic information in robotic surgery can significantly improve clinical outcomes and help detect hard soft-tissue inclusions that indicate potential abnormalities. Visual representation of tissue stiffness information is a cost-effective technique. Meanwhile, direct force feedback, although considerably more expensive than visual representation, is an intuitive method of conveying information regarding tissue stiffness to surgeons. In this study, real-time visual stiffness feedback by sliding indentation palpation is proposed, validated, and compared with force feedback involving human subjects. In an experimental tele-manipulation environment, a dynamically updated color map depicting the stiffness of probed soft tissue is presented via a graphical interface. The force feedback is provided, aided by a master haptic device. The haptic device uses data acquired from an F/T sensor attached to the end-effector of a tele-manipulated robot. Hard nodule detection performance is evaluated for 2 modes (force feedback and visual stiffness feedback) of stiffness feedback on an artificial organ containing buried stiff nodules. From this artificial organ, a virtual-environment tissue model is generated based on sliding indentation measurements. Employing this virtual-environment tissue model, we compare the performance of human participants in distinguishing differently sized hard nodules by force feedback and visual stiffness feedback. Results indicate that the proposed distributed visual representation of tissue stiffness can be used effectively for hard nodule identification. The representation can also be used as a sufficient substitute for force feedback in tissue palpation.

The objective of this article is to provide a thorough examination of the current state of haptic information feedback in minimally invasive surgery (MIS), including research conducted between 1985 and the present. Despite the fact that haptic information input in MIS is currently scarce, the comprehensive analysis emphasises that it represents a potential option. Surgeons might potentially get substantial benefits from receiving more input about force information, which would therefore enhance their surgical accuracy and control. The extant literature highlights a significant knowledge deficit regarding the intricate ramifications of haptic feedback, specifically with regard to the sense of slip and gripping forces. It is crucial to investigate the possible advantages of incorporating supplementary haptic information in order to protect against tissue damage when performing manipulation treatments. It is of the utmost importance to fill this void, since doing so might revolutionise the field of MIS via the mitigation of unintended tissue injury and the improvement of surgical results. By capitalising on technology progress to provide more extensive haptic feedback, surgeons have the ability to enhance their tactile perception, optimise the application of force, and reduce the likelihood of tissue injuries when performing complex surgical procedures.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

Teleoperated minimally invasive surgical robots can significantly enhance a surgeon's accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon's performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through "virtual fixtures" and the automatic acquisition of tissue properties.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

The usability of virtual reality (VR) controller interfaces are often complex and difficult for first time users. Most controllers provide minimal feedback which relegates the potential for heightened interaction and feedback within virtual experiences. This research explores how haptic technology systems partnered with VR can deliver immersive interactions between user and virtual environment (VE). This research involves the development of a haptic glove interface prototype that incorporates a force feedback and vibrotactile feedback system. It focuses on determining a workflow that communicates in real-time user interaction and environmental feedback using Unreal Engine and the produced haptic glove system. Testing and calibrating the prototype feedback system provided a baseline for developers to rationalise and improve accuracy of current real-time virtual feedback systems. The evaluation of this research in industry unfolds new technical knowledge for implementing a wider range of haptic technologies within VR. This further development would involve reviewing the usability and interaction standards for VR users in the design process.

When humans are asked to palpate a soft tissue to locate a hard nodule, they regulate the stiffness, speed, and force of the finger during examination. If we understand the relationship between these behavioral variables and haptic information gain (transfer entropy) during manual probing, we can improve the efficacy of soft robotic probes for soft tissue palpation, such as in tumor localization in minimally invasive surgery. Here, we recorded the muscle co-contraction activity of the finger using EMG sensors to address the question as to whether joint stiffness control during manual palpation plays an important role in the haptic information gain. To address this question, we used a soft robotic probe with a controllable stiffness joint and a force sensor mounted at the base to represent the function of the tendon in a biological finger. Then, we trained a Markov chain using muscle co-contraction patterns of human subjects, and used it to control the stiffness of the soft robotic probe in the same soft tissue palpation task. The soft robotic experiments showed that haptic information gain about the depth of the hard nodule can be maximized by varying the internal stiffness of the soft probe.

Camera drones allow exploring remote scenes that are inaccessible or inappropriate to visit in person. However, these exploration experiences are often limited due to insufficient scene information provided by front cameras, where only 2D images or videos are supplied. Combining a camera drone vision with haptic feedback would augment users' spatial understandings of the remote environment. But such designs are usually difficult for users to learn and apply, due to the complexity of the system and unfluent UAV control. In this paper, we present a new telepresence system for remote environment exploration, with a drone agent controlled by a VR mid-air panel. The drone is capable of generating real-time location and landmark details using integrated Simultaneous Location and Mapping (SLAM). The SLAMs' point cloud generations are produced using RGB input, and the results are passed to a Generative Adversarial Network (GAN) to reconstruct 3D remote scenes in real-time. The reconstructed objects are taken advantage of by haptic devices which could improve user experience through haptic rendering. Capable of providing both visual and haptic feedback, our system allows users to examine and exploit remote areas without having to be physically present. An experiment has been conducted to verify the usability of 3D reconstruction result in haptic feedback rendering.

As robotics prehension systems and virtual reality applications are in constant evolution, the need for high-fidelity haptic interaction increases. This helps ensure and enhance user immersion and handling precision. While commercial haptic interfaces offer high performance, their prohibitive cost limits their widespread adoption in general-purpose robotics. Furthermore, many low-cost solutions suffer from limited transparency, where the operator constantly fights the friction of the actuator even during free motion. This article presents the design and development of an innovative, cost-effective master-slave robotic system aimed at democratizing efficient haptic feedback devices. The solution is intended for remote manipulation of objects with a maximum mass of 1 kg, while limiting the gripping force to 50 N, thus ensuring the integrity of objects being manipulated. The device includes a master haptic module in the form of a clamp that reproduces the thumb-index-middle finger gripping motion performed by the user. The system relies on a custom haptic interface measuring the angular position of the master gripper, which is transmitted in real time to the slave gripper, so as to adjust the position of the clamp accordingly, thus optimizing the grasping control loop. As soon as an object is detected, using a force sensor integrated into the slave gripper, the master motor renders a resistive force, preventing the user from closing the haptic module. The other part of the system is the slave mechanical gripper with three fingers, each with three phalanges based on human anatomy, allowing the clamp to mechanically conform to irregular object geometries with a single actuator. The last but not least innovative aspect lies in the implementation of a current sensor, which provides the haptic feedback. The force applied by the user is reproduced by the slave gripper using current sensors, eliminating the need for expensive force-torque sensors while maintaining a responsive feedback loop.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Haptic feedback provides an essential sensory stimulus crucial for interaction and analyzing three-dimensional spatio-temporal phenomena on surface visualizations. Given its ability to provide enhanced spatial perception and scene maneuverability, virtual reality (VR) catalyzes haptic interactions on surface visualizations. Various interaction modes, encompassing both mid-air and on-surface interactions-with or without the application of assisting force stimuli-have been explored using haptic force feedback devices. In this paper, we evaluate the use of on-surface and assisted on-surface haptic modes of interaction compared to a no-haptic interaction mode. A force-based haptic stylus is used for all three modalities; the on-surface mode uses collision based forces, whereas the assisted on-surface mode is accompanied by an additional snapping force. We conducted a within-subjects user study involving fundamental interaction tasks performed on surface visualizations. Keeping a consistent visual design across all three modes, our study incorporates tasks that require the localization of the highest, lowest, and random points on surfaces; and tasks that focus on brushing curves on surfaces with varying complexity and occlusion levels. Our findings show that participants took almost the same time to brush curves using all the interaction modes. They could draw smoother curves using the on-surface interaction modes compared to the no-haptic mode. However, the assisted on-surface mode provided better accuracy than the on-surface mode. The on-surface mode was slower in point localization, but the accuracy depended on the visual cues and occlusions associated with the tasks. Finally, we discuss participant feedback on using haptic force feedback as a tangible input modality and share takeaways to aid the design of haptics-based tangible interactions for surface visualizations.

In both teleoperation in real space and exploration in virtual space, ‘passive’ and ‘active’ haptic feedback can help to improve the performance of the task, especially in object handover and exploring. However, the current wearable haptic devices are hard to display continuous omnidirectional motion feedback simultaneously, which makes it not yet achieved. In this study, we thus propose a cutaneous haptic device, which enables continuous omnidirectional motion feedback for exhibiting ‘active’ and ‘passive’ haptic feedback. By applying small smart actuators (i.e., piezo actuators), the device can obtain contact force and be wearable. By arranging the closed loop with a plain-woven structure, our device makes continuous omnidirectional motion feedback possible. Our 35 g device can generate 0.94 N contact force and 0.5 N shear force. The passive and active haptic evaluations also proved its haptic capability. In conclusion, our proposed device with ‘active’ and ‘passive’ haptic feedback can provide continuous omnidirectional motion making it possible to be used for precise teleoperation.

Mid-air haptic interfaces have several advantages - the haptic information is delivered directly to the user, in a manner that is unobtrusive to the immediate environment. They operate at a distance, thus easier to discover; they are more hygienic and allow interaction in 3D. We validate, for the first time, in a preliminary study with sighted and a user study with blind participants, the use of mid-air haptics for conveying Braille. We tested three haptic stimulation methods, where the haptic feedback was either: a) aligned temporally, with haptic stimulation points presented simultaneously (Constant); b) not aligned temporally, presenting each point independently (Point-By-Point); or c) a combination of the previous methodologies, where feedback was presented Row-by-Row. The results show that mid-air haptics is a viable technology for presenting Braille characters, and the highest average accuracy (94% in the preliminary and 88% in the user study) was achieved with the Point-by-Point method.

Haptic feedback is important to improve the operability of virtual objects. However, AR systems with haptic feedback are still not many. Moreover, users cannot perform tasks that need large workspaces in those systems because of physical constraints from haptic feedback devices. In this paper, we combined a wearable haptic device and a marker-less AR technology to develop an AR system in which users can perform activities that need large workspaces. Differences of the operability of virtual objects in an assembly task of large machines are examined to evaluate effects of the haptic feedback in this system. The result indicates that haptic feedback is effective and users can perform activities that need large space in this system.