Remote Environment Exploration with Drone Agent and Haptic Force Feedback

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

Haptic feedback provides an essential sensory stimulus crucial for interaction and analyzing three-dimensional spatio-temporal phenomena on surface visualizations. Given its ability to provide enhanced spatial perception and scene maneuverability, virtual reality (VR) catalyzes haptic interactions on surface visualizations. Various interaction modes, encompassing both mid-air and on-surface interactions-with or without the application of assisting force stimuli-have been explored using haptic force feedback devices. In this paper, we evaluate the use of on-surface and assisted on-surface haptic modes of interaction compared to a no-haptic interaction mode. A force-based haptic stylus is used for all three modalities; the on-surface mode uses collision based forces, whereas the assisted on-surface mode is accompanied by an additional snapping force. We conducted a within-subjects user study involving fundamental interaction tasks performed on surface visualizations. Keeping a consistent visual design across all three modes, our study incorporates tasks that require the localization of the highest, lowest, and random points on surfaces; and tasks that focus on brushing curves on surfaces with varying complexity and occlusion levels. Our findings show that participants took almost the same time to brush curves using all the interaction modes. They could draw smoother curves using the on-surface interaction modes compared to the no-haptic mode. However, the assisted on-surface mode provided better accuracy than the on-surface mode. The on-surface mode was slower in point localization, but the accuracy depended on the visual cues and occlusions associated with the tasks. Finally, we discuss participant feedback on using haptic force feedback as a tangible input modality and share takeaways to aid the design of haptics-based tangible interactions for surface visualizations.

Teleoperated minimally invasive surgical robots can significantly enhance a surgeon's accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon's performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through "virtual fixtures" and the automatic acquisition of tissue properties.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

Passing an object between two people is a common event that happens in various forms for example when giving someone a cup of coffee. An experimental study is presented where passing objects between two people in a virtual environment with haptic feedback was compared to passing objects in a nonhaptic virtual environment. The aim of the experiment was to investigate if and how added haptic feedback in such an environment affects perceived virtual presence, perceived social presence and perceived task performance. A within subject design was used, were nine pairs of subjects performed a hand off task with six differently sized cubes without audio communication. Results showed that haptic force feedback significantly improved perceived virtual presence, perceived social presence and perceived performance in this experiment.KeywordsHapticPresenceSocial presenceCollaborative

This paper presents the research work on developing a virtual sculpting system with haptic interface integrated with PowerWall system for complex product design. The PowerWall is a large scale (10 ft by 7.5 ft) immersive Virtual Environment (VE). The approach is to apply virtual sculpting method by interactively carving a workpiece using a virtual tool. With the implementation of stereoscopic visual feedback and haptic force feedback in the PowerWall, the designer would appreciate a much better understanding of the 3D shape geometry and can explore through the 3D scene like he/she can do in the real world. The “hybrid interaction technique” is presented as solution to solve the mismatch between the small workspace of the haptic device and the large size of PowerWall system.

The Earth’s surface is mostly water-covered and the ocean is the source of a significant slice on natural resources and renewable energies. However, only a small fraction of the ocean has been surveyed. Being able to estimate the 3D model of the environment from a single video eases the task of surveying the underwater environment, saves costs and opens doors to autonomous exploration of unknown environments. In order to estimate the 3D structure of a vehicle’s surrounding environment, we propose a deep learning based Simultaneous Localization and Mapping (SLAM) method. With our method, it is possible to predict a depth map of a given video frame while, at the same time, estimate the movement of the vehicle between different frames. Our method is completely self-supervised, meaning that it only requires a dataset of videos, without ground truth, to be trained. We propose a novel learning based depth map prior using Generative Adversarial Networks (GANs) to improve the depth map prediction results. We evaluate the performance of our method on the KITTI dataset and on a private dataset of subsea inspection videos. We show that our method outperforms state of the art SLAM methods in both depth prediction and pose estimation tasks. In particular, our method achieves a mean Absolute Trajectory Error of 1.6 feet in our private subsea test dataset.

In both teleoperation in real space and exploration in virtual space, ‘passive’ and ‘active’ haptic feedback can help to improve the performance of the task, especially in object handover and exploring. However, the current wearable haptic devices are hard to display continuous omnidirectional motion feedback simultaneously, which makes it not yet achieved. In this study, we thus propose a cutaneous haptic device, which enables continuous omnidirectional motion feedback for exhibiting ‘active’ and ‘passive’ haptic feedback. By applying small smart actuators (i.e., piezo actuators), the device can obtain contact force and be wearable. By arranging the closed loop with a plain-woven structure, our device makes continuous omnidirectional motion feedback possible. Our 35 g device can generate 0.94 N contact force and 0.5 N shear force. The passive and active haptic evaluations also proved its haptic capability. In conclusion, our proposed device with ‘active’ and ‘passive’ haptic feedback can provide continuous omnidirectional motion making it possible to be used for precise teleoperation.

Haptic feedback is important to improve the operability of virtual objects. However, AR systems with haptic feedback are still not many. Moreover, users cannot perform tasks that need large workspaces in those systems because of physical constraints from haptic feedback devices. In this paper, we combined a wearable haptic device and a marker-less AR technology to develop an AR system in which users can perform activities that need large workspaces. Differences of the operability of virtual objects in an assembly task of large machines are examined to evaluate effects of the haptic feedback in this system. The result indicates that haptic feedback is effective and users can perform activities that need large space in this system.

Periodic Kinesthetic Guidance Cannot Expedite Learning Surgical Skills.

Although more than 1 in 10 people experience pruritus, there are limited medical technologies that can accurately and continuously quantify and simultaneously reduce scratching behaviors through nonpharmacological methods. To evaluate the accuracy and efficacy of an artificial intelligence-enabled wearable sensor with closed-loop haptic feedback to decrease nocturnal scratch in patients with mild atopic dermatitis who report a moderate to severe degree of scratching. This single-arm 2-stage cohort study with a within-participants design was conducted at a single center and carried out in an at-home environment. Adult patients with atopic dermatitis were recruited from the Northwestern University Department of Dermatology in Chicago, Illinois. Participants were fluent in English, 18 years old or older, had a diagnosis of atopic dermatitis, and self-reported moderate or severe scratching behaviors. Each participant's disease at time of recruitment was scored via the Validated Investigator Global Assessment for Atopic Dermatitis. Data were collected from April to July 2023. Haptic feedback delivered by a wearable sensor mounted on the hand triggered whenever nocturnal scratch was detected by an artificial intelligence algorithm. Participants initially wore the sensor for sensing only for 7 nights to assess baseline nocturnal scratching and sleep parameters. This was followed by an additional 7 nights of wearing the sensor with haptic feedback activated. Retrospective analysis was performed for scratch events and scratch duration per night and per hour of sleep opportunity. Paired t tests were used to compare changes in patient scratching behaviors before and after use of the artificial intelligence-enabled haptic feedback devices. Of 10 included patients, 6 were female, and the mean (SD) age was 36 (12) years. All patients had a Validated Investigator Global Assessment for Atopic Dermatitis score of 0 to 2 (clear to mild) who contributed a total of 104 sleep nights and 831 monitoring hours. No patients were lost to follow-up. There was a significant decrease in mean (SD) scratch events nightly (45.6 [24.0] vs 32.8 [13.0]; P = .03), a 28% difference, and mean (SD) scratch duration per hour of sleep opportunity (15.8 [10.7] seconds vs 7.9 [3.7] seconds; P = .01), a 50% difference, when haptic feedback was activated in the second week without a decrease in total sleep opportunity. This study found that haptic feedback may be used as a nonpharmacological intervention to reduce nocturnal scratching in patients with mild atopic dermatitis. Future randomized studies are needed to confirm.

Retinal microsurgery is one of the most technically difficult surgeries since it is performed at the threshold of human capability. If certain retinal conditions are left untreated, they can lead to severe damage, including irreversible blindness. Thus, techniques for reliable retinal microsurgery operations are critical. Recent research shows promise for improving surgical safety by implementing various types of sensory input and output. Sensory information is used to inform the surgeon about the environment inside the eye in real time. This review examines literature that discusses human factors and ergonomics (HFE) of sensory inputs and outputs of retinal microsurgery instrumentation with a focus on force and haptic feedback. Thirty-four studies were reviewed on the following topics: (1) variation between different input sensory devices and their performance, (2) variation between alternative output sensory devices and their performance, and (3) variation between alternative output sensory devices and their user satisfaction. This review finds that the implementation of HFE is important for the consideration of retinal microsurgery devices, but it is largely missing from current research. The addition of direct comparisons between devices, measures of user acceptance, usability evaluations, and greater realism in testing would help advance the use of haptic sensory feedback for retinal microsurgery instruments.

Using the spatial structure of various indoor environments as prior knowledge, the robot would construct the map more efficiently. Autonomous mobile robots generally apply simultaneous localization and mapping (SLAM) methods to understand the reachable area in newly visited environments. However, conventional mapping approaches are limited by only considering sensor observation and control signals to estimate the current environment map. This paper proposes a novel SLAM method, map completion network-based SLAM (MCN-SLAM), based on a probabilistic generative model incorporating deep neural networks for map completion. These map completion networks are primarily trained in the framework of generative adversarial networks (GANs) to extract the global structure of large amounts of existing map data. We show in experiments that the proposed method can estimate the environment map 1.3 times better than the previous SLAM methods in the situation of partial observation.

During traditional surgery, the surgeons’ hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn’t include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator’s side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.

A new controller design method for bilateral force-reflecting teleoperators orientated to telesurgical applications is presented. The impedance of the remote environment is estimated online and the parameters of the system controller are adaptively updated accordingly. The proposed method shows quite visible improvement of haptic (force) feedback fidelity under the condition of stability compared to traditional passivity-based approaches.