Visual and Haptic Feedback in Multitasking Drone Operation: Effects of Task Complexity on Performance

We routinely have physical interactions with others, whether it be handing someone a glass of water or jointly moving a heavy object together. These sensorimotor interactions between humans typically rely on visual feedback and haptic feedback. Recent single-participant studies have highlighted that the unique noise and time delays of each sense must be considered to estimate the state, such as the position and velocity, of one's own movement. However, we know little about how visual feedback and haptic feedback are used to estimate the state of another person. Here, we tested how humans utilize visual feedback and haptic feedback to estimate the state of their partner during a collaborative sensorimotor task. Across two experiments, we show that visual feedback dominated haptic feedback during collaboration. Specifically, we found that visual feedback led to comparatively lower task-relevant movement variability, smoother collaborative movements, and faster trial completion times. We also developed an optimal feedback controller that considered the noise and time delays of both visual feedback and haptic feedback to estimate the state of a partner. This model was able to capture both lower task-relevant movement variability and smoother collaborative movements. Taken together, our empirical and modeling results support the idea that visual accuracy is more important than haptic speed to perform state estimation of a partner during collaboration.NEW & NOTEWORTHY Physical collaboration between two or more individuals involves both visual and haptic feedback. Here, we investigated how visual and haptic feedback is used to estimate the movements of a partner during a collaboration task. Our experimental and computational modeling results parsimoniously support the notion that greater visual accuracy is more important than faster yet noisier haptic feedback when estimating the state of a partner.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

We introduce a novel posture guidance office chair and evaluate the effectiveness of vibrotactile and visual feedback methods for guiding seated postures. For visually dominant office work such as typing on the computer, it is possible that delivering posture feedback visually may overload the visual sense while haptic feedback may be a viable alternative. We performed two experiments to compare vibrotactile and visual feedback--posture compliance and dual-task cognitive workload assessment. In the first experiment, our results showed no statistically significant difference in effectiveness between using vibrotactile and visual feedback to obtain postural compliance to a reference posture. In the second experiment, participants experienced typing performance and response time degradations from both types of feedback. However the differences in performance degradation were not statistically significant between the two feedback methods. We conclude that vibrotactile and visual feedback are similarly effective for guiding quasistatic postures in routine tasks such as seated office work.

Most prosthetic hand users are limited to visual feedback of movement performance. To characterize the benefit of vibrotactile feedback for a task that lacks haptic feedback, a virtual environment was used to experimentally manipulate visual, task-relevant haptic, and remote vibrotactile feedback on simple object manipulation for unimpaired subjects. The combination of visual and remote vibrotactile feedback was compared to visual feedback alone, and to simultaneous visual and direct haptic feedback to represent ideal performance. Visual and vibrotactile feedback resulted in improvement of most performance variables including difficulty ratings relative to visual feedback alone. However addition of sensory cues to visual feedback increased trial times and the increase was steeper for vibrotactile than for haptic feedback. Specifically, during vibrotactile feedback the velocity did not change, but the duration of execution increased due to improved performance, resulting in increased trial times. This result suggests future exploration of performance improvement and execution speed for augmented sensory feedback.

Controlling the motion of a prosthetic upper limb without visual feedback is extremely difficult because the wearer does not know the prosthesis' configuration. This paper describes an experiment designed to determine the relative importance of visual and haptic position feedback during targeted force-based motion by non-amputee human subjects as an analogy to prosthetic use. Subjects control the angle of a virtual proxy through an admittance relationship by generating torque at the MCP joint of the right index finger. During successive repetitions of a target acquisition task, the proxy's state is selectively conveyed to the user through graphical display, finger motion, and tactile stimulation. Performance metrics for each feedback condition will provide insights on the role of haptic position feedback and may help guide the development of future upper-limb prostheses

The focus of this research is on the perception and control of forces produced by the hand. It is known that the change in force that subjects can reliably discriminate is approximately 6%, and that subjects are able to match forces generated by two corresponding muscle groups with an accuracy of approximately 15%. The present study was concerned with measuring the ability to control finger force using only haptic feedback and both haptic and visual feedback. It was found that subjects were surprising good at controlling forces for up to 120 s using only haptic feedback with mean absolute errors of less than 0.3 N over a force range of 2–6 N. The variation in force averaged 10.5% when haptic feedback was available and decreased to 6% when visual feedback was also provided. These results provide a framework for specifying the characteristics of force-reflecting haptic interfaces that should optimize human perception.

In this study we compared tactile and visual feedbacks for the motor imagery-based brain–computer interface (BCI) in five healthy subjects. A vertical green bar from the center of the fixing cross to the edge of the screen was used as visual feedback. Vibration motors that were placed on the forearms of the right and the left hands and on the back of the subject’s neck were used as tactile feedback. A vibration signal was used to confirm the correct classification of the EEG patterns of the motor imagery of right and left hand movements and the rest task. The accuracy of recognition in the classification of the three states (right hand movement, left hand movement, and rest) in the BCI without feedback exceeded the random level (33% for the three states) for all the subjects and was rather high (67.8% ± 13.4% (mean ± standard deviation)). Including the visual and tactile feedback in the BCI did not significantly change the mean accuracy of recognition of mental states for all the subjects (70.5% ± 14.8% for the visual feedback and 65.9% ± 12.4% for the tactile feedback). The analysis of the dynamics of the movement imagery skill in BCI users with the tactile and visual feedback showed no significant differences between these types of feedback. Thus, it has been found that the tactile feedback can be used in the motor imagery-based BCI instead of the commonly used visual feedback, which greatly expands the possibilities of the practical application of the BCI.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

Concurrent augmented feedback has been shown to be less effective for learning simple motor tasks than for complex tasks. However, as mostly artificial tasks have been investigated, transfer of results to tasks in sports and rehabilitation remains unknown. Therefore, in this study, the effect of different concurrent feedback was evaluated in trunk-arm rowing. It was then investigated whether multimodal audiovisual and visuohaptic feedback are more effective for learning than visual feedback only. Naïve subjects (N = 24) trained in three groups on a highly realistic virtual reality-based rowing simulator. In the visual feedback group, the subject's oar was superimposed to the target oar, which continuously became more transparent when the deviation between the oars decreased. Moreover, a trace of the subject's trajectory emerged if deviations exceeded a threshold. The audiovisual feedback group trained with oar movement sonification in addition to visual feedback to facilitate learning of the velocity profile. In the visuohaptic group, the oar movement was inhibited by path deviation-dependent braking forces to enhance learning of spatial aspects. All groups significantly decreased the spatial error (tendency in visual group) and velocity error from baseline to the retention tests. Audiovisual feedback fostered learning of the velocity profile significantly more than visuohaptic feedback. The study revealed that well-designed concurrent feedback fosters complex task learning, especially if the advantages of different modalities are exploited. Further studies should analyze the impact of within-feedback design parameters and the transferability of the results to other tasks in sports and rehabilitation.

Virtual reality can help to learn basic laparoscopic tasks. However, no haptic feedback, which alerts for tissue slippage, is provided by most simulators, although, it might be of influence for the decrease of errors. This study explored whether visual or tactile feedback can be used to alert the surgeon of tissue slippage. Twenty-four participants performed a laparoscopic grasping task and where provided with either visual or tactile feedback about tissue slippage. The reaction time with the visual feedback was compared to the reaction time with tactile feedback signal. The results showed that when tissue slippage is simulated, tactile feedback shows significant faster reaction times (269ms) than visual feedback signals (398ms).

The perceptual deadband (PD) data reduction scheme allows for a significant reduction of network load in real-time haptic communication scenarios. Aiming to ascertain how the optimum PD data reduction parameter k is best determined, an experiment was conducted in which a possible interaction of PD data reduction and visual feedback on haptic feedback quality and task performance was investigated. The results show that when haptic feedback is complemented with visual feedback, haptic data reduction affects haptic feedback quality at a lower data reduction rate than it affects task performance. When visual feedback is missing, however, this effect is reversed. These results imply that the perception of haptic feedback quality is markedly influenced by visual sensory input as well as task requirements; hence, it is recommended to consider feedback quality and task performance in the optimization of PD reduction before such schemes may be applied to industrial telepresence and teleaction systems.

Online courses are becoming ubiquitous and increasingly tend to use authentic learning tasks as the driving force for teaching and learning. Nevertheless, designing online courses that incorporate real– world tasks is more challenging as these problems require more cognitive processes (van Merriënboer Sluijsmans, 2009). This phenomenon can be explained by Cognitive Load Theory (CLT) introduced by Sweller (1994). CLT distinguishes three types of cognitive load: intrinsic, extraneous and germane load. The level of intrinsic load is assumed to be determined by the level of element interactivity. An element can be a definition, concept, formula and procedure that needs to be or has been learned. Extraneous load is mainly imposed by instructional procedures that are suboptimal, whereas germane load refers to the learners’ working memory resources available to deal with the complexity of the task or learning material (Sweller, 2010). Accordingly, the experienced cognitive load is mainly dependent of students’ prior knowledge. Nevertheless, cognitive load can also be determined by students’ motivation (Feldon, Franco, Chao, Peugh, Maahs-Fladung, 2018; Verhoeven, Schnotz, Paas, 2009). As a consequence, when designing an online course for complex tasks, it is important to understand how the different types of cognitive load are affected by students’ cognitive and motivational characteristics. Therefore, in the current study, a high and low complex task was developed relating to the learning and teaching of geometry. The complexity of the task was manipulated by increasing the element interactivity for the high complex task (Sweller, 2010). In the low complex task one element was questioned each time, and consequently students had to apply a single rule, formula or procedure. By contrast, the high complex task was based on a real-life context (e.g., teaching geometry), and had higher element interactivity. Subsequently, the high complex task required learners to engage in a series of cognitive activities such as analysing, decision making, implementing and evaluating, while holding several procedures and rules in mind. Accordingly, we expected the high complex task to induce more cognitive load. The same amount of support containing the same content, was provided during both tasks. Consequently, in this context, students could take initiative in diagnosing their learning needs by identifying appropriate support. Since students could consult different amounts of support, this self-directed learning strategy could also influence the perceived cognitive load (van Merriënboer Sluijsmans, 2009). Accordingly, the amount of consulted support was also taken into account during the analyses. The aim of the study was twofold. First, as a manipulation check of task complexity, we investigated differences in the experienced cognitive load while solving a high and low complex task. Secondly, we examined whether students’ cognitive and motivational characteristics influence the different types of perceived cognitive load, when taking into account the amount of consulted support for both the high and low complex task. A multivariate approach was chosen to assess the degree of interplay that may exist among students’ cognitive, motivational characteristics, consultation of support and the different types of perceived cognitive load. By conducting this study, we wanted to gain insight into whether the cognitive, motivational characteristics and consultation of support influence the perceived cognitive load differently for a high and low complex task.

The Simulation-Based Open Surgery Training System (SOSTS) is a virtual reality simulator designed to provide training on a simulated orthopedic bone pinning procedure. The present version of SOSTS offers five distinct combinations of multi-sensory feedback during training. The purpose of this study was to investigate the effects of visual and haptic feedback on participants' ability to place a pin through a fractured tibia on the simulator. Enhanced visual feedback resulted in a significant reduction of pin alignment error, although participants in these conditions took five times longer to achieve proficiency than those in the moderate visual feedback conditions. Any form of haptic feedback resulted in significantly greater pin placement error than no haptic feedback. The results provide partial support for the proximity compatibility principle for display design.

This chapter examines the use of haptic feedback in a multimedia simulation as a means of conveying information about physical science concepts. The case study presented herein investigates the effects of force feedback on children’s conceptions of gravity, mass, and related concepts following experimentation with a force-feedback-enabled simulation. Two groups of 17 children conducted experiments with the simulation; the experimental group used both visual and force feedback, and the control group used visual feedback only. Evidence of positive gains by the experimental group who used the simulation with force feedback is presented. Guidelines for applying these technologies effectively for educational purposes are discussed. This chapter adds to the limited research on the application of haptic feedback for conceptual learning and provides a basis for further research into the effects of computer-based haptic feedback on children’s cognition.

Robotic-assisted surgery requires an intuitive and effective human-machine interface. Providing haptic feedback for pinching and dissecting motions of bipolar forceps, a tool commonly used in neurosurgery, could potentially improve the surgeon's experience. Current haptic hand controllers have limited actuation and feedback capability, requiring surgeons to hold the handle differently compared to a conventional tool. This paper presents a new master design that provides 1-DOF force feedback by adding a Hall-effect sensor and a voice coil actuator directly onto a bipolar forceps. Twenty participants used this interface to perform a remote stiffness perception test that employed the method of constant stimuli. Ten participants pinched the samples, and the other ten dissected them. Each participant did two blocks of 35 trials with only visual feedback or with visual and haptic feedback in random order. Psychometric functions were created from the results to compare perceptual capabilities, metrics were calculated from the force and position data, and participant survey responses were analyzed. The results show that providing the force feedback made the task seem easier, increased the participant's confidence, and reduced the total tip distance traveled in the pinching task. The haptic feedback slightly improved stiffness perception in the pinching task but did not improve perception in the dissection task. These results support the utility of a force-feedback attachment to conventional forceps for pinching and motivate further investigation into the design for dissection.