Development of Innovative Feedback Device for Swallowing Therapy

Visual representations are used in various digitization processes to display objects in immersive environments. In industrial environments, the self-representation can be very helpful when reviewing 3D models to conform realistic proportions, particularly in combi-nation with haptic feedback that goes beyond vibrating controllers. For instance, haptic feedback in combination with a virtual representation supports working on use cases where collision is an important part to maintain data quality. To understand the dependency of both, we conducted a pilot study with 15 users from the automotive sector to examine the influence of upper limb representations on haptic feedback in Virtual Reality. Each participant was assigned with one of three upper limb representations which was used in different scenarios using haptic feedback devices. Overall, we found that the realistic arm representation was rated highest in terms of perceived realism and achieved the best task performance.

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state of the art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronized and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through, e.g., amplitude modulation (AM) or spatiotemporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

While feedback is frequently emphasized as a crucial principle of presentation courses in higher education, previous studies revealed that teachers outperform peers in terms of impact on students’ development of oral presentation competence. Further, presentation research showed that the lack of quality of peer feedback can be considered as an essential argumentation for the identified differences in effect. Follow-up field experiments demonstrated that Virtual Reality (VR) can be considered as a valuable alternative feedback source for developing public speaking skills, since this technology is able to simulate real-life presentation situations as well as to deliver feedback from the VR system to the individual learner. Recent technological developments allowed to convert quantitative information from VR systems into qualitative feedback messages that directly relate to the standards for high-quality feedback. If students are able to individually interpret the feedback messages without the intervention of a human feedback source, it could enrich the quality of feedback in peer and self-learning and further increase students’ oral presentation competence development. This chapter provides a synthesis of the literature in presentation research with the aim to construct a research agenda on computer-mediated feedback in VR for peer learning in this field. Further, two recent VR experiments in presentation research are discussed with the aim to effectively construct feedback messages in VR for improving peer learning.

Tactile codes and devices are widely used in virtual reality, robotics, and telemanipulation by blind and sighted people. Based upon the Braille code, we provide a rigorous analysis of tactile spatial codes, and propose a code that is improved in the information theoretical sense. Based upon the criterion of a minimum average code word weight, we derive a code that provides improved reliability and longer lifetime of tactile devices. Such an approach can be also employed for robot hands, and tactile feedback in virtual reality and dextrous telemanipulation.

This paper presents the first results on a user study in which people with visual impairments (PVI) explored a virtual environment (VE) by walking in a virtual reality (VR) treadmill. As recently suggested, we have now acquired first results from our feasibility study investigating this walk-in-place interaction. This represents a new, more intuitive way of for example virtually exploring unknown spaces in advance. Our prototype consists of off-the-shelf VR components (i.e., treadmill, headphones, glasses, and controller) providing a simplified white cane simulation and was tested by six visually impaired subjects. Our results indicate that this interaction is yet difficult, but promising and an important step to make VR more and better usable for PVIs. As an impact on the CHI community, we would like to make this research field known to a wider audience by sharing our intermediate results and suggestions for improvements, on some of which we are already working on.

We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user's finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.

In recent years, providing additional visual feedback about the interaction forces has been found to offer benefits to haptic-assisted teleoperation. However, there is limited insight into the effects of the design of force feedback-related visual cues and the type of visual display on the performance of teleoperation of robotic arms executing industrial tasks. In this study, we provide new insights into this interaction by extending these findings to the haptic assistance teleoperation of a simulated robotic arm in a virtual environment, in which the haptic assistance is comprised of a set of virtual fixtures. We design a novel method for providing visual cues about the interaction forces to complement the haptic assistance and augment visual feedback in virtual reality with a head-mounted display. We evaluate the visual cues method and head-mounted display method through human factors experiments in a teleoperated dross removal use case. The results show that both methods are beneficial for task performance, each of them having stronger points in different aspects of the operation. The visual cues method was found to significantly improve safety in terms of peak collision force, whereas the head-mounted display additionally improves the performance significantly. Furthermore, positive scores of the subjective analysis indicate an increased user acceptance of both methods. This work provides a new study on the importance of visual feedback related to (interaction) forces and spatial information for haptic assistance and provides two methods to take advantage of its potential benefits in the teleoperation of robotic arms.

This paper outlines a study into the effects of co-location (the term `co-location' is used throughout to refer to the co-location of haptic and visual sensory modes, except where otherwise specified) of haptic and visual sensory modes in VR simulations. The study hypothesis is that co-location of these sensory modes will lead to improved task performance within a VR environment. Technical challenges and technological limitations are outlined prior to a description of the implementation adopted for this study. Experiments were conducted to evaluate the effect on user performance of co-located haptics (force feedback) in a 3D virtual environment. Results show that co-location is an important factor, and when coupled with haptic feedback the performance of the user is greatly improved.

It is known that systematic visual distortions using virtual reality technologies, prisms or mirrors, may have therapeutic effects for patients suffering from stroke or body image identity disorders. However, there are few studies which directly investigate neural activity changes during visual feedback manipulation. In the present study we created an experimental setup for investigating the effects of systematic virtual reality-mediated visual feedback manipulation of finger movements on cortical activity. We performed tests with two healthy female subjects who performed a line-tracking task under four conditions manipulating visual feedback of their own hand. To investigate hemodynamic responses in motor areas during the line tracking task we used functional near-infrared spectroscopy (fNIRS). We predicted that viewing larger or smaller virtual movements of fingers, compared to the real movements, would affect activity in motor areas and thus the hemodynamic response. Our preliminary results showed changes in the hemodynamic responses between stimulation period and baseline. There were indications of possible differences between conditions, and also of adaptation effects within conditions. However these effects were not significant in our preliminary data and we therefore need to collect additional data to draw further conclusions.

In this paper, we develop possible realizations of pseudo-haptic feedback in teleoperation systems based on existing works for pseudo-haptic feedback in virtual reality and the intended applications. We derive four potential factors affecting the performance of haptic feedback (calculation operator, maximum displacement, offset force, and scaling factor), which are analyzed in three compliance identification experiments. First, we analyze the principle usability of pseudo-haptic feedback by comparing information transfer measures for teleoperation and direct interaction. Pseudo-haptic interaction yields well above-chance performance, while direct interaction performs almost perfectly. In order to optimize pseudo-haptic feedback, in the second study we perform a full-factorial experimental design with 36subjects performing 6,480trials with 36different treatments. Information transfer ranges from 0.68bit to 1.72bit in a task with a theoretical maximum of 2.6bit, with a predominant effect of the calculation operator and a minor effect of the maximum displacement. In a third study, short- and long-term learning effects are analyzed. Learning effects regarding the performance of pseudo-haptic feedback cannot be observed for single-day experiments. Tests over 10 days show a maximum increase in information transfer of 0.8bit. The results show the feasibility of pseudo-haptic feedback for teleoperation and can be used as design basis for task-specific systems.

The potential of robotics: A systematic review of neuroplastic changes following advanced lower limb rehabilitation in neurological disorders.

Virtual reality (VR) and augmented reality (AR) systems are increasingly available to neurosurgeons. These systems may provide opportunities for technical rehearsal and assessments of surgeon performance. The assessment of neurosurgeon skill in VR and AR environments and the validity of VR and AR feedback has not been systematically reviewed. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was conducted through MEDLINE and PubMed. Studies published in English between January 1990 and February 2021 describing the use of VR or AR to quantify surgical technical performance of neurosurgeons without the use of human raters were included. The types and categories of automated performance metrics (APMs) from each of these studies were recorded. Thirty-three VR studies were included in the review; no AR studies met inclusion criteria. VR APMs were categorized as either distance to target, force, kinematics, time, blood loss, or volume of resection. Distance and time were the most well-studied APM domains, although all domains were effective at differentiating surgeon experience levels. Distance was successfully used to track improvements with practice. Examining volume of resection demonstrated that attending surgeons removed less simulated tumor but preserved more normal tissue than trainees. More recently, APMs have been used in machine learning algorithms to predict level of training with a high degree of accuracy. Key limitations to enhanced-reality systems include limited AR usage for automated surgical assessment and lack of external and longitudinal validation of VR systems. VR has been used to assess surgeon performance across a wide spectrum of domains. The VR environment can be used to quantify surgeon performance, assess surgeon proficiency, and track training progression. AR systems have not yet been used to provide metrics for surgeon performance assessment despite potential for intraoperative integration. VR-based APMs may be especially useful for metrics that are difficult to assess intraoperatively, including blood loss and extent of resection.

In this dissertation different aspects from research in the fields of Tangible User Interfaces, encounter-type devices and Passive Haptics are combined to investigate the benefits that robots offer for providing haptic feedback in Virtual and Augmented Reality. Robotic elements like micro drives and robotic arms are employed for the actuation of passive or active physical objects. In that way physical props can be collocated with virtual counterparts to allow high fidelity, natural interaction.

Motor imagery combining virtual reality (VR) technique has recently been reported to have an increasingly positive impact on post-stroke rehabilitation. However, there is a common problem that the engagement of patients cannot be confirmed during motor imagery training due to a lack of effective feedback control. This paper proposes a VR-based motor imagery training system for post-stroke rehabilitation, using surface electromyographic (EMG)-based real-time feedback to enable the personalized training and quantitative assessment of participation degree. Three different experiments including assessment experiment, action observation (AO), combined motor imagery and action observation (MI+AO) experiment were performed on 4 post-stroke patients to verify the system. The immersive scenario of the VR system provides a shooting basketball training for bilateral upper limbs. The EMG data of assessment of each participant was collected to calculate the thresholds, which was utilized in the subsequent experiments based on real-time feedback of EMG. The result reveals significant differences of the muscle strength between AO and MI+AO experiments. This demonstrates that the EMG-based feedback is effective to be of use in assessment of participation degree. The primary application shows that VR-assisted motor imagery system has potential to provide personalized and more engaged training for post-stroke rehabilitation.

Clinical and kinematic evaluation of the H-Man arm robot for post-stroke upper limb rehabilitation: Preliminary findings of a randomised controlled trial