A multimodal tactile interface for immersive virtual experience

Virtual field trips are transforming education in biology and other place-based disciplines – and not just since Covid-19 has imposed social distancing constraints. Efficient methods for content creation, combined with affordable immersive technology, provide the opportunity to integrate immersive experiences into ever-larger classes. However, many aspects of how immersive experiences can enrich and improve learning in place-based education are still not well understood. We present an empirical study, conducted in a large undergraduate ecology class, comparing students, that experienced an immersive virtual version of a field trip to a biological preserve as preparation for a later actual field trip to the site, with a control group that visited the real site but without having had the virtual experience. Within the virtual experience group, we also varied the quality of the headset. The VR field trip was well received for raising spatial awareness and also motivation and interest. Students ranked it higher than any traditional means with regard to achieving a high level of preparedness for the actual field trip. In contrast, no statistically significant evidence was found for differences in perceived learning outcome and enjoyment of the actual field trip.

Multimodal human–computer interfaces utilize sensory information from different modalities to create richer and more immersive user experiences. The use of ultrasound haptics in a multimodal interface is compelling: It requires no physical contact, has high spatial and temporal resolution, and can even be physically co-located with other sensory information (e.g., visual content in mid-air). In this chapter, we explore examples of multimodal interface that use ultrasound haptics, and reflect on the technical and design challenges of creating a high-quality multimodal ultrasound haptic interface.

This study is to propose a realistic game system using a multi-modal interface, including gaze tracking, hand gesture recognition and bio-signal analysis. Our research is novel in the following four ways, compared to previous game systems. First, a highly immersive and realistic game is implemented on a head mounted display (HMD), with a gaze tracker, a gesture recognizer and a bio-signal analyzer. Second, since the camera module for eye tracking is attached below the HMD, a user's gaze position onto the HMD display can be calculated without wearing any additional eye tracking devices. Third, an aiming cursor in the game system is controlled by the gaze tracking. The grabbing and throwing behaviors toward a target are performed by the user's hand gestures using a data glove. Finally, the level of difficulty in the game system is adaptively controlled according to the measurement and analysis of a user's bio-signals. Experimental results show that the proposed method provides more effect on experience of immersion and interest than conventional device such as a keyboard and or a mouse.

Over a number of decades, studies have revealed compelling relationships between experiences of the natural environment and positive health outcomes in adult communities. These psychosocial health benefits have typically been described via key theoretical frameworks in the health sciences, such as the biophilia hypothesis. Despite the body of evidence for nature design and well-being, propositions for immersive virtual nature experiences are still emerging across the fields of creativity and health. In this article, we identify the potential for immersive virtual nature art as a creative well-being intervention, articulated through a discussion of background literature and concepts leading to the development of our artwork, Inner Forest (2020–21). This article incorporates a transdisciplinary suite of perspectives across three key parts; in the first section, we review key health and design research around beneficial nature experiences, with particular emphasis on designing technological nature. Secondly, we propose considerations for immersive virtual nature experiences, as distinct from prior iterations of technological nature; these considerations are framed through discussion of our artistic and well-being rationale for designing the collaborative artwork Inner Forest. This extended reality (XR) project was developed in response to well-being challenges such as social isolation and restricted nature access- of particular valence during the COVID-19 pandemic. The artwork incorporates multisensory, aesthetic elements drawn from biophilic design guidelineswhich support creative, evidence-based approaches to designed nature and societal health. To conclude this article, we report on prospects for further scaling of the Inner Forest artwork, with ongoing scope to contribute to both nature-health design and immersive virtual nature art practices.

In this study we explore the impact of multisensory stimuli in virtual reality on users’ emotional responses, addressing a knowledge gap in this rapidly evolving field. Utilizing a range of sensory inputs, including taste, haptics, and smell, in addition to audiovisual cues, this study aims to understand how different combinations of these stimuli affect the users’ emotional experience. Two immersive virtual experiences have been developed for this purpose. One included a scenario to evoke positive emotions through selectively chosen pleasant multisensory stimuli, validated in a focus group. The other sought the contrary: to trigger negative emotions by integrating selected combinations of unpleasant multisensory stimuli, also validated in the same focus group. Through a comparative analysis, our findings revealed significant differences in emotional responses between the groups exposed to positive and negative stimuli combinations. Results indicated that combinations involving haptics and taste were particularly effective in eliciting intense emotions using positive stimuli, but their impact was less significant with negative stimuli. This investigation suggests that a fully multisensory virtual environment integrating positive stimuli might lead to cognitive overload, reducing overall emotional responses. In contrast, environments with negative stimuli could enhance emotional engagement and be more likely to avoid cognitive overload. These findings have important implications for designing emotionally resonant and compelling virtual reality experiences. This research enhances the understanding of sensory integration in virtual reality and its effects on emotional engagement, offering valuable insights for developing more impactful virtual experiences.

Artistic design using emerging technologies provides new carriers and inspiration for the Red Culture ceramic exhibition and innovates the aesthetic and technical aspects of artistic creation. This paper primarily utilizes augmented reality technology to enhance the spatial design of Red Culture ceramic exhibitions, thereby optimizing the immersive experience. Multimodal interaction is proposed, and finger recognition based on skin color segmentation is mainly analyzed. Creating a multi-sensory interactive space, combined with the principle of holographic imaging and designing a depth-of-field viewing area AR display method based on holographic optical elements, emphasizes the multi-viewpoint depth of field. Closely follow the red culture and use the ceramic landscape to convey the essence of red culture. Set the SLM and HOE parameters and simulate the experiment to analyze the normalized light intensity distribution and focus coordinates of the multi-viewpoint. Conduct user evaluation research to analyze the advantages and shortcomings of the digital exhibition of the ceramic landscape of red culture. The order of the dimensional ratings for the ceramic landscape digital exhibition is: cultural experience (91.52) > emotional experience (84) > functional experience (80.39) > interactive experience (79.95) > aesthetic experience (77.22). The ceramic exhibition using augmented reality technology can meet the cultural and emotional needs of visitors, and the AR function, as well as the interactive experience, meet the design requirements.

When we use a tablet style handheld device such as a smart phone as a part of a virtual really system, its most outstanding future: the touch screen dominating the most area of the front face should be incorporated into the system effectively and beneficially. For example, the visual information displayed on the screen can be merged with the surrounding or background scenes and the intuitive touch operation can be performed in a suitable scenario. However, if the means of the interaction is limited to the touch operation, finger operation on the front screen must be performed even for unsuitable scenarios, and the fingers or hands occluding the visual information disturb our immersive experience. To deal with this situation, we propose a multi-modal interactive tablet that uses its cameras, accelerometer, track ball and pressure sensors implemented on its rear and side for operations ensuring visibility. The pressing and ball-rotating operation on the rear and the side and the tactile feedback generated by voice-coil-based actuators assist and guide the multi-modal interaction. The effectiveness of the multi-modality with the rear and the side operation and the tactile feedback is evaluated by an experiment.

Tecnológico de Monterrey’s Tec21 framework structures student development through Exploration, Focus, and Specialization phases, supported by a triad of peer mentors, program directors, and student mentors. Despite this robust design, two decision points, leaving the Exploration phase and selecting a specialization, emerge as peaks of stress and anxiety, driven by fears of irreversible mistakes, financial uncertainty, and external pressures. A mixed-methods survey of 78 sixth-semester students revealed that while most report low stress when choosing a university or major, a significant minority experiences moderate to severe anxiety. Open responses highlighted concerns about costly tuition, heavy course loads, confusing curricular options, and worry over disappointing family or peers. Traditional interventions, detailed program overviews, mindfulness exercises, and proactive advising, provide some relief but may not fully engage Generation Z learners, who gravitate toward immediate, immersive experiences. Simultaneously, Tec’s Mostla innovation hub has established VR Zones on twenty campuses, each equipped with HTC Vive Pro stations and a proven track record of five thousand visits within the first ten months. Yet, existing VR applications do not address the specific decision-making challenges faced by students. Anchored by the university’s 2030 vision, “education not just for work but for life, and continuously”, and its credo of “leadership, innovation, and entrepreneurship for human flourishing,” a tailored VR prototype offers a compelling solution. By recreating campus tours, career pathways, and specialization scenarios in a controlled virtual environment, students can test outcomes without real-world stakes. This experiential rehearsal bridges information gaps, alleviates financial and logistical anxieties, and builds emotional resilience and self-efficacy, core components of human flourishing. Discussion of survey findings alongside VR’s proven capacity for social-emotional learning suggests that immersive simulations can transform high-pressure academic choices into guided explorations. Students who feared choosing the wrong discipline or losing scholarships could, for instance, virtually navigate financial aid processes or “walk through” a corporate project before committing. By aligning Tec21’s phased curriculum and well-being dimensions with interactive VR modules, the university can deliver decision-making workshops and in-simulation prompts that reinforce confidence at those pivotal junctures. In conclusion, implementing bespoke VR experiences, grounded in Tec21 pedagogy and deployed through Mostla’s infrastructure, promises to convert moments of acute anxiety into opportunities for growth. Such an approach not only prepares students for professional success but also nurtures their capacity for continuous learning and balanced living, satisfying Tecnológico de Monterrey’s aspiration to cultivate resilient individuals ready for the challenges of an ever-changing world.

Virtual reality (VR) is an interactive simulation environment based on computer technology that enables users to immerse themselves in a virtual world. With the continuous advancement of technology, VR has shown great potential in various fields. This article aims to review the application of VR in different fields and its impact on society. First, this paper introduces VR's basic principles and technical characteristics. VR systems are usually composed of head-mounted display devices, trackers, joysticks, etc., enabling users to experience immersive virtual environments by simulating real-world perception and interaction. Secondly, this paper discusses the application of VR in the field of education. VR technology can provide an immersive learning experience, enabling students to personally participate in the course content, greatly enhancing the effect of learning. At the same time, VR can also simulate experimental scenes, train operational skills, etc., to provide students with more abundant and practical learning resources. Then, this paper introduces the application of VR in the medical field. VR can be used for rehabilitation training of patients, psychological treatment, and other aspects, effectively improving the quality and effect of medical services. In addition, this paper also discusses the application of VR in entertainment, tourism, and other fields. VR technology can bring users immersive gaming experience, virtual tourism, etc., and inject new vitality into the entertainment industry. In tourism, VR can provide tourists with an immersive sightseeing experience, so they can feel the distant scenery at home.

Humans are multi-sensory beings who make sense of theworld via multiple sensory modalities. However, most everyday technologies, including smartphones, tablets, and personal computers, often fail to engage all the senses effectively. Instead, vision is heavily relied on to interface with everyday technologies, and as a result, it is disproportionately stimulated when compared to the rest of our senses. This hegemony of vision in design not only excludes the visually challenged but imbalanced sensory experiences also create sensory overload and fatigue inthose who are not. Limiting the senses that are engaged while interfacing with digital devices also increases the cognitive burden placed on the user. When complex information is conveyed unimodally in highly stimulating environments, people must expend a large amount ofenergy and cognitive resources to attend to and grasp information to commit it to memory and recall it.This thesis aims to address contemporary sensory imbalance challenges by investigating and hypothesizing effective and accessible multimodal interfaces for everyday technologies and providing considerations for designers who seek to leverage multiple senses to convey information. Given the fact that taste often requires people to consume substances, the effects are difficult to control when designing for larger populations. Therefore, this thesis intentionally focuses on designing for touch, smell, hearing, and vision as a means of creating experiences that effectively disperse the intake of information across multiplesenses. In particular, it aims to investigate how interactionswith everyday technology can be designed to effectively engage the senses beyond vision to create balanced, immersive, and inclusive experiences.

Purpose To understand the views of people with dysphagia and their supporters on the feasibility of using 3D food printing to improve the visual appeal of texture-modified foods and their mealtime experiences. Materials and Methods Nine people with dysphagia and four of their supporters engaged in a virtual 3D food printing experience and interview over Zoom© about their impressions and usability of the printer and potential for it to improve their mealtime experiences. The in-depth interviews were analysed using thematic analysis and usability heuristics. Results Four content themes in the interviews impacted on the feasibility of 3D food printing. They related to the practicality, design acceptability, population suitability, and cost of 3D food printing. Usability heuristic analysis revealed that print quality, user control, error prevention, and handling of food consistencies would impact on use. Perceived low efficiency and increased time required to create food shapes meant participants were cautious in their overall impressions of the role of the device. Conclusion After an immersive virtual experience with a 3D food printer, people with dysphagia and their supporters identified a wide range of usability issues that would need to be addressed prior to implementation and in the future design of user-friendly 3D food printers for people with dysphagia. Future research should include people with dysphagia and their supporters in 3D food printer design and implementation trials. Implications for Rehabilitation 3D food printing may provide people with dysphagia who require texture-modified food a way to produce visually appealing texture-modified food if usability issues are addressed. 3D food printing could improve participation in meal preparation if the person with dysphagia chooses the food and the shape and size of the printed food shape.

Immersing Patients in a Virtual Reality Environment for Brain Mapping During Awake Surgery: Safety Study

This project aimed to investigate the correlation between virtual reality (VR) imagery and ambisonic sound. With the increasing popularity of VR applications, understanding how sound is perceived in virtual environments is crucial for enhancing the immersiveness of the experience. In the experiment, participants were immersed in a virtual environment that replicated a concert hall. Their task was to assess the correspondence between sound scenes (which differed in reverberation times and their characteristics) and the observed invariant visual scene. The research was conducted using paired tests. Participants were asked to identify the sound scene they considered more closely matched the concert hall seen in the VR goggles for each pair. Each sound scene differed in the employed impulse response. All the impulse responses were recorded in real venues such as concert halls, auditoriums, churches, etc. To provide a realistic auditory experience, the sound scenes were processed using third-order ambisonics and decoded using binaural techniques with HRTFs. The virtual concert hall was generated using the Unreal Engine and was the same for all the tests. One of the major conclusions drawn from the conducted research was confirming the role of spatial sound in creating immersive VR experiences. The study demonstrated that appropriately matching spatial sound to the VR visual scene is essential for achieving complete immersion. Additionally, expectations and preferences regarding reverberation characteristics in different types of spaces were discovered. These findings have significant implications for the design of virtual environments, and understanding these aspects can contribute to improving VR technology and creating more immersive and realistic virtual experiences for users.

The Edu-Metaverse, a vast ensemble of different technologies, has initiated a great and unprecedented change in the field of education. This change has been effected through the following Edu-Metaverse characteristics: embodied and multimodal interaction; immersive teaching scenarios, which can accelerate learning and skill acquisition; and the emergence of AI-enabled agents. In comparison to traditional classroom teaching models, smart education is a collaborative and visual model that adopts the latest AI technologies to reach a learning outcome. However, a problem that should be considered is how a smart education model, enabled by the Edu-Metaverse, can be developed to enhance better learning outcomes for students. Such a model should highlight smart pedagogy in the context of the Edu-Metaverse, together with a smart teaching environment, multimodal teaching resources, and AI-enabled assessment. In this study, we focused on the teaching of college English to 60 students from Zhejiang Open University. We investigated the effectiveness of a smart education model, which was empowered by the Edu-Metaverse, in enhancing better learning outcomes for the students, using a combination of qualitative and quantitative research. After the one-semester-long experiment, questionnaires were sent out to complement the interview findings. It was found that the students who engaged in the smart education model in the Edu-Metaverse yielded higher scores in oral English, vocabulary and grammar, reading comprehension, English-to-Chinese translation, and writing than those who engaged in traditional instruction. Therefore, this study suggests that a smart education model enabled by the Edu-Metaverse, which is characterized by a highly immersive experience, multimodal interaction, and a high degree of freedom for resource sharing and creation can help learners to realize deep learning, develop their capacity for high-order thinking, and help them to become intelligent individuals in an online learning space. In order to facilitate this smart learning, we make the following suggestions for educational institutions: (1) teachers should improve the design of teaching scenarios, (2) teachers should focus on learning assessment that is based on core literacy, and (3) teachers’ knowledge of the architecture of the Edu-Metaverse should be enhanced.

Evidence demonstrates the quality of advanced and serious illness care is substantively improved by integration of Palliative Medicine practices. The demand for trained palliative care professionals remains unmet with pronounced national workforce gaps in rural communities. We developed and implemented an interdisciplinary primary care team training initiative that leverages the experience of well-established palliative care teams to provide an immersive virtual training experience that blends live case-based learning embedded in clinical encounters with interdisciplinary role appreciation, teamwork, prognostication, breaking bad news, pain, symptom management, team dynamics, and value-based goals of care discussions. Between May and September 2024 six multidisciplinary cohorts of palliative scholars (N = 32) completed the 3-day practicum. In-depth pre-post practicum metrics measured change in knowledge, attitudes, perceived skill, comfort, confidence with recognizing, addressing, and holding end-of-life conversations with patients and families. The scholars post evaluations demonstrated learners’ experienced significant gains in knowledge of and comfort with providing essential palliative and supportive care in the primary care setting. Knowledge increased by 85%, 100% indicated having gained valuable communication skills to discuss advance care planning and goals of care, 90% ranked their experience as highly satisfactory, and 100% indicated intention to recommend the practicum to others. Scholars reported feeling empowered to continue learning as champions for Palliative Care principles at their primarily rural settings of care. Initial evaluation of the Palliative Scholars model supports it viability as one effective solution to expanding the primary palliative care competency of our workforce to meet the needs of older adults with life-limiting illness.