Editorial: Cutting-edge technologies for multi-sensory research at the frontier between disciplines

The integration of technology into cooking and eating routines often leads to distractions, hindering mindfulness. Mindful eating involves awareness of the present moment and attention to sensory experiences. While mindful eating has been promoted, mindful cooking remains underexplored. As conversational agents become increasingly popular in diverse fields like education, healthcare, and e-commerce, we are leveraging this technology to promote mindfulness practices that enhance the cooking and eating experience. This paper introduces MyndFood, a conversational agent designed to promote mindful cooking and eating. We describe the development process of MyndFood and present the outcomes from a user experiment involving 40 participants who engaged with the system while preparing sushi. The study employed a between-subjects design featuring two conditions for the agent: mindful and non-mindful. Results indicate that participants in the mindful condition experienced greater enjoyment and heightened food awareness compared to those in the non-mindful condition. These findings suggest that conversational agents can enhance hedonic experiences and improve food awareness by promoting mindful practices.

Decades of research on visual information for speech has demonstrated the informativeness of visual cues to enhance (Cho et al., 2020; Kawase et al., 2014) and influence (MacDonald and McGurk, 1978) speech perception. However, it is unclear which specific visual cues are spatially and temporally correlated to certain features of speech segments. This study explored visual cues of voicing during the production of Canadian English stops using facial recognition technology (Baltrušaitis et al., 2018) and manual coding (using ELAN 2022). We recorded the audio along with two videos, capturing both front and side views simultaneously, from six native Canadian English speakers. We paid special attention to the throat (larynx), chin, and neck areas which have been understudied by the previous literature. Preliminary data shows expanding movement in the submental triangle and throat during the production of voiced stops compared to voiceless stops. This finding supports tongue body lowering and larynx lowering found in the production of English voiced stops (Westbury, 1983). The comparison between utterance initial stops with post-vocalic stops shows that certain visual cues may be related to phonological voicing categorization irrespective of actual voicing during closure, while others reflect phonetic voicing reality.

Maximizing educational impacts with learning technologies is one of the areas that researchers and practitioners are concerned about in the field of Human-Computer Interaction (HCI) and human-centered artificial intelligence (HCAI). In this case study, we report user experiences and lessons learned of the Enactive Virtual Environment for teaching practice (EVETeach) with AI-powered virtual student agents called Evelyn. We conducted a user study with a case study research design. We collected multiple sources of data from 24 student teachers, including participatory observations, field notes, semi-structured interviews, computer-based conversation logs, audio-, video-, and screen-recordings, and a cognitive walkthrough. We identified the following salient emerging findings as lessons learned: 1) Student teachers value and relate to the teaching practices in virtual reality simulation with AI-powered conversational agents, 2) AI-powered conversational agents inject humor to facilitate situational and social teaching practice, and 3) AI-powered conversational student agents maintain authentic discourse to promote student teachers’ pedagogical reasoning.

Audiovisual information reaches the brain via both sustained and transient input channels, representing signals’ intensity over time or changes thereof, respectively. To date, it is unclear to what extent transient and sustained input channels contribute to the combined percept obtained through multisensory integration. Based on the results of two novel psychophysical experiments, here we demonstrate the importance of the transient (instead of the sustained) channel for the integration of audiovisual signals. To account for the present results, we developed a biologically-inspired, general-purpose model for multisensory integration, the Multisensory Correlation Detectors, which combines correlated input from unimodal transient channels. Besides accounting for the results of our psychophysical experiments, this model could quantitatively replicate several recent findings in multisensory research, as tested against a large collection of published datasets. In particular, the model could simultaneously account for the perceived timing of audiovisual events, multisensory facilitation in detection tasks, causality judgments, and optimal integration. All-in-all, this study demonstrates that several phenomena in multisensory research that were previously considered unrelated, all stem from the integration of correlated input from unimodal transient channels.

Pooling and synthesizing signals across different senses often enhances responses to the event from which they are derived. Here we examine whether multisensory response enhancements are attributable to a redundant target effect (two stimuli rather than one), or if there is some special quality inherent in the combination of cues from different senses. To test these possibilities, the performance of animals in localizing and detecting spatiotemporally concordant visual and auditory stimuli was examined when these stimuli were presented individually (visual or auditory), or in cross-modal (visual-auditory) and within-modal (visual-visual, auditory-auditory) combinations. Performance enhancements proved to be far greater for combinations of cross-modal than within-modal stimuli and support the idea that the behavioral products derived from multisensory integration are not attributable to simple target redundancy. One likely explanation is that while cross-modal signals offer statistically independent samples of the environment, within-modal signals can exhibit substantial covariance, and consequently multisensory integration can yield more substantial error reduction than unisensory integration.

Nonverbal and other visual cues are well established as a critical component of human communication. Under most circumstances, visual information is available to aid in the comprehension and interpretation of spoken language. Citing these facts, many L2 assessment researchers have studied video-mediated listening tests through score comparisons with audio tests, by measuring the amount of time spent watching, and by attempting to determine examinee viewing behavior through self-reports. However, the specific visual cues to which examinees attend have heretofore not been measured objectively. The present research employs eye-tracking methodology to determine the amounts of time 12 participants viewed specific visual cues on a six-item, video-mediated L2 listening test. Seventy-two scanpath-overlaid videos of viewing behavior were manually coded for visual cues at 0.10-second intervals. Cued retrospective interviews based on eye-tracking data provided reasons for the observed behaviors. Faces were found to occupy the majority (81.74%) of visual dwell time, with participants largely splitting their time between the speaker’s eyes and mouth. Detected gesture viewing was negligible. The reason given for most viewing behavior was determining characters’ emotional states. These findings suggest that the primary difference between audio- and video-mediated L2 listening tests of conversational content is the absence or presence of facial expressions.

Natural and intuitive interaction between users and complex systems is a crucial research topic in human‐computer interaction. A major direction is the definition and implementation of systems with natural language understanding capabilities. The interaction in natural language is often performed by means of systems called chatbots. A chatbot is a conversational agent with a proper knowledge base able to interact with users. Chatbots appearance can be very sophisticated with 3D avatars and speech processing modules. However the interaction between the system and the user is only performed through textual areas for inputs and replies. An interaction able to add to natural language also graphical widgets could be more effective. On the other side, a graphical interaction involving also the natural language can increase the comfort of the user instead of using only graphical widgets. In many applications multi‐modal communication must be preferred when the user and the system have a tight and complex interaction. Typical examples are cultural heritages applications (intelligent museum guides, picture browsing) or systems providing the user with integrated information taken from different and heterogenous sources as in the case of the iGoogle™ interface. We propose to mix the two modalities (verbal and graphical) to build systems with a reconfigurable interface, which is able to change with respect to the particular application context. The result of this proposal is the Graphical Artificial Intelligence Markup Language (GAIML) an extension of AIML allowing merging both interaction modalities. In this context a suitable chatbot system called Graphbot is presented to support this language. With this language is possible to define personalized interface patterns that are the most suitable ones in relation to the data types exchanged between the user and the system according to the context of the dialogue.

Anthropomorphism, the attribution of human-like characteristics to non-human entities, and Human-Computer Interaction (HCI) or Human-AI Interaction (HAI) have become a significant topic of interest in the field of Artificial Intelligence (AI) with the rise of Large Language Models (LLMs). From this scenario, this article examines the concept of anthropomorphism in the context of chatbots, especially ChatGPT. Drawing on a range of contemporary research and use cases, various implications, benefits, challenges, and ethical considerations associated with anthropomorphism in AI are explored. The findings highlight the potential benefits of anthropomorphism in enhancing user engagement, trust, and acceptance. However, challenges such as overreliance, privacy concerns, and accuracy issues need to be addressed. Answer-Bot Effect and other psychological mechanisms experienced in HCI with ChatGPT reinforce the Computer-Are-Social-Actors paradigm, the Attachment Theory and the Media Equation hypothesis. Ethical considerations are crucial in ensuring responsible development and deployment of anthropomorphic AI systems. Frameworks such as HCAI, IFA, SPADE or SEEK can be of help in ensuring that the development and deployment of such models are responsible and ethical. The need for further research to understand the full potential and limitations of anthropomorphism in Conversational Agents is emphasized, particularly in the context of ChatGPT and Smart Personal Assistants (SPAs).

Mid-air haptic feedback technology produces tactile sensations that are felt without the need for physical interactions. However, mid-air haptic experiences need to be congruent with visual cues to reflect user expectations. To overcome this, we investigate how to visually present properties of objects, so that what one feels is a more accurate prediction of what one sees. Specifically, this paper investigates the relationship between 8 visual parameters of a point-cloud representation of a surface (particle color, size, distribution, etc.) and 4 mid-air haptic spatial modulation frequencies (20, 40, 60 and 80Hz). Our results and analysis reveal a statistical significance between low and high-frequency modulations and particle density, particle bumpiness (depth) and particle arrangement (randomness).

Sensory feedback in upper limb amputees is crucial for improving movement decoding and also to enhance embodiment of the prosthetic limb. Recently, an increasing number of invasive and noninvasive solutions for sensory stimulation have demonstrated the capability of providing a range of sensations to upper limb amputees. However, the cortical impact of restored sensation is not clearly understood. Particularly, understanding the cortical connectivity changes at multiple scales (nodal and modular) in response to sensory stimulation, can reveal crucial information on how amputees brain process the sensory stimuli. Using Electroencephalography (EEG) signals, we compared the cortical connectivity network in response to sensory feedback provided by targeted transcutaneous electrical nerve stimulation (tTENS) in an upper limb amputee during phantom upper limb movements. We focused our cortical connectivity analysis on four functional modules comprising of 20 brain regions that are primarily associated with a visually guided motor task (visual, motor, somatosensory and multisensory integration (MI)) used in this study. At the modular level, we observed that the hubness (a graph theoretic measure quantifying the importance of brain regions in integrating brain function) of the motor module decreases whereas that of the somatosensory module increases in presence of tTENS feedback. At the nodal level, similar observations were made for the visual and MI regions. This is the first work to reveal the impact of sensory feedback at multiple scales in the cortex of amputees in response to sensory stimulation.

Artificial intelligence (AI) and Human Computer Interaction (HCI) share common roots and early work on conversational agents has laid the foundation for both fields. However, in subsequent decades the initial tight connection between the fields has become less pronounced. The recent rise of deep learning has revolutionized AI and has led to a raft of practical methods and tools that significantly impact areas outside of core-AI. In particular, modern AI techniques now power new ways for machines and humans to interact. Thus it is timely to investigate how modern AI can propel HCI research in new ways and how HCI research can help direct AI developments. This workshop offers a forum for researchers to discuss new opportunities that lie in bringing modern AI methods into HCI research, identifying important problems to investigate, showcasing computational and scientific methods that can be applied, and sharing datasets and tools that are already available or proposing those that should be further developed. The topics we are interested in including deep learning methods for understanding and modeling human behaviors and enabling new interaction modalities, hybrid intelligence that combine human and machine intelligence to solve difficult tasks, and tools and methods for interaction data curation and large-scale data-driven design. At the core of these topics, we want to start the conversation on how data-driven and data-centric approaches of modern AI can impact HCI.

Self-perception of body posture and movement is achieved through multi-sensory integration, particularly the utilisation of vision, and proprioceptive information derived from muscles and joints. Disruption to these processes can occur following a neurological accident, such as stroke, leading to sensory and physical impairment. Rehabilitation can be helped through use of augmented visual and auditory biofeedback to stimulate neuro-plasticity, but the effective design and application of feedback, particularly in the auditory domain, is non-trivial. Simple auditory feedback was tested by comparing the stepping accuracy of normal subjects when given a visual spatial target (step length) and an auditory temporal target (step duration). A baseline measurement of step length and duration was taken using optical motion capture. Subjects (n=20) took 20 ‘training’ steps (baseline ±25%) using either an auditory target (950 Hz tone, bell-shaped gain envelope) or visual target (spot marked on the floor) and were then asked to replicate the target step (length or duration corresponding to training) with all feedback removed. Visual cues (mean percentage error=11.5%; SD ± 7.0%); auditory cues (mean percentage error = 12.9%; SD ± 11.8%). Visual cues elicit a high degree of accuracy both in training and follow-up un-cued tasks; despite the novelty of the auditory cues present for subjects, the mean accuracy of subjects approached that for visual cues, and initial results suggest a limited amount of practice using auditory cues can improve performance.

Although multisensory integration is crucial for sensorimotor function, it is unclear how visual and proprioceptive sensory cues are combined in the brain during motor behaviors. Here we characterized the effects of multisensory interactions on local field potential (LFP) activity obtained from the superior parietal lobule (SPL) as non-human primates performed a reaching task with either unimodal (proprioceptive) or bimodal (visual-proprioceptive) sensory feedback. Based on previous analyses of spiking activity, we hypothesized that evoked LFP responses would be tuned to arm location but would be suppressed on bimodal trials, relative to unimodal trials. We also expected to see a substantial number of recording sites with enhanced beta band spectral power for only one set of feedback conditions (e.g. unimodal or bimodal), as was previously observed for spiking activity. We found that evoked activity and beta band power were tuned to arm location at many individual sites, though this tuning often differed between unimodal and bimodal trials. Across the population, both evoked and beta activity were consistent with feedback-dependent tuning to arm location, while beta band activity also showed evidence of response suppression on bimodal trials. The results suggest that multisensory interactions can alter the tuning and gain of arm position-related LFP activity in the SPL.

Sensory experience is the result of a multisensory analysis of the environment around us. When information is properly integrated, visual cues facilitate auditory localization. To investigate the spatial and temporal rules of contingency in multisensory integration, a majority of previous studies have focused on sensory space within the field of vision. But the spaces encoded by vision and audition do not always align with each other. For foveal species such as humans and monkeys, the visual field is restricted to frontal space, whereas the auditory field is panoramic, covering the entire frontal and rear space. The rear sensitivity provided by spatial hearing is critical for avoiding unseen dangers coming from behind. The rear space, however, has been largely overlooked in multisensory research. In this talk, I will present recent work related to vision’s role in panoramic spatial hearing in humans, the changes in visual bias observed in human listeners with chronic unilateral hearing loss, and findings concerning visual modulation of spatial responses of single neurons in the marmoset auditory cortex. Based on these results, I will discuss the challenges of implementing existing theories of multisensory spatial perception in neural circuits.

Integrating conversational generative AI agents into mobile health platforms represents a transformative approach to personalized digital health interventions, fundamentally reshaping how individuals engage with wellness technologies. This article examines how large language models are deployed across fitness coaching, nutritional guidance, and mental wellness applications to create dynamic, contextually aware interactions that adapt to individual needs and preferences. Unlike their rule-based predecessors, these sophisticated agents can process natural language inputs alongside physiological data from wearables to deliver personalized recommendations, emotional support, and behavioral guidance that evolves. The technical architecture enabling these capabilities spans multiple dimensions, including privacy-preserving processing methods, multimodal data integration frameworks, and emotion-aware interaction design. The article demonstrates promising improvements in user engagement, behavioral adherence, and health outcomes across diverse populations; significant challenges remain regarding information accuracy, health equity, appropriate boundaries with clinical care, and potential dependency risks. As this technology continues to evolve, thoughtful attention to ethical implementation, regulatory frameworks, and evidence-based design principles will be essential to realize the full potential of conversational agents as accessible, scalable tools for health behavior change while mitigating risks to vulnerable populations. Increasingly sophisticated multimodal capabilities will likely define the future trajectory of this field, as well as seamless healthcare system integration and personalization approaches that respect both individual autonomy and the irreplaceable value of human connection in health and wellness.