Conversational interfaces that interact with humans need to continuously establish, maintain and repair common ground in task-oriented dialogues. Uncertainty, repairs and acknowledgements are expressed in user behaviour in the continuous efforts of the conversational partners to maintain mutual understanding. Users change their behaviour when interacting with systems in different forms of embodiment, which affects the abilities of these interfaces to observe users’ recurrent social signals. Additionally, humans are intellectually biased towards social activity when facing anthropomorphic agents or when presented with subtle social cues. Two studies are presented in this paper examining how humans interact in a referential communication task with wizarded interfaces in different forms of embodiment. In study 1 (N = 30), we test whether humans respond the same way to agents, in different forms of embodiment and social behaviour. In study 2 (N = 44), we replicate the same task and agents but introduce conversational failures disrupting the process of grounding. Findings indicate that it is not always favourable for agents to be anthropomorphised or to communicate with non-verbal cues, as human grounding behaviours change when embodiment and failures are manipulated.

Interactive simulations are tools that can help students understand and learn about complex relationships. While most simulations are primarily visual due to mostly historical reasons, sounds can be used to add to the experience. In this work, we evaluated sets of audio designs for two different, but contextually- and visually-similar simulations. We identified key aspects of the audio representations and the simulation content which needed to be evaluated, and compared designs across two simulations to understand which auditory designs could generalize to other simulations. To compare the designs and explore how audio affected a user’s experience, we measured preference (through usability, user experience, and open-ended questions) and interpretation accuracy for different aspects of the simulation (including the main relationships and control feedback). We suggest important characteristics to represent through audio for future simulations, provide sound design suggestions, and address how overlap between visual and audio representations can support learning opportunities.

Social signal processing and virtual social interaction technologies have allowed the creation of social skills training applications, and initial studies have shown that such solutions can lead to positive training outcomes and could complement traditional teaching methods by providing cheap, accessible, safe tools for training social skills. However, these studies evaluated social skills training systems as a whole and it is unclear to what extent which components contributed to positive outcomes. In this paper, we describe an experimental study where we compared the relative efficacy of real-time interactive feedback and after-action feedback in the context of a public speaking training application. We observed that both components provide benefits to the overall training: the real-time interactive feedback made the experience more immersive and improved participants’ motivation in using the system, while the after-action feedback led to positive training outcomes when it contained personalized feedback elements. Taken in combination, these results confirm that both social signal processing technologies and virtual social interactions are both contributing to social skills training systems’ efficiency. Additionally, we observed that several individual factors, here the subjects’ initial level of public speaking anxiety, personality and tendency to immersion significantly influenced the training experience. This finding suggests that social skills training systems could benefit from being tailored to participants’ particular individual circumstances.

Addressee detection is a fundamental task for seamless dialogue management and turn taking in human-agent interaction. Though addressee detection is implicit in dyadic interaction, it becomes a challenging task when more than two participants are involved. This article proposes multiple addressee detection models based on smart feature selection and focus encoding schemes. The models are trained using different machine learning and deep learning algorithms. This research work improves existing baseline accuracies for addressee prediction on two datasets. In addition, the article explores the impact of different focus encoding schemes in several addressee detection cases. Finally, an implementation strategy for addressee detection model in real-time is discussed.

Remote collaboration has been studied for more than two decades and now there is the possibilities for new types of collaboration with the recent advances in immersive technologies such as Virtual, Augmented, Mixed Reality (VR/AR/MR). However, despite the increasing research interest in remote collaboration study with VR/AR/MR technologies, there is still a lack of academic venues specifically focusing on VR/AR/MR remote collaboration research. This special issue provides high-quality papers on the topic of remote collaboration research and increases visibility of this timely interesting and important research area. We particularly focus on three research aspects in remote collaboration: (1) use of multimodal communication cues, (2) awareness of the task space, and (3) human factors understanding. In this editorial, we first describe five essential factors for remote collaboration: task, local user, remote user, communication, and tool/interface, and then summarize a brief history of the research areas. We also cover the feature papers accepted in this issue, which introduce novel multimodal interfaces for remote collaboration and the effects on task performance and perceptual factors. Finally, we discuss some potential future research directions while concluding the editorial.

An extensive metamorphosis is currently taking place in the education industry due to the rapid adoption of different technologies and the proliferation of new student-instructor and student–student interaction models. While traditional face-to-face interaction is still the norm, mobile, online and virtual augmentations are increasingly adopted worldwide. Moreover, with the advent of gaming technology besides the 3D visual paradigm, the “touch” and “feel” paradigm is slowly taking its place in the user interface design through gamification. While haptic (force feedback) devices were barely available a decade ago outside research laboratories, the rapid rise in gaming technology has driven the cost significantly lower enabling the spread of these devices in many households and the wide public. This article presents a novel haptic-based training tool implemented as a gaming scenario to assist students in learning of abstract concepts in Physics. The focus is on electromagnetism as one of the fundamental forces in nature and specifically the abstractions used as building blocks around the Lorentz force. Experimental results suggest that by introducing well designed visual-haptic interfaces in presenting abstract concepts, students become better engaged in the classrooms and superior learning outcomes can be achieved.

Data analytics-driven solutions are widely used in various intelligent systems, where humans and machines make decisions collaboratively based on predictions. Human factors such as personality and trust have significant effects on such human–machine collaborations. This paper investigates effects of personality traits on user trust in human–machine collaborations under uncertainty and cognitive load conditions. A user study of 42 subjects in a repeated factorial design experiment found that uncertainty presentation led to increased trust but only under low cognitive load conditions when users had sufficient cognitive resources to process the information. Presentation of uncertainty under high load conditions led to a decrease in trust. When further drilling down into personality trait groups of users, overall, users with low Openness showed the highest trust. Furthermore, under the low cognitive load condition, it was found that the trust was enhanced under ambiguity uncertainty with low Agreeableness, low Neuroticism, high Extraversion, high Conscientiousness, and high Openness. Under the high cognitive load condition, high Neuroticism and low Extraversion benefitted the trust without the uncertainty presentation. The results demonstrated that different personality traits affected trust differently under uncertainty and cognitive load conditions. A framework of user trust feedback loop was set up to incorporate the study results into human–machine collaborations for the meaningful participatory design.

This paper describes a new web-based system specially adapted to the education of Czech pupils with visual impairment. The system integrates speech and language technologies with a web framework in lower secondary education, especially in mathematics and physics subjects. A new interface utilized the text-to-speech (TTS) synthesis for online automatic reading of educational texts. The interface provides several TTS voices, synthesized data caching, and automatic processing of formulas in mathematics and physics. The system was designed to enable teachers create and manage teaching materials. It also enables the pupils to view and listen to the read forms of these documents online. A school for pupils with visual impairment participated in the development and implementation of the system. After one year of using the system daily, the user experience and evaluation data were collected. The results indicate a positive reception and frequent use of the system as well as a preference over classical educational materials.

When designing movement sonifications, their effect on people’s movement must be considered. Recent work has shown how real-time sonification can be designed to alter the way people move. However, the mechanisms through which these sonifications alter people’s expectations of their movement is not well explained. This is especially important when considering musical sonifications, to which people bring their own associations and musical expectation, and which can, in turn, alter their perception of the sonification. This paper presents a Movement Expectation Sonification Model, based on theories of motor-feedback and expectation, to explore how musical sonification can impact the way people perceive their movement. Secondly, we present a study that validates the predictions of this model by exploring how harmonic stability within sonification interacts with contextual cues in the environment to impact movement behaviour and perceptions. We show how musical expectancy can be built to either reward or encourage movement, and how such an effect is mediated through the presence of additional cues. This model offers a way for sonification designers to create movement sonifications that not only inform movement but can be used to encourage progress and reward successes.