Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations “in-the-wild”. These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers’ tasks. These practicalities have been rarely investigated. This article addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators in the context of nursing education. The lessons learnt were synthesised into topics related to (i) technological/physical aspects of the deployment; (ii) multimodal data and interfaces; (iii) the design process; (iv) participation, ethics and privacy; and (v) sustainability of the deployment.

Today’s knowledge workers face cognitively demanding tasks and blurred work-life boundaries amidst rising stress and burnout in the workplace. Holistic approaches to supporting workers, which consider both productivity and well-being, are increasingly important. Taking this holistic approach, we designed an intervention inspired by cognitive behavioral therapy that consists of: (1) using the term “Time Well Spent” (TWS) in place of “productivity”, (2) a mobile self-logging tool for logging activities, feelings, and thoughts at work, and (3) a visualization that guides users to reflect on their data. We ran a 4-week exploratory qualitative comparison in the field with 24 graduate students to examine ourTherapy-inspiredintervention alongside a classicBaselineintervention. Participants who used our intervention often shifted toward a holistic perspective of their primary working hours, which included an increased consideration of breaks and emotions. No such change was seen by those who used theBaselineintervention.

While developing a story, novices and published writers alike have had to look outside themselves for inspiration. Language models have recently been able to generate text fluently, producing new stochastic narratives upon request. However, effectively integrating such capabilities with human cognitive faculties and creative processes remains challenging. We propose to investigate this integration with a multimodal writing support interface that offers writing suggestions textually, visually, and aurally. We conduct an extensive study that combines elicitation of prior expectations before writing, observation and semi-structured interviews during writing, and outcome evaluations after writing. Our results illustrate the individual and situational variation in machine-in-the-loop writing approaches, suggestion acceptance, and ways the system is helpful. Centrally, we report how participants performintegrative leaps, by which they do cognitive work to integrate suggestions of varying semantic relevance into their developing stories. We interpret these findings, offering modeling and design recommendations for future creative writing support technologies.

Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user’s domain expertise, mental models of an AI system, trust in recommendations, and more. This article reports on a study that examines users’ interactions with three simulated algorithmic models, all with equivalent accuracy rates but each tuned differently in terms of true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Users completed 140 trials across multiple stages, first without an AI and then with recommendations from an AI-Assistant. Although all users had prior experience with the task, their levels of proficiency varied widely.Our results demonstrated that while recommendations from an AI-Assistant can aid in users’ decision making, several underlying factors, including user base expertise and complementary human-AI tuning, significantly impact the overall team performance. First, users’ base performance matters, particularly in comparison to the performance level of the AI. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. Second, tuning an AI algorithm to complement users’ strengths and weaknesses also significantly impacted users’ performance. For example, users in our study were better at detecting flowing blood vessels, so when the AI was tuned to reduce false negatives (at the expense of increasing false positives), users were able to reject those recommendations more easily and improve in accuracy. Finally, users’ perception of the AI’s performance relative to their own performance had an impact on whether users’ accuracy improved when given recommendations from the AI. Overall, this work reveals important insights on the complex interplay of factors influencing Human-AI collaboration and provides recommendations on how to design and tune AI algorithms to complement users in decision-making tasks.

Computational media describes a vision of software, which, in contrast to application-centric software, is (1) malleable, so users can modify existing functionality, (2) computable, so users can run custom code, (3) distributable, so users can open documents across different devices, and (4) shareable, so users can easily share and collaborate on documents. Over the last ten years, the Webstrates and Codestrates projects aimed at realizing this vision of computational media. Webstrates is a server application that synchronizes the DOM of websites. Codestrates builds on top of Webstrates and adds an authoring environment, which blurs the use and development of applications. Grounded in a chronology of the development of Webstrates and Codestrates, we present eight tensions that we needed to balance during their development. We use these tensions as an analytical lens in three case studies and a game challenge in which participants created games using Codestrates. We discuss the results of the game challenge based on these tensions and present key takeaways for six of them. Finally, we present six lessons learned from our endeavor to realize the vision of computational media, demonstrating the balancing act of weighing the vision against the pragmatics of implementing a working system.

Hybridity in immersive technologies has not been studied for factors that are likely to influence engagement. A noticeable factor is the spatial enclosure that defines where users meet. This involves a mutual object of interest, contents that the users may generate around the object, and the proximity between users. This study examines these factors, namely how object interactivity, user-generated contents (UGC) and avatar proximity influence engagement. We designed a Hybrid Virtual and Augmented Reality (HVAR) environment that supports paired users to experience cultural heritage in both Virtual Reality (VR) and Augmented Reality (AR) . A user study was conducted with 60 participants, providing assessments of engagement and presence via questionnaires, together with mobile electroencephalogram (mEEG) and user activity data that measures VR user engagement in real-time. Our findings provide insights into how engagement between users can occur in HVAR environments for the future hybrid reality with multi-device connectivity.

Ingestible sensors have become smaller and more powerful and allow us to envisage new human-computer interactions and bodily play experiences inside our bodies. Users can swallow ingestible sensors, which facilitate interior body sensing functions that provide data on which play experiences can be built. We call bodily play that uses ingestible sensors as play technologies “ingestible play”, and we have adopted a research-through-design (RtD) approach to investigate three prototypes. For each prototype, we conducted a field study to understand the player experiences. Based upon these results and practical design experiences, we have developed a design framework for ingestible play. We hope this work can guide the future design of ingestible play; inspire the design of play technologies inside the human body to expand the current bodily play design space; and ultimately extend our understanding of how to design for the human body by considering the bodily experience of one’s interior body.

We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.

This systematic mapping review sheds light on how emerging technologies have been introduced and taught in various K–12 learning settings, particularly with regard to artificial intelligence (AI), machine learning (ML), the internet of things (IoT), augmented reality (AR), and virtual reality (VR). These technologies are rapidly being integrated into children's everyday lives, but their functions and implications are rarely understood due to their complex and distributed nature. The review provides a rigorous overview of the state of the art based on 107 records published across the fields of human-computer interaction, learning sciences, computing education, and child–computer interaction between 2010 and 2020. The findings show the urgent need on a global scale for inter- and transdisciplinary research that can integrate these dispersed contributions into a more coherent field of research and practice. The article presents nine discussion points for developing a shared agenda to mature the field. Based on the HCI community's expertise in human-centred approaches to technology and aspects of learning, we argue that the community is ideally positioned to take a leading role in the realisation of this future research agenda.

Picture interactions are key to daily and long-term social connections between families and communities, especially through reminiscence. Across the nearly 200-year history of domestic photography, this social reminiscence has been accomplished largely through photo albums. However, in the now common digital setting, albums are pushed aside for the endless film roll metaphor. In this article, we explore this metaphor through the 20-year history of proposed digital picture interactions from Human–Computer Interaction research, and compare this to ongoing interactions with modern picture tools. Through these, we reveal that this prominent design metaphor does not create space for social reminiscence, but does fit the novel use of immediate sharing seen across social networking. Furthermore, the endless and ever-growing nature of digital film rolls are not meaningfully browsable for either intended use. We close by reconnecting to the past works that explore the broad potential of digital pictures.