Adaptive User Interface Design Based on Keystroke Dynamics for Enhanced Digital Well-being

Adapting the user interface of a software system to the requirements of the context of use continues to be a major challenge, particularly when users become more demanding in terms of adaptation quality. A considerable number of methods have, over the past three decades, provided some form of modelling with which to support user interface adaptation. There is, however, a crucial issue as regards in analysing the concepts, the underlying knowledge, and the user experience afforded by these methods as regards comparing their benefits and shortcomings. These methods are so numerous that positioning a new method in the state of the art is challenging. This paper, therefore, defines a conceptual reference framework for intelligent user interface adaptation containing a set of conceptual adaptation properties that are useful for model-based user interface adaptation. The objective of this set of properties is to understand any method, to compare various methods and to generate new ideas for adaptation. We also analyse the opportunities that machine learning techniques could provide for data processing and analysis in this context, and identify some open challenges in order to guarantee an appropriate user experience for end-users. The relevant literature and our experience in research and industrial collaboration have been used as the basis on which to propose future directions in which these challenges can be addressed.

In this work, a novel computational approach for the dynamic adaptation of User Interfaces (UIs) is proposed, which aims at enhancing the Situational Awareness (SA) of users by leveraging the current context and providing the most useful information, in an optimal and efficient manner. By combining Ontology modeling and reasoning with Combinatorial Optimization, the system decides <italic>what</italic> information to present, <italic>when</italic> to present it, <italic>where</italic> to visualize it in the display - and <italic>how</italic>, taking into consideration contextual factors as well as placement constraints. The main objective of the proposed approach is to optimize the SA associated with the displayed UI <italic>at run-time</italic>, while avoiding information overload and induced stress. In the context of this work, we have deployed our computational approach to the use case of an Augmented Reality (AR) system for Law Enforcement Agents (LEAs). To explore the benefits and limitations of the developed system, two evaluations have been conducted. The first one was an expert-based evaluation with LEAs and User Experience (UX) experts, assessing the appropriateness of the system&#x2019;s decisions. The second one was a user-based evaluation involving LEAs from different agencies, estimating the SA, the mental workload and the overall UX associated with the system, through an AR simulation. The results indicate that the system enhances SA, and while not imposing workload, it provides an overall positive UX.

This study explores the creation of emotion- responsive user interfaces (UIs) that adapt to users' mental states in stressful settings in real time. Utilizing human- computer interaction (HCI) principles and artificial intelligence, it introduces an innovative framework for UIs that adjusts based on fatigue, concentration, and emotional cues. The method incorporates sophisticated facial recognition, eye tracking, and physiological sensors to continuously evaluate users' cognitive and emotional conditions. The adaptive UI system uses machine learning algorithms to optimize cognitive load and improve remote interactions in various high-pressure situations. Through a series of experiments and user studies, notable enhancements in task performance, user satisfaction, and overall cognitive efficiency have been demonstrated. The results underscore the potential of emotion-responsive UIs to transform human-computer interaction, particularly in areas such as healthcare, emergency response, and mission-critical operations. Keywords — HCI, Emotion-Adaptive UI, Cognitive Efficiency, Real-Time Adaptation, Fatigue, Focus, Emotional Expression, Artificial Intelligence, Machine Learning, User Interface, High- Stress Environments, Remote Interaction.

Speech has become increasingly important in human–computer interaction. Spoken dialog interfaces rely on automatic speech recognition, speech synthesis, language understanding, and dialog management. A main issue in dialog systems is that they typically are limited to pre-programmed vocabularies and sets of sentences. The research reported here focuses on developing an adaptive spoken dialog interface capable of acquiring new linguistic units and their corresponding semantics during the human–computer interaction. The adaptive interface identifies unknown words and phrases in the users utterances and asks the user for the corresponding semantics. The user can provide the meaning or the semantic representation of the new linguistic units through multiple modalities, including speaking, typing, pointing, touching, or showing. The interface then stores the new linguistic units in a semantic grammar and creates new objects defining the corresponding semantic representation. This process takes place during natural interaction between user and computer and, thus, the interface does not have to be rewritten and compiled to incorporate the newly acquired language. Users can personalize the adaptive spoken interface for different domain applications, or according to their personal preferences. [Work supported by NSF.]

Technological progress in the fields of ubiquitous sensing and machine learning has been fueling the development of user-aware human-computer interaction in recent years. Especially natural user interfaces, like digital voice assistants, can benefit from understanding their users in order to provide a more naturalistic experience. Such systems can, for example, detect the emotional state of users and accordingly act in an empathic way. One major research field working on this topic is Affective Computing, where psycho-physiological measures, speech input, and facial expressions are used to sense human emotions. Affective data allows natural user interfaces to respond to emotions, providing promising perspectives not only for user experience design but also for safety aspects. In automotive environments, informed estimations of the driver’s state can potentially avoid dangerous errors and evoking positive emotions can improve the experience of driving. This dissertation explores Affective Automotive User Interfaces using two basic interaction paradigms: firstly, emotion regulation systems react to the current emotional state of the user based on live sensing data, allowing for quick interventions. Secondly, emotional interaction synthesizes experiences which resonate with the user on an emotional level. The constituted goals of these two interaction approaches are the promotion of safe behavior and an improvement of user experience. Promoting safe behavior through emotion regulation: Systems which detect and react to the driver’s state are expected to have great potential for improving road safety. This work presents a model and methods needed to investigate such systems and an exploration of several approaches to keep the driver in a safe state. The presented methods include techniques to induce emotions and to sample the emotional state of drivers. Three driving simulator studies investigate the impacts of emotionaware interventions in the form of implicit cues, visual mirroring and empathic speech synthesis. We envision emotion-awareness as a safety feature which can detect if a driver is unfit or in need of support, based on the propagation of robust emotion detection technology. Improving user experience with emotional interaction: Emotional perception is an essential part of user experience. This thesis entails methods to build emotional experiences derived from a variety of lab and simulator studies, expert feedback, car-storming sessions and design thinking workshops. Systems capable of adapting to the user’s preferences and traits in order to create an emotionally satisfactory user experience do not require the input of emotion detection. They rather create value through general knowledge about the user by adapting the output they generate. During this research, cultural and generational influences became evident, which have to be considered when implementing affective automotive user interfaces in future cars. We argue that the future of user-aware interaction lies in adapting not only to the driver’s preferences and settings but also to their current state. This paves the way for the regulation of safe behavior, especially in safety-critical environments like cars, and an improvement of the driving experience.

The user interface (UI) layer is considered an important component in software applications since it links the users to the software's functionality. Enterprise applications such as enterprise resource planning and customer relationship management systems have very complex UIs that are used by users with diverse needs in terms of the required features and layout preferences. The inability to cater for the variety of user needs diminishes the usability of these applications. One way to cater for those needs is through adaptive UIs. Some enterprise software providers offer mechanisms for tailoring UIs based on the variable user needs, yet those are not generic enough to be used with other applications and require maintaining multiple UI copies manually. A generic platform based on a model-driven approach could be more reusable since operating on the model level makes it technology independent. The main objective of this research is devising a generic, scalable, and extensible platform for building adaptive enterprise application UIs based on a runtime model-driven approach. This platform primarily targets UI simplification, which we defined as a mechanism for increasing usability through adaptive behavior by providing users with a minimal feature-set and an optimal layout based on the context-of-use. This paper provides an overview of the research questions and methodology, the results that were achieved so far, and the remaining work.

Mobile computing devices and their applications that encompass context aware components are becoming increasingly more prevalent. The context-awareness of these types of applications typically focuses on the services offered. In this paper we describe a framework that supports the monitoring and analysis of mobile application usage patterns with the goal of updating user models for adaptive services and user interface personalisation. This paper focuses on two aspects of the framework. The first is the modelling and storage of the usage data. The second focuses on the data mining component of the framework, outlining the five different capabilities of the adaptation in addition to the algorithms used. The proposed framework has been evaluated through specific case studies, with the results attained demonstrating the effectiveness of the data mining capabilities and in particular the adaptation of the User Interface. The accuracy and efficiency of the algorithms used are also evaluated with three users. The results of the evaluation show that the aims of the data mining component were achieved with the personalisation and adaptation of content and user interface, respectively.

This work presents the design and analysis of an Adaptive User Interface (AUI) for a desktop application that uses a novel solution for the recognition of the emotional state of a user through both facial expressions and body posture from an RGB-D sensor. Six basic emotions are recognized through facial expressions in addition to the physiological state, which is recognized through the body posture. The facial expressions and body posture are acquired in real-time from a Kinect sensor. A scoring system is used to improve recognition by minimizing the confusion between the different emotions. The implemented solution achieves an accuracy rate of above 90%. The recognized emotion is then used to derive an Automatic AUI where the user can use speech commands to modify the User Interface (UI) automatically. A comprehensive user study is performed to compare the usability of an Automatic, Manual, and a Hybrid AUI. The AUIs are evaluated in terms of their efficiency, effectiveness, productivity, and error safety. Additionally, a comprehensive analysis is performed to evaluate the results from the viewpoint of different genders and age groups. Results show that the hybrid adaptation improves usability in terms of productivity and efficiency. Finally, a combination of both automatic and hybrid AUIs result in significantly positive user experience compared to the manual adaptation.

Human Computer Interaction (HCI) performance can be impaired by several HCI obstacles. Cognitive adaptive systems should dynamically detect such obstacles and compensate them with suitable User Interface (UI) adaptation. In this paper, we discuss the detection of two main HCI obstacles: memory-based and visual obstacles. A sequential model based on Long-Short Term Memory (LSTM) is suggested for such a detection of HCI obstacles. UI adaptations for both types of obstacles are discussed and analyzed. We investigate the classification performance on data from a user study with 17 participants. Furthermore, we also investigate the influence of different adaptation mechanisms on performance and subjective assessment. Results show advantages of the proposed sequential LSTM model: on the one hand, the LSTM outperforms the baseline random guess and also a baseline static model LDA in the detection of visual obstacles with 70.6% as an average accuracy. On the other hand, the evaluation of HCI sessions impeded by obstacles but supported with different UI adaptations shows that LSTM results well match the subjective assessment as a plausible detector of behaviour changes.

User expertise and skill affects the way users interact with software. It is envisaged that an adaptive user interface (AUI) which dynamically changes from a novice user interface (UI) to an expert UI could possibly improve users' performance. Contact centres (CCs), are the primary interaction point between a company and its customers and an important challenge is to increase the speed at which contact centre agents (CCAs) retrieve information to answer customer queries. CCAs however have different capabilities, experience and expertise. Studies have provided empirical support that user performance can be increased when the computer UI characteristics match the user skill level. As a result, software systems have to become more individualised and cater for different users. This research focuses on applying this concept of AUIs to the domain of CCs and investigates whether an AUI for CCAs could improve the CCA's performance and enhance their interaction with UI. The main aim of this research study was to propose a model for AUIs which could be applied to the domain of CCs. The implementation of the proposed model was completed as proof of concept and the model was evaluated. The results indicate that the proposed AUI model significantly improves CCAs' performance and most users were satisfied with the adaptation provided by the model.

Adapting the user interface (UI) of software systems to meet the needs and preferences of users is a complex task. The main challenge is to provide the appropriate adaptations at the appropriate time to offer value to end-users. Recent advances in Machine Learning (ML) techniques may provide effective means to support the adaptation process. In this paper, we instantiate a reference framework for Intelligent User Interface Adaptation by using Reinforcement Learning (RL) as the ML component to adapt user interfaces and ultimately improving the overall User Experience (UX). By using RL, the system is able to learn from past adaptations to improve the decision-making capabilities. Moreover, assessing the success of such adaptations remains a challenge. To overcome this issue, we propose to use predictive Human-Computer Interaction (HCI) models to evaluate the outcome of each action (i.e., adaptations) performed by the RL agent. In addition, we present an implementation of the instantiated framework, which is an extension of OpenAI Gym, that serves as a toolkit for developing and comparing RL algorithms. This Gym environment is highly configurable and extensible to other UI adaptation contexts. The evaluation results show that our RL-based framework can successfully train RL agents able to learn how to adapt UIs in a specific context to maximize the user engagement by using an HCI model as rewards predictor.

One of the problems adaptive interfaces must solve is the issue of stability---users must be able to complete a familiar task reliably. Split Adaptive Interfaces, where a limited part of the screen contains copies of the interface elements predicted to be of immediate use, are one technique for resolving this difficulty. While prior work demonstrated that Split Adaptive Interfaces improve performance on average, the results of our study demonstrate systematic individual differences in the utilization of the adaptive features, which correlate with the stable user traits of Need for Cognition and Extraversion. Specifically, higher Need for Cognition (a willingness to undertake difficult mental activities) is correlated with increased utilization rates, while higher Extraversion (a general orientation towards seeking gratification from the external world) is negatively correlated with utilization rates. Our results also demonstrate a significant negative correlation between cognitive load induced by a secondary task and the utilization of the adaptive features. This effect, however, is very small (less than two percentage points). Together, these results provide additional evidence of the usefulness of the split adaptive interface approach and a negligible effect of additional cognitive load, but also demonstrate that the approach does not benefit all users equally.

Human-computer interaction for virtual-real fusion

ion and Activity in Computer-Mediated Music Production Matthew Duignan,∗ James Noble,∗ and Robert Biddle† ∗School of Engineering and Computer Science Victoria University of Wellington Wellington, New Zealand matthew@duignan.net kjx@ecs.vuw.ac.nz †Human Oriented Technology Laboratory Carleton University Ottawa, Canada robert biddle@carleton.ca Digital audio workstations (DAWs) such as Digidesign Pro Tools, Apple Logic, and Ableton Live are the cornerstone of composition, recording, editing, and performing activities for producers working in popular music (Theberge 1997). Human– computer interaction (HCI) research has a unique challenge in understanding the activities of professional music producers and in designing DAW user interfaces to support this work. Unlike many other user-interface design domains, in computermediated music production the user is principally engaged in the process of building and editing immensely complex digital representations (Lerdahl and Jackendoff 1983; Pope 1986; Dannenberg, Rubine, and Neuendorffer 1991; Dannenberg 1993). In this case, those representations model the intricate structure and synthesis parameters of the musical composition the producer is creating. Determining the right vocabulary of abstract representations to build into the user interface of DAWs is a difficult problem, and these design decisions have a critical impact on the activity of professional producers. The design of these abstraction mechanisms has been primarily informed by the historical origins of DAWs in multitrack tape recorders and mixing desks (Theberge 2004), which together we refer to as the “multitrack-mixing model.” Our research has identified many ways in which these user-interface metaphors (Barr 2003) from the past often do not support the activities of professional producers. The evolutionary reliance of DAWs on the multitrack-mixing model can be contrasted with more radical algorithmic approaches to composition and performance, such as those found in visual or Computer Music Journal, 34:4, pp. 22–33, Winter 2010 c © 2010 Massachusetts Institute of Technology. textual programmatic tools that are common in the experimental and avant-garde computer-music traditions. Because these tools have a different set of abstraction mechanisms and corresponding tradeoffs, their procedural rather than declarative nature (Dannenberg 1993) and their focus on “generative” music-making render them less well suited to the work of the participants in this study, and therefore outside the scope of this discussion. This article outlines findings from our detailed qualitative investigation (Duignan 2008) into the activity of computer-mediated music-making in the popular idiom, and the abstraction mechanisms that professional producers use to manage the complex digital representations of their compositions for studio work and live performance. We present a framework that articulates the key interactions and tensions between professional producers and the abstraction mechanisms provided by the tools on which they so heavily rely. This framework helps us understand and clearly identify issues that need to be resolved in the next generation of DAW user interfaces. Theoretical Perspective This research was conducted as a collective case study based on Creswell’s Qualitative Inquiry and Research Design (1998), and it exploits the emphasis of case-study inquiry on developing an in-depth analysis of a particular activity. We employed comprehensive semi-structured interviews of professional producers and conducted extended observations of them in the field. We focused on five participants for the in-depth case study, with twelve additional participants who added breadth of 22 Computer Music Journal Figure 1. Music production tools play a vital mediating role between users and their composition. perspective to better support meaningful triangulation and context for the issues that we discovered. We drew from two theoretical traditions with associated research tools in creating our semistructured interview protocol and in analyzing the findings: activity theory and cognitive dimensions of notations. Activity theory (Vygotskii 1978; Nardi 1996) provides us with a useful terminology, a framework for analysis, and a clear catalog of the important components of human activity (Kaptelinin, Nardi, and Macaulay 1999) that we used to develop an interview protocol (Duignan, Noble, and Biddle 2006). Cognitive dimensions of notations (Blackwell and Green 2003) provides a common vocabulary for describing recurring trade-offs in the conceptual models employed by notational systems. Cognitive dimensions have obvious relevance to our inquiry, having been applied to music typesetting and “live-coding” musical systems in the past (Blackwell and Green 2000; Blackwell and Collins 2005). The cognitive dimensions framework provided us with further content for our interview protocol (Blackwell and Green 2000) and a terminology and framework for analyzing the tensions between music production abstractions and the activity of producers.

User interface adaptation has been proposed in recent years as a means to achieve personalized accessibility and usability of user interfaces. Related user interface architectures, as well as a variety of related development method and tools, have also been elaborated. Admittedly, however, despite the recognized validity of the approach, which has been proved in practice in a series of prototype systems, the wider adoption and uptake of user interface adaptation approaches are still limited. One of the identified obstacles is the complexity intrinsic in designing such interfaces and the need of radically revising the current user interface design practice to account for (a) the alternative designs required for adaptation, (b) the parameters involved in driving adaptations (i.e., selecting among alternatives at a given point during interaction), and (c) the logic of adaptation at runtime. This article proposes a twofold tool-based support strategy for user interface adaptation development, based on (a) an adaptation development toolkit and related widget library, which directly embeds lexical level adaptations into common interactive widgets, and (b) embedding such a library in a common integrated development environment, thus allowing designers to define and view alternative adaptations at design time and create adaptable user interfaces through traditional prototyping. The aforementioned approach has been implemented in the domain of adaptable applications for older users, producing tools that are currently in use in the development of a large suite of interactive applications in various domains. The approach presented in this article is claimed to be the first and so far unique supporting rapid prototyping of adaptable user interfaces, thus minimizing the divergence between typical development practices and user interface adaptation development.