Authoring context-dependent cross-device user interfaces based on trigger/action rules

Controlling assistive robots can be challenging for some users, especially those lacking relevant experience. Augmented Reality (AR) User Interfaces (UIs) have the potential to facilitate this task. Although extensive research regarding legged manipulators exists, comparatively little is on their UIs. Most existing UIs leverage traditional control interfaces such as joysticks, Hand-Held (HH) controllers and 2D UIs. These interfaces not only risk being unintuitive, thus discouraging interaction with the robot partner, but also draw the operator’s focus away from the task and towards the UI. This shift in attention raises additional safety concerns, particularly in potentially hazardous environments where legged manipulators are frequently deployed. Moreover, traditional interfaces limit the operators’ availability to use their hands for other tasks. Towards overcoming these limitations, in this article, we provide a user study comparing an AR Head-Mounted Display (HMD) UI we developed for controlling a legged manipulator against off-the-shelf control methods for such robots. This user study involved 27 participants and 135 trials, from which we gathered over 405 completed questionnaires. These trials involved multiple navigation and manipulation tasks with varying difficulty levels using a Boston Dynamics’s Spot, a 7 df Kinova robot arm and a Robotiq 2F-85 gripper that we integrated into a legged manipulator. We made the comparison between UIs across multiple dimensions relevant to a successful human–robot interaction. These dimensions include cognitive workload, technology acceptance, fluency, system usability, immersion and trust. Our study employed a factorial experimental design with participants undergoing five different conditions, generating longitudinal data. Due to potential unknown distributions and outliers in such data, using parametric methods for its analysis is questionable, and while non-parametric alternatives exist, they may lead to reduced statistical power. Therefore, to analyse the data that resulted from our experiment, we chose Bayesian data analysis as an effective alternative to address these limitations. Our results show that AR UIs can outpace HH-based control methods and reduce the cognitive requirements when designers include hands-free interactions and cognitive offloading principles into the UI. Furthermore, the use of the AR UI together with our cognitive offloading feature resulted in higher usability scores and significantly higher fluency and Technology Acceptance Model scores. Regarding immersion, our results revealed that the response values for the AR Immersion questionnaire associated with the AR UI are significantly higher than those associated with the HH UI, regardless of the main interaction method with the former, i.e., hand gestures or cognitive offloading. Derived from the participants’ qualitative answers, we believe this is due to a combination of factors, of which the most important is the free use of the hands when using the HMD, as well as the ability to see the real environment without the need to divert their attention to the UI. Regarding trust, our findings did not display discernible differences in reported trust scores across UI options. However, during the manipulation phase of our user study, where participants were given the choice to select their preferred UI, they consistently reported higher levels of trust compared to the navigation category. Moreover, there was a drastic change in the percentage of participants that selected the AR UI for completing this manipulation stage after incorporating the cognitive offloading feature. Thus, trust seems to have mediated the use and non-use of the UIs in a dimension different from the ones considered in our study, i.e., delegation and reliance. Therefore, our AR HMD UI for the control of legged manipulators was found to improve human–robot interaction across several relevant dimensions, underscoring the critical role of UI design in the effective and trustworthy utilisation of robotic systems.

The success of context-aware systems depends on intuitive user interface designs which facilitate communication between multiple devices. This paper presents Throw It, a natural user interface for sharing objects between mobile devices using gaze and hand gestures. For enabling gesture control, Throw It includes an off-the-shelf monochrome camera with infrared lights and a hand-held 6 degree of freedom sensor. To share an object, the user first locks their gaze on the object, then selects the object by doing a short 'grab' gesture, and finally, sends the object to another mobile device by doing a 'throwing' gesture. The mobile devices locate each other via a context server. According to the results of our user study, the idea of sharing digital objects using Throw It was found both useful and fascinating. However, sharing objects to many people at the same time raised security concerns.

The popular view of software engineering focuses on managing teams of people to produce large systems. This paper addresses a different angle of software engineering, that of development for re-use and portability. We consider how an essential part of most software products - the user interface - can be successfully engineered so that it can be portable across multiple platforms and on multiple devices. Our research has identified the structure of the problem domain, and we have filled in some of the answers. We investigate promising solutions from the model-driven frameworks of the 1990s, to modern XML-based specification notations (Views, XUL, XIML, XAML), multi-platform toolkits (Qt and Gtk), and our new work, Mirrors which pioneers reflective libraries. The methodology on which Views and Mirrors is based enables existing GUI libraries to be transported to new operating systems. The paper also identifies cross-cutting challenges related to education, standardization and the impact of mobile and tangible devices on the future design of UIs. This paper seeks to position user interface construction as an important challenge in software engineering, worthy of ongoing research.

It takes movie camera operators years of professional training to follow an object in an aesthetically pleasing way, both by classical means (boom, slider) and by drones. As this complex task requires a high workload and situation awareness while controlling the camera, an uncluttered and efficient user interface (UI) is preferred. The emergence of mobile devices and motion control devices incorporating automation made touch-based UIs attractive to operators. Much work has already been done on UI adaptation strategies. However, little work is trying to solve the problem of combining manual control and automation within a UI. Especially with a central premise of minimising occlusion and visual clutter in a cinematic context. We, therefore, conducted a first user study (N=15) to evaluate different design alternatives regarding occlusion and preference. Afterwards, we created a functional prototype of the most promising design. To further reduce the occlusion we applied a progressive reduction adaption strategy. We evaluated the influence of different reduction levels on workload, control, creativity support and precision in a second user study (N=24). While we could reduce the clutter, due to our design decisions we found no negative effects affecting the measured variables.

Portraits, also known as images of people, constitute an important part of consumer photos. Existing methods manage portraits based on either explicit objectives, e.g., a specified person or event, or aesthetics, i.e., the aesthetic quality of portraits. This paper presents a novel system for personalized portraits ranking. First, four kinds of personalized features, i.e., composition, clothing style, affection and social relationship are proposed to quantify users' intent. Then, example-based and sketch-based user interfaces (UI) are developed, which are capable of capturing users' personal intent hardly described by queries or aesthetics. Finally, portraits ranking is implemented by combing these features together with the developed user interfaces. Experimental results show that the system performs well in providing personalized preferences and the proposed features are effective for portraits ranking. From the user study, our system gets promising results.

This study explores the creation of emotion- responsive user interfaces (UIs) that adapt to users' mental states in stressful settings in real time. Utilizing human- computer interaction (HCI) principles and artificial intelligence, it introduces an innovative framework for UIs that adjusts based on fatigue, concentration, and emotional cues. The method incorporates sophisticated facial recognition, eye tracking, and physiological sensors to continuously evaluate users' cognitive and emotional conditions. The adaptive UI system uses machine learning algorithms to optimize cognitive load and improve remote interactions in various high-pressure situations. Through a series of experiments and user studies, notable enhancements in task performance, user satisfaction, and overall cognitive efficiency have been demonstrated. The results underscore the potential of emotion-responsive UIs to transform human-computer interaction, particularly in areas such as healthcare, emergency response, and mission-critical operations. Keywords — HCI, Emotion-Adaptive UI, Cognitive Efficiency, Real-Time Adaptation, Fatigue, Focus, Emotional Expression, Artificial Intelligence, Machine Learning, User Interface, High- Stress Environments, Remote Interaction.

Nowadays several Computer-Aided Software Engineering environments exploit Model-Driven Engineering (MDE) techniques in order to generate a single user interface for a given computing platform or multi-platform user interfaces for several computing platforms simultaneously. Therefore, there is a need to assess the usability of those generated user interfaces, either taken in isolation or compared to each other. This paper describes an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that will be subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) has been examined in terms of satisfaction, effectiveness and efficiency. An experiment with a factorial design for repeated measures was conducted for 31 participants, i.e., postgraduate students and professors selected by convenience sampling. The data were collected with the help of questionnaires and forms and were analyzed using parametric and non-parametric tests such as ANOVA with repeated measures and Friedman's test, respectively. Efficiency was significantly better in large screens than in small ones as well as in the desktop platform rather than in the web platform, with a confidence level of 95%. The experiment also suggests that satisfaction tends to be better in standard size screens than in small ones. The results suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

XML Based Graphical User Interface Editor and Runtime Parser for ISO 11783 Machine Automation Systems

Quickly accessing the contents of a video is challenging for users, particularly for unstructured video, which contains no intentional shot boundaries, no chapters, and no apparent edited format. We approach this problem in the domain of lecture videos though the use of machine learning, to gather semantic information about the videos; and through user interface design, to enable users to fully utilize this new information. First, we use machine learning techniques to gather the semantic information. We develop a system for rapid automatic semantic tagging using a heuristic-based feature selection algorithm called Sort-Merge, by using large initial heterogeneous low-level feature sets (cardinality greater than 1K). We explore applying Sort-Merge to heterogeneous feature sets though two methods: early fusion and late fusion. Each takes different approaches to handling the different kinds of features in the heterogeneous set. We determine the most predictive feature sets for key-frame filters such as “has text”, “has computer source code”, or “has instructor motion”. Specifically we explore the usefulness of Harr Wavelets, Fast Fourier Transforms, Color Coherence Vectors, Line Detectors, Ink Features and Pan/Tilt/Zoom detectors. For evaluation, we introduce a “keeper” heuristic for feature sets, which provides a method of performance comparison against a baseline. Second, we create a user interface to allow the user to make use of the semantic tags we gathered though our computer vision and machine learning process. The interface is integrated into an existing video browser, which detected shot-like boundaries and presented a multitimeline view. The content within shot-like boundaries is represented by frames to which our new interface applies the generated semantic tags. Specifically, we make accessible the semantic concepts of 'text', 'code', 'presenter', and 'person motion.' The tags are detected in the simulated shots using the filters generated with our machine learning approach and are displayed to users using a user-customizable multi-timeline view. We also generate tags based on ASRgenerated transcripts that have been limited to the words provided in the index of the course text book. Each of these occurrences is aligned with the simulated shots. Each spoken word becomes a tag analogous to the visual concepts. A full Boolean algebra over the tags is provided to enable new composite tags such as 'text or code, but no presenter.' Finally, we quantify the effectiveness of our features and our browser through user studies, both observational and task driven. We find that users that use the full suite of tools performed a search task in 60% of the time of users without access to tags. We find that when users are asked to perform search tasks they follow a nearly fixed pattern of accesses, alternating between the use of tags and Keyframes, or between the use of Word Bubbles and the media player. Based on user behavior and feedback, we redesigned the interface to group spatially interface components that are used together, removed un-used components, and redesigned the display of Word Bubbles to match that of the Visual Tags. We found that users strongly preferred the Keyframe tool, as well as both kinds of tags. Users also either found the algebra very useful or not useful at all.

A novel software application, Blip, was created to combine and display diabetes data from multiple devices in a uniform, user-friendly manner. The objective of this study was to test the usability of this application by adults and caregivers of children with type 1 diabetes (T1D). Patients (n = 35) and caregivers of children with T1D (n = 30) using an insulin pump for >1 year ± CGM were given access to the software for 3 months. Diabetes management practices and the use of diabetes data were assessed at baseline and at study end, and feedback was gathered in a concluding questionnaire. At baseline, 97% of participants agreed it was important for patients to know how to interpret glucose data. Most felt that clinicians and patients should share the tasks of reviewing data, finding patterns, and making changes to their insulin plans. However, despite valuing shared responsibility, at baseline, 43% of participants never downloaded pump data, and only 9% did so at least once per month. At study end, 72% downloaded data at least once during the 3-month study, and 38% downloaded at least once per month. Regarding the software application, participants liked the central repository of data and the user interface. Suggestions included providing tools for understanding and interpreting glucose patterns, an easier uploading process, and access with mobile devices. Collaboration between developers and researchers prompted iterative, rapid development of data visualization software and improvements in the uploading process and user interface, which facilitates clinical integration and future clinical studies.

We present a new user interface for the common morphing tool found in animation packages. Previously this interface has been based on the features of the underlying algorithm, with little regard to how artists actually use this feature. By careful design and analysis of a user study, we were able to design a novel user interface that greatly enhances the usability of the morphing tool for animation. Our improvements come in three areas: First, we replicate the artists' own ad-hoc annotation language and interaction techniques in the user interface. Second, we make the user experience more fluid and editable, to support exploration and iteration. Finally, we use the artists' morph expectations to redesign the morph algorithm itself to be more predictable. We conclude by discussing how our user study technique could help other interface design tasks.

An exploration of techniques for developing intuitive, and efficient user interfaces for virtual reality systems. Work seeks to understand which paradigms from the better-understood world of 2D user interfaces remain viable within 3D environments. In order to establish this a new user interface was created that applied various understood principles of interface design. A user study was then performed where it was compared with an earlier interface for a series of medical visualization tasks.

Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descriptions are constantly missing in mobile UIs. We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces. We collected a large-scale dataset for widget captioning with crowdsourcing. Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.

User interfaces are important for streamlining the interactions between humans and computers. However, there are few effective approaches for collecting users’ preferences implicitly and objectively for the purpose of user interface (UI) design optimization. This paper presents an effective approach to interactive genetic algorithm (IGA) optimization-based UI design via eye tracking, including eye-movement data based users’ preference inferring, gene coding for real UI components and design features, and the visualization and interaction mechanisms. Then we design and build a prototype system that applies IGA to generate and optimize music player UI solutions automatically. An evaluation of our prototype system suggests that it can generate and identify personalized UI designs reliably with minimal user intervention and high efficiency and user satisfaction.

Human-aware robot navigation is an essential aspect to increase the acceptance of mobile service robots in human-centered environments, e.g. home environments. Robots need to navigate in a human-acceptable way according to the users’ conventions, presence and needs. In order to address the users’ needs, we employ virtual borders, which are non-physical borders and respected by the robots while working, to effectively restrict the workspace of a mobile robot and change its navigational behavior. To this end, we consider different user interfaces, i.e. visual markers, a laser pointer, a graphical user interface and a RGB-D Google Tango tablet with augmented reality application, to allow non-expert users the flexible and interactive definition of virtual borders. These user interfaces were evaluated with respect to their correctness, flexibility, accuracy, teaching effort and user experience. Experimental results show that the RGB-D Google Tango tablet as user interface yields the best overall results compared to the other user interfaces. Apart from a low teaching effort and high flexibility and accuracy, it features the highest user ratings acquired from a comprehensive user study with 25 participants for intuitiveness, comfort, learnability and its feedback system.