Axis- plus Content-based Control for Camera Drones

Controlling assistive robots can be challenging for some users, especially those lacking relevant experience. Augmented Reality (AR) User Interfaces (UIs) have the potential to facilitate this task. Although extensive research regarding legged manipulators exists, comparatively little is on their UIs. Most existing UIs leverage traditional control interfaces such as joysticks, Hand-Held (HH) controllers and 2D UIs. These interfaces not only risk being unintuitive, thus discouraging interaction with the robot partner, but also draw the operator’s focus away from the task and towards the UI. This shift in attention raises additional safety concerns, particularly in potentially hazardous environments where legged manipulators are frequently deployed. Moreover, traditional interfaces limit the operators’ availability to use their hands for other tasks. Towards overcoming these limitations, in this article, we provide a user study comparing an AR Head-Mounted Display (HMD) UI we developed for controlling a legged manipulator against off-the-shelf control methods for such robots. This user study involved 27 participants and 135 trials, from which we gathered over 405 completed questionnaires. These trials involved multiple navigation and manipulation tasks with varying difficulty levels using a Boston Dynamics’s Spot, a 7 df Kinova robot arm and a Robotiq 2F-85 gripper that we integrated into a legged manipulator. We made the comparison between UIs across multiple dimensions relevant to a successful human–robot interaction. These dimensions include cognitive workload, technology acceptance, fluency, system usability, immersion and trust. Our study employed a factorial experimental design with participants undergoing five different conditions, generating longitudinal data. Due to potential unknown distributions and outliers in such data, using parametric methods for its analysis is questionable, and while non-parametric alternatives exist, they may lead to reduced statistical power. Therefore, to analyse the data that resulted from our experiment, we chose Bayesian data analysis as an effective alternative to address these limitations. Our results show that AR UIs can outpace HH-based control methods and reduce the cognitive requirements when designers include hands-free interactions and cognitive offloading principles into the UI. Furthermore, the use of the AR UI together with our cognitive offloading feature resulted in higher usability scores and significantly higher fluency and Technology Acceptance Model scores. Regarding immersion, our results revealed that the response values for the AR Immersion questionnaire associated with the AR UI are significantly higher than those associated with the HH UI, regardless of the main interaction method with the former, i.e., hand gestures or cognitive offloading. Derived from the participants’ qualitative answers, we believe this is due to a combination of factors, of which the most important is the free use of the hands when using the HMD, as well as the ability to see the real environment without the need to divert their attention to the UI. Regarding trust, our findings did not display discernible differences in reported trust scores across UI options. However, during the manipulation phase of our user study, where participants were given the choice to select their preferred UI, they consistently reported higher levels of trust compared to the navigation category. Moreover, there was a drastic change in the percentage of participants that selected the AR UI for completing this manipulation stage after incorporating the cognitive offloading feature. Thus, trust seems to have mediated the use and non-use of the UIs in a dimension different from the ones considered in our study, i.e., delegation and reliance. Therefore, our AR HMD UI for the control of legged manipulators was found to improve human–robot interaction across several relevant dimensions, underscoring the critical role of UI design in the effective and trustworthy utilisation of robotic systems.

Situational Awareness (SAW) is a concept widely used in areas that require critical decision making, such as in the field of emergency management. SAW is related to the level of perception and understanding that an individual has about real events occurring in complex scenarios, which must be managed by critical systems. Such critical systems require specialized user interfaces (UI) to give operators a dynamic understanding of what is happening in the environment. A challenging issue in the design of SAW-oriented interfaces is to determine how the human-computer interface process can be constructed for SAW enrichment, considering environments with heterogeneous data sources, limitations in data quality, and ever-changing situations. The problem increases when information is subject to uncertainties, which may compromise the situation analysis process. In addition, humans make decisions based on their own understanding of the event, which allied with experience and knowledge can be valuable assets to be used to process situational information about emergencies for the acquisition of SAW. The objective of this work is to demonstrate how to include a SAI-oriented UI in the process of evaluating emergency situations and to present the development of a UI that promotes the management of situational information of emergencies to promote the acquisition of SAW. The results present a specified routine for employing specialized UIs in SAW as part of a situation assessment process, which supports a strong integration between the human operator and other phases of the process, such as quality assessment, data fusion and information visualization , As well as a prototype interface that meets the process. A case study with a critical scenario of a theft event is also presented to demonstrate the applicability of the proposed approach

The successful design of user interfaces of the synchronous eLearning systems depends on HD display and efficient switching of many windows. A new improved design is made by our team. With the consideration of the specific requirement of teaching and learning process, as well as the concept of user centered design, we combine all of the functionalities in a simple and organic manner to ensure the simplicity of the interface. We divide the functionalities into front channel and back channel categories. Based on the partition, a simple interface integrates them organically by using quick buttons in the front channel window for easy and rapid control on the showing or hiding of the back channel windows, thus ensure the operating efficiency, New messages arisen in the back channel windows will be displayed in the front channel windows by making the corresponding quick buttons twinkle and highlighted, thus ensure the effectiveness of the interface. User study has shown that this interface has a good user experience and usability.

Current authoring environments provide the possibility of developing user interfaces with limited adaptation capacities. The most widely adopted tools follow the responsive design approach and allow developers to obtain user interfaces that can adapt mainly to the screen size and orientation. We present a solution able to support development of user interfaces able to adapt to the various types of contextual events (that can be related to users, devices, environments, and social relationships), with the added possibility of distributing the user interface elements across multiple devices. The context-dependent behavior is modelled through trigger / action rules, and can even be applied to Web applications that were not originally designed to be context-aware. This paper describes the design and main features of the novel authoring environment and reports on a first user study.

This study explores the creation of emotion- responsive user interfaces (UIs) that adapt to users' mental states in stressful settings in real time. Utilizing human- computer interaction (HCI) principles and artificial intelligence, it introduces an innovative framework for UIs that adjusts based on fatigue, concentration, and emotional cues. The method incorporates sophisticated facial recognition, eye tracking, and physiological sensors to continuously evaluate users' cognitive and emotional conditions. The adaptive UI system uses machine learning algorithms to optimize cognitive load and improve remote interactions in various high-pressure situations. Through a series of experiments and user studies, notable enhancements in task performance, user satisfaction, and overall cognitive efficiency have been demonstrated. The results underscore the potential of emotion-responsive UIs to transform human-computer interaction, particularly in areas such as healthcare, emergency response, and mission-critical operations. Keywords — HCI, Emotion-Adaptive UI, Cognitive Efficiency, Real-Time Adaptation, Fatigue, Focus, Emotional Expression, Artificial Intelligence, Machine Learning, User Interface, High- Stress Environments, Remote Interaction.

Quickly accessing the contents of a video is challenging for users, particularly for unstructured video, which contains no intentional shot boundaries, no chapters, and no apparent edited format. We approach this problem in the domain of lecture videos though the use of machine learning, to gather semantic information about the videos; and through user interface design, to enable users to fully utilize this new information. First, we use machine learning techniques to gather the semantic information. We develop a system for rapid automatic semantic tagging using a heuristic-based feature selection algorithm called Sort-Merge, by using large initial heterogeneous low-level feature sets (cardinality greater than 1K). We explore applying Sort-Merge to heterogeneous feature sets though two methods: early fusion and late fusion. Each takes different approaches to handling the different kinds of features in the heterogeneous set. We determine the most predictive feature sets for key-frame filters such as “has text”, “has computer source code”, or “has instructor motion”. Specifically we explore the usefulness of Harr Wavelets, Fast Fourier Transforms, Color Coherence Vectors, Line Detectors, Ink Features and Pan/Tilt/Zoom detectors. For evaluation, we introduce a “keeper” heuristic for feature sets, which provides a method of performance comparison against a baseline. Second, we create a user interface to allow the user to make use of the semantic tags we gathered though our computer vision and machine learning process. The interface is integrated into an existing video browser, which detected shot-like boundaries and presented a multitimeline view. The content within shot-like boundaries is represented by frames to which our new interface applies the generated semantic tags. Specifically, we make accessible the semantic concepts of 'text', 'code', 'presenter', and 'person motion.' The tags are detected in the simulated shots using the filters generated with our machine learning approach and are displayed to users using a user-customizable multi-timeline view. We also generate tags based on ASRgenerated transcripts that have been limited to the words provided in the index of the course text book. Each of these occurrences is aligned with the simulated shots. Each spoken word becomes a tag analogous to the visual concepts. A full Boolean algebra over the tags is provided to enable new composite tags such as 'text or code, but no presenter.' Finally, we quantify the effectiveness of our features and our browser through user studies, both observational and task driven. We find that users that use the full suite of tools performed a search task in 60% of the time of users without access to tags. We find that when users are asked to perform search tasks they follow a nearly fixed pattern of accesses, alternating between the use of tags and Keyframes, or between the use of Word Bubbles and the media player. Based on user behavior and feedback, we redesigned the interface to group spatially interface components that are used together, removed un-used components, and redesigned the display of Word Bubbles to match that of the Visual Tags. We found that users strongly preferred the Keyframe tool, as well as both kinds of tags. Users also either found the algebra very useful or not useful at all.

The effectiveness of a computer application depends on, among other things, an efficient user interface. In order to visualize subsurface geologic phenomena using the Google Earth™ application, we initially employed the built-in Google Earth time slider. Dragging the slider's right thumb elevated a COLLADA model that initially loads at a subsurface altitude. However, the double-thumb feature of the time slide caused users some difficulties. It is not possible to turn off this feature when not required, so it can be misleading to users. Because of this and because of the need for more control, we transitioned from the stand-alone Google Earth application to the web-based Google Earth plug-in. To overcome some of the limitation for the existing user interface, such as the inability to make controls appear semitransparent, we designed and implemented a screen overlay using the plug-in's application programming interface. This approach opened new possibilities to build more customizable user interfaces. A demonstration of the approach and sample usage of JavaScript to create buttons, draggable images, slides, and slider controls is presented.

This paper presents a set of situation awareness (SA) requirements for an operator who monitors critical infrastructure (CI). The requirements consist of pieces of information that the operator needs in order to be successful in their work. The purpose of this research was to define a common requirement base that can be used when designing a CI monitoring system or a user interface to support SA. The requirements can also be used during system or user interface evaluation, and as a guide for what aspects to emphasize when training new CI monitoring operators. To create the SA requirements, goal-directed task analysis (GDTA) was conducted. For GDTA, nine interview sessions were held during the research. For a clear understanding of a CI monitoring operator's work, all interviewees were subject matter experts (SMEs) and had extensive experience in CI monitoring. Before the interviews, a day-long observation session was conducted to gather initial input for the GDTA goal hierarchy and the SA requirements. GDTA identified three goals an operator must achieve in order to be successful in their work, and they were used to define the final SA requirements. As a result, a hierarchy diagram was constructed that includes three goals: monitoring, analysis and internal communication, and external communication. The SA requirements for a CI monitoring operator include information regarding ongoing incidents in the environment and the state of systems and services in the operator's organization.

Experiment on sentiment embedded comparison interface

Due to the rapid advancements in the digital avionics systems, the aircraft operation is getting more automated. However, this does not completely exclude the operations carried out by the pilot via interaction with the automation through the User Interface (UI) (or more commonly known as ight deck instruments). At times, this interaction raises situational awareness issues to the pilot about the behavior of the automation. One such critical issue is mode confusion where a mismatch occurs between the behavior of the automation expected by the pilot and the actual behavior of the automation. This paper examines the mode confusion detection problem from the UI design perspective by taking a control-theoretic perspective on dynamical systems. The automation is modeled as a hybrid system and the pilot is modeled as an intent-based discrete-event system for the given UI which is modeled as a lter of ight information. By inferring and comparing the intents of automation and pilot using a host of information such as the aircraft’s continuous states and ight modes, and automation’s and pilot’s inputs, this paper attempts to validate the given UI via mode confusion detection. The causes responsible for the detected mode confusion are used to suggest improvements to the existing UI design, thereby improving the situational awareness of the pilot and aircraft safety. The framework is tested on two representative mode confusion incidents/accidents and the associated UIs are validated.

Graphs are often visualized using node-link representations: vertices are depicted as dots, edges are depicted as (poly)lines connecting two vertices. A directed edge running from vertex A to B is generally visualized using an arrow representation: a (poly)line with a triangular arrowhead at vertex B. Although this representation is intuitive, it is not guaranteed that a user is able to determine edge direction as quickly and unambiguously as possible; alternative representations that exhibit less occlusion and visual clutter might be better suited. To investigate this, we developed five additional directed-edge representations using combinations of shape and color. We performed a user study in which subjects performed different tasks on a collection of graphs using these representations and combinations thereof to investigate which representation is best in terms of speed and accuracy. We present our initial hypotheses, the outcome of the user studies, and recommendations regarding directed-edge visualization.

The user interface is an influential element in software applications. A well-designed user interface will potentially increase the usability of software applications. Therefore, user interface designers should deliberate when considering which approach and method to implement for designing user interfaces. Design thinking is currently a widely followed approach in user interface design practices. Hence, this study aimed to explore research trends and current practices of design thinking approach for user interface design. This study employed a systematic literature review following the Kitchenham method. This study found 39 articles deemed relevant to the design thinking approach. In general, our study found five common stages broadly mentioned in the articles, i.e., empathize, define, ideate, prototype, and test. The most widely practiced method during those five stages is interview, user persona, brainstorming, user interface, and usability testing. However, there is no consensus on what kind of stage(s) and which method(s) should be employed when following the design thinking approach for user interface design. Therefore, it will depend on the designer's decision in choosing which stage(s) and their related method(s) will be employed.

This paper presents a preliminary work of a simulation study for the evaluation of human-machine team (HMT) performance with assistance of a sentinel system under cyberattacks. Sentinel system is an intelligent agent of cyberattack-detection kit, whose user interface (UI) was designed and implemented as a proof-of-concept in the simulation. The goal for this sentinel system is to improve the pilots’ Situation Awareness (SA) of the system vulnerability and resiliency, such that the HMT performance can be improved against disruptive events. Based on the literature of cyberattacks on unmanned aerial vehicle (UAV) control systems, realistic mission operation and cyberattack scenarios were identified and implemented on a simulated UAV ground control station (GCS). In the follow-up experiment, up to twenty-four Air Force pilots will be instructed to supervise the UAVs under specific mission scenarios, and to respond for system recovery solutions generated on Sentinel UI after detection of cyberattacks. Understanding the interactive behaviors of the pilot and Sentinel under mission contexts and cyberattacks is expected to help improve HMT performance.

Suppose we are given an autonomous vehicle that has limitations, meaning that it may need to transfer control back to the human driver to guarantee safety in certain situations. This paper presents work on designing a user interface to assist this hand off by considering the effects of the expression of internal and external awareness. Internal awareness is the concept of knowing whether or not the system is confident in its ability to handle the current situation. External awareness is the concept of being able to identify the limitations as the car is driving in terms of situational anomalies. We conduct a user study to examine what information should be presented to the driver, as well as the effects of expressing these levels of awareness on the driver's situational awareness and trust in the automation. We found that expressing uncertainty about the autonomous system (internal awareness) had an adverse effect on driver experience and performance. However, by effectively conveying the automation's external awareness on the anomaly, improvements were found in the driver's situational awareness, increased trust in the system, and performance after the transfer of control.

In this paper, we present the design case for Gate Reminder, a family shared home appliance located at the front door area that represents informative messages, reminding users of things they need to take and know before leaving home. For this project, we built a working prototype and conducted a number of usability and user experience evaluations. In the paper we describe (1) why we chose reminder as our research topic (2) what we found from the early phases of user research for the Gate Reminder (3) what design requirements and decisions we have established from our user study (4) how we designed the working prototype based on our design decisions and (5) what we have learned from our user experience evaluation. The requirements for effective reminding, the usability challenges in ubicomp application, issues in the current prototype and future developments will be presented throughout the paper as well.