Adaptive click-and-cross

Every scientist and engineer dreams of reducing a problem to a few basic principles. Although our understanding of the human interface will continue to grow, it is unlikely that we will ever see a time when there is a simple, comprehensive model for the user interface. User interfaces become increasingly more diverse in their construction. Computer users, unless they are programmers, use languages designed for specific applications or languages based on interaction techniques, such as point-and-click, which require very little training to use. The interfaces commonly used now by computer users, the mouse, keyboard and screen, will be confined to those with desk jobs, and others will use a variety of input devices, from voice and pen to virtual reality interfaces. In the future, many of our interfaces will receive input through observing users rather than waiting for human action as in interfaces for tracking and teleconferencing.

This chapter will show you how to use and specialise UML diagrams for describing the user interfaces of a software system. In order to accomplish the description of user interfaces, the proposed technique considers three specialised UML diagrams called user-interaction, user-interface, and GUI-class diagrams, which will be built following a model-driven development (MDD) perspective. These diagrams can be seen as the UML-based UI models of the system. In addition, this chapter is concerned with code-generation to implement the user interfaces of the system by using GUI-class diagrams and user-interaction diagrams. A case study of an Internet book shopping system is introduced in this chapter to proof and illustrate the proposed user interaction and interface design technique.

Purpose State: An innovative mobile health system called iMHere (interactive Mobile Health and Rehabilitation) has been developed at the University of Pittsburgh to support self-care and adherence to self-care regimens for patients with chronic conditions. The goal of this study is to explore and to identify the accessibility needs and preferences of individuals with dexterity impairments when they use the iMHere system. Method: Participants were asked to perform tasks after a one-week field trial. The time for a participant to complete each task, the number of possible errors a participant made and the number of errors a participant was able to self-correct were recorded and analyzed. The Telehealth Usability Questionnaire was collected to rate participants' experiences. Results: Nine participants with various levels of dexterity abilities were included in the study. A statistically significant negative correlation was identified between participants' dexterity levels and their error ratios by using a Pearson product-moment correlation (r = −0.434, n = 36, p = 0.004). Approximately 51% of errors detected was self-corrected without any help, but other errors called for resolution from a researcher. Conclusions: Due to the diversity of participants' dexterity impairments, their needs and preferences differ one from another. Personalized design may be the key to approaching these challenges in improving accessibility. Implications for Rehabilitation An innovative mobile health system called iMHere has been developed to support self-care and adherence to self-care regimens as part of rehabilitation for patients with chronic conditions. Before populations with these limitations can harness the potential of mHealth, accessibility of mHealth has to be addressed to ensure its quality and value. Dexterity limitations are commonly associated with chronic disease, accidents or aging. The study presented here identified the potential issues and barriers to accessibility related to user interface components for persons with dexterity impairments. The findings should be of interest for those working with mobile health, accessibility, smartphone apps, wellness and self-care.

At the time of this paper, the advances in light field technology already offer 3D displays that immerse the users without the need for additional viewing devices. Despite the numerous advantages and attractive capabilities of such glasses-free 3D displays, their user interface methods are quite complicated and they are currently underwhelming when compared to conventional 2D displays, due to the fact that visual feedback can only be rendered sharply on the emission surface of light field displays. The sharp rendering of user interfaces is a necessity, as blur may hinder their fundamental functions. When it comes to 2D displays, many user interaction techniques and interfaces have been devised. Rendering a user interface on a 2D display could be done in various ways, such as rendering overlays on top of the rendered scene, or by using billboards. These are extensively used in modern video games. User interaction methods have proven their importance and added efficiency to virtual environments throughout the years. Due to their overall value and usefulness, interaction techniques develop immediately as new types of displays arise. With the recent advancements in visualization technologies, user interfaces have been redesigned for use in AR, VR and MR visualization. This includes on-screen augmentation, which enables interaction with visual content on the screen. Although light field displays contain immense potentials, only basic user interfaces have been devised thus far, including FOX (Focus Sliding surface), which grants users the option to scale and to rotate 3D objects. In this paper, we visualize the theater model on real light field displays and we test the different interactions by means of a monitor room. The theatre model is analogous to real-life theatres, where viewers may observe the theatrical presentation on the stage from various angles. The motivation to choose the theater model was the fact that light field visualization similarly allows multiple simultaneous viewers within its field of view, in which the content can be observed in an angle- dependent manner. Moreover, from the users’ perspective, the theater model is thus familiar and it provides high-quality visual feedback. Furthermore, theater stages encompass a lot of interactions, including rigging and flying systems, pulleys, rotating stages, lights, curtains etc. In order to test the different interaction methods on light field displays, a theater model depicting the virtual environment was implemented. Methods for rendering the monitor room and the results of the interactions are discussed in the paper, illustrated by images of the actual visualization on light field displays. It is shown that producing plausible results with no noticeable visual artifacts is challenging, yet possible. The scientific contributions of the paper also highlight the various novel user interfaces for future light field systems and services.

The ongoing utilization of computer technologies in all areas of life leads to the development of smart environments comprising numerous networked devices and resources. Interacting in and with such environments requires new interaction paradigms, abstracting from single interaction devices to utilize the environment as interaction space. Using a networked set of interaction resources allows supporting multiple modalities and new interaction techniques, but also requires the consideration of the set of devices and the adaptation to this set at runtime. While the generation of user interfaces based on UI models, although still challenging, has been widely researched, the runtime processing and delivery of the derivable user interfaces has gained less attention. Delivering distributed user interfaces while maintaining their interdependencies and keeping them synchronized is not a trivial problem. In this paper we present an approach to realize a runtime environment, capable of distributing user interfaces to a varying set of devices to support multimodal interaction based on a user interface model and the management of interaction resources.Keywordsmultimodal distributed user interfacesmodel-based user interfaceshuman-computer interactionsmart environments

The paper describes how 3D-based visualization and interaction techniques can be used in information retrieval user interfaces. It demonstrates that information retrieval systems based on state of the art retrieval models can be supplied with an intuitive interface functionality by applying the cone-tree metaphor for the visualization of content spaces. The natural ability of humans for spatial perception, orientation and spatial memories is outlined as an advantage in the process of perceiving information spaces by means of spatial metaphors. It is shown how the models, concepts and mechanisms of the retrieval system underlying its user interface can become more transparent and perceptible for the user at the interface level and how some of the cognitive costs of navigations in information spaces can be reduced. This goal is achieved by transforming document-term networks with two levels of abstraction into hierarchical and directed cone trees that use the spatial depth of 3D to achieve an easy perception of their topological structure. Finally the paper presents a 3D-based information system user interface which gives the user intuitive control over content-oriented search paths resulting in a query generation for the underlying automatic retrieval mechanisms. In order to improve the presentation of the query results as well, it is shown that 3D-based graphical metaphors can provide very intuitive ways of perceiving the relevance of the result in accordance to the query.

User interface adaptation has been proposed in recent years as a means to achieve personalized accessibility and usability of user interfaces. Related user interface architectures, as well as a variety of related development method and tools, have also been elaborated. Admittedly, however, despite the recognized validity of the approach, which has been proved in practice in a series of prototype systems, the wider adoption and uptake of user interface adaptation approaches are still limited. One of the identified obstacles is the complexity intrinsic in designing such interfaces and the need of radically revising the current user interface design practice to account for (a) the alternative designs required for adaptation, (b) the parameters involved in driving adaptations (i.e., selecting among alternatives at a given point during interaction), and (c) the logic of adaptation at runtime. This article proposes a twofold tool-based support strategy for user interface adaptation development, based on (a) an adaptation development toolkit and related widget library, which directly embeds lexical level adaptations into common interactive widgets, and (b) embedding such a library in a common integrated development environment, thus allowing designers to define and view alternative adaptations at design time and create adaptable user interfaces through traditional prototyping. The aforementioned approach has been implemented in the domain of adaptable applications for older users, producing tools that are currently in use in the development of a large suite of interactive applications in various domains. The approach presented in this article is claimed to be the first and so far unique supporting rapid prototyping of adaptable user interfaces, thus minimizing the divergence between typical development practices and user interface adaptation development.

Older adults benefit from information and communication technology solutions and services in the Ambient Assisted Living (AAL) domain. Unfortunately offered user interfaces for these ICT solutions often do not take the special needs, preferences and the physical and mental capabilities of older adults into account. The project AALuis focuses on solutions to increase accessibility, adaptability and usability of user interfaces in the AAL domain. The general idea is to build a layer which can be used for different platforms to connect different types of user interfaces on different I/O devices with various AAL services. The layer shall take the burden from the service developers to design a target group friendly user interface by providing mechanisms for an automatic user interface generation. First results have shown that the chosen approach is very powerful, but as well that the automatic user interface generation process needs optimization and additional semantic information to be able to build richer user interfaces.

LanguageGame is a tool for a non-professional computer user to make a new programming language. Playing with LanguageGame allows a user to make an original new programming language. In order to give an instruction to a computer, now we have many ways such as selecting menu, push button, drag icon or so. Recent researches of computer user interface have developed various methods of interacting with computer. Above all, with using WYSIWYG (What You See Is What You Get) way, user interface becomes very rich and useful for non-engineer users. But yet, the most fundamental way of having a dialog with computer is to input a sequence of command kind of programming language. Using WYSIWIG style has advantages when a user uses a computer instead of real paper. But to deal with more abstract or involved matter, using a programming language has more benefit because symbols are more stable and we can use powerful idea as a recursiveness or a reference with such language. The importance of programming language could never change however GUI will be more improved. There are various languages in the world, and there are many interesting aspects and tastes in each language. As the grammars of the programming language are very interesting, developing a language must be more fun. Of course making a language seemed to be difficult. And it was planned to develop a parser generator that common people can make their own language easily. A parser generator is one of the most basic software in computer. Historically, various programming languages have been developed. And to make developing their parser easily, the technique of automatic parser generation was found. Now the parser generator becomes established technology. LanguageGame is such a parser generator that has graphical user interface. Contrary to popular professional parser generator like yacc/lex in UNIX platform, it is intended to use non-professional computer users, students and children. Playing with LanguageGame, users can learn and build simple parser with pretty graphical interface, drag-and-drop and graphical syntax tree representation. In current implementation, LanguageGame is developed on Squeak - a free open environment of Smalltalk. Hereby, the user grammar with LanguageGame can use all objects like a sketch and sound in Squeak.

XML Based Graphical User Interface Editor and Runtime Parser for ISO 11783 Machine Automation Systems

Computer users who handle complex tasks like air traffic control (ATC) need to quickly detect updated information from multiple displays of graphical user interface. The objectives of this study are to investigate how much computer users can segment GUI display into distinctive objects within very short glances and whether human perceives complexity differently after different durations of exposure. Subjects in this empirical study were presented with 20 screenshots of web pages and software interfaces for different short durations (100ms, 500ms, 1000ms) and were asked to recall the visual objects and rate the complexity of the images. The results indicate that subjects can reliably recall 3-5 objects regardless of image complexity and exposure duration up to 1000ms. This result agrees with the magic number 4 of visual short-term memory (VSTM). Perceived complexity by subjects is consistent among the different exposure durations, and it is highly correlated with subjects' rating on the ease to segmentation as well as the image characteristics of density, layout, and color use.

In this paper we propose vrLib, a toolkit which aims to make 3D graphical user interface and interaction very easy to design in virtual environments. This project provides well known and new interaction techniques and ready to use widgets, which are totally hardware independent. The main contribution allows to elaborate complex 3D user interfaces in same way it is done in 2D. This toolkit provides adaptive interaction, which means interaction techniques can automatically adapt their behavior to 2D and 3D widgets. This short paper describes the vrLib architecture for construction of 3D interactive applications.

We present a new user interface for the common morphing tool found in animation packages. Previously this interface has been based on the features of the underlying algorithm, with little regard to how artists actually use this feature. By careful design and analysis of a user study, we were able to design a novel user interface that greatly enhances the usability of the morphing tool for animation. Our improvements come in three areas: First, we replicate the artists' own ad-hoc annotation language and interaction techniques in the user interface. Second, we make the user experience more fluid and editable, to support exploration and iteration. Finally, we use the artists' morph expectations to redesign the morph algorithm itself to be more predictable. We conclude by discussing how our user study technique could help other interface design tasks.

Modeling tools typically have their own interaction methods for combining virtual objects. For realistic composition in 3D space, many researchers from the fields of virtual and augmented reality have been trying to develop intuitive interactive techniques using novel interfaces. However, many modeling applications require a long learning time for novice users because of unmanageable interfaces. In this paper, we propose two-handed tangible augmented reality interaction techniques that provide an easy-to-learn and natural combination method using simple augmented blocks. We have designed a novel interface called the cubical user interface, which has two tangible cubes that are tracked by marker tracking. Using the interface, we suggest two types of interactions based on familiar metaphors from real object assembly. The first, the screw-driving method, recognizes the user’s rotation gestures and allows them to screw virtual objects together. The second, the block-assembly method, adds objects based on their direction and position relative to predefined structures. We evaluate the proposed methods in detail with a user experiment that compares the different methods.

Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descriptions are constantly missing in mobile UIs. We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces. We collected a large-scale dataset for widget captioning with crowdsourcing. Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.