Multimodal Interaction with Hypermedia Structures

The ongoing utilization of computer technologies in all areas of life leads to the development of smart environments comprising numerous networked devices and resources. Interacting in and with such environments requires new interaction paradigms, abstracting from single interaction devices to utilize the environment as interaction space. Using a networked set of interaction resources allows supporting multiple modalities and new interaction techniques, but also requires the consideration of the set of devices and the adaptation to this set at runtime. While the generation of user interfaces based on UI models, although still challenging, has been widely researched, the runtime processing and delivery of the derivable user interfaces has gained less attention. Delivering distributed user interfaces while maintaining their interdependencies and keeping them synchronized is not a trivial problem. In this paper we present an approach to realize a runtime environment, capable of distributing user interfaces to a varying set of devices to support multimodal interaction based on a user interface model and the management of interaction resources.Keywordsmultimodal distributed user interfacesmodel-based user interfaceshuman-computer interactionsmart environments

The strength and stiffness of a composite rockmass (intact material, defects, joints, bedding, etc.) are primary inputs for the engineering analysis of rock slopes and underground excavations. The Geological Strength Index (GSI) has served rock engineering for three decades as a rockmass assessment system based on the blockiness of the rockmass and discontinuity condition. GSI is used to factor the Hoek–Brown strength envelope for intact rock to represent jointed rockmasses in conventional equivalent-continuum numerical modelling. Modern numerical tools can represent networks of discrete (explicit) structure with assigned discontinuity properties. This provides an opportunity to compare explicit structural modelling with the conventional implicit equivalent-continuum approach. In this paper, explicit structural models are developed using elasto-plastic (constant-strength) constitutive models for intact rock and realistic parametric ranges for explicit structure properties. Explicit model results are compared to equivalent-continuum results, validating the classical implicit approach while also identifying key limitations. Explicit models with post-yield weakening of the intact and structural elements are then developed and compared to post-yield weakening implicit models that use post-yield dilation and empirical relationships between peak and residual GSI. The results provide guidance for practical modelling in strain-weakening rockmasses, including recommendations for post-yield dilation in the traditional implicit approach.

Software product line (SPL) is a software engineering paradigm for software development. A software product within a product line often has specific functionalities that are not common to all other products within the product line. Those specific functionalities are termed “variant features” in a product line. SPL paradigm involves the modeling of variant features. However, little work in SPL investigates and addresses the modeling of variant features specific to user interface (UI). Unified Modeling Language (UML) is the de facto modeling language for object-oriented software systems. It is known that UML needs better support in modeling UIs. Thus, much research developed UML extensions to improve UML support in modeling UIs. Yet little of this work is related to developing such extensions for modeling UIs for SPLs in which variant features specific to UI modeling must be addressed. This research develops a UML extension -Web User Interface Modeling Language (WUIML) to address these problems. WUIML defines elements for modeling variant features specific to user interfaces for Web-based SPLs. The model elements in WUIML extend from the metaclass and BasicActivity of the UML2.0 metamodel. WUIML integrates the modeling of variant features specific to user interfaces to UML. For example, in a Web-based patient registration software product line, member products targeting British users may use British date format in the user interface, while member products targeting United States users may use United States date format in the user interface. Thus, this is a variant feature for this product line. WUIML defines a model element, XOR, to represent such exclusive or conditions in a product line user interface model. WUIML would reduce SPL engineers’ efforts needed in UI development. To validate the WUIML research outcome, a case study was conducted. The results of this empirical study indicate that modeling UIs for Web-based SPLs using WUIML is more effective and efficient than using standard UML.

Nowadays, most of the research developed by industry and academia has led to several object-oriented methods that are highly adequate for the development of systems, but most of these methods do not include mechanisms (models) for identifying and specifying user needs and requirements, as well as testing and validating requirements with end users before, during and after development. These considerations are especially important in interactive systems, where the user interaction is very high and the user interface is a significant part of the system. As a result of this weakness, interactive systems developed using such methods can meet all technical requirements, be very robust, and yet be unusable by the end user. This problem explains a large part of the frequently-observed phenomenon whereby large numbers of change requests to modify the services of an application are made after its deployment. As a solution to this problem, we propose a method for integrating a user interface model into the software life-cycle, taking into account user needs and requirements. Also, this approach to the development process ensures the quality of the delivered applications from the end user's point of view.

User interfaces represent a major part in software products. Generally, it is difficult to document the content and the structure of the user interface. However, there is a need to save and preserve the overall state and structure of the user interface for later communication and modifications in requirements, design and implementation. The Graphical User Interface (GUI) storage format should be universal and can be transferred from one application to another (such as XML). GUI states are usually defined as the overall combination of properties of all the components or widgets of the GUI This paper suggests an alternative back end representation of user interfaces and their state from their original format within the applications. The user interface model is transformed to a schema saved in an XML file. The schema structure represents the user interface state. This state considers only the structure of the user interface and ignores controls' specific properties. XML format is widely used and accepted by many applications. It is the infrastructure language for web pages and database management systems. User interface documentation is useful for future evaluation and comparison, useful for stakeholders' communication. It is also for some GUI testing activities such as regression testing where we trigger testing if there is a change in the user interface.

This chapter presents the visual Formal Interaction Logic Language (FILL) for the description of executable user interface models. This modeling approach is based on an architecture called 3LA, which separates a user interface from its outward appearance as a set of interaction elements called physical representation, and its functional part called interaction logic, which connects the physical representation with the system to be controlled. The latter refers in general terms to a business model of an interactive system, where the interaction logic is further meant to describe dialog-specific aspects, e.g., the availability of widgets depending on the system state or other widgets. FILL is algorithmically transformed to reference nets, a special type of Petri nets, which makes it executable and equips FILL with formal semantics through the formal semantics of reference nets. FILL models are modularization which enables the description of multidevice user interface models as well as the reuse and exchange of parts of the model. For the creation and execution of FILL-based user interface models, the UIEditor tool is presented. It offers editors for modeling the physical representation and the modularized FILL-based interaction logic. It further implements the transformation algorithm for generating reference nets out of a given FILL model and finally the execution environment for the user interface model. The applicability of this modeling approach will be shown by means of a case study for the control of a simplified nuclear power plant. The chapter will conclude with a broader view to future challenges this approach is facing.

Portable devices come with different limitations in user interaction like limited display size, small keyboard, and different sorts of input and output capabilities. With the advance of speech recognition and speech synthesis technologies, their complementary use becomes attractive for mobile devices in order to implement real multimodal user interaction. However, current systems and formats do not sufficiently integrate advanced multimodal interactions. We introduce an advanced generic multimodal interaction and rendering system (MIRS) dedicated for mobile devices. MIRS incorporates efficient processing of XML specification languages for limited, mobile devices and comes with the XML-based dialog and interface specification language (DISL). DISL can be considered as an UIML subset, which is enhanced by the means of state-oriented dialog specifications. The dialog specification is based on ODSN (object oriented dialog specification notation), which has been introduced to define user interface control by means of interaction states with transition rules.

This paper discusses the development of Virtual Control Interface (VCI), a Virtual Reality (VR) system for simulating and exploring multimodal context-sensitive interactions between Augmented Reality (AR) glasses and Internet of Things (IoT) devices.Current trends in controlling IoT devices focus on using existing control interfaces such as smartphones. Parallel technological trends indicate that AR glasses are likely to become the next consumer terminal. Therefore, there is a need to explore a new paradigm for IoT control interfaces that leverages the benefits of AR.Towards developing future-proof AR-IoT interfaces, we observed the HCI community’s challenges in using existing hardware to explore the full potential and the limitations of future AR-IoT interfaces. That became the foundation for developing VCI.This paper provides an overview of the advantages and shortcomings of developing such interfaces on smartphones, state-of-the-art AR/Mixed Reality (MR) glasses, and our proposed VR simulated AR environment. This overview outlines how our proposed system could be a superior choice for researchers to investigate, challenge assumptions, prototype, develop, and experimentally test AR-IoT interactions and interfaces with currently available technologies.Furthermore, we present the three pillars we followed to guide the design and development of VCI to make it resemble an actual AR-IoT framework as much as possible while abstracting parts that have little to no effect on AR-IoT HCI research. These pillars are: design transferable architecture, use viable multimodal interactions and explore user interfaces in real-life scenarios.These three principles informed the VCI’s system architecture built upon three main subsystems: handling multi-modal user input, IoT device emulation and connectivity, and managing the AR user interface (UI).Finally, we provide a heuristic evaluation and future work for VCI.KeywordsSmart homeInternet of ThingsAugmented realityVirtual realityAR-IoTMultimodal interactionVirtual Control Interface

In this paper we present a solution to compose synchronized multimodal computer human interaction with a pattern-oriented approach. The focus of this paper is how to synchronize the different modalities of interaction.In our application scenario the system consists of mobile agents and the user interface is generated from a dialog description language, which is XML-encoded. The generated User Interface is executed by one or more UI-engines, which can use one or more modalities like a text chat system, a speech-based system or a graphical, window-oriented user interface. The user interface is separated and also structured by so-called dialog moves which are suitable to be adapted to different UI-engines. The system supports more than one UI-engine at the same time which results in a possibly multimodal interaction. The interaction is synchronized by observing the events produced by the dialog moves of each UI-engine.The system is evaluated in a project called MAP1 which deals with new human computer interaction methods in future mobile work environments.

This paper discusses the design and applications of an object-oriented user interface management system (UIMS). Specifically, the Lockheed User Interface System (LUIS) is described. LUIS is based on a user interface model that includes declarative and procedural components. The package has been used by both non-programmers and programmers in several applications at Lockheed. Experiences derived from applications of the package are used to address several key issues in the UIMS field, such as procedural versus declarative specification, separation of the user interface from applications, UIMS flexibility, and UIMS support for design evaluation.

An emergency can occur at anytime, anywhere and at anyone. The local government offers an application called “Panic Button” as a solution to these problems. This application aims to help people who experience or see emergency conditions so that they can be followed up by the authorities. However, this application has not been targeted for disability users. The procedure for using this application still has weaknesses that make it difficult for users with hearing loss, especially when an emergency event is confirmed. Normally, emergency events confirmation uses telephone calls to the incident reporter (user). This confirmation method is a big problem for those users with disabilities to be able to confirm the details of emergency events. In this research, a user interface re-design was carried out at the application interface to be able to accommodate the users necessary. The user interface modeling is done using the User-Centered Design (UCD) method to produce a user interface design that suits the user’s characteristics and limitations. USE Questionnaire was used to test the resulting prototype and results in a usability value of 87% with an excellent category after the third iteration. These results indicate that the results of the re-design that have been done have met the needs of the deaf, with the use of appropriate communication for people with hearing loss so that they can confirm emergencies properly.

Eye movements can carry a rich set of information about someone's intentions. In the case of physically impaired people gaze can be the only communication channel they can use. People with severe disabilities are usually assisted by helpers during everyday life activity, which in time can lead to a development of an effective visual communication protocol between helper and disabled. This protocol allows them to communicate at some extent only by glancing one towards the other. Starting from this premise, we propose a new model of attentive user interface featured with some of the visual comprehension abilities of a human helper. The purpose of this user interface is to be able to identify user's intentions, and so to assist him/her in the process of achieving simple interaction goals (i.e. object selection, task selection). Implementation of this attentive interface is accomplished by way of statistical analysis of user's gaze data, based on a hidden Markov model.

This paper describes a wearable multi-modal user interface design and its implementation for a teleoperated field robot system. Recently some teleoperated field robots are employed for hazard environment applications (e.g. rescue, explosive ordnance disposal, security). To complete these missions in outdoor environment, the robot system must have appropriate functions, accuracy and reliability. However, the more functions it has, the more difficulties occur in operation of the functions. To cope up with this problem, an effective user interface should be developed. Furthermore, the user interface is needed to be wearable for portability and prompt action. This research starts at the question: how to teleoperate the complicated slave robot easily. The main challenge is to make a simple and intuitive user interface with a wearable shape and size. This research provides multi-modalities such as visual, auditory and haptic sense. It enables an operator to control every functions of a field robot more intuitively. As a result, an EOD (explosive ordnance disposal) demonstration is conducted to verify the validity of the proposed wearable multi-modal user interface.

This paper describes the user interface and interaction model of a system for Personal Environment Control -- the PECo System.The Personal Environment Controller (PECo) is a digital assistant system that provides the user with a unified facility for organizing and accessing the different media repositories at his disposal and for direct manipulation of his physical environment (projectors, TV sets, hi-fi systems, Lights, shutters, etc.).

User interfaces play an increasingly important role in robot applications. This paper presents the design and implementation of a multi-modal user interface for a mobile manipulator, consisting of an autonomous vehicle with a manipulator on it. It is capable of navigating in a real world and doing useful manipulations. The system is intended to provide services for handicapped people. The user interface is crucial for such a user to benefit from the services. The user interface designed for the system makes advantage of multimedia technology and combines graphics, speech and visualization into a coherent multi-modal interface. It serves as a user command interpreter, robot monitor and simulator. A user-centered design approach is adopted for enhanced understandability and usability.