Multimodal User Interaction in Smart Environments: Delivering Distributed User Interfaces

Software product line (SPL) is a software engineering paradigm for software development. A software product within a product line often has specific functionalities that are not common to all other products within the product line. Those specific functionalities are termed “variant features” in a product line. SPL paradigm involves the modeling of variant features. However, little work in SPL investigates and addresses the modeling of variant features specific to user interface (UI). Unified Modeling Language (UML) is the de facto modeling language for object-oriented software systems. It is known that UML needs better support in modeling UIs. Thus, much research developed UML extensions to improve UML support in modeling UIs. Yet little of this work is related to developing such extensions for modeling UIs for SPLs in which variant features specific to UI modeling must be addressed. This research develops a UML extension -Web User Interface Modeling Language (WUIML) to address these problems. WUIML defines elements for modeling variant features specific to user interfaces for Web-based SPLs. The model elements in WUIML extend from the metaclass and BasicActivity of the UML2.0 metamodel. WUIML integrates the modeling of variant features specific to user interfaces to UML. For example, in a Web-based patient registration software product line, member products targeting British users may use British date format in the user interface, while member products targeting United States users may use United States date format in the user interface. Thus, this is a variant feature for this product line. WUIML defines a model element, XOR, to represent such exclusive or conditions in a product line user interface model. WUIML would reduce SPL engineers’ efforts needed in UI development. To validate the WUIML research outcome, a case study was conducted. The results of this empirical study indicate that modeling UIs for Web-based SPLs using WUIML is more effective and efficient than using standard UML.

Perspectives in human-media interaction We are now in the third decade of the 21st Century, and, especially in the last years, the achievements made by scientists have been exceptional, leading to major advancements in the fast-growing field of Human-Media Interaction. This article collection is aimed at promoting a critical reflection on the research conducted so far, as well as the future perspectives on all aspects of human interaction with computers and related media. This Research Topic has solicited perspective articles that describe the state of the art, outlining recent developments and major accomplishments that have been achieved and that need to occur to move the field forward. Authors have been encouraged to identify the greatest challenges in the sub-disciplines, and how to address those challenges. The goal of this special edition Research Topic is to shed light on the progress made in the past decade in the Human-Media Interaction field, and on its future challenges to provide a thorough overview of the field. This article collection is meant to inspire, inform, and provide direction and guidance to researchers in the field. Although not limited to perspective articles, original research and review papers were welcomed as well, we focussed on receiving articles that cover all aspects of human-media interaction. Therefore, topics of interest included but were not limited to human-computer interaction and interaction design, multi-modal and multisensory interaction, verbal and non-verbal behavior, affective and physiological computing, augmented and virtual reality, human-food interaction, embodied interaction, tangible interfaces and wearables, conversational agents, interaction in smart environments, and social media interaction. Many of these topics were covered by the articles submitted to this Research Topic. In total, 20 articles were submitted, and 12 articles, including desk rejects, were rejected. In this editorial, we discuss the articles that were accepted. We can distinguish between original research, perspective, and review articles. Two "Original Research" articles, four "Perspective" articles, and two "Review" articles were accepted. We introduce and summarize these contributions in the next section. In section 3 we have some concluding remarks on these articles and how they fit in the development of human-media interaction research.

Point & Click - Interaction in Smart Environments

Portable devices come with different limitations in user interaction like limited display size, small keyboard, and different sorts of input and output capabilities. With the advance of speech recognition and speech synthesis technologies, their complementary use becomes attractive for mobile devices in order to implement real multimodal user interaction. However, current systems and formats do not sufficiently integrate advanced multimodal interactions. We introduce an advanced generic multimodal interaction and rendering system (MIRS) dedicated for mobile devices. MIRS incorporates efficient processing of XML specification languages for limited, mobile devices and comes with the XML-based dialog and interface specification language (DISL). DISL can be considered as an UIML subset, which is enhanced by the means of state-oriented dialog specifications. The dialog specification is based on ODSN (object oriented dialog specification notation), which has been introduced to define user interface control by means of interaction states with transition rules.

User interfaces represent a major part in software products. Generally, it is difficult to document the content and the structure of the user interface. However, there is a need to save and preserve the overall state and structure of the user interface for later communication and modifications in requirements, design and implementation. The Graphical User Interface (GUI) storage format should be universal and can be transferred from one application to another (such as XML). GUI states are usually defined as the overall combination of properties of all the components or widgets of the GUI This paper suggests an alternative back end representation of user interfaces and their state from their original format within the applications. The user interface model is transformed to a schema saved in an XML file. The schema structure represents the user interface state. This state considers only the structure of the user interface and ignores controls' specific properties. XML format is widely used and accepted by many applications. It is the infrastructure language for web pages and database management systems. User interface documentation is useful for future evaluation and comparison, useful for stakeholders' communication. It is also for some GUI testing activities such as regression testing where we trigger testing if there is a change in the user interface.

A service robot should be able to detect and identify the user in order to personalize its services and guarantee security, it should recognize the user’s emotions to allow affective interaction, and it should be able to communicate easily with the user and understand given commands by recognizing speech and gestures. Our research is motivated, for example, by the emerging needs of elderly care, health care, safety, and logistics. We have developed a distributed system for affective human–robot interaction which combines all the basic sensory elements and can collaborate with a smart environment and obtain further knowledge from the Internet. The realized HRI robot, Minotaurus, runs in real time and is capable of interacting with human beings. Minotaurus forms a rather generic platform for experimenting with human–robot collaboration in different applications and environments.

The Graphical User Interface (GUI) – although developed in research laboratories in the late 1970s – is still the dominant interaction paradigm in Information Visualization. We propose a new interaction paradigm called Blended Interaction. It combines ideas of Embodied Cognition, Multimodal Interaction, Reality-Based Interaction & Ubiquitous Computing. This is intended to stress that a single increase in the reality aspect of the interaction cannot go far enough. The particular challenge – and from the user's standpoint, the key advantage – lies in a meaningful marriage between the tested real-world options and the digital world. As a minimum this marriage must exist on the levels of the interaction, communication, of the way we solve problems with conventional tools (workflows), and of the design of the space or the architecture of buildings and places. The digital world often offers entirely new possibilities and takes the form of interactive devices of various shapes but also of intelligent everyday objects (e.g. the 'Internet of things'). In our view, interaction concepts can indeed offer a new quality of interaction, but only when the design of the interaction includes all these domains at the same time and with equal weighting. We test the suitability of our ideas of Blended Interaction concepts by using specific application examples that are being worked on as part of current research projects. Our experiences show that this new interaction paradigm has also great potential for interacting with visualization. For example, we have developed multi-touch scatter plots & facet maps for tangible user interfaces supporting the searching & browsing in Digital Libraries. We have embedded different visualizations into a Zoomable Object-oriented Information Landscape (ZOIL), which supports our vision of using visualizations on different displays of different size at the same time. We have developed specific kind of search tokens that supports collaborative search activities. For example, we try to address the following research questions: * How can future interactive InfoVis tools look like, especially in the light of the idea Blended Interaction? * How can future interactive InfoVis tools benefit from Multi-Displays & Multimodal environments used by Multiple Users? * What are the specific design requirements for multi-touch visualizations? * How can we support the collaborative use visualization tools?

Hobbit, a care robot supporting independent living at home: First prototype and lessons learned

This chapter presents the visual Formal Interaction Logic Language (FILL) for the description of executable user interface models. This modeling approach is based on an architecture called 3LA, which separates a user interface from its outward appearance as a set of interaction elements called physical representation, and its functional part called interaction logic, which connects the physical representation with the system to be controlled. The latter refers in general terms to a business model of an interactive system, where the interaction logic is further meant to describe dialog-specific aspects, e.g., the availability of widgets depending on the system state or other widgets. FILL is algorithmically transformed to reference nets, a special type of Petri nets, which makes it executable and equips FILL with formal semantics through the formal semantics of reference nets. FILL models are modularization which enables the description of multidevice user interface models as well as the reuse and exchange of parts of the model. For the creation and execution of FILL-based user interface models, the UIEditor tool is presented. It offers editors for modeling the physical representation and the modularized FILL-based interaction logic. It further implements the transformation algorithm for generating reference nets out of a given FILL model and finally the execution environment for the user interface model. The applicability of this modeling approach will be shown by means of a case study for the control of a simplified nuclear power plant. The chapter will conclude with a broader view to future challenges this approach is facing.

It is now a common belief that the truths of our lives are hidden in the databases streamed from our interactions in smart environments. In this current hype of big data, the Internet of Things has been suggested as the idea to embed small sensors and actuators everywhere to unfold the truths beneath the surfaces of everything. However, remaining the technology that promises more than it can provide thus far, more important for the IoT’s actual expansion to various social domains than the actual discovery of hidden truths has been people’s speculations about the unknown problems, such as hidden security issues or lifestyle concerns, beyond the narrow human knowability but assumed to leave their traces in the IoT-collected big data. This paper discuss this speculation as the concealed cognitive labor of IoT users that projects some fictitious values to the big data IoT companies accumulate. By the term pan-kinetics, the systemic operation of smart actuators is analyzed as the process through which fictitious values of data are converted to the real values as these actuators draw some profitable correlations from physical domains of the IoT. Analyzing smart electroencephalogram (EEG) headsets as the unique IoT devices operating on human brains, it argues how the IoT translates this speculative realism of unknown problems into its big data, which the IoT developers believe to be full of machine-learnable correlations that would lead to the smart solutions of the problems.

The automatic monitoring of specific Activities of Daily Living (ADL) can be a useful tool for Human-Robot Interaction in smart environments and Assistive Robotics applications. The qualitative definition that is given for most ADL and the lack of well-defined benchmarks, however, are obstacles toward the identification of the most effective monitoring approaches for different tasks. The contribution of the article is two-fold: (i) we propose a taxonomy of ADL allowing for their categorization with respect to the most suitable monitoring approach; (ii) we present a freely available dataset of acceleration data, coming from a wrist-worn wearable device, targeting the recognition of 14 different human activities.

This paper presents ArmNet, a mobile sensing system for capturing the trajectory of the wrist using measurements from wrist-worn IMU devices, specifically targeting robust interaction within Internet of Things (IoT) ecosystems. Unlike existing solutions that directly map IMU data to joint positions, ArmNet integrates physical kinematic constraints with neural network modeling. This hybrid approach is crucial for resource-constrained IoT devices where computational overhead and sensor limitations are primary concerns. Specifically, kinematic priors capture spatial dependencies between the elbow and wrist, generating physics-guided intermediate features that reduce the solution space. These features, together with raw IMU data, are fed into a recurrent neural network to learn joint displacement vectors, which are then integrated into continuous trajectories. Extensive experiments show that ArmNet achieves robust and accurate arm tracking, generalizing well across users and motion patterns, thereby enabling a new modality for seamless human-computer interaction in smart environments.

Multimodal interfaces provide users with a number of different ways of interacting with a system. This paper proposes the integration of an explicit interaction model as an extension of the well-known MVC architectural pattern. It builds upon previous extensions regarding explicit structure, user interface (UI), and environmental models. In addition, this paper proposes an explicit UI layer in a service-oriented hypermedia infrastructure to address requirements from multimodal interfaces.

The way we communicate with speech goes far beyond the words we use and nonverbal cues play a pivotal role in, e.g., conveying emphasis or expressing emotion. Furthermore, speech can also serve as a biomarker for a range of health conditions, e.g., Alzheimer’s. With a strong evolution of speech technologies, in recent years, speech has been increasingly adopted for interaction with machines and environments, e.g., our homes. While strong advances are being made in capturing the different verbal and nonverbal aspects of speech, the resulting features are often made available in standalone applications and/or for very specific scenarios. Given their potential to inform adaptability and support eHealth, it’s desirable to increase their consideration as an integral part of interactive ecosystems taking profit of the rising role of speech as an ubiquitous form of interaction. In this regard, our aim is to propose how this integration can be performed in a modular and expandable manner. To this end, this work presents a first reflection on how these different dimensions may be considered in the scope of a smart environment, through a seamless and expandable integration around speech as an interaction modality by proposing a first iteration of an architecture to support this vision and a first implementation to show its feasibility and potential.

This paper discusses the design and applications of an object-oriented user interface management system (UIMS). Specifically, the Lockheed User Interface System (LUIS) is described. LUIS is based on a user interface model that includes declarative and procedural components. The package has been used by both non-programmers and programmers in several applications at Lockheed. Experiences derived from applications of the package are used to address several key issues in the UIMS field, such as procedural versus declarative specification, separation of the user interface from applications, UIMS flexibility, and UIMS support for design evaluation.