Optimizing the Utility Function-Based Self-adaptive Behavior of Context-Aware Systems Using User Feedback

Computing and communication technology is widely used and integrated in devices, environments, and everyday objects. Even with major advances in technology the vision of ubiquitous computing - from a user perspective - is not yet achieved. In this paper we look at new forms of interaction that will help to interact with the ubiquitous computer. In particular we introduce the concept of embedded interaction, and implicit use. The focus of the research is on embedding information into people's environments. Currently massive amounts of information are available. However, delivering it to the user in a way that is pleasant and not annoying is still a challenge. Observing mobile phone information push services, it appears that endless information is available; however, much of the information is interesting only in a very specific context of use. We investigate how information can be provided to users - exactly when and where it is needed. Our approach is based on a variety of information displays unobtrusively embedded into the user's everyday environment. We place the information displays in context. In contrast to the traditional approach on context-awareness where a context is recognized and then the appropriate information is delivered, we look at providing information already in context. It is up to the user to make use of the provided information or not.

A particular vision of ubiquitous computing is offered to contribute to the burgeoning, dominant interaction paradigm in human-computer interaction (HCI). An engaged vision of ubiquitous computing (UbiComp) can take advantage of natural human abilities and tendencies for interaction. The HCI literature is reviewed to provide a brief overview of promising interaction styles and paradigms in order to situate them within ubiquitous computing. Embodied interaction is introduced as a key theoretical framework for moving UbiComp forward as an engaged interaction paradigm.Keywordsubiquitous computingHCIembodied interactiontangible interaction

In the vision of ubiquitous computing, there are billions of context sources that continuously publish their contextual information, and even more user agents that search for and consume such information. These context producers and consumers form a large-scale, wide-area, context-aware network which is both dynamic and heterogeneous. Existing service discovery mechanisms do not address the unique challenges imposed by such environments. In this paper we identify these challenges and propose a large scale context service discovery infrastructure for context-aware computing. We describe an approach for context service discovery and description that addresses these challenges.

The vision of ubiquitous computing is a network of small, inexpensive, robust processing devices, distributed throughout everyday life. Wireless Sensor Networks (WSNs) are an essential part to realize this dream. Sensor Networks can be seen as sources of input, delivering data from the real world in

Weiser's vision of Ubiquitous Computing has inspired many researchers and an active research community has been established in the past years. Various aspects have been investigated from novel user interfaces, applications to specialized hardware devices and system software helping application developers utilizing the capabilities of a smart world. In my talk I want to address the challenges researchers have addressed in the past. Based on a number of products and prototypes I will identify the problems where concepts and solutions exist and will also highlight open research challenges with a focus on adaptation and self-organization.

Mobile applications experience context changes during their execution, moreover their success depends on the user satisfaction that is strictly related to application quality. However, context changes may affect system quality and therefore user satisfaction, hence mobile applications may greatly benefit from context awareness that allows the application to adapt to context changes. In this paper we propose a context aware approach for mobile applications that targets both software quality and user satisfaction. In particular, we focus on service availability and develop a methodology that, leveraging on a novel reward formula that contains the notion of software availability and users feedback, determines the best application adaptation to be applied in a given context. We show the devised methodology on a motivating example.

To realize the vision of ubiquitous computing, it is important to develop a context-aware system which can help ubiquitous agents, services, and devices become aware of their contexts. Context-aware system is required to be capable of configuring appropriate context model and managing it through the context manipulation. In this paper, we propose the context space and the conceptual structure for context acquisition, representation, and utilization. We also describe how this conceptual structure is implemented in the service robot area. As the result of the research, we developed the CAMUS (Context-Aware Middleware for URC Systems) which is a context-aware server framework for a network-based intelligent robot system and applied it to the field test.

Quickly accessing the contents of a video is challenging for users, particularly for unstructured video, which contains no intentional shot boundaries, no chapters, and no apparent edited format. We approach this problem in the domain of lecture videos though the use of machine learning, to gather semantic information about the videos; and through user interface design, to enable users to fully utilize this new information. First, we use machine learning techniques to gather the semantic information. We develop a system for rapid automatic semantic tagging using a heuristic-based feature selection algorithm called Sort-Merge, by using large initial heterogeneous low-level feature sets (cardinality greater than 1K). We explore applying Sort-Merge to heterogeneous feature sets though two methods: early fusion and late fusion. Each takes different approaches to handling the different kinds of features in the heterogeneous set. We determine the most predictive feature sets for key-frame filters such as “has text”, “has computer source code”, or “has instructor motion”. Specifically we explore the usefulness of Harr Wavelets, Fast Fourier Transforms, Color Coherence Vectors, Line Detectors, Ink Features and Pan/Tilt/Zoom detectors. For evaluation, we introduce a “keeper” heuristic for feature sets, which provides a method of performance comparison against a baseline. Second, we create a user interface to allow the user to make use of the semantic tags we gathered though our computer vision and machine learning process. The interface is integrated into an existing video browser, which detected shot-like boundaries and presented a multitimeline view. The content within shot-like boundaries is represented by frames to which our new interface applies the generated semantic tags. Specifically, we make accessible the semantic concepts of 'text', 'code', 'presenter', and 'person motion.' The tags are detected in the simulated shots using the filters generated with our machine learning approach and are displayed to users using a user-customizable multi-timeline view. We also generate tags based on ASRgenerated transcripts that have been limited to the words provided in the index of the course text book. Each of these occurrences is aligned with the simulated shots. Each spoken word becomes a tag analogous to the visual concepts. A full Boolean algebra over the tags is provided to enable new composite tags such as 'text or code, but no presenter.' Finally, we quantify the effectiveness of our features and our browser through user studies, both observational and task driven. We find that users that use the full suite of tools performed a search task in 60% of the time of users without access to tags. We find that when users are asked to perform search tasks they follow a nearly fixed pattern of accesses, alternating between the use of tags and Keyframes, or between the use of Word Bubbles and the media player. Based on user behavior and feedback, we redesigned the interface to group spatially interface components that are used together, removed un-used components, and redesigned the display of Word Bubbles to match that of the Visual Tags. We found that users strongly preferred the Keyframe tool, as well as both kinds of tags. Users also either found the algebra very useful or not useful at all.

Adapting intelligent context-aware systems (CAS) to future operating rooms (OR) aims to improve situational awareness and provide surgical decision support systems to medical teams. CAS analyzes data streams from available devices during surgery and communicates real-time knowledge to clinicians. Indeed, recent advances in computer vision and machine learning, particularly deep learning, paved the way for extensive research to develop CAS. In this work, a deep learning approach for analyzing laparoscopic videos for surgical phase recognition, tool classification, and weakly-supervised tool localization in laparoscopic videos was proposed. The ResNet-50 convolutional neural network (CNN) architecture was adapted by adding attention modules and fusing features from multiple stages to generate better-focused, generalized, and well-representative features. Then, a multi-map convolutional layer followed by tool-wise and spatial pooling operations was utilized to perform tool localization and generate tool presence confidences. Finally, the long short-term memory (LSTM) network was employed to model temporal information and perform tool classification and phase recognition. The proposed approach was evaluated on the Cholec80 dataset. The experimental results (i.e., 88.5% and 89.0% mean precision and recall for phase recognition, respectively, 95.6% mean average precision for tool presence detection, and a 70.1% F1-score for tool localization) demonstrated the ability of the model to learn discriminative features for all tasks. The performances revealed the importance of integrating attention modules and multi-stage feature fusion for more robust and precise detection of surgical phases and tools.

Understanding the work context of future system users is essential in requirements engineering. It is particularly crucial when developing ubiquitous systems that react on context changes. This paper discusses the need for in-situ requirements elicitation approaches to build mobile and context-aware systems. We identify three different levels of support: The first level covers contextual techniques without tool support. Second level support is based on existing RE approaches and mobile tools. Third level support utilizes context-aware tools receiving context-specific information to guide analysts in the field. These tools enhance requirements gathering for ubiquitous systems. We present a context-aware tool prototype for on-site scenario walkthroughs and discuss how the underlying scenario-based approach needs to be adapted. Our tool-based approach was tested in an initial evaluation study. Finally, the paper presents requirements for RE approaches supporting ubiquitous system development based on lessons learned from using level II and III tools.

Configuration files, command-line arguments and environment variables are the dominant tools for local configuration management today. When accessing such program execution environments, however, most applications do not take context, e.g. the system they run on, into account. The aim of this paper is to integrate unmodified applications into a coherent and context-aware system by instrumenting the getenv API. We propose a global database stored in configuration files that includes specifications for contextual interpretations and a novel matching algorithm. In a case study we analyze a complete Debian operating system where every getenv API call is intercepted. We evaluate usage patterns of 16 real-world applications and systems and report on limitations of unforeseen context changes. The results show that getenv is used extensively for variability. The tool has acceptable overhead and improves context-awareness of many applications.

QoS contract preservation through dynamic reconfiguration: A formal semantics approach

Modeling is essential to integrating knowledge of human physiology. Comprehensive self-consistent descriptions expressed in quantitative mathematical form define working hypotheses in testable and reproducible form, and though such models are always "wrong" in the sense of being incomplete or partly incorrect, they provide a means of understanding a system and improving that understanding. Physiological systems, and models of them, encompass different levels of complexity. The lowest levels concern gene signaling and the regulation of transcription and translation, then biophysical and biochemical events at the protein level, and extend through the levels of cells, tissues and organs all the way to descriptions of integrated systems behavior. The highest levels of organization represent the dynamically varying interactions of billions of cells. Models of such systems are necessarily simplified to minimize computation and to emphasize the key factors defining system behavior; different model forms are thus often used to represent a system in different ways. Each simplification of lower level complicated function reduces the range of accurate operability at the higher level model, reducing robustness, the ability to respond correctly to dynamic changes in conditions. When conditions change so that the complexity reduction has resulted in the solution departing from the range of validity, detecting the deviation is critical, and requires special methods to enforce adapting the model formulation to alternative reduced-form modules or decomposing the reduced-form aggregates to the more detailed lower level modules to maintain appropriate behavior. The processes of error recognition, and of mapping between different levels of model complexity and shifting the levels of complexity of models in response to changing conditions, are essential for adaptive modeling and computer simulation of large-scale systems in reasonable time.

Two limitations of the current implementations of adaptive QoS behaviors are complexity associated with inserting them into common application contexts and lack of reusability across applications. What is needed is a way of bundling all the specification for systemic behavior into one place, and making it feasible to insert the collection of related QoS management artifacts into applications as a single, reusable behavior. As a means of addressing these issues, we have developed a new abstraction intended specifically to bundle together the lower level abstractions, specifications and implementations associated with providing adaptive QoS behaviors in a manner which is both easier to package and makes the behavior bundle reusable in a variety of applications and styles of use. We introduce the concept of "qoskets" as reusable systemic behaviors, lay out elements of our design for qoskets within the QuO framework, and use a case study example to highlight the issues of using qoskets in practice.

One of the fundamental problems of computer vision is to extract 3D shape and motion from images. This can be achieved when a scene or object is observed from different viewpoints or over a period of time. There is a wide range of applications, ranging from digitizing cultural heritage to vision-based autonomous robot navigation. This talk will present several approaches to solve this problem. First, we’ll discuss techniques for 3D shape recovery for static objects and scenes. One particular case is the 3D mapping and localization in large environments from images, e.g. urban 3D reconstruction from vehicleborne cameras or localization from cell-phone images. Next, we’ll shift our focus to modeling dynamic scenes, e.g. people who are moving around. In addition to explicitly 3D modeling an event, we'll consider the possibility to perform video-based rendering from casually captured videos. Speaker Bio-Sketch: Marc Pollefeys is a full professor in the Dept. of Computer Science of ETH Zurich since 2007 where he is the head of the Institute for Visual Computing and leads the Computer Vision and Geometry lab. He currently also remains associated with the Dept. of Computer Science of the University of North Carolina at Chapel Hill where he started as an assistant professor in 2002 and became an associate professor in 2005. Before this he was a postdoctoral researcher at the Katholieke Universiteit Leuven in Belgium, where he also received his M.S. and Ph.D. degrees in 1994 and 1999, respectively. His main area of research is computer vision. One of his main research goals is to develop flexible approaches to capture visual representations of real world objects, scenes and events. Dr. Pollefeys has received several prizes for his research, including a Marr prize, an NSF CAREER award, a Packard Fellowship and a European Research Council Starting Grant. He is the author or co-author of more than 130 peer-reviewed publications. He is the General Chair for the European Conference on Computer Vision 2014 (ECCV), was a Program Co-Chair for the IEEE Conference on Computer Vision and Pattern Recognition 2009 (CVPR), was general/program co-chair of the Third Symposium on 3D Data Processing, Visualization and Transmission and has organized workshops and courses at major vision and graphics conferences and has served on the program committees of many conferences. Prof. Pollefeys is/was on the Editorial Board of the IEEE Transactions on Pattern Analysis and Machine Intelligence and the International Journal of Computer Vision as well as several other journals in computer vision, graphics and robotics.