An architecture to provide adaptive, synchronized and multimodal human computer interaction

We present a design and implementation of a prototype system of the hard disk based portable entertainment device called photo jukebox based on an ARM-based system for managing thousands of personal photo/music collections anywhere you go. This photo jukebox system is designed as a music-centric entertainment player, slideshow player, and digital photo storage with built-in metadata database. In addition, this photo jukebox comes with a host application on computer for managing and synchronizing the contents seamlessly. In addition, based on this system, we have applied for a process of user-centered design. Therefore, the ease of use of its user interface was investigated using the human computer interaction (HCI) methods of heuristic evaluation and usability testing. The user interface was subsequently modified, and a follow-up usability test confirmed improvements to ease of use. These findings demonstrate that HCI methods can and should be used to enhance the usability of handheld-based entertainment hardware. At last, we will explore how we conduct the configurable dynamic power management based on different user profiles to enhance the battery life on such multimedia devices.

We have successfully designed, developed and deployed a pressure sensing floor system with a higher frame rate, less latency, high sensor resolution, large sensing area that can provide us with real time data about the location and amount of pressure exerted on the floor. The floor has been integrated and synchronized with the marker based motion capture system to create a smart environment for movement based human computer interaction. Future direction shall be towards extending the context aware capabilities of the floor system. An algorithm that can clearly distinguish between the left foot and the right foot shall find extensive usage of the floor to numerous applications. Shape descriptors such as Fourier and Hu moments to distinguish left foot and right foot on the basis of shape come in handy for such an analysis. Such intelligence to the floor to recognize and distinguish the left and right foot shall pave way for recognizing gestures with varying foot contact. The above work may be further extended to make a distinction of the heel and toe of a particular foot as well. This shall find extensive use in diagnosis of various gait pathologies as most disorders are reflected by abnormal pressure patterns localized to either the toe or heel. AME Floor-III as it stands now is not portable. Further work is also being done in the design of the interfacing hardware to make a portable system. The creation of a wireless pressure sensing system is a possible alternative towards a portable system. Integration of other sensing modalities such as audio-based sensing (microphone arrays), a wireless EMG system into the existing multimodal framework to create a very powerful tool for movement based human computer interaction is another major challenge in the future. The establishment of statistical models and machine learning techniques to model the

With the rapid development of intelligent information technology, human–computer interaction for unmanned aerial vehicle (UAV) systems has gradually developed from single-mode interaction to multi-modal interaction. Due to the problems of information ambiguity and operational redundancy in multi-modal interaction, and the inability to effectively realize the coupling of human–computer interaction of different modalities, a priority strategy for multi-modal interaction for UAV systems is proposed. Living cells or tissues, whether in a static or active state, will produce regular electrical phenomena called bioelectricity. Human–computer interaction intention recognition technology based on bioelectrical signals can significantly enhance the real-time performance and flexibility of human–computer interaction and has great development potential. The characteristics of human–computer interaction intention recognition methods based on bioelectrical signals, such as electroencephalograms (EEGs), electromyograms (EMGs), electrooculograms (EOGs), and other conventional methods are introduced and analyzed, and then the basic characteristics of single-mode interaction and multi-modal interaction are compared. Furthermore, the priority strategies for UAV systems under different interaction modes are studied based on the analysis of task value evaluation parameters. The cumulative value, urgency, and task complexity are comprehensively analyzed and evaluated, and a set of priority strategies for multi-modal interaction is integrated. Through the simulation and verification of the multi-modal human–computer interaction software of the UAV system, it is determined that the proposed priority strategy conforms to the operation intention of the user, which improves the interaction efficiency and the reliability. In noisy environments, the UAV system control can be combined with the complementary information of various interaction methods to achieve precise control, which has obvious advantages.

This paper contends that Human-Computer Interaction (HCI) methods have the potential to improve person-centered fall prevention for older adults. One-size-fits-all solutions are not appropriate for everybody, and a considered, individualised approach to fall prevention is needed. A qualitative content analysis of sixty community based falls risk assessment forms and person-centered care documents was conducted. The results presented provide insights into the care goals of older adults as well as various reasons underpinning low acceptance and compliance with fall preventative home modifications and assistive technologies. These interventions are based on the goals of older adults that commonly extend beyond compensatory approaches to improve physical performance within the home environment. HCI methods and new technologies provide an opportunity to reshape person-centered fall prevention, resulting in higher satisfaction and subsequent compliance rates amongst older adults. Furthermore, a series of HCI inspired person-centered design guidelines for designing home modifications and assistive technologies solutions are presented.

The development of computer technology has materialized into the term of human computer interaction. Research studies in human computer interaction studied young age groups that are educated and technically inclined. This paper will study the mental model in Human Computer Interaction that could be useful in some area. There are a number of types of this review paper on the Internet. One will lay out the current approach, result, trends, in the human computer interaction and the other approach will look into research construction that have been around long before but are now forgotten. In this paper, the author is also looking at the landscape in the emotional intelligence of a user so that they are more user like, fidelity prototyping. The development and design of an automated system that performs this task is still being undertaken. The notion of human-computer interaction is conceived as a result of the evolution of the development criteria of computer technology. The next generation of individuals (those at a younger age cohort) in which is termed an educated, technical, or at least knowledgeable individual will also be the focus of individual (tech) research experiments in human-computer interaction. Human-computer interaction (HCI) refers to both technical and human behaving aspects. The central tenets of practical studies in human-computer interaction are to reveal implicit knowledge about human behaviour and human interaction with technology. Resilience is set of behaving contextually taken collectively reacts to challenges. The term resilience has been applied to virtually everything from the economy, real estate, sports, psychology, business, and education, and the list could and can go on. Resilience is mainly comprised or made of a number of different abilities and skills as a way to be able to relate effectively to others, build relationships, self-efficacy, optimism, and self-awareness, creating meaning from other experiences form these are essential combination of competencies. In this manner people should use this in order to optimize the value of the resilience of an organization. This becomes important resourcefulness knowledge of the help available to increasing capacity to deal with the challenges facing change, overcoming (wasting time) relative to everything that could be done alternatively.

The ongoing integration of IT systems is offering computer users a wide range of new (networked) services and the access to an explosively growing host of multimedia information. Consequently, apart from the currently established computer tasks, the access and exchange of information is generally considered as the driving computer application of the next decade. Major advances are required in the development of the human-computer interface, in order to enable end-users with the most varied backgrounds to master the added functionalities of their computers with ease, and to make the use of computers in general an enjoyable experience. Research efforts in computer science are concentrated on user interfaces that support the highly evolved human perception and interaction capabilities better than today's 2D graphic user interfaces with a mouse and keyboard. Thanks to the boosting computing power of general purpose computers, highly interactive interfaces are becoming feasible supporting a variety of input and output modalities. Multimodal interaction not only makes working with a computer more natural and intuitive but can substantially help to disambiguate the exchange of information in both directions between the user and the computer. Recent approaches aim to create a (virtual) three-dimensional interaction environment where users find a clear arrangement of the media objects, paths and links used, as well as highly responsive tools supporting continuous interaction. The article overviews the current concepts in general and describes an approach developed by the authors in greater detail as a case study of a 3D PC (the mUltimo3D project). This particular approach uses 3D displays which do not require stereo glasses to present a 3D graphic user interface. A newly developed 3D display makes it possible to seamlessly integrate the virtual interaction space into the real working space. Video trackers remotely sensing the user offer new modalities for unencumbered interaction with the displayed objects. Interface agents watch and interpret the user's actions with regard to the task context, fuse the input from different interaction modalities, and organize the visual interaction space accordingly. The concept aims at the vision of a command-free, continuous human-computer interaction. A test bed has been implemented and used for system evaluation with users. First results are reported.

Multi-modal human computer interaction indicates that human can be interacted with computer by use of keyboard and mouse, also by using gesture, brow and speech. This paper introduces a formal model of perception agent, discusses the structure of multi-modal human computer interaction system based on agent. Finally, network environment of system execution and the methods of information fusion are presented.

The continuous evolution of technology has significantly transformed the way humans interact with computers, moving beyond traditional keyboard and mouse inputs toward more natural and adaptive interfaces. This literature review examines the current and emerging trends in user interface (UI) development within the context of Human-Computer Interaction (HCI), with a specific focus on future-oriented innovations. Key areas of discussion include voice-user interfaces (VUIs), gesture-based interaction systems, adaptive and context-aware UIs, artificial intelligence (AI) integration, and neuroadaptive technologies. The study employs a qualitative analysis method, drawing upon scholarly publications from 2020 to 2025 and emphasizing the contributions of researchers from Universitas Islam Negeri Sumatera Utara, particularly M. Khalil Gibran. These technologies are assessed for their usability, accessibility, efficiency, and their ability to enhance user satisfaction through personalization and contextual awareness. Results from reviewed studies suggest that adaptive and AI-driven UIs are becoming essential in delivering personalized user experiences

The challenges of designing computer support for education have shifted considerably in recent years, with, e.g., the rapid growth of the Web, online learning and social networking. New human–computer interaction (HCI) design approaches, methods, tools and theories are now required to analyze and understand interactions and learning of online groups. This chapter first reviews a number of issues related to the new software and pedagogy challenges. It then presents the approach of the VMT research project to address these issues by combining support for integrated synchronous and asynchronous collaboration media. The VMT system integrates a lobby, small-group chat rooms, multiple shared work spaces and community wiki pages to foster learning at the individual, small-group and community levels. The use of this system for a college HCI course is reported. The VMT Project illustrates the application of design-based research to system development, the theory of group cognition as a conceptual framework and an adaptation of chat interaction analysis for HCI design.

We have great pleasure in welcoming you to the 15th ACM International Conference on Multimodal Interaction -- ICMI 2013. ICMI is the premier international forum for multidisciplinary research on multimodal human-human and human-computer interaction, interfaces, and system development. The conference focuses on theoretical and empirical foundations, component technologies, and combined multimodal processing techniques that define the field of multimodal interaction analysis, interface design, and system development. Similarly to recent years, 2013 saw a very strong number of submissions: a record 133 long and short paper submissions, 11 demonstration papers, 5 multimodal grand challenge proposals and 3 workshop proposals. From the 133 long and short paper submissions, 49 were accepted for oral (27) or poster (22) presentation after a rigorous peer review process, bringing the acceptance rate to 37% overall and 20% for oral presentation. In addition, the program includes three invited keynote talks from Professors James Rehg, Stefan Kopp and Mark Billinghurst. The general themes that dominated this year's accepted papers focused on communication, interaction, virtual and robotic agents, and social-personality topics. Other heavily represented topics also focused on different communication modalities and related processing strategies, including spoken interaction, audio-visual, gesture and body movement, gaze, and pen or handwriting. In addition, other common themes included user-centered design & modeling, mobile interfaces, spatiotemporal processing, machine learning, adaptive processing, emotion, cognition, and language. Finally, diversity in this year's themes included a smaller number of papers on the nonetheless important topics of error handling, retrieval, tangible interfaces, brain-computer interfaces, and visualization tools. The review process was organized using the PCS submission and review system, and the review process included a rebuttal step. The process was assisted by 24 Associate Chairs (ACs), who helped the Program Chairs in defining the Program Committee. The papers were allocated to ACs in areas via a bidding process overseen by the Program Chairs, and then checked for conflicts. The Associate Chairs distributed the papers to members of the program committee and volunteer reviewers for comments. Once reviews were submitted, the ACs provided meta-reviews for all papers which were sent to the authors for rebuttal consideration. After hearing the authors' arguments, the scores of the papers were then collected and tabulated. All papers and their reviews were finally discussed at length by the Program Chairs in order to decide on the list of accepted submissions. The program was formed by grouping papers into main topics of interest for this year's conference. Following the trend of ICMI and many other academic meetings, to minimize paper consumption, the conference proceedings are distributed on USB flash drives only. A committee of senior ICMI leaders has been formed to select among the very best papers for Outstanding Paper Awards. These awards include best papers that were first authored by a student, and also papers by ICMI professionals. The final award decisions will be announced at the conference banquet. As in previous events, ICMI has been organized with the support of ACM and SIGCHI. In addition, despite the difficult financial period, many sponsors have lent their support to the event. Significant funding has been provided by Openstream and by National ICT Australia, which each deserve particular mention for their dedication to ICMI 2013. The US National Science Foundation (NSF) and Tateishi Foundation have generously provided us with travel, accommodation and living expense support for several students to allow them to travel the great distance to Australia to participate in ICMI and particularly to present at the Doctoral Consortium. Similar support for Australasian participants has been provided by the Australasian Speech Science and Technology Association. Significant support was also provided by the US Office of Naval Research and Microsoft Research. These organizations collectively deserve our warmest gratitude; without their generous support, this conference would not have been possible.

Nowadays several Computer-Aided Software Engineering environments exploit Model-Driven Engineering (MDE) techniques in order to generate a single user interface for a given computing platform or multi-platform user interfaces for several computing platforms simultaneously. Therefore, there is a need to assess the usability of those generated user interfaces, either taken in isolation or compared to each other. This paper describes an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that will be subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) has been examined in terms of satisfaction, effectiveness and efficiency. An experiment with a factorial design for repeated measures was conducted for 31 participants, i.e., postgraduate students and professors selected by convenience sampling. The data were collected with the help of questionnaires and forms and were analyzed using parametric and non-parametric tests such as ANOVA with repeated measures and Friedman's test, respectively. Efficiency was significantly better in large screens than in small ones as well as in the desktop platform rather than in the web platform, with a confidence level of 95%. The experiment also suggests that satisfaction tends to be better in standard size screens than in small ones. The results suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

BackgroundIt is challenging to design usable and effective digital health interventions (DHIs). The person-based approach (PBA) has been proposed to incorporate users’ perspectives for the design of DHIs. However, it does not explicitly describe the iterative stages of design and evaluation that are essential in moving from early planning to deployment. For this, we draw on methods from human computer interaction (HCI) that have been developed for various situations.ObjectiveThis study aimed to reflect on the adaptation and synthesis of PBA and HCI approaches to developing DHIs. We present a case study applying both approaches to develop Empowering People through Informed Choices for HIV (EPIC-HIV1), a DHI designed for men living in rural KwaZulu-Natal, South Africa, intended to support them in making an informed choice about whether to take an HIV test and, if necessary, engage in care.MethodsWe conducted a retrospective analysis of the documentation generated during the development of EPIC-HIV1 including findings about requirements, design representations, and the results of iterative rounds of testing. We developed an account of the process, the outcomes, and the strengths and limitations of the design and evaluation techniques applied. We also present the design of EPIC-HIV1 and summarize considerations when designing for hard-to-reach people in such settings.ResultsThe PBA was applied to deliver a first prototype. This helped identify key messages to convey and how to manage issues such as user privacy, but the resulting prototype was judged by the team not to be engaging for potential users, and it was unclear whether the design was inclusive of people with low digital or health literacy. We therefore introduced methods from HCI to iteratively test and refine the app. Working with local community representatives, we conducted four refinement cycles with 29 participants, adapting and retesting the app until no further changes were needed. Key changes included making it clearer what the consequences of selecting options in the app were and changing wording to minimize misconceptions (eg, that the app would test for HIV) while addressing common concerns about testing and emphasizing long-term benefits of engaging with care, if needed.ConclusionsTechniques for developing DHIs need to be situationally appropriate. The PBA enabled us to establish both empirical data and theory to design the content of EPIC-HIV1, but it did not directly inform interaction design to make the app usable and effective for the intended users; HCI techniques tailored to the setting enabled us to refine the app to be easy for men with little familiarity with digital technologies to use within the constraints of the setting. Iterative testing ensured the app was easy to use and that the intended clinical messages were communicated effectively.

Portable devices come with different limitations in user interaction like limited display size, small keyboard, and different sorts of input and output capabilities. With the advance of speech recognition and speech synthesis technologies, their complementary use becomes attractive for mobile devices in order to implement real multimodal user interaction. However, current systems and formats do not sufficiently integrate advanced multimodal interactions. We introduce an advanced generic multimodal interaction and rendering system (MIRS) dedicated for mobile devices. MIRS incorporates efficient processing of XML specification languages for limited, mobile devices and comes with the XML-based dialog and interface specification language (DISL). DISL can be considered as an UIML subset, which is enhanced by the means of state-oriented dialog specifications. The dialog specification is based on ODSN (object oriented dialog specification notation), which has been introduced to define user interface control by means of interaction states with transition rules.

This paper discusses the development of Virtual Control Interface (VCI), a Virtual Reality (VR) system for simulating and exploring multimodal context-sensitive interactions between Augmented Reality (AR) glasses and Internet of Things (IoT) devices.Current trends in controlling IoT devices focus on using existing control interfaces such as smartphones. Parallel technological trends indicate that AR glasses are likely to become the next consumer terminal. Therefore, there is a need to explore a new paradigm for IoT control interfaces that leverages the benefits of AR.Towards developing future-proof AR-IoT interfaces, we observed the HCI community’s challenges in using existing hardware to explore the full potential and the limitations of future AR-IoT interfaces. That became the foundation for developing VCI.This paper provides an overview of the advantages and shortcomings of developing such interfaces on smartphones, state-of-the-art AR/Mixed Reality (MR) glasses, and our proposed VR simulated AR environment. This overview outlines how our proposed system could be a superior choice for researchers to investigate, challenge assumptions, prototype, develop, and experimentally test AR-IoT interactions and interfaces with currently available technologies.Furthermore, we present the three pillars we followed to guide the design and development of VCI to make it resemble an actual AR-IoT framework as much as possible while abstracting parts that have little to no effect on AR-IoT HCI research. These pillars are: design transferable architecture, use viable multimodal interactions and explore user interfaces in real-life scenarios.These three principles informed the VCI’s system architecture built upon three main subsystems: handling multi-modal user input, IoT device emulation and connectivity, and managing the AR user interface (UI).Finally, we provide a heuristic evaluation and future work for VCI.KeywordsSmart homeInternet of ThingsAugmented realityVirtual realityAR-IoTMultimodal interactionVirtual Control Interface

Human-computer interaction for virtual-real fusion