Collaborating with LLMs Through a Voice and Graphical User Interface

Visual and aural are two most important channels of information processing. While most of the interaction with computers have been designed around the visual channel, there are circumstances where voice based man-machine interaction becomes preferable, and in some cases, necessary, given that voice based interaction comes naturally to humans and can be used by illiterate people easily.

This article investigates the potential of artificial intelligence (AI) in optimizing interface design to enhance task completion time across various types of interfaces. By examining the characteristics, strengths, and weaknesses of each interface type and the potential integration of AI, we aim to provide insights into how AI can be leveraged to improve user experience and task efficiency. We will discuss the influence of AI on Graphical User Interfaces (GUIs), Command Line Interfaces (CLIs), Voice User Interfaces (VUIs), Gesture-based Interfaces, Haptic Interfaces, and Mixed Reality Interfaces (Augmented Reality and Virtual Reality). Through a review of existing research and case studies, we explore the current state of AI integration in interface design and future opportunities for enhancing task completion time and overall system usability.

Mobile applications use Accessibility Services to assist users with disabilities (UWDs) in using Graphical User Interface (GUI)-based apps. However, such mobile assistive technologies are not fully secure as these can obtain data from the GUI objects, enhance this data for UWDs and transmit it to a server, thereby potentially exposing sensitive data to the external world. SEAPHISH (SEcuring Accessibility using PHISHing) is a platform aimed towards protecting against such an attack by providing defense by deception. This platform generates a phishing app i.e. an app similar to the original app installed on the user’s smartphone by extracting GUI elements and properties from the original app. A simulation for SEAPHISH can help determine the situations when an attack against a particular app can be performed with a high degree of probability. But performing effective simulations requires a fundamental understanding of the properties of GUI layouts of apps at large. This thesis aims at providing a framework that analyzes GUI layouts and their transitions using a large base of approximately three million Android apps. Various state-of-the-art tools that use different strategies in traversing layouts are explored. We created a framework where the tool Backstage performs static analysis and another tool AndroidRipper performs dynamic analysis of the layouts to help build a GUI model of the app. Using these models, we investigate the layouts of Android apps by collecting statistics on various GUI elements and screens. This investigation enhances SEAPHISH with statistically significant real-world constraints thereby providing defense against malwares and reducing the security vulnerabilities faced by UWDs.

The contemporary theory of metaphor defines the metaphor as a major and indispensable part of our ordinary, conventional way of conceptualizing the world and it asserts that the metaphors are used to define our relationship with the physical environment and to create a context of communication. The metaphors are the core idioms of the contemporary graphical user interface (GUI) and have an important role in helping users interact with computer systems. The desktop metaphor began with the first GUI Xerox Star and was popularized by the Lisa of Apple Macintosh. Realizing the physical-office metaphor and reflecting the familiar world of the potential users by using icons of file, folder and trashcan, the metaphor-based GUIs made it relatively easy for all to use the computer productively. Nowadays almost all of the software industry is based on the desktop metaphor paradigm. However, besides the researchers that favor the use of metaphors in the development of the system image by accentuating their assistance to learning, there are also other researchers, who reject the metaphor-based design approach in GUIs by referring to their limitations. Considering these discussions and various approaches, this study aims to make an outline of the historical development of metaphor-based GUIs; to determine the current state of metaphor within GUI design by brief examples and finally to identify future prospects and research themes. The study claims that metaphor still remains to be an inevitable part of the GUI to communicate it with the users and despite its potential limitations -rather than rejecting the metaphor totally- it seems better-to try compatible and supplementary solutions like visual formalism.

Human-computer interaction (HCI) is a branch of computer science that is concerned with the design, evaluation, and implementation of interactive computing systems for human use. It is focused on the interfaces between people and computers and involves several fields including computer science, cognitive psychology, design, and communication. It has grown over the decades to include text-based interaction systems, graphical user interfaces (GUI), and voice user interfaces (VUI) for speech recognition and speech synthesis.

Mobility for graphical user interfaces (GUIs) is a challenging problem, as different GUIs need to be constructed for different device capabilities and changing context, preferences and users’ locations. GUI developers frequently create multiple user interface versions for different devices. The solution lies in using a single, abstract, user interface description that is used later to automatically generate user interfaces for different devices. Various techniques are proposed to adapt GUIs from an abstract specification to a concrete interface. Design-time techniques have the possibility of creating better performing GUIs but, in contrast to run-time techniques, lack flexibility and mobility. Run-time techniques’ mobility and autonomy can be significantly improved by using mobile agent technology and an indirect GUI generation paradigm. Using indirect generation enables analysis of computer-human interaction and application of artificial intelligence techniques to be made at run-time, increasing GUIs’ performance and usability.

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.

Dynamic Software Product Line (DSPL) Engineering has gained interest through its promise of being able to unify software adaptation whereby software adaptation can be realised at compile time and runtime. While previous work has enabled program logic adaptation by the use of language extensions and platform support, little attention has been placed on Graphical User Interface (GUI) variability. Different design patterns including the Model View Controller are commonly used in GUI implementation, with GUI documents being used for declaring the GUI. To handle dynamic GUI variability currently, the developer needs to implement GUI refinements using multiple techniques. This paper proposes a solution for dealing with GUI document variability, statically and dynamically, in a unified way. In our approach, we currently use a compile time method for producing GUI variants, and code transformations to handle these variants within the application at runtime. To avoid GUI duplicates, only GUI variants that are unique, and related to a valid product configuration, are produced. To validate our approach, we implemented tool support to enable this for Android based applications.

An important component of Early Warning Systems (EWS) for man-made and natural hazards is the command and control unit’s Graphical User Interface (GUI). All relevant information of an EWS is concentrated in this GUI and offered to human operators. However, when designing the GUI, not only the user experience and the GUI’s screens are relevant, but also the frameworks and technologies that the GUI is built on and the implementation of the GUI itself are of great importance. Implementations differ based on their applications in different domains but the design and approaches to implement the GUIs of different EWS often show analogies. The design and development of such GUIs are performed repeatedly on some parts of the system for each EWS. Thus, the generic GUI framework of a geospatial EWS for tsunamis is introduced to enable possible synergistic effects on the development of other new related technology. The results presented here could be adopted and reused in other EWS for man-made and natural hazards.

AI (Artificial Intelligence) and GUI (Graphical User Interface) are the two most cutting-edge technologies. These technologies individually can make a comprehensive change in industries and alter economic and social interactions and connections. In general, Now-a-days AI is integrated with human technology and used in some wide range of sectors like healthcare, finance to education, banking, manufacturing, transportation, etc. GUI is used in gaming, designing, AR (Augmented reality) and VR (Virtual reality), media, smart home devices, etc., Integration of Artificial Intelligence and Graphical User Interface is beneficial for both users and app developers. The convergence of AI and GUI gives dynamic applications like Data Visualization, Insights, Smart assistants, Adaptions, Navigations, and Predictive analytics. This integration will increase the ease of access for new information to users. Now we are facing some general problems based on these technologies like Adaptive User Interfaces, Automated user interface testing, AI-Enhanced data visualization, Intelligent GUI Design Assistance, Real-Time Emotion Recognition for GUI Adaptation, etc., User interfaces are often designed with a one-size-fits-all approach, which does not account for individual user preferences and abilities. Designing perceptive and attractive GUIs requires significant space and time. This chapter focus on how can we develop a UI (User Interface) with base of AI which can be capable of solving the above obstacles and gives the simple objectives and technical solutions. By this integration we can execute some future projects like AI-Enhanced Customer Support Portals, Adaptive Gaming Interfaces, Interactive Financial Management Systems, and more.

The paper provides the design of a distributed test system for testing distributed GUI (Graphical User Interface) applications. The distributed GUI application that is considered allows multiple operators or users to interact with a distributed system in parallel. The distributed GUI application allows the use of existing GUI test frameworks for local test execution and offers additional services of coordination and synchronization among the deployed GUI test frameworks, test verdict arbitration, and debugging support, all of which are needed to deal with the concurrent nature of the system under test. In addition, the paper offers a test specification method for test cases that is based on the semantic model of a concurrent event flow graph. This method allows the automatic test case execution on top of the suggested distributed test system. The overall approach allows the extension of testing of single GUI application systems towards testing of distributed GUI application systems by preserving investments in test automation for single GUI testing.

Graphical user interfaces (GUIs) are becoming increasingly important in software development because they are user friendly. Thus, a well-developed GUI is an important factor for software quality, while the reliability of GUIs is a relatively young research field. The existing software reliability assessment techniques are based on particular assumptions and preconditions that are not suitable for GUIs. Compared with traditional software, the profile of a GUI is quite complicated, and so event interactions and event states make GUI reliability very complex. This paper introduces a model of GUI reliability and discusses related issues encountered in the modeling process. Then a simple case is introduced based on the GUI reliability model.

In software development process, the last step is usually the Graphic User Interface(GUI) test, which is part of the final user experience (UE) test. Traditionally, there exist some GUI test tools in the market, such as Abbot Java GUI Test Framework and Pounder, in which testers pre-configure in the script all desired actions and instructions for the computer, nonetheless requiring too much of invariance of GUI environment; and they require reconfiguration in case of GUI changes, therefore still to be done mostly manually and hard for non-programmer testers to. Consequently, we proposed GUI tests by image recognition to automate the last process; we managed to innovate upon current algorithms such as SIFT and Random Fern, from which we develop the new algorithm scheme retrieving most efficient feature and dispelling inefficient part of each algorithm. Computers then apply the algorithm, to search for target patterns themselves and take subsequent actions such as manual mouse, keyboard and screen I/O automatically to test the GUI without any manual instructions. Test results showed that the proposed approach can accelerate GUI test largely compared to current benchmarks.

Sketching out Graphical User Interface (GUI) layout is part of the pipeline of designing a GUI and a crucial task for the success of a software application. Arranging all components inside a GUI layout manually is a time-consuming task. In order to assist designers, we developed a method named GUILGET to automatically generate GUI layouts from positional constraints represented as GUI arrangement graphs (GUIAGs). The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. The existing image layout generation techniques often cannot incorporate GUI design constraints. Thus, GUILGET needs to adapt existing techniques to generate GUI layouts that obey to constraints specific to GUI designs. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Moreover, the model learns constraints through the minimization of losses responsible for placing each component inside its parent layout, for not letting components overlap if they are inside the same parent, and for component alignment. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG and has the best performances in most of evaluation metrics. Therefore, our work contributes to improved GUI layout generation by proposing a novel method that effectively accounts for the constraints on GUI elements and paves the road for a more efficient GUI design pipeline.

In the situation of monitoring an experiment, it is often necessary to control several independently running tasks from one graphical user interface (GUI). Such a GUI must be able to execute commands in the tasks even if they are busy, i.e., getting data, analyzing data, or waiting for data. Moreover, the tasks, being controlled by data streams (i.e., event data samples or slow control data), must be able to send data asynchronously to the GUI for visualization. A multitasking package (C++ class library) that meets these demands has been developed at the Gesellschaft fur Schwerionenforschung (GSI), Darmstadt, Germany, in the framework of a new analysis system, Go4, which is based on the ROOT system [CERN, R. Brun et al]. The package provides a thread manager, a task handler, and asynchronous intertask communication between threads through sockets. Hence, objects can be sent at any time from a task to the GUI or vice versa. At the GUI side, an incoming object is accepted by a thread and processed. In a task, an incoming command is queued by the accepting thread and executed in the execution thread. Utilizing the package one can implement nonblocking GUIs to control one or several tasks processing data in parallel and updating graphical elements in the GUI. The package could also be useful in building data dispatchers or in slow control applications. All components have been tested with Go4 analysis tasks and a very preliminary GUI.