Using XML files to document the user interfaces of applications

XML Based Graphical User Interface Editor and Runtime Parser for ISO 11783 Machine Automation Systems

This paper presents an interactive design project of a remote clinical robot for ophthalmology, for use in support of telemedicine. The project is a case study of finding the best organizational structure of a Graphical User Interface (GUI) and Solid User Interface (SUI) in the age of ubiquitous computing. At the early stage of design, we carried out usability tests with paper-based prototypes and obtained requirements from ophthalmologists. Results showed that paper prototyping is applicable for GUI- and SUI-combined products, even they have manual control input and continuously changing visual feedback. We examine lessons learned from the design process and designed product.Keywordsdesign methodGraphical User Interface (GUI)interaction designophthalmologypaper prototypingremote clinical robotSolid User Interface (SUI)telemedicine

AI (Artificial Intelligence) and GUI (Graphical User Interface) are the two most cutting-edge technologies. These technologies individually can make a comprehensive change in industries and alter economic and social interactions and connections. In general, Now-a-days AI is integrated with human technology and used in some wide range of sectors like healthcare, finance to education, banking, manufacturing, transportation, etc. GUI is used in gaming, designing, AR (Augmented reality) and VR (Virtual reality), media, smart home devices, etc., Integration of Artificial Intelligence and Graphical User Interface is beneficial for both users and app developers. The convergence of AI and GUI gives dynamic applications like Data Visualization, Insights, Smart assistants, Adaptions, Navigations, and Predictive analytics. This integration will increase the ease of access for new information to users. Now we are facing some general problems based on these technologies like Adaptive User Interfaces, Automated user interface testing, AI-Enhanced data visualization, Intelligent GUI Design Assistance, Real-Time Emotion Recognition for GUI Adaptation, etc., User interfaces are often designed with a one-size-fits-all approach, which does not account for individual user preferences and abilities. Designing perceptive and attractive GUIs requires significant space and time. This chapter focus on how can we develop a UI (User Interface) with base of AI which can be capable of solving the above obstacles and gives the simple objectives and technical solutions. By this integration we can execute some future projects like AI-Enhanced Customer Support Portals, Adaptive Gaming Interfaces, Interactive Financial Management Systems, and more.

Mobility for graphical user interfaces (GUIs) is a challenging problem, as different GUIs need to be constructed for different device capabilities and changing context, preferences and users’ locations. GUI developers frequently create multiple user interface versions for different devices. The solution lies in using a single, abstract, user interface description that is used later to automatically generate user interfaces for different devices. Various techniques are proposed to adapt GUIs from an abstract specification to a concrete interface. Design-time techniques have the possibility of creating better performing GUIs but, in contrast to run-time techniques, lack flexibility and mobility. Run-time techniques’ mobility and autonomy can be significantly improved by using mobile agent technology and an indirect GUI generation paradigm. Using indirect generation enables analysis of computer-human interaction and application of artificial intelligence techniques to be made at run-time, increasing GUIs’ performance and usability.

Implementation of a Chatbot System using AI and NLP

This paper discusses the design and applications of an object-oriented user interface management system (UIMS). Specifically, the Lockheed User Interface System (LUIS) is described. LUIS is based on a user interface model that includes declarative and procedural components. The package has been used by both non-programmers and programmers in several applications at Lockheed. Experiences derived from applications of the package are used to address several key issues in the UIMS field, such as procedural versus declarative specification, separation of the user interface from applications, UIMS flexibility, and UIMS support for design evaluation.

This paper presents a distributed, multi-user virtual reality testbed for a mobile robot. The main components of the testbed include user interface, networking, simulation and robot control. The user interface is implemented as a combination of a virtual environment and a standard graphical user interface (GUI). The virtual environment is developed using the Virtual Reality Modeling Language (VRML) and it corresponds to the laboratory environment where the robot is placed and tested. The standard GUI is implemented using Java Programming language. The World Wide Web (WWW) browser is used to integrate the virtual environment (embedded VRML) and the standard GUI (Java applet) in a single user interface. Users can access a dedicated WWW server and download the user interface. Networking component (communication among users and robots) is implemented using the Distributed Interactive Simulation (DIS) protocol. The existing robot development system has been utilized to provide simulation capabilities. A separate stand-alone Java application has been developed to communicate and control the robot using a wireless Ethernet link. In addition to user's direct control, control algorithms (e.g. fuzzy logic based) can be used to replace the user. A Nomad200 mobile robot has been used for initial measurements and evaluations of the testbed.

Mobile applications use Accessibility Services to assist users with disabilities (UWDs) in using Graphical User Interface (GUI)-based apps. However, such mobile assistive technologies are not fully secure as these can obtain data from the GUI objects, enhance this data for UWDs and transmit it to a server, thereby potentially exposing sensitive data to the external world. SEAPHISH (SEcuring Accessibility using PHISHing) is a platform aimed towards protecting against such an attack by providing defense by deception. This platform generates a phishing app i.e. an app similar to the original app installed on the user’s smartphone by extracting GUI elements and properties from the original app. A simulation for SEAPHISH can help determine the situations when an attack against a particular app can be performed with a high degree of probability. But performing effective simulations requires a fundamental understanding of the properties of GUI layouts of apps at large. This thesis aims at providing a framework that analyzes GUI layouts and their transitions using a large base of approximately three million Android apps. Various state-of-the-art tools that use different strategies in traversing layouts are explored. We created a framework where the tool Backstage performs static analysis and another tool AndroidRipper performs dynamic analysis of the layouts to help build a GUI model of the app. Using these models, we investigate the layouts of Android apps by collecting statistics on various GUI elements and screens. This investigation enhances SEAPHISH with statistically significant real-world constraints thereby providing defense against malwares and reducing the security vulnerabilities faced by UWDs.

Analyses some of the components of graphical user interfaces (GUI), applied to CD-ROM databases (windows, dialogue boxes, menus, commands, buttons, check boxes and icons) and remarks on the degree to which these are based on Windows software. Notes the constraints to GUIs that still remain even after databases have appeared in the marketplace: particularly mouse based prob lems. Discusses ways of evaluating GUIs and their specific components and reports a study of databases in which these types of interfaces are used. Notes the significant variability between products in the way that they apply GUIs and discusses whether database users would benefit from a certain degree of standardization in the basic interface structure using GUIs, or whether users would prefer greater control over the design of the local user interface. Concludes that the introduction of Windows based CD-ROM databases with graphical user interfaces will increasingly revolutionize the design of such data bases. Makes 3 recommendations that would aim to enhance the user friend liness of CD-ROM interfaces: research to develop criteria for evaluating interfaces in the GUI environment; standardization of interface design between some products, particularly for novice users; and facilities to allow experienced users to develop and tailor their own interface design.

Software product line (SPL) is a software engineering paradigm for software development. A software product within a product line often has specific functionalities that are not common to all other products within the product line. Those specific functionalities are termed “variant features” in a product line. SPL paradigm involves the modeling of variant features. However, little work in SPL investigates and addresses the modeling of variant features specific to user interface (UI). Unified Modeling Language (UML) is the de facto modeling language for object-oriented software systems. It is known that UML needs better support in modeling UIs. Thus, much research developed UML extensions to improve UML support in modeling UIs. Yet little of this work is related to developing such extensions for modeling UIs for SPLs in which variant features specific to UI modeling must be addressed. This research develops a UML extension -Web User Interface Modeling Language (WUIML) to address these problems. WUIML defines elements for modeling variant features specific to user interfaces for Web-based SPLs. The model elements in WUIML extend from the metaclass and BasicActivity of the UML2.0 metamodel. WUIML integrates the modeling of variant features specific to user interfaces to UML. For example, in a Web-based patient registration software product line, member products targeting British users may use British date format in the user interface, while member products targeting United States users may use United States date format in the user interface. Thus, this is a variant feature for this product line. WUIML defines a model element, XOR, to represent such exclusive or conditions in a product line user interface model. WUIML would reduce SPL engineers’ efforts needed in UI development. To validate the WUIML research outcome, a case study was conducted. The results of this empirical study indicate that modeling UIs for Web-based SPLs using WUIML is more effective and efficient than using standard UML.

Visual and aural are two most important channels of information processing. While most of the interaction with computers have been designed around the visual channel, there are circumstances where voice based man-machine interaction becomes preferable, and in some cases, necessary, given that voice based interaction comes naturally to humans and can be used by illiterate people easily.

The contemporary theory of metaphor defines the metaphor as a major and indispensable part of our ordinary, conventional way of conceptualizing the world and it asserts that the metaphors are used to define our relationship with the physical environment and to create a context of communication. The metaphors are the core idioms of the contemporary graphical user interface (GUI) and have an important role in helping users interact with computer systems. The desktop metaphor began with the first GUI Xerox Star and was popularized by the Lisa of Apple Macintosh. Realizing the physical-office metaphor and reflecting the familiar world of the potential users by using icons of file, folder and trashcan, the metaphor-based GUIs made it relatively easy for all to use the computer productively. Nowadays almost all of the software industry is based on the desktop metaphor paradigm. However, besides the researchers that favor the use of metaphors in the development of the system image by accentuating their assistance to learning, there are also other researchers, who reject the metaphor-based design approach in GUIs by referring to their limitations. Considering these discussions and various approaches, this study aims to make an outline of the historical development of metaphor-based GUIs; to determine the current state of metaphor within GUI design by brief examples and finally to identify future prospects and research themes. The study claims that metaphor still remains to be an inevitable part of the GUI to communicate it with the users and despite its potential limitations -rather than rejecting the metaphor totally- it seems better-to try compatible and supplementary solutions like visual formalism.

This research project investigates the connections between user identity, product emotion and interface design. It investigates whether mobile communication devices can better reflect user identity and lifestyle through the provision of a customizable Graphical user interface (GUI) that users can actively manipulate. The study proposes a user and design-driven interaction process whereby users can easily personalize the interface of their devices to reflect their idiosyncratic needs and preferences, the designed interface making interaction more individually useful, meaningful and empowering. The project extends the scope of end user customization beyond basic cosmetic changes to the physical appearance of the GUI to the modification of the actual process of interaction. The research focuses on the capacity to employ the user interface to support more intimate and flexible patterns of use that foster personal identification and emotional investment with devices, as against those that are preordained, device-driven and technologically deterministic.

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.

The World Wide Web is increasingly becoming an important medium for business growth. The migration of existing applications to the web is a core problem in software reengineering, since applications are frequently deemed to be useful when their functionalities are accessible via the Web. A substantial class of applications that could benefit from migration to the Web is that of Java Graphical User Interface (GUI) desktop applications. An emerging approach to Web migration is interaction-based reengineering, in which the User Interface (UI) of the existing application is wrapped with a software layer that exports a Web accessible interface and hides the original one. A major barrier to such migration approaches is the reverse engineering of the User Interface, which is usually a time consuming and highly manual task. We present a new approach to automatically Reverse Engineer various structural and behavioral features of a GUI from an executing Java-Swing desktop application. The approach is based on source code independent techniques, such as Aspect J. In this paper, we describe our GUI reverse engineering approach and we illustrate it with a case study.