Code generation from a graphical user interface via attention-based encoder–decoder model

End-to-end (E2E) models, including the attention-based encoder-decoder (AED) models, have achieved promising performance on the automatic speech recognition (ASR) task. However, the supervised training process of the E2E model needs a large amount of speech-text paired data. In contrast, self-supervised pre-training can pre-train the model on the unlabeled data and then fine-tune it on the limited labeled data to realize better performance. Most of the previous self-supervised pre-training methods focus on learning hidden representations from speech but ignore how to utilize the unpaired text. As a result, previous works often pre-train an acoustic encoder and then fine-tune it as a classification based ASR model, such as Connectionist Temporal Classification (CTC) based model, rather than an AED model. In this paper, we propose a self-supervised pre-training method for the AED model (SP-AED). The SP-AED method contains acoustic pre-training for the encoder, linguistic pre-training for the decoder, and an adaptive combination fine-tuning for the whole system. We first design a linguistic pre-training method for decoder by utilizing the text-only data. The decoder will be pre-trained as a noise-condition language model to learn the prior distribution of the text. Then, we pre-train the AED encoder with the wav2vec2.0 method with some modifications. Finally, we combine the pre-trained encoder and decoder and fine-tune them on the limited labeled data. We design an adaptive combination method during fine-tuning by modifying the decoder's input and output to prevent catastrophic forgetting. Experiments prove that compared with the random initialized models, the SP-AED pre-trained models can realize up to 17% relative improvement. And with similar model size or computational cost, we can get comparable results to other classification-based models on both English and Chinese corpus.

The deployment of a client–server-based distributed intelligent system involves application development in both the network domain and the device domain. In the network domain, an application server (typically in the cloud) is deployed to execute the network applications. In the device domain, several Internet of Things (IoT) devices may be configured as, for example, wireless sensor networks (WSNs), and interact with each other through the application server. Developing the network and the device applications are tedious tasks that are the major costs for building a distributed intelligent system. To resolve this issue, a low-code or no-code (LCNC) approach has been purposed to automate code generation. As traditional LCNC solutions are highly generic, they tend to generate excess code and instructions, which will lack efficiency in terms of storage and processing. Fortunately, optimization of automated code generation can be achieved for IoT by taking advantage of the IoT characteristics. An IoT-based distributed intelligent system consists of the device domain (IoT devices) and the network domain (IoT server). The software of an IoT device in the device domain consists of the Device Application (DA) and the Sensor Application (SA). Most IoT LCNC approaches provide code generation in the network domain. Very few approaches automatically generate the DA code. To our knowledge, no approach supports the SA code generation. In this paper, we propose DeviceTalk, an LCNC environment for the DA and the SA code development. DeviceTalk automatically generates the code for IoT devices to speed up the software development in the device domain for a distributed intelligent system. We propose the DeviceTalk architecture, design and implementation of the code generation mechanism for the IoT devices. Then, we show how a developer can use the DeviceTalk Graphical User Interface (GUI) to exercise LCNC development of the device software.

Mobile applications use Accessibility Services to assist users with disabilities (UWDs) in using Graphical User Interface (GUI)-based apps. However, such mobile assistive technologies are not fully secure as these can obtain data from the GUI objects, enhance this data for UWDs and transmit it to a server, thereby potentially exposing sensitive data to the external world. SEAPHISH (SEcuring Accessibility using PHISHing) is a platform aimed towards protecting against such an attack by providing defense by deception. This platform generates a phishing app i.e. an app similar to the original app installed on the user’s smartphone by extracting GUI elements and properties from the original app. A simulation for SEAPHISH can help determine the situations when an attack against a particular app can be performed with a high degree of probability. But performing effective simulations requires a fundamental understanding of the properties of GUI layouts of apps at large. This thesis aims at providing a framework that analyzes GUI layouts and their transitions using a large base of approximately three million Android apps. Various state-of-the-art tools that use different strategies in traversing layouts are explored. We created a framework where the tool Backstage performs static analysis and another tool AndroidRipper performs dynamic analysis of the layouts to help build a GUI model of the app. Using these models, we investigate the layouts of Android apps by collecting statistics on various GUI elements and screens. This investigation enhances SEAPHISH with statistically significant real-world constraints thereby providing defense against malwares and reducing the security vulnerabilities faced by UWDs.

WebDraw: A machine learning-driven tool for automatic website prototyping

Visual and aural are two most important channels of information processing. While most of the interaction with computers have been designed around the visual channel, there are circumstances where voice based man-machine interaction becomes preferable, and in some cases, necessary, given that voice based interaction comes naturally to humans and can be used by illiterate people easily.

The contemporary theory of metaphor defines the metaphor as a major and indispensable part of our ordinary, conventional way of conceptualizing the world and it asserts that the metaphors are used to define our relationship with the physical environment and to create a context of communication. The metaphors are the core idioms of the contemporary graphical user interface (GUI) and have an important role in helping users interact with computer systems. The desktop metaphor began with the first GUI Xerox Star and was popularized by the Lisa of Apple Macintosh. Realizing the physical-office metaphor and reflecting the familiar world of the potential users by using icons of file, folder and trashcan, the metaphor-based GUIs made it relatively easy for all to use the computer productively. Nowadays almost all of the software industry is based on the desktop metaphor paradigm. However, besides the researchers that favor the use of metaphors in the development of the system image by accentuating their assistance to learning, there are also other researchers, who reject the metaphor-based design approach in GUIs by referring to their limitations. Considering these discussions and various approaches, this study aims to make an outline of the historical development of metaphor-based GUIs; to determine the current state of metaphor within GUI design by brief examples and finally to identify future prospects and research themes. The study claims that metaphor still remains to be an inevitable part of the GUI to communicate it with the users and despite its potential limitations -rather than rejecting the metaphor totally- it seems better-to try compatible and supplementary solutions like visual formalism.

Mobility for graphical user interfaces (GUIs) is a challenging problem, as different GUIs need to be constructed for different device capabilities and changing context, preferences and users’ locations. GUI developers frequently create multiple user interface versions for different devices. The solution lies in using a single, abstract, user interface description that is used later to automatically generate user interfaces for different devices. Various techniques are proposed to adapt GUIs from an abstract specification to a concrete interface. Design-time techniques have the possibility of creating better performing GUIs but, in contrast to run-time techniques, lack flexibility and mobility. Run-time techniques’ mobility and autonomy can be significantly improved by using mobile agent technology and an indirect GUI generation paradigm. Using indirect generation enables analysis of computer-human interaction and application of artificial intelligence techniques to be made at run-time, increasing GUIs’ performance and usability.

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.

Dynamic Software Product Line (DSPL) Engineering has gained interest through its promise of being able to unify software adaptation whereby software adaptation can be realised at compile time and runtime. While previous work has enabled program logic adaptation by the use of language extensions and platform support, little attention has been placed on Graphical User Interface (GUI) variability. Different design patterns including the Model View Controller are commonly used in GUI implementation, with GUI documents being used for declaring the GUI. To handle dynamic GUI variability currently, the developer needs to implement GUI refinements using multiple techniques. This paper proposes a solution for dealing with GUI document variability, statically and dynamically, in a unified way. In our approach, we currently use a compile time method for producing GUI variants, and code transformations to handle these variants within the application at runtime. To avoid GUI duplicates, only GUI variants that are unique, and related to a valid product configuration, are produced. To validate our approach, we implemented tool support to enable this for Android based applications.

An important component of Early Warning Systems (EWS) for man-made and natural hazards is the command and control unit’s Graphical User Interface (GUI). All relevant information of an EWS is concentrated in this GUI and offered to human operators. However, when designing the GUI, not only the user experience and the GUI’s screens are relevant, but also the frameworks and technologies that the GUI is built on and the implementation of the GUI itself are of great importance. Implementations differ based on their applications in different domains but the design and approaches to implement the GUIs of different EWS often show analogies. The design and development of such GUIs are performed repeatedly on some parts of the system for each EWS. Thus, the generic GUI framework of a geospatial EWS for tsunamis is introduced to enable possible synergistic effects on the development of other new related technology. The results presented here could be adopted and reused in other EWS for man-made and natural hazards.

Daily multistep soil moisture forecasting by combining linear and nonlinear causality and attention-based encoder-decoder model

Recently, attention based sequence-to-sequence model become a research hotspot in speech recognition. The attention model has the problem of slow convergence and poor robustness. In this paper, a model that jointed a bottleneck feature extraction network and attention model is proposed. The model is composed of a Deep Belief Network as bottleneck feature extraction network and an attention-based encoder-decoder model. DBN can store the priori information from Hidden Markov Model so that increasing convergence speed of and enhancing both robustness and discrimination of features. Attention model utilizes the temporal information of feature sequence to calculate the posterior probability of phoneme. Then the number of stack recurrent neural network layers in attention model is reduced in order to decrease the calculation of gradient. Experiments in the TIMIT corpus showed that the phoneme error rate is 17.80% in test set, the average training iteration decreased 52%, and the number of training iterations decreased from 139 to 89. The word error rate of WSJ eval92 is 12.9% without any external language model.

AI (Artificial Intelligence) and GUI (Graphical User Interface) are the two most cutting-edge technologies. These technologies individually can make a comprehensive change in industries and alter economic and social interactions and connections. In general, Now-a-days AI is integrated with human technology and used in some wide range of sectors like healthcare, finance to education, banking, manufacturing, transportation, etc. GUI is used in gaming, designing, AR (Augmented reality) and VR (Virtual reality), media, smart home devices, etc., Integration of Artificial Intelligence and Graphical User Interface is beneficial for both users and app developers. The convergence of AI and GUI gives dynamic applications like Data Visualization, Insights, Smart assistants, Adaptions, Navigations, and Predictive analytics. This integration will increase the ease of access for new information to users. Now we are facing some general problems based on these technologies like Adaptive User Interfaces, Automated user interface testing, AI-Enhanced data visualization, Intelligent GUI Design Assistance, Real-Time Emotion Recognition for GUI Adaptation, etc., User interfaces are often designed with a one-size-fits-all approach, which does not account for individual user preferences and abilities. Designing perceptive and attractive GUIs requires significant space and time. This chapter focus on how can we develop a UI (User Interface) with base of AI which can be capable of solving the above obstacles and gives the simple objectives and technical solutions. By this integration we can execute some future projects like AI-Enhanced Customer Support Portals, Adaptive Gaming Interfaces, Interactive Financial Management Systems, and more.

The paper provides the design of a distributed test system for testing distributed GUI (Graphical User Interface) applications. The distributed GUI application that is considered allows multiple operators or users to interact with a distributed system in parallel. The distributed GUI application allows the use of existing GUI test frameworks for local test execution and offers additional services of coordination and synchronization among the deployed GUI test frameworks, test verdict arbitration, and debugging support, all of which are needed to deal with the concurrent nature of the system under test. In addition, the paper offers a test specification method for test cases that is based on the semantic model of a concurrent event flow graph. This method allows the automatic test case execution on top of the suggested distributed test system. The overall approach allows the extension of testing of single GUI application systems towards testing of distributed GUI application systems by preserving investments in test automation for single GUI testing.

Graphical user interfaces (GUIs) are becoming increasingly important in software development because they are user friendly. Thus, a well-developed GUI is an important factor for software quality, while the reliability of GUIs is a relatively young research field. The existing software reliability assessment techniques are based on particular assumptions and preconditions that are not suitable for GUIs. Compared with traditional software, the profile of a GUI is quite complicated, and so event interactions and event states make GUI reliability very complex. This paper introduces a model of GUI reliability and discusses related issues encountered in the modeling process. Then a simple case is introduced based on the GUI reliability model.