Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Mobile user interface (UI) design plays a very important role in determining an application that can be adopted by all types of users, especially those residing in rural areas. This review aims to provide a mapping of previously published studies on mobile UI design application for smallholder agriculture to be a smart farmer. The focus of this scoping review is on the design elements of the proposed UI, strategies of inquiry used and to find out if there is a UI design that involves geospatial context by published articles. The results show that interview and observation are the most common field study methods used to obtain information in designing mobile UI, and the most widely applied UI elements are audio and Interactive Voice Response (IVR). There are many user interface elements that can be used by farmers who have different skills level in dealing with mobile technology. Icon interface is an important element in designing mobile UI for rural farmers based on previous results. Other than that, the use of location function in mobile UI design shows an upward trend where three recent articles used location function to report crop pest and disease and determine the nearest market location. In addition, there is a need to design geospatial elements such as map for mobile UI that match the skills of rural farmers to support the fourth Industrial Revolution in agriculture to be a smart farmer.

There is substantial evidence that incorporating interactive environments for game-based instruction has a significant potential to support the development of computational thinking and programming skills in primary education students. However, it is not clear whether a simulation game (SG) with different user interface elements, created via three-dimensional (3D) virtual worlds and visual programming environments to project various problem-solving exercises in a simulated reality, can significantly influence students to think and practice "computationally" their solution plans into code. The current study aims to identify any possible added value of each instructional approach by measuring students’ game experience and learning performance. This quasi-experimental study involved ninety participants (n = 90) aged between 10 and 11 years that consisted of two comparison conditions. The experimental group (n = 45) received training using OpenSimulator in combination with Scratch4SL, whereas the control group (n = 45) was trained using Scratch. The findings indicate considerable differences in students’ game experience and satisfaction, but no statistically significant difference in their learning performance and knowledge gain was identified. This study provides several design implications for user interface and game elements to inform educational practitioners and instructors about the benefits that each approach can offer students for better knowledge acquisition and deeper disciplinary understanding.

The data binding pattern is an established technique to couple user interface (UI) elements and data objects. Various markup languages (e.g. Microsoft XAML, Adobe MXML) integrate advanced data binding concepts in order to ease application development. However, the HTML standard does not embrace means for data binding although being the Web markup language supported by millions of Web programmers. Therefore, we propose a standard-compliant WebSocket-based Data Binding (WebSoDa) framework. The WebSoDa framework synchronizes data objects and UI elements by orchestrating a Microdata-based data binding language as well as a client-side and a server-side messaging component. Thus, developers may speed up the tedious task of implementing binding associations in Web applications.

XML Based Graphical User Interface Editor and Runtime Parser for ISO 11783 Machine Automation Systems

This article is focused on the system side of the multidisciplinary problem of building User Interface Management Systems (UIMS) for distributed and heterogeneous I/O devices. It presents a new architecture that decouples applications from their interfaces by using distributed synthetic file system interfaces (similar to /proc on UNIX) to export user interface elements and a new UIMS, O/live, following this approach. The UIMS has been in use for several years on a daily basis, in our laboratory and personal computers. It supports transparent distribution, replication, and migration of user interface elements among highly heterogeneous devices. Moreover, it is highly programmable without the need for special tools, which (a) facilitates experimentation and iteration for new human–computer interaction techniques and (b) enables the creation of orthogonal services to manipulate programmatically and independently the elements of the distributed UI. This article describes both the approach and the O/live UIMS and window system.

Games can be evolved into a creature that can condition the users enjoy doing activities especially for the kids. User interface elements are a major factor to attract kids to play. This subject aim to identify the preschoolers' propensity to game's user interface, particularly the character, emotion and reward. The method applied is the research and development focusing on the user interface necessity analysis in the game with survey. The tendency of character, emotion and reward game that can captivate preschoolers will be described in this paper.

A mobile user interface (UI) is a graphical and usually touch-sensitive display on a mobile device that allows the user to interact with the device’s apps, features, contents, and functions. Since people of the elderly age group were not as exposed to technology as much as the people of the present generation, they face hurdles while using mobile apps. Furthermore, designers also face difficulties while designing user interfaces for elderly users. It is difficult for them to determine whether a designed UI component will be perceived by the elderly users as per its purpose. An elderly user may perceive a tappable UI component as not tappable and vice versa. Therefore, the objective of this research is to propose a conceptual framework for developing an efficient system to evaluate whether an UI is usable to the elderly people or not considering the tappability of its (UI) components. As outcomes, a deep learning-based conceptual framework for such a system is proposed based on the identified user-requirements through semi-structured interviews. An initial UI prototype of the system is also developed and evaluated by the real end-users. It is expected that the conceptual framework will be able to pave the way for developing a system to help the elderly user by increasing the usability of the UIs and to aid the designers in designing intuitive UIs considering the limitations of the elderly user. KeywordsUsabilityMobile user interfaceTappabilityDeep learningElderly users

Multi-touch gestures like swipe and pinch have introduced a new interaction paradigm that is currently used in developing mobile applications. Such applications can include different user gestures that cause different results for different interactions with the same user interface element. Testing such user interaction is complex. We present a formal task-based approach to automate testing of user gestures in mobile applications. In previous work, we used TaMoGolog, a formal task modelling language built on the top of the Golog family of high-level programming languages, for evaluating user interfaces along the development process. In this work, we present MobiGolog that extends TaMoGolog to support the testing of users' multi-touch gestures in the developing and developed mobile applications.

This paper contributes to the understanding of graphical user interfaces for illiterate users. It looks at text-free user interface elements for software applications from a semiotic perspective. A field test was conducted (N=40) with literate and illiterate participants of both genders in Egypt to evaluate the preferences and performance with different types of graphical representations. The participants were shown five groups of different signs, each group contained one index, one icon and one symbol to identify meanings and create stories with their preferred signs. The signs were associated with common user interface elements and with objects that the participants observed in their daily life. The results show superior performance of literates regarding sign interpretation and the use of working memory. No significant differences were found regarding the preferences of icons, indexes or symbols between literates and illiterates.

We introduce the grab ‘n’ drop toolglass, an extension of the toolglass bi-manual interaction technique. It enables users to create and configure their own toolglasses from existing user interfaces that were not designed for toolglasses. Users compose their own toolglass interactions at runtime from an application’s user interface elements, bringing interaction closer to the objects of interest in a workspace. Through a proof-of-concept implementation for Mac OS X, we show how grab ‘n’ drop capabilities could be added to existing applications at the toolkit level, without modifying application source code or UI design. Finally, we evaluate the power and flexibility of this approach by applying it to a variety of applications. We further identify limitations and risks associated with this approach and propose changes to existing toolkits to foster such user-reconfigurable interaction.

Tokopedia is one of the companies in Indonesia that has a unicorn status. With a vision to build a platform that allows anyone to start and find anything and anywhere, Tokopedia also supports Micro, Small and Medium Enterprises (MSMEs) and individuals to develop their businesses through application. Even so, this does not guarantee that all Tokopedia users can really easily use the application. The baby boomer generation is a generation that faces more difficulties in processing new or complex information, thus affecting their learning about new technologies, including the user interface design of an application. The purpose of this study is to understand how the user interface elements are applied to the Tokopedia application and how the baby boomer affordance to the Tokopedia application user interface design. The method used is a case study with a qualitative approach. The results show that the user interface design elements of the Tokopedia application in the perception of baby boomer generation are always related to design elements, use of color, and layout. Some of these elements have been understood by the user's function. Nevertheless, there are some technical things, such as the use of free space, that users still don't understand. Thus, Tokopedia still has to make some improvements to the application's user interface design .

The approach describe in the paper unifies three main approaches to the development of hypermedia systems: language, screen and information based. The approach presumes providing a small set of conventional standard tools for this development and a wide range of tools for generating non-standard user's interface elements and control mechanisms for navigation. The concept and implementation details of development tools for these elements in HyperF system are considered.

This paper presents version 1.1.0 of the DLDiagnosis software, which serves as an automated decision support tool for disease detection and classification through imaging analysis using deep learning. The latest update of DLDiagnosis includes significant enhancements, expanding the system to manage a total of eight different diseases. Three new diseases have been added: keratoconus, breast cancer, and pneumonia. This update focuses on refining user interface elements, identifying gaps in code logic, and improving mobile and web interface elements. Additionally, various functionalities have been improved, including reduced diagnostic latency, faster results, enhanced responsiveness, and a more consistent and intuitive diagnostic experience.

This is the second of a two‐part article on user interfaces and bridges as they relate to library user interface design. This article takes a closer look at the concept and elements of the user interface, and the role of librarians in planning and implementation.

Existing speech recognition systems have claimed high accuracy for specific tasks such as dictation. What is new in Windows Speech recognition for Vista is a combination of high accuracy and high usability for the end-to-end speech experience. This paper describes the architecture, user interface, and key technologies that make up the speech system incorporated in Microsoft Windows Vista. It outlines some of the challenges encountered in providing a speech-based interface to a system as complex and extensible as the modern desktop PC, as well as the technology developments that have made this possible. In particular, the paper describes key elements of the speech user interface and how the user's ability to control the system is maintained despite limitations in the underlying recognition technology. The paper also explains how feedback and adaptation systems are used to tailor the experience to each user and their particular style of speaking/use of language.