Chapter 6 - Trends in Model-based GUI Testing

Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modifications in the AUT or in its GUI arrangement and definition. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modifications in the AUT that caused such modifications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the ground up a taxonomy of causes for modifications of test code. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modified often – specifically, about 8% of developers’ modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modifications were induced by modifications in GUI definition and arrangement. Test code written with GUI automation fromeworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintainance needed by test code wrtitten with GUI automation frameworks, and the taxonomy of modification causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.

With the enrichment of social activities, people are more inclined to group activities in life, study and work. Therefore, how to make a recommendation for group user has become a research hotspot. As the result of the existed recommendation systems are designed for individuals, this paper proposes a group recommendation method based on user's interaction behavior. First, a global trust estimation model based on the interaction between users is proposed, which can be used to estimate the global trust of each user. Second, each user's score of the project can be obtained through collaborative filtering. In the third step, the results obtained in the above two parts are combined to obtain the scoring prediction model proposed in this paper. Finally, according to the characteristics of the group itself, a model GROUP-MAE which is more suitable for evaluating the accuracy of group recommendation systems is proposed in this paper. By comparing experiments on EPINIONS_EXTENDED dataset, it is shown that the proposed model is superior to the traditional cooperative fusion strategy in precision and accuracy.

Demographic information has a rich context from which to make decisions about how to filter or individualize computer users in forensic analysis. Although current explorations into technologies such as face and fingerprint analysis have seen varying rates of success, two main problems limit their applicability in the context of computer crimes: they can be intrusive, and they can require costly equipment. Our solution is to determine users’ demographic traits by analyzing the interactions between users and computers. We conducted a field study that gathered users’ keystroke and mouse data during interaction with a computer. From user interaction data, we extracted keystroke timing and mouse movement features, and developed weighted random forest classifiers for five demographic traits: gender, age, ethnicity, handedness, and language. Experiments showed that these demographics can be accurately inferred from user interaction behavior, with recognition rates expressed by the area under the ROC curve (AUC) ranging from 82.11 % to 87.32 %.

Social recommendation leverages both user interaction behaviors and social relations to support personalized information services. However, jointly modeling complex social structures and the dynamic evolution of user preferences remains challenging under sparse and noisy data settings. To address this issue, we propose Spatio-Temporal Graph Mamba Network (STG-Mamba), a unified spatiotemporal graph learning framework for social recommendation. The model adopts a dual-view architecture to jointly model user behavior sequences and social relations, where a Mamba-based sequential encoder captures long-range temporal dependencies, and a Light Graph Convolutional Network (LightGCN) based graph encoder models social influence propagation over trust networks. Unlike existing fusion methods that rely on static weighting or simple feature concatenation, STG-Mamba introduces a spatiotemporal uncertainty calibration mechanism to adaptively regulate the contribution of sequential signals during fusion, thereby alleviating behavioral volatility and noise. Extensive experiments on the Ciao and Epinions datasets demonstrate the superiority of our approach. STG-Mamba consistently outperforms sequential-only baselines by a significant margin and achieves competitive or superior performance compared to state-of-the-art graph-based models. Notably, on the Epinions dataset, STG-Mamba outperforms the strongest baseline, LightGCN, with relative improvements of 7.0% in Recall@10 and 3.5% in Normalized Discounted Cumulative Gain (NDCG)@20. These results validate the importance of jointly modeling temporal dynamics and social structures to enhance recommendation precision.

ABSTRACTUser Interface (UI) and User Experience (UX) are two crucial factors in the success of information systems, especially web-based applications and mobile applications. Design elements in web or mobile applications can be made more atrractive, by applying several elements in human-computer interaction theory based on user segmentation, product knowledge, consumer behavior, latest marketing theories, both in terms of visuals, after-effects, display sets, content, multimedia, lay-out menus on the web or mobile applications. In the recent state development of tourism in Bali, tourism transactions and promotions are already based on E-Tourism using web media and mobile applications. Making it easier for tourists to access information and services related to ticketing, transportation, accommodation and even travel packages. This study uses the User Interface (UI) and User Experience (UX) methods to view usability and web-quality from a web-based E-Tourism websites that promotes tourism in Bali.Keywords: User Interface, User Experience, E-Tourism, Informastion System, Bali Tourism.ABSTRAKUser Interface (UI) dan User Experience (UX) adalah dua faktor krusial dalam keberhasilan sistem informasi, terutama aplikasi berbasis web dan aplikasi mobile. Elemen desain dalam web dapat dibuat lebih menarik, dengan menerapkan beberapa elemen dalam teori interaksi manusia-komputer berdasarkan segmentasi user, pengetahuan produk, perilaku konsumen, teori pemasaran terbaru, baik dari segi visual, after-effect, display set, konten, multimedia, lay-out menu di web atau aplikasi mobile. Dalam perkembangan pariwisata di Bali saat ini, transaksi dan promosi pariwisata sudah berbasis E-Tourism dengan menggunakan media web dan aplikasi mobile. Sehingga mempermudah wisatawan dalam mengakses informasi dan layanan terkait dengan ticketing, transportasi, akomodasi bahkan paket perjalanan wisata. Penelitian ini menggunakan metode User Interface (UI) dan User Experience (UX) untuk melihat usability dan web-quality dari website layanan E-Tourism yang berbasis web yangmempromosikan pariwisata di Bali.Kata kunci: User Interface, User Experience, E-Tourism, Sistem Informasi, Pariwisata Bali.

Ensuring the quality of mobile applications (apps) needs to explore the GUI thoroughly. In practice, exhaustively exploring every GUI widget is unscalable on large real-world apps since it usually suffers from the problem of widgets explosion. To mitigate the problem, many existing testing tools usually detect and group homogeneous widgets heuristicly with different level of model abstraction since these widgets behave the same. However, no heuristic always works well. Heterogeneous widgets with divergent behaviors can be mistakenly grouped, which largely limits the testing effectiveness. This paper proposes a technique to effective GUI testing of Android apps with dynamic feedback-directed widgets grouping. Initially, we group the widgets according to the structure of the GUI. During testing, we observe behaviors of widgets in a group and regroup improperly-grouped widgets dynamically. Then, we apply a feedback-directed strategy to effectively accelerate the GUI exploration. The proposed technique is implemented as a practical tool for Android apps, named WGDroid. We evaluated WGDroid on 17 widely-used Android apps and compared it with the state-of-the-art GUI testing tools, i.e., AimDroid, SAPIENZ, and Monkey on both emulators and real devices. WGDroid outperformed the three tools in all testing coverages and also detected the most unique crashes. In particular, WGDroid discovered 208 more activities on 12 large benchmark apps on real devices and 11 more activities on another 5 benchmark apps on emulators, than the best of the other tools. These results show that WGDroid can significantly accelerate the GUI exploration.

Modeling approaches based on standards are of paramount importance in the field of front-end design for web and mobile applications. Problems are often encountered during the selection of tools for designing applications for developers and researchers, particularly applications that are related to time and cost in the market and academia. The Interaction Flow Modeling Language (IFML) is a recently standardized modeling language designed for managing the content expression, user interaction and behavior control of front-end applications. IFML brings several benefits to the development process of web and mobile front-end applications. Thus, several tools have been developed for the exploitation of technical artifacts offered by current specifications. In this study, we discuss the tools proposed for modeling IFML to provide a comparative analysis while considering various criteria. The result can be used as a basis of tool selection for specific aspects.

In recent years, with the growing popularity of smartphones, the number of Android malware shows explosive growth. As malicious apps may steal users' sensitive data and money from mobile and bank accounts, it's important to detect potential malicious behavior in real time. To achieve this goal, we propose a dynamic behavior inspection and analysis framework for malicious behavior detection in Android apps. A customized Android system is built to record apps' API (Application Programming Interface) calls, permission uses, and some other runtime features such as user operations. We also develop an automated testing platform to test massive samples so as to collect dynamic app behavior records. Then we exploit these records to extract apps' runtime features of both user interaction and app dynamic behavior for benign and malicious behavior classification. The experimental results show that the app behavior classification can reach an accuracy of 99.0%, identifying 71.8% instances of malware samples by running each app for only 18 minutes.

Many software systems are executable on various computing devices. No one denies that each of these devices has its own user interfaces. Nevertheless, this trend of computing everywhere is not accompanied by a solution that can be used to abstractly express the content, user interaction and control behavior of the software application front end without focusing on the implementation platform. Applying the concept of abstract models to user interfaces become a necessity. Accordingly, OMG adopted (in March 2013) the new Interaction Flow Modeling Language (IFML) for abstractly describing the system front end. It ensures executability in order to be mapped into executable applications for different kind of devices. In this paper, we propose a new model driven development approach to execute the logical description of UIs components and their interactions captured with IFML. We define IFVM, a virtual machine for executing IFML models with focus on the content-dependent navigation specification for passing parameters between the ViewElements, and the Data binding specification to specify the source of the published content.

Self-learning functions, an evolving field in modern technology, are increasingly being integrated into a multitude of applications. They primarily rely on data-driven learning techniques, such as supervised, unsupervised machine learning and reinforcement learning. In the field of autonomous vehicles, self-learning functions are important for real-time decision-making, as they adapt to dynamic scenarios by collecting extensive data from sensors. Likewise, self-learning functions with user-interaction, a subset of self-learning functions, are asserting their influence in the automotive industry, as they observe driver behavior and recognize user-specific interactions with the system. In addition to data-driven learning, these functions incorporate real-time user interactions, such as activating seat heating or ventilation, and autonomously execute these interactions, enhancing overall comfort of the driver. The growing integration and interaction of self-learning functions underscore the importance of conducting research and refining testing methodologies to ensure their reliability and effectiveness. To meet the growing need for trusted and reliable self-learning features, effective testing methods are essential for validating the accuracy and robustness of self-learning functions with user interaction. In contrast to traditional software, self-learning functions change and adjust themselves based on data and interactions. This causes challenges to predict and verify their intended behavior. Furthermore, each potential user exhibits distinctive individual behavior that differentiates them from other users. As a consequence, attempting to address every potential user interaction with traditional testing methods and predefined test case specifications becomes impractical. Moreover, the behavior of a self-learning function adapts over time to that of the respective user. As a result, the user behavior to be tested evolves, rendering traditional testing through predefined test cases unfeasible. Consequently, adapted testing methods are indispensable to effectively address the test-oracle problem. This paper presents a solution to address the test-oracle problem by leveraging metamorphic testing as a method for validating self-learning functions with user interaction. Metamorphic testing approaches the problem of the test-oracle from a perspective not typically employed by other testing strategies: instead of focusing on individual test cases, metamorphic tests examine the outcomes of multiple test cases within a testing system and their relationships with each other. Metamorphic testing assesses whether the test inputs and outputs fulfill specific metamorphic relationships upon multiple test executions. These metamorphic relationships describe the essential properties of the intended functionality. They transform existing input-output test cases into new follow-up test cases. If the behavior of the self-learning functions deviates from the metamorphic relationship in these original and follow-up test cases, the testing system is considered faulted. The effectiveness of fault detection significantly relies on metamorphic relations. Thus, the analysis of metamorphic relations becomes an essential task and a creative endeavor for the tester. Furthermore, it will be a significant contribution of this publication.The proposed paper offers an analysis of the insights gained from the application of metamorphic testing to a self-learning comfort function. This underscores how effective the testing methodology is in identifying inaccuracies in the self-learning function's interpretation of user behaviors, thus contributing to our understanding of their reliability and adaptability in simulated scenarios. In conclusion, the utilization of metamorphic testing in the context of self-learning functions, with a specific emphasis on user interactions, emerges as a promising and efficient strategy.

Although mobile application ecosystems have experienced tremendous growth in recent years, retrieving content of mobile applications that serves a key to mobile content search engines still faces grand challenges. Compared to web content retrieval, it is much more difficult to capture content in mobile applications due to the diversity of applications and the lack of Uniform Resource Locator indices. In this study, we propose and implement a <underline>u</underline>ser interaction-driven <underline>m</underline>obile <underline>c</underline>ontent <underline>r</underline>etrieval (UMCR) system to address such issues, which is the first mobile content crawler in the current literature. UMCR is a distributed system that contains many measurement nodes, each of which combines the user interaction path traversing (UIPT) and Deep Package Inspection (DPI) together to obtain mobile content. UIPT determines the events of user interactions in various applications to capture the static content such as text and images, in which a traversal depth termination scheme and an optional cut-off component are adopted to balance the content coverage and traversing efficiency. Meanwhile, the analysis based on DPI is responsible for extracting the videos as well as digging the infrastructural information and performance metrics. In addition, a distributed traversal scheduling method is designed for UIPT tasks to improve the throughput and scalability in large-scale content retrieval. Experiments on retrieving content of 64 real mobile applications demonstrate that UMCR can handle diverse mobile applications efficiently. The scheduler can improve throughput by 3 times compared to the legacy arbitrary task assignment strategy.

It is an undeniable fact that software plays an important role in our lives. We use the software to play our music, to check our e-mail, or even to help us drive our car. Thus, the quality of software directly influences the quality of our lives. However, the traditional Software Engineering paradigm is not able to cope with the increasing demands in quantity and quality of produced software. Thus, a new paradigm of Model Driven Software Engineering (MDSE) is quickly gaining ground. MDSE promises to solve some of the problems of traditional Software Engineering (SE) by raising the level of abstraction. Thus, MDSE proposes the use of models and model transformations, instead of textual program files used in traditional SE, as means of producing software. The models are usually graph-based, and are built by using graphical notations – i.e. the models are represented diagrammatically. The advantages of using graphical models over text files are numerous, for example it is usually easier to deduce the relations between different model elements in their diagrammatic form, thus reducing the possibility of defects during the production of the software. Furthermore, formal model transformations can be used to produce different kinds of artifacts from models in all stages of software production. For example, artifacts that can be used as input for model checkers or simulation tools can be produced. This enables the checking or simulation of software products in the early phases of development, which further reduces the probability of defects in the final software product. However, methods and techniques to support MDSE are still not mature enough. In particular methods and techniques for model configuration management (MCM) are still in development, and no generic MCM system exists. In this thesis, I describe my research which was focused on developing methods and techniques to support generic model configuration management. In particular, during my research, I focused on developing methods and techniques for supporting model evolution and model co-evolution. Described methods and techniques are generic and are suitable for a state-based approach to model configuration management. In order to support the model evolution, I developed methods for the representation, calculation, and visualization of state-based model differences. Unlike in previously published research, where these three aspects of model differences are dealt with in separation, in my research all these three aspects are integrated. Thus, the result of model differences calculation algorithm is in the format which is described by my research on model differences representation. The same representation format of model differences is used as a basis of my approach to differences visualization. It is important to notice that the developed representation format for model differences is metamodel independent, and thus is generic, i.e., it can be used to represent differences between all graph-based models. Model co-evolution is a term that describes the problem of adapting models when their metamodels evolve. My solution to this problem has three steps. In the first step a special metamodel MMfMM is introduced. Unlike in traditional approaches, where metamodels are represented as instances of a metametamodel, in my approach the metamodels are represented by models which are instances of an MMfMM. In the second step, since metamodels are represented by models, previously defined methods and techniques for model evolution are reused to represent and calculate the metamodel differences. In the final step I define an algorithm that uses the calculated metamodel differences to adapt models conforming to the evolved metamodel. In order to validate my approaches to model evolution and model co-evolution, I have developed a tool for comparing models and visualizing resulting differences, and a tool for model co-evolution. Moreover, I have developed a method to compare tools for model comparison, and using this method I have conducted a series of experiments in which I compared the tool I developed to an industrial tool called EMFCompare. The results of these experiments are also presented in the thesis. Furthermore, in order to validate my tool and approach to model co-evolution, I have also specified and conducted several experiments. The results of these experiments are also presented in the thesis.

Search systems use context to effectively satisfy a user's information need as expressed by a query. Tasks are important factors in determining user context during search and many studies have been conducted that identify tasks and task stages through users' interaction behavior with search systems. The type of interaction available to users, however, depends on the type of search interface features available. Queries are the most pervasive input from users to express their information need regardless of the input method, e.g., typing keywords or clicking facets. Instead of characterizing interaction behavior in terms of interface specific components, we propose to characterize users' search behavior in terms of two types of query modification: (i) direct modification, which refers to reformulations of queries; and (ii) indirect modification, which refers to user operations on additional input components provided by various search interfaces. We investigate the utility of characterizing task stages through direct and indirect query reformulations in a case study and find that it is possible to effectively differentiate subsequent stages of the search task. We found that describing user interaction behavior in such a generic form allowed us to relate user actions to search task stages independent from the specific search interface deployed. The next step will then be to validate this idea in a setting with a wider palette of search tasks and tools.

Patterns of user interaction behavior in online social networks have attracted increasing academic interest, since the quantitative understanding of user behavior is helpful to uncover the origins of many socioeconomic phenomena. In order to gain insights into user interaction behavior characteristics in microblogging, we have crawled Sina Microblog data and obtained user profiles and statuses. Six features of users are studied in our experiments, which are followers and friends, statuses and favorites, retweets and comments. Empirical analysis reveals some common statistical features of user behavior in microblogging. First, all of these properties follow different power-law distributions. Especially, followers and statuses exhibit a double-power-law distribution and friends follow a specific interval power law distribution. Second, there exists a strong positive correlation between statuses and followers, retweets and comments, but the relation between followers and friends is weak. Whats more, there exists a moderate correlation between the number of friends and statuses, retweets and followers, and comments and followers. Third, only a small number of statuses have extreme dissemination and concern. From a quantitative view, the result shows that the strong effect of celebrity cannot be ignored in Sina Microblogs. We believe this paper has added rich empirical materials to our knowledge and understanding of user behavior in microblogging.

Sequential recommendation aims to predict the next item based on users' historical interaction behaviors.Most existing methods enhance representations by generating different views of user sequences via random augmentation and maximizing their mutual information.However, user interaction behaviors are influenced by underlying intents, and random augmentation may alter the original intents of users.Furthermore, the embedding representations based on Transformer exhibit high similarity, which can obscure short-term intents.To address these challenges, we propose an approach called Diffusion Model-based Intent Contrastive Learning for Sequential Recommendation (DICLR).DICLR leverages contextual information to guide the diffusion model in generating positive samples that are consistent in intent, and uses the Gated Linear Unit (GLU) to enhance context understanding and denoising effects.Additionally, DICLR uses the Fourier transform to separate the frequency-domain features of user behavior, thereby capturing both long-term and short-term user intents.DICLR integrates the recommendation task with intent contrastive learning and diffusion tasks within a multi-task learning framework.Experiments conducted on four datasets demonstrate the superiority of the proposed model.