An Exploratory Study of Automated GUI Testing: Goals, Issues, and Best Practices

Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modifications in the AUT or in its GUI arrangement and definition. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modifications in the AUT that caused such modifications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the ground up a taxonomy of causes for modifications of test code. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modified often – specifically, about 8% of developers’ modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modifications were induced by modifications in GUI definition and arrangement. Test code written with GUI automation fromeworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintainance needed by test code wrtitten with GUI automation frameworks, and the taxonomy of modification causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.

Database Management System (DBMS) developers have implemented extensive test suites to test their DBMSs. For example, the SQLite test suites contain over 92 million lines of code. Despite these extensive efforts, test suites are not systematically reused across DBMSs, leading to wasted effort. Integration is challenging, as test suites use various test case formats and rely on unstandardized test runner features. We present a unified test suite, SQuaLity, in which we integrated test cases from three widely-used DBMSs, SQLite, PostgreSQL, and DuckDB. In addition, we present an empirical study to determine the potential of reusing these systems' test suites. Our results indicate that reusing test suites is challenging: First, test formats and test runner commands vary widely; for example, SQLite has 4 test runner commands, while MySQL has 112 commands with additional features, to, for example, execute file operations or interact with a shell. Second, while some test suites contain mostly standard-compliant statements (e.g., 99% in SQLite), other test suites mostly test non-standardized functionality (e.g., 31% of statements in the PostgreSQL test suite are nonstandardized). Third, test reuse is complicated by various explicit and implicit dependencies, such as the need to set variables and configurations, certain test cases requiring extensions not present by default, and query results depending on specific clients. Despite the above findings, we have identified 3 crashes, 3 hangs, and multiple compatibility issues across four different DBMSs by executing test suites across DBMSs, indicating the benefits of reuse. Overall, this work represents the first step towards test-case reuse in the context of DBMSs, and we hope that it will inspire follow-up work on this important topic.

GUI testing is one of the most important and significant testing approaches among all different software testing techniques. Most software errors are captured and detected through the software GUI layer. Manual testing for GUIs has its problems. It lacks in capturing all different cases and takes a huge time from the software tester to plan, design and re-design the testing suites in case of UI change. Old techniques in the area of test-case generation are not fully-automated or dependent on human inputs. This paper presents a test-case generation model to build a testing suite for webpages using its HTML file. The proposed model has two branches. The first one focuses on generating test cases for each web-element individually based on its type. The other branch focuses on generating test cases based on different paths between web-elements in the same webpage. It is also concerned with eliminating redundant test-cases using a supervised learning, feed-forward, dynamic artificial neural network that changes number of inputs according to generated cases per web page. The proposed system has been evaluated using several datasets. Results show a significant enhancement in the test-case generation procedure.

Context: 2nd generation (Layout-based) and 3rd generation (Visual) GUI testing are two approaches for testing mobile GUIs, both with individual benefits and drawbacks. Previous research has presented approaches to translate 2nd generation scripts to 3rd generation scripts but not the vice versa. Goal: The objective of this work is to provide Proof of Concept of the effectiveness of automatic translation between existing 3rd generation test scripts to 2nd generation test scripts. Method: A tool architecture is presented and implemented in a tool capable of translating most 3rd generation interactions with the GUI of an Android app into 2nd generation instructions and oracles for the Espresso testing tool.Results: We validate our approach on two test suites of our own creation, consisting of 30 test cases each. The measured success rate of the translation is 96.7% (58 working test cases out of 60 applications of the translator). Conclusion: The study provides support for the feasibility of a translation-based approach from 3rd to 2nd generation test cases. However, additional work is needed to make the approach applicable in real-world scenarios or larger open-source test suites.

Redundant test cases in newly generated test suites often remain undetected until execution and waste scarce project resources. In model-based testing, the testing process starts early on in the developmental phases and enables early fault detection. The redundancy in the test suites generated from models can be detected earlier as well and removed prior to its execution. The article presents a novel model-based test suite optimization technique involving UML activity diagrams by formulating the test suite optimization problem as an Equality Knapsack Problem. The aim here is the development of a test suite optimization framework that could optimize the model-based test suites by removing the redundant test cases. An evolution-based algorithm is incorporated into the framework and is compared with the performances of two other algorithms. An empirical study is conducted with four synthetic and industrial scale Activity Diagram models and results are presented.

Debugging and maintaining pragmatically reused test suites

There exists a direct need to automate the process of test suite generation to get the most optimal results as testing accounts for more than 40% of total cost. One method to solve this problem is the use of meta-heuristic algorithms which iteratively improve the test data to reach the most optimized test suites. This study focuses on the performance evaluation of six meta-heuristic algorithms namely: hill-climbing algorithm (HCA), particle swarm optimization (PSO), firefly algorithm (FA), cuckoo search algorithm (CS), bat algorithm (BA) and artificial bee colony algorithm (ABC) by using their standard implementation to optimize the path coverage and branch coverage produced by the test data. The goal of the study was to find the best-suited algorithm to narrow down the future research in the field of test automation for path coverage-based optimization approaches. Each algorithm was first implemented to automatically generate test suites based on the program under test. This was followed by the performance evaluation of each algorithm for five programs written in Java. The algorithms were compared using process metrics: average time, best time, worst time and product metrics: path coverage & objective function values of the generated test suites. Results indicated ABC as the best-suited algorithm as it gave the most optimal test suites in reasonable time. BA was found to be the fastest but produced less optimal results. FA was found to be the slowest algorithm, while CS, PSO and HCA performed in between. These results show the relative performance of the six algorithms for this scenario and may be used by the future researchers to narrow down and improve the best performing algorithms for path coverage-based optimization approaches.

Mutation-oriented test data augmentation for GUI software fault localization

An integral part of test case is the verification phase (also called `test oracle'), which verifies program's state, output or behavior. In automated testing, the verification phase is often implemented using test assertions which are usually developed manually by testers. More precisely, assertions are used for checking the unit or the system's behavior (or output) which is reflected by the changes in the data fields of the class under test, or the output of the function under test. Originated from human (testers') error, test suites are prone to having inadequate assertions. The paper reports an empirical study on the Inadequate-Assertion (IA) problem in the context of automated test suites developed for open-source projects. In this study, test suites of three active open-source projects have been chosen. To investigate IA problem occurrence among the sampled test suites, we performed mutation analysis and coverage analysis. The results indicate that: (1) the IA problem is common among the sampled open-source projects, and the occurrence varies from project to project and from package to package, and (2) the occurrence rate of the IA problem is positively co-related with the complexity of test code.

Software developers use testing to gain and maintain confidence in the correctness of a software system. Automated reduction and prioritization techniques attempt to decrease the time required to detect faults during test suite execution. This paper uses the Harrold Gupta Soffa, delayed greedy, traditional greedy, and 2-optimal greedy algorithms for both test suite reduction and prioritization. Even though reducing and reordering a test suite is primarily done to ensure that testing is cost-effective, these algorithms are normally configured to make greedy choices with coverage information alone. This paper extends these algorithms to greedily reduce and prioritize the tests by using both test cost (e.g., execution time) and the ratio of code coverage to test cost. An empirical study with eight real world case study applications shows that the ratio greedy choice metric aids a test suite reduction method in identifying a smaller and faster test suite. The results also suggest that incorporating test cost during prioritization allows for an average increase of 17% and a maximum improvement of 141% for a time sensitive evaluation metric called coverage effectiveness.

The feedback provided by current testing education tools about the\ndeficiencies in a student's test suite either mimics industry code coverage\ntools or lists specific instructor test cases that are missing from the\nstudent's test suite. While useful in some sense, these types of feedback are\nakin to revealing the solution to the problem, which can inadvertently\nencourage students to pursue a trial-and-error approach to testing, rather than\nusing a more systematic approach that encourages learning. In addition to not\nteaching students why their test suite is inadequate, this type of feedback may\nmotivate students to become dependent on the feedback rather than thinking for\nthemselves. To address this deficiency, there is an opportunity to investigate\nalternative feedback mechanisms that include a positive reinforcement of\ntesting concepts. We argue that using an inquiry-based learning approach is\nbetter than simply providing the answers. To facilitate this type of learning,\nwe present Testing Tutor, a web-based assignment submission platform that\nsupports different levels of testing pedagogy via a customizable feedback\nengine. We evaluated the impact of the different types of feedback through an\nempirical study in two sophomore-level courses. We use Testing Tutor to provide\nstudents with different types of feedback, either traditional detailed code\ncoverage feedback or inquiry-based learning conceptual feedback, and compare\nthe effects. The results show that students that receive conceptual feedback\nhad higher code coverage (by different measures), fewer redundant test cases,\nand higher programming grades than the students who receive traditional code\ncoverage feedback.\n

Android GUI testing is an important research field to maintain software quality of Android apps. Although many GUI testing schemes have been investigated in the past, the discussion of exploration event scheduling has not been investigated comprehensively for the event replay process. Based on the component priority relationship, this study proposes a novel exploration event scheduling scheme called CPR to reduce the number of the testing events in the replay process. Compared with the breadth-first traversal scheme, the proposed CPR scheme can reduce up to 62% of testing events for achieving the same component coverage, and up to 69% of testing events for layout traversal. Compared with the depth-first traversal scheme, CPR can reduce up to 15% of testing events for achieving the same component coverage, and up to 42% of testing events for layout traversal. With respect to the testing time, CPR can achieve the best performance for most of the AUTs in the empirical study. The results of the empirical experiments show that the proposed CPR scheduling scheme can have the benefits in improving the testing performance.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.

Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to decrease the size of the test suite. Yet, the difference in the execution cost of the tests is often significant and it may be costly to use a test suite consisting of a few long-running test cases. Thus, this paper proposes an algorithm, based on the concept of test irreplaceability, which creates a reduced test suite with a decreased execution cost. Leveraging widely used benchmark programs, the empirical study shows that, in comparison to existing techniques, the presented algorithm is the most effective at reducing the cost of running a test suite.

The fully automatic Graphical User Interface tool for any application using novel model based test suite generation techniques for a GUI. They are unable to control response time and time intervals are based on relationship between GUI events handlers and test cases with their responsibilities. We present a novel prioritization algorithm that enhances event handlers for the automated GUI tool. The proposed tool generates GUI events, it Captures and Playback event responses to automatic verification point of the results for the test cases which are written to a log file and corresponding report will be generated. This novel algorithm was able to detect new test suite and ordering of test cases to reduce a GUI fault integration defects. The number of faults detected for a single event are found after generating test cases for the application. The Average Percentage of Fault Detection (APFD) and charts has been used to show the effectiveness of proposed algorithm to find fault detection rate.