Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes

Chapter 6 - Trends in Model-based GUI Testing

Automation of acceptance test suites becomes necessary in the context of agile software development practices, which require rapid feedback on the quality of code changes. To this end, companies try to automate their acceptance tests as much as possible. Unfortunately, the growth of the automated test suites, by several automation testers, gives rise to potential test smells, i.e., poorly designed test code, being introduced in the test code base, which in turn may increase the cost of maintaining the code and creating new one. In this paper, we investigate this problem in the context of our industrial partner, BGL BNP Paribas, and introduce Ukwikora, an automated tool that statically analyzes acceptance test suites, enabling the continuous inspection of the test code base. Ukwikora targets code written in the Robot Framework syntax, a popular framework for writing Keyword-Driven tests. Ukwikora has been successfully deployed at BGL BNP Paribas, detecting issues otherwise unknown to the automation testers, such as the presence of duplicated test code, dead test code and dependency issues among the tests. The success of our case study reinforces the need for additional research and tooling for acceptance test suites.

Manually testing GUIs can be expensive and complex, so the creation of automated GUI test suites has been an area of significant interest. However, to our knowledge, the motivations of testers and the problems they encounter when attempting to create and use automated GUI tests have not been explored. We used Grounded Theory to investigate the goals motivating automated GUI testing, the issues testers encounter, and the best practices applied to overcome these issues. Through this study, we demonstrate that automated GUI test suite evolution and architecture are extremely important to the success of automated GUI testing and describe techniques that can be of use to practitioners. In addition to these best practices, this study identifies additional areas in which future research should be concentrated. Keywordsautomated GUI test; test suite architecture; test suite evolution; best practices;empirical study

Ensuring the quality of mobile applications (apps) needs to explore the GUI thoroughly. In practice, exhaustively exploring every GUI widget is unscalable on large real-world apps since it usually suffers from the problem of widgets explosion. To mitigate the problem, many existing testing tools usually detect and group homogeneous widgets heuristicly with different level of model abstraction since these widgets behave the same. However, no heuristic always works well. Heterogeneous widgets with divergent behaviors can be mistakenly grouped, which largely limits the testing effectiveness. This paper proposes a technique to effective GUI testing of Android apps with dynamic feedback-directed widgets grouping. Initially, we group the widgets according to the structure of the GUI. During testing, we observe behaviors of widgets in a group and regroup improperly-grouped widgets dynamically. Then, we apply a feedback-directed strategy to effectively accelerate the GUI exploration. The proposed technique is implemented as a practical tool for Android apps, named WGDroid. We evaluated WGDroid on 17 widely-used Android apps and compared it with the state-of-the-art GUI testing tools, i.e., AimDroid, SAPIENZ, and Monkey on both emulators and real devices. WGDroid outperformed the three tools in all testing coverages and also detected the most unique crashes. In particular, WGDroid discovered 208 more activities on 12 large benchmark apps on real devices and 11 more activities on another 5 benchmark apps on emulators, than the best of the other tools. These results show that WGDroid can significantly accelerate the GUI exploration.

An integral part of test case is the verification phase (also called `test oracle'), which verifies program's state, output or behavior. In automated testing, the verification phase is often implemented using test assertions which are usually developed manually by testers. More precisely, assertions are used for checking the unit or the system's behavior (or output) which is reflected by the changes in the data fields of the class under test, or the output of the function under test. Originated from human (testers') error, test suites are prone to having inadequate assertions. The paper reports an empirical study on the Inadequate-Assertion (IA) problem in the context of automated test suites developed for open-source projects. In this study, test suites of three active open-source projects have been chosen. To investigate IA problem occurrence among the sampled test suites, we performed mutation analysis and coverage analysis. The results indicate that: (1) the IA problem is common among the sampled open-source projects, and the occurrence varies from project to project and from package to package, and (2) the occurrence rate of the IA problem is positively co-related with the complexity of test code.

GUI testing is one of the most important and significant testing approaches among all different software testing techniques. Most software errors are captured and detected through the software GUI layer. Manual testing for GUIs has its problems. It lacks in capturing all different cases and takes a huge time from the software tester to plan, design and re-design the testing suites in case of UI change. Old techniques in the area of test-case generation are not fully-automated or dependent on human inputs. This paper presents a test-case generation model to build a testing suite for webpages using its HTML file. The proposed model has two branches. The first one focuses on generating test cases for each web-element individually based on its type. The other branch focuses on generating test cases based on different paths between web-elements in the same webpage. It is also concerned with eliminating redundant test-cases using a supervised learning, feed-forward, dynamic artificial neural network that changes number of inputs according to generated cases per web page. The proposed system has been evaluated using several datasets. Results show a significant enhancement in the test-case generation procedure.

Context: 2nd generation (Layout-based) and 3rd generation (Visual) GUI testing are two approaches for testing mobile GUIs, both with individual benefits and drawbacks. Previous research has presented approaches to translate 2nd generation scripts to 3rd generation scripts but not the vice versa. Goal: The objective of this work is to provide Proof of Concept of the effectiveness of automatic translation between existing 3rd generation test scripts to 2nd generation test scripts. Method: A tool architecture is presented and implemented in a tool capable of translating most 3rd generation interactions with the GUI of an Android app into 2nd generation instructions and oracles for the Espresso testing tool.Results: We validate our approach on two test suites of our own creation, consisting of 30 test cases each. The measured success rate of the translation is 96.7% (58 working test cases out of 60 applications of the translator). Conclusion: The study provides support for the feasibility of a translation-based approach from 3rd to 2nd generation test cases. However, additional work is needed to make the approach applicable in real-world scenarios or larger open-source test suites.

It is the demand of our ever-advancing IT industry that software be updated in order to continue its use. Such a modification should not introduce any unwanted new faults in the system. For this, the existing test suite needs to be rerun, often called as regression testing. The main challenge during the regression testing process is not to exceed the desired time and budge deadlines. As a consequence various techniques such as test case selection, minimization and prioritization are used. This paper proposes and analyzes the effect of time constraint on an ant colony optimization based technique for Regression test selection and prioritization. It has been found that with an increase in the applied time constraint, there are more chances to get an optimum selected and prioritized test suite. Also it was found that the complexity of our algorithm depends on the size of the test suite and the applied time constraint and is independent of the number of faults being mutated or any other input variable.

Refactoring as a process is aimed at improving the quality of a software system while preserving its external behavior. In practice, refactoring comes in the form of many specific and diverse refactoring operations, which have different scopes and thus a different potential impact on both the production and the test code. We present a large-scale quantitative study complemented by a qualitative analysis involving 615,196 test cases to understand how and to what extent different refactoring operations impact a system's test suites. Our findings show that while the vast majority of refactoring operations do not or very seldom induce test breaks, some specific refactoring types (e.g., “RENAME Attribute” and “RENAME Class”) have a higher chance of breaking test suites. Meanwhile, “ADD Parameter” and “CHANGE Return Type” refactoring operations often require additional lines of changes to fix the test suite they break. While some modern IDEs provide features to automatically apply these two types of refactoring operations, they are not always able to avoid test breaks, thus demanding extra human efforts.

Mutation testing is a technique in which faults (mutants) are injected into a program or application to assess its test suite effectiveness. It works by inserting mutants and running the application’s test suite to identify if the mutants are detected (killed) or not (survived) by the tests. Although computationally expensive, it has proven to be an effective method to assess application test suites. Several mutation testing frameworks and tools have been built for the various programing languages, however, very few tools have been built for the JavaScript language, more specifically, there is a lack of mutation testing tools for the Node.js runtime and npm based applications. The npm Registry is a public collection of modules of open-source code for Node.js, front-end web applications, mobile applications, robots, routers, and countless other needs of the JavaScript community. The over 700,000 packages hosted in npm are downloaded more than 5 billion times per week. More and more software is published in npm every day, representing a huge opportunity to share code and solutions, but also to share bugs and faulty software. In this paper, we briefly describe prior work for mutation operators in JavaScript and Node.js, and propose Mutode, an open source tool which leverages the npm package ecosystem to perform mutation testing for JavaScript and Node.js applications. We empirically evaluated Mutode effectiveness by running it on 12 of the top 20 npm modules that have automated test suites.

Many programming languages have a system of garbage collection to automate memory management. Problems such as unexpected memory leak still may occur due to the automation of memory management. Weak reference has been proposed as a solution to such problems. However, the use of weak reference often requires consideration of memory release timing, which is difficult for developers. In this study, we investigate the use of weak reference on open source projects. More concretely, we investigated the domain of software projects where the weak reference was used, the timing of the introduction of the weak reference, the usage method, the presence or absence of the test code for weak reference, and the case of failure to introduce weak reference in the Java language. The survey showed that weak references were used in 73 repositories, about one-third of the total 202 repositories, and that weak references were most common in test code.

In this chapter we present Verification-Aided Regression Testing, a novel extension of regression testing that is significantly less sensitive to the completeness of the validation test suite due to the use of model checking. We extend the use of test case executions from conventional direct fault discovery to generation of behavioral properties specific to the new version by (i) automatically producing properties that are proved to hold for the base version of a program, (ii) automatically identifying and checking on the upgraded program only the properties that, according to the developers’ intention, must be preserved by the upgrade, and (iii) reporting the faults and the corresponding counterexamples that are not revealed by the regression tests. Our empirical study on both open-source and industrial software systems show that Verification-Aided Regression Testing produces properties that can be extremely beneficial in increasing the effectiveness of regression testing by timely and automatically detecting faults unnoticed by existing test suites.

In the software industry, software testing becomes more important in the entire software development life cycle. Software testing is one of the fundamental components of software quality assurances. Software Testing Life Cycle (STLC)is a process involved in testing the complete software, which includes Regression Testing, Unit Testing, Smoke Testing, Integration Testing, Interface Testing, System Testing & etc. In the STLC of Regression testing, test case selection is one of the most important concerns for effective testing as well as cost of the testing process. During the Regression testing, executing all the test cases from existing test suite is not possible because that takes more time to test the modified software. This paper proposes new Hybrid approach that consists of modified Greedy approach for handling the test case selection and Genetic Algorithm uses effective parameter like Initial Population, Fitness Value, Test Case Combination, Test Case Crossover and Test Case Mutation for optimizing the tied test suite. By doing this, effective test cases are selected and minimized the tied test suite to reduce the cost of the testing process. Finally the result of proposed approach compared with conventional greedy approach and proved that our approach is more effective than other existing approach.

This paper presents a selective regression testing technique and an associated tool for object-oriented software. The technique is based on the concept of Control Call Graphs, which are a reduced form of traditional Control Flow Graphs. It uses static analysis of the source code of the program. The developed tool (1) identifies the Control Call Paths potentially impacted by changes, (2) selects, from an existing test suite, the appropriate test cases, and (3) generates new JUnit test cases for control call paths that are not covered by existing tests (new ones, or those whose structure has been modified after changes). In this way, the approach supports an incremental update of the test suite. The selected JUnit test cases, including the new ones, are automatically executed. Three concrete case studies are reported to provide evidence of the feasibility of the approach and its benefits in terms of reduction of regression testing effort.

Most software companies have extensive test suites and re-run parts of them continuously to ensure that recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This article analyses two decades of TCP research and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to system under test code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The TePIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.