Graph-aided directed testing of Android applications for checking runtime privacy behaviours

Although Android apps are becoming more and more popular, apps tend to contain defects which can ultimately manifest as failures (or crashes) to end-users. In order to improve software quality, people have proposed different automated tools for testing Android applications. The automatic generation of test cases is an important means to improve the efficiency of software testing, but many tools related to test case generation do not seem to be fully adapted to the problem of Android test case generation. Therefore, this article aims to design and evaluate the automatic generation method of Android application test cases based on the multi-dimensional coverage. We conducted test experiments on 48 self-developed Android apps through the Gradle plugin, and the final average target coverage reached about 32%. Considering that the current component modules are not yet complete, we believe that tools will reach even higher coverage in the future.

Automated-test-generation tools generate test cases to enable dynamic analysis of Android apps, such as functional testing. These tools build a GUI model to describe the app states during the app execution, and generate a script that performs actions on UI widgets to form a test case. However, when the test cases are re-executed, the apps under analysis often do not behave consistently. The major reasons for such limited reproducibility are due to (1) backend-service dependencies that cause non-determinism in app behaviors and (2) the severe fragmentation of Android platform (i.e., the alarming number of different Android OS versions in vendor-customized devices). To address these challenges, we design and implement Paladin, a novel system that generates reproducible test cases for Android apps. The key insight of Paladin is to provide a GUI model that leverages the structure of the GUI view tree to identify equivalent app states, since the structure can tolerate the changes on the UI contents for an app behavior performed in different test executions. Based on the model, Paladin can search the view tree to locate the desired UI widgets to trigger events and drive the app exploration to reach the desired app states, making the test cases reproducible. Evaluation results on real apps show that Paladin could reach a much higher reproduction ratio than the state-of-the-art tools when the generated test cases are re-executed across different device configurations. In addition, benefiting from the reproducible capability, Paladin is able to cover more app behaviors compared with the existing tools.

As we know, software testing is an important part of software development lifecycle. More than 50% of the whole system development work and total cost were spend on the software testing. And it’s estimated that almost 60% of the total test time and cost were spent on the design of test cases. In recent years, the automated testing gradually replaces the traditional manual testing to become an important branch of software testing. In automated testing, the most important thing is to design and generate valid test case automatically. In this paper, In order to reduce the workload of testers on the test case generation, improve the test efficiency and makes test experiences be passed on, we design a common description method for the test case based on ontology; propose a semantic similarity measure method to retrieve the usable test cases from test case library based on WordNet; establish the relationship between the test case and test requirement though rules to modify the retrieved test case; generate the final test case sequence. At last, the practical applicability of the approach is evaluated through an experiment. Keywords-software testing; ontology; test case generation; semantic similarity measure

Maximum coverage with minimum testing time is the main objective of a test case generation activity which leads to a multi-objective problem. Search-Based Testing (SBT) technique is a demanding research area for test case generation. Researchers have applied various metaheuristic (searching) algorithms to generate efficient and effective test cases in many research works. Out of these existing search-based algorithms, Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are the most widely used algorithms in automatic test case generation. In this paper, a test case generation approach is proposed using Cuckoo Search (CS) algorithm. CS has a controlling feature, Levy flights, which makes it more efficient in searching the best candidate solution. It helps to generate efficient test cases in terms of code coverage and execution time. In our proposed method, test cases are generated based on path coverage criteria. Fitness of a test case is evaluated using branch distance and approximation level combined functions. The result is compared with PSO and with its variant Adaptive PSO (APSO). The experimental result shows that both the algorithms give nearly equal to the same result. Though the results are nearly equal, the implementation of CS is simple as it requires only one parameter to be tuned.

Services are highly reusable, flexible and loosely coupled, which makes the evolution and the maintenance of composite services more complex. Evolution of BPEL composite service covers changes of processes, bindings and interfaces. In this paper, an approach is proposed to select and generate test cases during the evolution of BPEL composite service. The approach identifies the changes by using control-flow analysis technique and comparing the paths in new composite service version and the old one using extensible BPEL flow graph (or XBFG). Message flow is appended to the control flow so that XBFG can describe the behavior of composite service integrally. The binding and predicate constraint information added in XBFG elements can be used in path selection and test case generation. Theory analysis and case study both show that the approach is effective, and test cases coverage rate is high for the changes of processes, bindings and interfaces.

Test cases are essential parts to perform software testing as they play very significant role defining the test steps to work. Test cases can be generated either manually or automatically. Creating test cases manually has several disadvantages, such as higher probability in producing inaccurate results. Furthermore, generating test cases manually is ineffective since manual methods will take longer and require high human efforts. Therefore, generating test cases automatically is more preferable to limit the previous problems. Automated testing can also provide more accurate results than manual testing because it is less prone to human errors encouraging software developers to generate test cases automatically. Test cases can be generated from various sources, including UML diagrams. In this paper, we propose a method to generate test cases automatically by utilising depth-first search algorithms. The UML diagram used in this study is an activity diagram. This paper also provides a method to parse activity diagram that is created using the visual paradigm application that has been converted into an XML format. Our method is able to parse and convert the XML format into graph or tree form. Using our proposed method, a test case search will be carried out using the depth-first search on the tree form to generate test cases. Lastly, we also evaluate our method by comparing the generated test cases with cyclomatic complexity, the method will be performed manually and automatically to validate the results of many scenarios. Our result showed that our method could generate the same scenario (100% coverage) as the manual validation method.

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

This work aims at defining a procedure and a set of mechanisms able to improve the quality of the code coverage in automated software reverse engineering processes, and specifically in automated GUI-driven testing of Android apps. Existing automated model-based testing techniques, based on reverse engineering, generate test cases which can be executed directly on the software's GUI. We propose to augment the code coverage of these techniques, by exploiting information from patterns, defined at different levels (application design, state-based model, interaction with Android services), and generating additional test cases that may increase the coverage capability of GUI-Ripping based testing technique. The generation of the additional test cases is accomplished by defining an automatable procedure which exploits an existing GUI testing approach and a pattern based approach used in a different context.

Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore their impacts need to be considered when testing the apps. Unfortunately, few test cases explicitly specify the assignments of valid values to the preferences, or configurations , under which they should be executed, and few existing mobile testing tools take the impact of preferences into account or provide help to testers in identifying and setting up the configurations for running the tests. This article presents the Prefest approach to effective testing of Android apps with preferences. Given an Android app and a set of test cases for the app, Prefest amplifies the test cases with a small number of configurations to exercise more behaviors and detect more bugs that are related to preferences. In an experimental evaluation conducted on real-world Android apps, amplified test cases produced by Prefest from automatically generated test cases covered significantly more code of the apps and detected seven real bugs, and the tool’s test amplification time was at the same order of magnitude as the running time of the input test cases. Prefest ’s effectiveness and efficiency in amplifying programmer-written test cases was comparable with that in amplifying automatically generated test cases.

SummaryThis article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Back's action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright © 2014 John Wiley & Sons, Ltd.

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as much as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5\times\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.

Automated testing techniques for software design specification include model-based approaches that generate test cases from a coding or state machine model of the interface, capture-replay methods that require the user to specify and demonstrate each test case individually, and modelling-language approaches that provide templates for abstract test cases. In this paper elucidates a method for automatic software analysis testing for IoT service. Nowadays real-time IoT software is used to satisfy the needs and functions of a user and it is assisted in this by the hardware. Software testing is generally used to detect bugs and errors and eliminate them to produce a high quality, reliable software. The automated testing depends on the kind of approach taken. This testing may be poorly implemented which results in a low quality software or it may be effectively implemented which results in a high quality software. Automated IoT software testing can decrease time and cost. When these techniques are identified, the cost and time can be reduced. The automated testing is based on three steps such as analyzing the test case, test execution and log analyzer. These steps provide user friendly software with high standard. The automated process provides test reusability and repeatability and saves the test coverage cost and time. In this paper compare the automated software testing for IoT transactions and system peripherals such as sensor, DMA controller, internal counters, physical layer, and virtual layer using IoT specification based software testing.

The amount of test cases in real life software testing is huge. There are many studies, such as classification tree and class vector, on generating all the test cases at a time. The practicability of these studies is low as the testers are difficult to manage that huge amount of test cases. When testing a subset of functions of a system or testing the part of systems that have been modified, it would be more practical and convenience for testers if only all the concerned test cases are generated on demand. In order to generate test cases on demand, first we need a modeling language which is capable to model the input. The modeling language should be elegant and convenient such that testers can focus in the testing. This modeling language should support generating test cases according to some criteria as well. Furthermore, the model language should be implementable for computer aided test cases generation. We found hypergraph is a good candidate as the modeling language. In this paper, we report our study on using hypergraph as the modeling language for generating test cases on demand.

Model-Driven Engineering (MDE) supports model evolution and refinement by means of model transformations at several abstraction levels. Validating these transformations is essential to ensure the quality and correctness of such models. However, MDE transformations become more complex to validate, for example, when they are implemented in different languages. One particular example is the transformation of the SyMPLES approach. SyMPLES is a development approach for embedded systems, which is based on concepts of both Software Product Lines (SPL) and MDE. SyMPLES has a model transformation process which creates Simulink models from SysML models. This paper presents a case study which applies test case generation based on SPL to validate this model transformation. An SPL was used to generate a set of test cases based on coverage criteria. The results showed that the test cases generated uncovered errors in the transformation of SyMPLES. In addition, a comparison with the test case generation based on metamodel is presented, in order to analyze the effectiveness of the techniques. The coverage criteria made it possible to reduce the number of test cases generated, thus minimizing test effort and time.