Abstract test case generation for behavioural testing of software product lines

Cost-effective test case generation with the hyper-heuristic for software product line testing

Welcome to the 13th edition of the Advances in Model-based testing (A-MOST 2017) held on March 17th, 2017 in beautiful Tokyo, Japan. Context of A-MOST: complexity of software systems is increasing rapidly and thus warrants the development of novel automated testing techniques that can manage the increased complexity of software. Model-Based Testing (MBT) is one such approach towards automated testing that allows managing the complexity of the Software under Test with the use of models. A recent survey shows that MBT has been increasing, being used both in academic and in industrial settings and thus still remains an interesting area of research. In the past 12 years, A-MOST has served as a good venue for interaction among model-based testing researchers from both academia and industry. In 2017, A-MOST received in total eight submissions, and six of them were accepted after a thorough review and discussion process. Each submission was reviewed by three members of the program committee. topics of accepted papers cover security testing, test case generation, conformance checking, software product line testing and usability testing. accepted papers are divided into two main sections, i.e., functional MBT and non-functional MBT that will be presented during the workshop. Moreover, we are also delighted to have two well-renowned MBT researchers as keynote speakers. first keynote speaker is Ina Schieferdecker who is the director of Fraunhofer FOKUS, Germany. topic of her keynote speech is The New Version of the UML Testing Profile. second keynote speaker is Shaukat Ali who is currently a senior research scientist from Simula Research Laboratory, Norway. His theme of the keynote speech is Uncertainty-Wise Testing. We thank them very much for accepting our invitations for their inspiring keynote speeches. We also appreciate the effort made by our program committee members for providing their timely reviews with high quality. Last but not least, we would like to thank the A-MOST steering committee for their support and advice. Last, we thank all the authors who submitted their high quality work to A-MOST 2017.

Test case generation, selection and coverage from natural language

Software product line (SPL) testing consists of two separate but closely related test engineering activities: domain testing and application testing. Various software product line testing approaches have been developed over the last decade, and surveys have been conducted on them. However, thus far none of them deeply addressed the questions of what researches have been conducted in order to overcome the challenges posed by the two separate testing activities and their relationships. Thus, this paper surveys the current software product line testing approaches by defining a reference SPL testing processes and identifying, based on them, key research perspectives that are important in SPL testing. Through this survey, we identify the researches that addressed the challenges and also derive open research opportunities from each perspective.

State-based testing (SBT) is known as deriving test cases from state machines and examining the dynamic behaviour of the system. It helps to identify various types of state-based faults within a system under test (SUT). For SBT, test cases are generated from state chart diagrams based on various coverage criteria such as All Transition, Round Trip Path, All Transition Pair, All Transition Pair with length 2, All Transition Pair with length 3, All Transition Pair of length 4 and Full Predicate. This article discuses a number of coverage criteria at the design level to find out various types of state-based faults in SBT. First, the intermediate graph is generated from a state chart diagram using an XML parser. The graph is traversed based on the given coverage criteria to generate a sequence of test cases. Then, mutation testing and sneak-path testing are applied on the generated test cases to check the effectiveness of the generated test suite. These two are common methods for checking the effectiveness of test cases. Mutation testing helps in the number of seeded errors covered whereas sneak-path testing basically helps to examine the unspecified behavior of the system. In round trip path (RTP), it is not possible to cover all paths. All transition is not an adequate level of fault detection with more execution time compared to all transition pair (ATP) with length 4 (LN4). In the discussion, ATP with LN4 is the best among all coverage criteria. SBT can able to detect various state-based faults-incorrect transition, missing transition, missing or incorrect event, missing or incorrect action, extra missing or corrupt state, which are difficult to detect in code-based testing. Most of these state-based faults can be avoided, if the testing is conducted at the early phase of design.

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.

Software product line (SPL) testing is a challenging task, due to the huge number of variants sharing common functionalities to be taken into account for efficient testing. By adopting the concept of regression testing, incremental SPL testing strategies cope with this challenge by exploiting the reuse potential of test artifacts between subsequent variants under test. In previous work, we proposed delta-oriented test case prioritization for incremental SPL integration testing, where differences between architecture test model variants allow for reasoning about the order of reusable test cases to be executed. However, the prioritization left two issues open, namely (1) changes to component behavior are ignored, which may also influence component interactions and, (2) the weighting and ordering of similar test cases result in an unintended clustering of test cases. In this paper, we extend the test case prioritization technique by (1) incorporating changes to component behavior allowing for a more fine-grained analysis and (2) defining a dissimilarity measure to avoid clustered test case orders. We prototyped our test case prioritization technique and evaluated its applicability and effectiveness by means of a case study from the automotive domain showing positive results.

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

The complexity of asynchronous model based testing

In model-based testing, test cases are generated from a specification model. To avoid an exhaustive search for all possible test cases that can be obtained, usually an expensive and infeasible activity, test case generation may be guided by a test selection criterion. The objective of a test selection criterion is to produce a minimal test suite and yet effective to reveal faults. However, the choice of a criterion is not straightforward specially for real-time systems, because most criteria presented in the literature are general-purpose. Moreover, the relationship between general-purpose and specific criteria for real-time systems is not clear. In this paper, we investigate the criteria that can be applied for test case generation in the scope of model-based testing of real-time systems, specifically of Timed Input-Output Symbolic Transition Systems (TIOSTS) models. We formalize a family of 19 test selection criteria ordered by strict inclusion relation for TIOSTS models. The family combines general-purpose data-flow-oriented and transition-based criteria with specific reactive and real-time systems criteria. We also perform an empirical study to compare the effectiveness of selected criteria. Results of the empirical study indicate that failure detection capability of the generated test suite may vary, but differences are not significant for time failures. We conclude that more effective criteria for the model-based testing of real-time systems are still needed.

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Software Product Lines (SPLs) are not intended to create one application, but a number of them: a product family. In contrast to one-off development, SPLs are based on the idea that the distinct products of the family share a significant amount of assets. This forces a change in how software is developed. Likewise, software testing should mimic its code counterpart: product testing should also be produced out of a common set of assets. Specifically, this paper addresses how model-driven testing, used for one-off development, can be moved to an SPL setting. We focus on feature-oriented software development as the SPL realization technique. UML sequence diagrams are used to represent the common and feature scenarios. This models are transformed through model transformations to obtain test cases that conform to the UML Testing Profile.

Variability of modern software systems increases potential sources of errors and demands appropriate quality assurance strategies. In order to reduce the test effort when testing software product lines, incremental model-based testing strategies have been proposed, based on the conceptual ideas of delta modeling. It requires executable system specifications to derive and classify test cases. However, in industrial practice such system models rarely exist, but requirements and test cases are captured in natural language. In order to make delta-oriented testing strategies applicable in this context, we transfer them to the requirements level and show how a delta-oriented classification of requirements and associated test cases reduce test effort in these less formal domains.

Software Product Lines (SPL) are inherently difficult to test due to the\ncombinatorial explosion of the number of products to consider. To reduce the\nnumber of products to test, sampling techniques such as combinatorial\ninteraction testing have been proposed. They usually start from a feature model\nand apply a coverage criterion (e.g. pairwise feature interaction or\ndissimilarity) to generate tractable, fault-finding, lists of configurations to\nbe tested. Prioritization can also be used to sort/generate such lists,\noptimizing coverage criteria or weights assigned to features. However, current\nsampling/prioritization techniques barely take product behavior into account.\nWe explore how ideas of statistical testing, based on a usage model (a Markov\nchain), can be used to extract configurations of interest according to the\nlikelihood of their executions. These executions are gathered in featured\ntransition systems, compact representation of SPL behavior. We discuss possible\nscenarios and give a prioritization procedure illustrated on an example.\n

On the order of test goals in specification-based testing