Towards statistical prioritization for software product lines testing

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.

Formal semantics, modular specification, and symbolic verification of product-line behaviour

Software Product Lines (SPLs) are families of similar software products built from a common set of features. As the number of products of an SPL is potentially exponential in the number of its features, the model checking problem is harder than for single software. A practical way to face this exponential blow-up is to reuse common behaviour between products. We previously introduced Featured Transition Systems (FTS), a mathematical model that serves as a basis for efficient SPL model checking techniques. In this paper, we present ProVeLines, a product line of verifiers for SPLs that incorporates the results of over three years of research on formal verification of SPLs. Being itself a product line, our tool is flexible and extensible, and offers a wide range of solutions for SPL modelling and verification.

In Model Based Testing (MBT), test cases are generated automatically from a partial representation of expected behaviour of the System Under Test (SUT) (i.e., the model). For most industrial systems, it is impossible to generate all the possible test cases from the model. The test engineer recourse to generation algorithms that maximize a given coverage criterion, a metric indicating the percentage of possible behaviours of the SUT covered by the test cases. Our previous work redefined classical Transition Systems (TSs) criteria for SPLs, using Featured Transition Systems (FTSs), a mathematical structure to compactly represent the behaviour of a SPL, as model for test case generation. In this paper, we provide one all-states coverage driven generation algorithm and discuss its scalability and efficiency with respect to random generation. All-states and random generation are compared on fault-seeded FTSs.

A Featured Transition System (FTS) models the behaviour of all products of a Software Product Line (SPL) in a single compact structure, by associating action-labelled transitions with features that condition their presence in product behaviour. It may however be the case that the resulting featured transitions of an FTS cannot be executed in any product (so called dead transitions) or, on the contrary, can be executed in all products (so called false optional transitions). Moreover, an FTS may contain states from which a transition can be executed only in some products (so called hidden deadlock states). It is useful to detect such ambiguities and signal them to the modeller, because dead transitions indicate an anomaly in the FTS that must be corrected, false optional transitions indicate a redundancy that may be removed, and hidden deadlocks should be made explicit in the FTS to improve the understanding of the model and to enable efficient verification—if the deadlocks in the products should not be remedied in the first place. We provide an algorithm to analyse an FTS for ambiguities and a means to transform an ambiguous FTS into an unambiguous one. The scope is twofold: an ambiguous model is typically undesired as it gives an unclear idea of the SPL and, moreover, an unambiguous FTS can efficiently be model checked. We empirically show the suitability of the algorithm by applying it to a number of benchmark SPL examples from the literature, and we show how this facilitates a kind of family-based model checking of a wide range of properties on FTSs.

Software product line (SPL) is a software engineering paradigm for software development. A software product within a product line often has specific functionalities that are not common to all other products within the product line. Those specific functionalities are termed “variant features” in a product line. SPL paradigm involves the modeling of variant features. However, little work in SPL investigates and addresses the modeling of variant features specific to user interface (UI). Unified Modeling Language (UML) is the de facto modeling language for object-oriented software systems. It is known that UML needs better support in modeling UIs. Thus, much research developed UML extensions to improve UML support in modeling UIs. Yet little of this work is related to developing such extensions for modeling UIs for SPLs in which variant features specific to UI modeling must be addressed. This research develops a UML extension -Web User Interface Modeling Language (WUIML) to address these problems. WUIML defines elements for modeling variant features specific to user interfaces for Web-based SPLs. The model elements in WUIML extend from the metaclass and BasicActivity of the UML2.0 metamodel. WUIML integrates the modeling of variant features specific to user interfaces to UML. For example, in a Web-based patient registration software product line, member products targeting British users may use British date format in the user interface, while member products targeting United States users may use United States date format in the user interface. Thus, this is a variant feature for this product line. WUIML defines a model element, XOR, to represent such exclusive or conditions in a product line user interface model. WUIML would reduce SPL engineers’ efforts needed in UI development. To validate the WUIML research outcome, a case study was conducted. The results of this empirical study indicate that modeling UIs for Web-based SPLs using WUIML is more effective and efficient than using standard UML.

Extensive work on Search-Based Software Testing for Software Product Lines has been published in the last few years. Salient among them is the use of similarity as a surrogate metric for t-wise coverage whenever higher strengths are needed or whenever the size of the test suites is infeasible because of technological or budget limitations. Though promising, this metric has not been assessed with real fault data. In this paper, we address this limitation by using Drupal, a widely used open source web content management system, as an industry-strength case study for which both variability information and fault data have been recently made available. Our preliminary assessment corroborates some of the previous findings but also raises issues on some assumptions and claims made. We hope our work encourages further empirical evaluations of Combinatorial Interaction Testing approaches for Software Product Lines.

A Software Product Line (SPL) is a set of systems that share a group of manageable features and satisfy the specific needs of a particular domain. The features of an SPL can be used in variable combinations to derive product variants in the SPL domain. Because SPLs promote product development through reuse, it is vital to have a means to measure their quality in terms of quality attributes like complexity, reusability,… In this paper, we propose a set of metrics to evaluate the quality of an SPL at three levels: the feature model, design and code. We adapted a set of metrics for software quality and defined new metrics to deal with the inherent characteristics of SPLs, specifically the feature model and the traceability between features, design and code. Furthermore, to assist in interpreting the quality of a given SPL, we conducted an empirical study over ten open source SPLs to identify thresholds for the proposed metrics.

Automating feature model refactoring: A Model transformation approach

Automated diagnosis of feature model configurations

Goal models can capture similarities and the variability of a Software Product Line (SPL), but they cannot describe the detailed behavior of the SPL functionality. Due to this limitation, a process called GS2SPL was defined to systematically obtain, from goal models, feature models and the specification of use case scenarios with variability. However, the variability and the configuration knowledge of the SPL are tangled in the scenarios description, jeopardizing the maintenance and reuse of this artifact. In order to handle with this problem, it was proposed a technique called MSVCM. It specifies the SPL variability and the configuration knowledge separately from the common behavior present in the scenarios, as well as it defines a process to configure the specification of a product. However, there is a lack of traceability between the MSVCM scenarios and the goals and quality that the stakeholder expects to achieve. Thus, this work proposes to obtain, systematically, the specification of MSVCM scenarios from goal models. Using goals in SPL allows a systematic derivation of the SPL requirements from these goals. This new approach is called GAS2SPL and it is illustrated by using the TaRGeT example.

Lately, the demand for mobile applications development has increased significantly mainly due to growth of use of mobile devices and the need to port existing web applications. To reduce development’s time and cost, Software Product Lines (SPLs) have also been used in the context of mobile applications. However, the existing SPLs do not worry about supporting the development of mobile applications corresponding to the existing Web applications, as it is desirable to have access to the information and main features of these applications in mobile devices. In face of this problem, this paper discusses the motivation and presents the evolution from a SPL in the e-Gov Web (e-Gov Web SPL) domain to a SPL in the mobile domain (e-Gov Mobile SPL) having in mind the need to supply market demand. The conducted evolution was supported by the PLUS approach (Product Line UML-Based Software Engineering) and by the features model. Furthermore, this work debates the main results obtained through some e-Gov Mobile SPL instantiations in the precision livestock domain.KeywordsSoftware Product LineEvolutione-GovWebMobile

Needs for mass customization and economies of scale have pushed engineers to develop Software Product Lines (SPLs). SPLs are families of products sharing commonalities and exhibiting differences, built by reusing software assets abstractly represented by features. Feature models describe the constraints that link the features and allow the configuration of tailored software products. Common SPLs involve hundreds, even thousands of features, leading to billions of possible software products. As a result, testing a product line is challenging due to the enormous size of the possible products. Existing techniques focus on testing based on the product line's feature model by selecting a limited set of products to test. Being created manually or reverse-engineered, feature models are prone to errors impacting the generated test suites. In this paper, we examine ability of test suites to detect such errors. In particular, we propose two mutation operators to derive erroneous feature models (mutants) from an original feature model and assess the capability of the generated original test suite to kill the mutants. Experimentation on real feature models demonstrate that dissimilar tests suites have a higher mutant detection ability than similar ones, thus validating the relevance of similarity-driven product line testing.

Test scenario generation for feature-based context-oriented software systems

Software Product Line Engineering aims to ensure the correctness, completeness and consistency among its artifacts and the specified domain, in order to prevent the spread of defects for the products derived from this domain. Among initial artifacts of a software product line, feature models are artifacts generated in various domain engineering approaches. Although software inspection is highlighted as an effective review activity for detection of defects in software artifacts, mainly in the early models of software projects, a recent quasi-systematic review of literature indicated a lack of techniques to support the inspection of software product line artifacts, which include features models. Thus, this paper presents FMCheck, a checklist-based inspection technique to support the detection of defects on feature models. This technique was developed to be configurable and to be applied on several extensions of the original feature model notation presented by FODA approach, including the Odyssey-FEX notation, in particular. FMCheck was submitted to a proof of concept and to a further in vitro feasibility study, in which it could be seen the feasibility of FMCheck application and also that inspections applying FMCheck are more effective than ad-hoc inspections, when applied on four distinct domains.