Fault-based product-line testing

Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highly-configurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. Many CIT algorithms lack a mechanism to avoid these. In recent work, automated constraint solving methods have been combined with search-based CIT construction methods to address the constraint problem with promising results. However, these techniques can incur a non-trivial overhead. In this paper, we build upon our previous work to develop a family of greedy CIT sample generation algorithms that exploit calculations made by modern Boolean satisfiability (SAT) solvers to prune the search space of the CIT problem. We perform a comparative evaluation of the cost-effectiveness of these algorithms on four real-world highly-configurable software systems and on a population of synthetic examples that share the characteristics of those systems. In combination our techniques reduce the cost of CIT in the presence of constraints to 30 percent of the cost of widely-used unconstrained CIT methods without sacrificing the quality of the solutions.

Creating products according to own needs is omnipresent on the Internet. We choose the properties we want to have in a product from a product configurator, such as a car or printer configurator. The underlying models from the configurators are hidden. The migration of legacy configurators into feature models is meaningful to use the advantages of software configurations for product configurators. We introduce an abstraction level and assumptions for product configurators to readout information needed for a feature model. Furthermore, we develop an algorithm called FeatureDiagramSynthesizer. The algorithm consists of two parts which are finding features and adding constraints. For the algorithm, different strategies are used to create the feature model from a product configurator. Moreover, we implement the extensible algorithm FeatureDiagramSynthesizer and a prototype for the product configurator from Ricoh. Based on our developed algorithm, it is only basically possible to readout a feature models from product configurators which fulfill our assumptions.

Combinatorial interaction testing (CIT) is an important technique for testing highly configurable software systems with demonstrated effectiveness in practice. The goal of CIT is to generate test cases covering the interactions of configuration options, under certain hard constraints. In this context, constrained covering arrays (CCAs) are frequently used as test cases in CIT. Constrained Covering Array Generation (CCAG) is an NP-hard combinatorial optimization problem, solving which requires an effective method for generating small CCAs. In particular, effectively solving t-way CCAG with t>=4 is even more challenging. Inspired by the success of automated algorithm configuration and automated algorithm selection in solving combinatorial optimization problems, in this paper, we investigate the efficacy of automated algorithm configuration and automated algorithm selection for the CCAG problem, and propose a novel, automated CCAG approach called AutoCCAG. Extensive experiments on public benchmarks show that AutoCCAG can find much smaller-sized CCAs than current state-of-the-art approaches, indicating the effectiveness of AutoCCAG. More encouragingly, to our best knowledge, our paper reports the first results for CCAG with a high coverage strength (i.e., 5-way CCAG) on public benchmarks. Our results demonstrate that AutoCCAG can bring considerable benefits in testing highly configurable software systems.

Highly-configurable software systems can be easily adapted to address user's needs. Modelling parameter configurations and their relationships can facilitate software reuse. Combinatorial Interaction Testing (CIT) methods are already often used to drive systematic testing of software system configurations. However, a model of the system's configurations not conforming with respect to its software implementation, must be repaired in order to restore conformance. In this paper we extend CIT by devising a new search-based technique able to repair a model composed of a set of constraints among the various software system's parameters. Our technique can be used to detect and fix faults both in the model and in the real software system. Experiments for five real-world systems show that our approach can repair on average 37% of conformance faults. Moreover, we also show it can infer parameter constraints in a large real-world software system, hence it can be used for automated creation of CIT models.

Testing product lines and similar software involves the important task of testing feature interactions. The challenge is to test all those feature interactions that result in testing of all variations across all dimensions of variation. In this context, we propose the use of combinatorial test generation, with Multi-Perspective Feature Models (MPFM) as the input model. MPFMs are a set of feature models created to achieve Separation of Concerns within the model. We believe that the MPFM is useful as an input model for combinatorial testing and it is easy to create and understand. This approach helps achieve a better coverage of variability in the product line. Results from an experiment on a real-life case show that up to 37% of the test effort could be reduced and up to 79% defects from the live system could be detected.

Software Product Lines (SPL) are inherently difficult to test due to the\ncombinatorial explosion of the number of products to consider. To reduce the\nnumber of products to test, sampling techniques such as combinatorial\ninteraction testing have been proposed. They usually start from a feature model\nand apply a coverage criterion (e.g. pairwise feature interaction or\ndissimilarity) to generate tractable, fault-finding, lists of configurations to\nbe tested. Prioritization can also be used to sort/generate such lists,\noptimizing coverage criteria or weights assigned to features. However, current\nsampling/prioritization techniques barely take product behavior into account.\nWe explore how ideas of statistical testing, based on a usage model (a Markov\nchain), can be used to extract configurations of interest according to the\nlikelihood of their executions. These executions are gathered in featured\ntransition systems, compact representation of SPL behavior. We discuss possible\nscenarios and give a prioritization procedure illustrated on an example.\n

Feature models are frequently used to capture the knowledge about configurable software systems and product lines. However, feature modeling of large-scale systems is challenging as many models are needed for diverse purposes. For instance, feature models can be used to reflect the perspectives of product management, technical solution architecture, or product configuration. Furthermore, models are required at different levels of granularity. Although numerous approaches and tools are available, it remains hard to define the purpose, scope, and granularity of feature models. In this paper we thus present experiences of developing feature models for two large-scale industrial automation software systems. Specifically, we extended an existing feature modeling tool to support models for different purposes and at multiple levels. We report results on the characteristics and modularity of the feature models, including metrics about model dependencies. We further discuss lessons learned during the modeling process.

Feature models are the de facto standard for modelling variabilities and commonalities in features and relationships in software product lines. They are the base artefacts in many engineering activities, such as product configuration, derivation, or testing. Concrete models in different domains exist; however, many are in private or sparse repositories or belong to discontinued projects. The dispersion of knowledge of feature models hinders the study and reuse of these artefacts in different studies. The Universal Variability Language (UVL) is a community effort textual feature model language that promotes a common way of serializing feature models independently of concrete tools. Open science principles promote transparency, accessibility, and collaboration in scientific research. Although some attempts exist to promote feature model sharing, the existing solutions lack open science principles by design. In addition, existing and public feature models are described using formats not always supported by current tools. This paper presents ▪ , a repository of feature models in UVL format. ▪ provides a front end that facilitates the search, upload, storage, and management of feature model datasets, improving the capabilities of discontinued proposals. Furthermore, the tool communicates with Zenodo – one of the most well-known open science repositories – providing a permanent save of datasets and following open science principles. ▪ includes existing datasets and is readily available to include new data and functionalities in the future. It is maintained by three active universities in variability modelling.

In the software engineering world, software product lines constitute an approach for building reliable software systems. These use feature models to capture, develop, and document shared software for a base system. One of the main challenges when using feature models to derive new products configuration is a way of selecting a configuration that takes under consideration the minimum number of steps and minimum decision-making cost, taking into account resource constraints. To satisfy the challenges of optimizing the configuration selection technique, in this paper, we present an assessment approach that makes use of genetic algorithms to generate then evaluate the best product configurations from feature models. Our empirical outcomes reveal the effectiveness of using the genetic algorithm in obtaining the product configurations that meet the best level of trading-off between steps and decisions at a reasonable production time, consequently, assisting stakeholders in selecting the product configuration that fits their requirements.

Software Product Lines (SPLs) make it possible to configure a single system based on features in order to create many different variants and cater to a wide range of customers with varying requirements. This configuration space is often modeled using Feature Models (FMs). However, in practice, the SPL (and consequently the FM) is often created after a set of variants has already been created manually. Automating the task of reverse engineering a feature model that describes a set of variants makes the process of adopting an SPL easier. The genetic programming pipeline is a good fit for feature models and has been shown to produce good reverse engineering results. In this paper, we replicate the results of such an existing approach with a larger set of feature models and investigate the effects of various genetic programming parameters and operators on the results. The design of our replication experiments employs three perspectives: duplicate the exact conditions using various features models, study the interaction of two parameters of the genetic programming approach, and optimize the values for the population and generation parameters and for the mutation and crossover operators. Results reinforce the previously obtained outcome, the original study being confirmed. The relations between the number of features and number of generations, respectively number of features and size of populations were also investigated and best values based on obtained results are provided. The current study also aimed to optimize various parameters of the genetic programming approach, the interpretation of those experiments discovering concrete values.

Feature models are a de facto standard for representing the commonalities and variability of product lines and configurable software systems. Requirements-level features are commonly implemented in multiple source code artifacts, which results in complex dependencies at the code level. As developers change and evolve features frequently, it is challenging to keep feature models consistent with their implementation. We thus present an approach combining feature-to-code mappings and code dependency analyses to inform engineers about possible inconsistencies. Our focus is on code-level changes requiring updates in feature dependencies and constraints. Our approach uses static code analysis and a variation control system to lift complex code-level dependencies to feature models. We present the suggested dependencies to the engineer in two ways: directly as links between features in a feature model and as a heatmap visualizing the dependency changes of all features in a model. We present results of an evaluation on the Pick-and-Place Unit system, which demonstrates the utility and performance of our approach and the quality of the suggestions.

Product lines are an established framework for software design. They are specified by special diagrams called feature models. For formal analysis, the latter are usually encoded by propositional theories with Boolean semantics. We discuss a major deficiency of this semantics, and show that it can be fixed by considering that a product is an instantiation process rather than its final result. We call intermediate states of this process partial products, and argue that what a feature model M really defines is a poset of partial products called a partial product line, PPLM. We argue that such PPLs can be viewed as special partial product Kripke structures ppKS specifiable by a suitable version of CTL partial product CTL or ppCTL. We show that any feature model M is representable by a ppCTL theory $$\varPhi M$$ such that for any ppKs K, $$K\models \varPhi M$$ iff $$K = PPL M$$; hence, $$\varPhi M$$ is a sound and complete representation of the feature model.

Needs for mass customization and economies of scale have pushed engineers to develop Software Product Lines (SPLs). SPLs are families of products sharing commonalities and exhibiting differences, built by reusing software assets abstractly represented by features. Feature models describe the constraints that link the features and allow the configuration of tailored software products. Common SPLs involve hundreds, even thousands of features, leading to billions of possible software products. As a result, testing a product line is challenging due to the enormous size of the possible products. Existing techniques focus on testing based on the product line's feature model by selecting a limited set of products to test. Being created manually or reverse-engineered, feature models are prone to errors impacting the generated test suites. In this paper, we examine ability of test suites to detect such errors. In particular, we propose two mutation operators to derive erroneous feature models (mutants) from an original feature model and assess the capability of the generated original test suite to kill the mutants. Experimentation on real feature models demonstrate that dissimilar tests suites have a higher mutant detection ability than similar ones, thus validating the relevance of similarity-driven product line testing.

Augmented Reality (AR) has recently found high attention in mobile shopping apps such as in domains like furniture or decoration. Here, the developers of the apps focus on the positioning of atomic 3D objects in the physical environment. With this focus, they neglect the configuration of multi-faceted 3D object composition according to the user needs and environmental constraints. To tackle these challenges, we present a model-based approach to support AR-assisted product configuration based on the concept of Dynamic Software Product Lines. Our approach splits products (e.g. table) into parts (e.g. tabletop, table legs, funnier) with their 3D objects and additional information (e.g. name, price). The possible products, which can be configured out of these parts, are stored in a feature model. At runtime, this feature model can be used to configure 3D object compositions out of the product parts and adapt to user needs and environmental constraints. The benefits of this approach are demonstrated by a case study of configuring modular kitchens with the help of a prototypical mobile-based implementation.

Test scenario generation for feature-based context-oriented software systems