Combinatorial Test Generation for Software Product Lines Using Minimum Invalid Tuples

Cost-effective test case generation with the hyper-heuristic for software product line testing

Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of bounded-exhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

Despite recent progress in automated test generation research, significant challenges remain for applying these techniques on large-scale software systems. These systems under test often require structurally complex test inputs within a large input domain. It is challenging to automatically generate a reasonable number of tests that are both legal and behaviorally-diverse to exercise these systems. Constraint-based test generation is an effective approach for generating structurally complex inputs for systematic testing. While this approach can typically generate large numbers of tests, it has limited scalability – tests generated are usually only up to a small bound on input size. Combinatorial test generation, e.g., pair-wise testing, is a more scalable approach but is challenging to apply on commercial software systems that require complex input structures that cannot be formed by using arbitrary combinations. This paper introduces comKorat, which unifies constraint-based generation of structurally complex tests with combinatorial testing. Specifically, comKorat integrates Korat and ACTS test generators to generate test suites for large scale software systems with structurally complex test inputs. We have successfully applied comKorat on four software applications developed at eBay and Yahoo!. The experimental results show that comKorat outperforms existing solutions in execution time and test coverage. Furthermore, comKorat found a total of 59 previously unknown bugs in the four applications.

Software Product Lines (SPLs) can aid modern ecosystems by rapidly developing large-scale software applications. SPLs produce new software products by combining existing components that are considered as features. Selection of features is challenging due to the large number of competing candidate features to choose from, with different properties, contributing towards different objectives. It is also a critical part of SPLs as they have a direct impact on the properties of the product. There have been a number of attempts to automate the selection of features. However, they offer limited flexibility in terms of specifying objectives and quantifying datasets based on these objectives, so they can be used by various selection algorithms. In this research we introduce a novel feature selection approach that supports multiple multi-level user defined objectives. A novel feature quantification method using twenty operators, capable of treating text-based and numeric values and three selection algorithms called Falcon, Jaguar, and Snail are introduced. Falcon and Jaguar are based on greedy algorithm while Snail is a variation of exhaustive search algorithm. With an increase in 4% execution time, Jaguar performed 6% and 8% better than Falcon in terms of added value and the number of features selected.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a model based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework. Prototypes of the tools have been tried in industrial settings providing important feedback for the creation of the next generation of tools in this area.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a model based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework. Prototypes of the tools have been tried in industrial settings providing important feedback for the creation of the next generation of tools in this area.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a UML based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework.

Software product lines (SPL) dynamic configuring process could use methods for recommendation system (RS) elaboration. An overview and analysis of such methods was done in this paper. SPL represent a set of software systems that have common and variable functional components and use a set of paradigms and methods for development. In the classical static SPL the process of configuring performed before executing and performing in the operation environment (OE), in contrast dynamic software product lines performs after executing in the OE. Through the use of which it is possible to customize software solutions in accordance with the needs of end users. The following possible methods to build RS were considered: clustering, Markov decision-making process, matrix factorization. According to the review of the intelligent RS method development and researching of the functionalities of such systems in some open-source projects it was proposed to use N-dimensional context-dependent tensor factorization method and CARSkit tool system. Functional requirements and software architecture of the RS were developed. It allows to automatize software components configuration in the „Smart Home” (SH) systems that could be implemented with CARSkit software toolkit and algorithms implemented with programming language Python. This implementation allows to build a process for tracking changes in the external environment and transfer information to the SH system and, after analyzing the input data, process it in the RS to track changes in the context information. In the future research some additional quantitative experiments will be performed considering the specifics of the SH systems, additionally quantitative metrics will be used for efficiency assessment of the tensor factorization algorithms to predict the dynamic configurations of software components in these systems. Keywords: recommendation system, software product line, variability, dynamic configuration, architecture.

The RC-5 algorithm is an algorithm with the encryption method using the symmetric method and processing in the form of a cipher block, the same keywords are used for the encryption and decryption processes. The testing and analysis phase consists of a). Testing the execution time of the key generation process (set up key) b). Testing the success of the encryption and decryption process on the client-server integration method c). Testing the success of the encryption and decryption process using the file operation method d). Comparison of overhead/file size before and after passing through the system. The testing phase and execution time analysis are carried out when using the file operation method where the test will be divided into 3 parts, namely the execution time for the key setup process, the encryption process, and the decryption process. Testing the success of the encryption and decryption process on the client-server integration method is divided into 2 parts, namely testing the encryption process and the decryption process. In testing the encryption process, the characters that will be sent to the server will be encrypted first with a key to becoming a ciphertext in the form of integer data. The test results obtained are 1). The execution time for key generation (set up key) is very fast, around 9-10 ns. 2). In the encryption and decryption process, the execution time depends on the size of the plaintext file. The larger the plaintext file size, the longer the execution time. Where for a file with a size of 1 kb requires an execution time of 50.6 ms in the encryption process. 3). There is no difference between the size of the file before entering encryption and after decryption. Where for files with a size of 500 bytes before entering the encryption process, a file of 500 bytes will be generated from the decryption process.

In online model-based testing, test execution is interleaved with test generation. Test cases should be generated and executed with minimal delay, while still achieving targeted coverage criteria quickly. Extensive model analysis in such case is not possible as any delays in choosing the next step will immediately impact the response times of test execution. The algorithms thus need to be as fast as possible, where a limiting factor is the available computing power. Experts working on the test models used for the generation often need to be able to quickly edit the models, generate test cases, and use the feedback to further evolve the models. Reserving large-scale computing resources while editing the model is unnecessary, but performing the analysis on them for test generation can improve the execution response time significantly. In this paper, we present an approach and algorithm for distributing the online test generation analysis part concurrently over the network, while enabling the expert to work on the models and execute the test cases locally at the same time.

An efficient sequential circuit test generation algorithm is presented. The algorithm is based on PODEM and uses a nine-valued logic model. Among the novel features of the algorithm are use of an initial time-frame algorithm and correct implementation of a solution to the previous state information problem. The initial time-frame algorithm determines the number of time-frames required to excite the fault under test and the number of time-frames required to observe the excited fault. This step saves the test generator from doing unnecessary search in the input space. Test generation is done strictly in forward time. The algorithm saves good machine circuit state after test generation to aid in future test generation. Faulty machine state is set to unknown whenever test generation for a fault is begun. This solves the previous state information problem, which has often been ignored by existing test generators. >

Test generation using deterministic fault-oriented algorithms is highly complex and time consuming. New approaches are needed to augment the existing techniques, both to reduce execution time and to improve fault coverage. Genetic algorithms (GA's) have been effective in solving many search and optimization problems. Since test generation is a search process over a large vector space, it is an ideal candidate for GA's. In this work, we describe a GA framework for sequential circuit test generation. The GA evolves candidate test vectors and sequences, using a fault simulator to compute the fitness of each candidate test. Various GA parameters are studied, including alphabet size, fitness function, generation gap, population size, and mutation rate, as well as selection and crossover schemes. High fault coverages were obtained for most of the ISCAS'89 sequential benchmark circuits, and execution times were significantly lower than in a deterministic test generator in most cases.

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

Current test generation algorithms for path delay faults assume a variable-clock methodology for test application. Two-vector test sequences assume that the combinational logic reaches a steady state following the first vector before the second vector is applied. While such tests may be acceptable for combinational circuits, their use for nonscan sequential circuit testing is impractical. A rated-clock path delay simulator shows a large drop in coverage for vectors obtained from existing test generators that assume a variable clock. A new test generation algorithm provides valid tests for uniform rated-clock test application. In this algorithm, signals are represented for three-vector sequences. The test generation procedure activates a target path from input to output using the three-vector algebra. For an effective backward justification, we derive an optimal 41-valued algebra. This is the first time, rated-clock tests for large circuits are obtained. Results for ISCAS-89 benchmarks show that rated-clock tests cover some longest, or close to longest, paths.

Threshold testing, which is an LSI testing method based on the acceptability of faults, is effective in yield enhancement of LSIs and selective hardening for LSI systems. In this paper, we propose test generation models for threshold test generation. Using the proposed models, we can efficiently identify acceptable faults and generate test patterns for unacceptable faults with a general test generation algorithm, i.e., without a test generation algorithm specialized for threshold testing. Experimental results show that our approach is practically effective.