Combinatorial generation of structurally complex test inputs for commercial software applications

Full scan designs are widely used for their indisputable benefits of predictably high test coverage, diagnosis and debug. However, for high-performance designs the cost of scan - area and delay - is not acceptable and partial scan is used instead. Unfortunately, partial scan significantly increases test generation complexity. We define a structured partial scan design methodology and specific test generation enhancements, which significantly enhance test coverage and reduce test data and cycles. Selective design areas use special types of nonscan cells which can capture a value in the last few scan load cycles. Combinational test generation is extended to work with this structured partial scan design, resulting in higher coverage and fewer patterns. Experimental results on industrial designs show consistent testability benefits.

We present novel algorithms for parallel testing of code that takes structurally complex test inputs. The algorithms build on the Korat algorithm for constraint-based generation of structurally complex test inputs. Given an imperative predicate that specifies the desired structural constraints and a finitization that bounds the desired input size, Korat performs a systematic search to generate all test inputs (within the bounds) that satisfy the constraints. We present how to generate test inputs with a parallel search in Korat and how to execute test inputs in parallel, both off-line (when the inputs are saved on disk) and on-line (when execution immediately follows generation).

While pair-wise testing has shown a high level of fault detection in many situations, it has not done so for testing logical expressions in software applications. The modified condition/decision coverage (MC/DC) approach was especially developed for testing logical expressions, but test generation according to this approach can be complicated and time-consuming. From a practical point of view, combining pair-wise and MC/DC testing would integrate the benefits of both approaches; however, this would require solving many research problems. As an initial step in this direction, our paper evaluates the level of MC/DC coverage for pair-wise test cases in different situations and compares this level to MC/DC coverage for random test cases. To confirm this, we conducted experimental testing of logical expressions of different sizes, complexities, and numbers of input variables. Our experimental results show that the pair-wise test cases had a higher level of MC/DC coverage compared to the random test cases, achieving more than 70% MC/DC coverage. This allows us to suggest pair-wise testing as a promising method to be used as the basis for MC/DC test generation.

Test generation using deterministic fault-oriented algorithms is highly complex and time consuming. New approaches are needed to augment the existing techniques, both to reduce execution time and to improve fault coverage. Genetic algorithms (GA's) have been effective in solving many search and optimization problems. Since test generation is a search process over a large vector space, it is an ideal candidate for GA's. In this work, we describe a GA framework for sequential circuit test generation. The GA evolves candidate test vectors and sequences, using a fault simulator to compute the fitness of each candidate test. Various GA parameters are studied, including alphabet size, fitness function, generation gap, population size, and mutation rate, as well as selection and crossover schemes. High fault coverages were obtained for most of the ISCAS'89 sequential benchmark circuits, and execution times were significantly lower than in a deterministic test generator in most cases.

According to a recent synthesis for testability proposal, a test function specified as a finite state machine with the same number of state variables as the given object machine, is incorporated into the state diagram prior to synthesis. Since a complete verification of the test machine is not practical, an often used heuristic sets and observes each state variable. The two machines share logic and a fault can result in partial or total loss of the test function. We show that the tests generated under the assumption that the entire test function is intact can become invalid. We propose a new method of synthesizing PLA-based finite state machines with fault tolerant test machines. Our approach eliminates testing of the test function. A constrained logic minimization phase insures that faults have predictable effect on the state diagram of the composite machine (object machine embedded with the test function). This allows effective use of the test function during test generation even in the presence of faults that effect both object and test machines. Only a combinational test generator is required for test generation. Each combinational vector is augmented by appropriate initialization and propagation sequences. Unlike prior approaches, ourO(log2 n) length test sequence isguaranteed to detect any targeted crosspoint fault. Experimental results on the MCNC Logic Synthesis Workshop finite state machine benchmark set are given as evidence of practicality of the proposed approach.

We present a technique for extracting functional (control/dataflow) information from register transfer level (RTL) controller/data path circuits and illustrate its use in design for hierarchical testability of these circuits. This testing procedure and design for testability (DFT) technique is general enough to handle RTL control flow intensive circuits like protocol handlers as well as data flow intensive circuits like digital filters. It makes the combined controller-data path highly testable and does not require any external behavioral information. This scheme has the advantages of low area/delay/power overheads (average of 3.2%, 0.9% and 4.1%, respectively, for benchmarks), high fault coverage (over 99% for most cases), very low test generation times (because it is independent of bit-width), and the advantage of at-speed testing. Experiments show a 2-to-4 (1-to-3) orders of magnitude test generation time advantage over an efficient gate-level sequential test generator (combinational test generator that assumes full scan).

Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of bounded-exhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

Verification and test are critical phases in the development of any hardware or software system. This article focuses on black box testing of the control part of hardware and software systems. Black box testing involves specification, test generation, and fault coverage. Finite state machines (FSMs) are commonly used for specifying controllers. FSMs may have shortcomings in modeling complex systems. With the introduction of X-machines, complex systems can be modeled at higher levels of abstraction. An X-machine can be converted into an FSM while preserving the level of abstraction. The fault coverage of a test sequence for an FSM specification provides a confidence level. We propose a fault coverage metric for an FSM specification based on the transition fault model, and using this metric, we derive the coverage of a test sequence. The article also presents a method which generates short test sequences that meet a specific coverage level and then extends this metric to determine the coverage of a test sequence for an FSM driven by an FSM network. We applied our FSM verification technique to a real-life FSM, namely, the fibre channel arbitrated loop port state machine, used in the field of storage area networks.

A software product line is a set of software systems that share some common features. Several recent works have been reported that apply combinatorial testing, a very effective testing strategy, to software product lines. A unique challenge in these efforts is dealing with a potentially large number of constraints among different features. In this paper, we propose a novel constraint-handling strategy that uses minimum invalid tuples (MITs) as an alternative to traditional constraint solvers. Our approach systematically derives all MITs from a software product line, and uses them to quickly determine the validity of a test configuration during test generation. We implemented a test generation research tool called LOOKUP that integrates the proposed constraint-handling strategy with a general test generation algorithm called IPOG-C. Experimental results show that LOOKUP performs considerably better than two existing test generation tools in terms of test size and execution time.

In this paper, a heuristic method based on a pre-calculated prediction of tests is presented. This method is for combinational test generation and applies to the combinational parts of sequential circuits equally wells. The proposed method is Directed Random Test Generation that is based on SCOAP observability and controllability parameters. RTG methods are usually evaluated based on fault coverage that grows exponentially with the number of test vectors. Instead of pure random testing with a massive number of test vectors, fewer test vectors are selected by Directed Random Test Generation, and more faults are detected in the early stages of test generation. As a result, fewer faults remain for the deterministic test stage, which is helpful where online testing is considered. A comparison between random testing and directed random testing is performed on combinational benchmarks, using MATLAB and ModelSim simulation environments. The focus is on the stuck-at fault models in random logic circuits. The simulation results show our method has a sharper increase in fault coverage when compared with pure random test generation.

This paper presents an efficient automatic test pattern generation technique for loop-free circuits. A partial scan technique is used to convert a sequential circuit (finite state machine) with arbitrary feedback paths into a pipelined circuit for testing. Test generation for these modified circuits can be performed with a modified combinational automatic test pattern generator (ATPG), which is much faster than a sequential ATPG. A combinational model is obtained by replacing all flipflops by buffers. It is shown that a test vector for a fault in this model obtained by a combinational test generator can be expanded into a sequence of identical vectors to detect the same fault in the original sequential circuit. This technique may abort a few faults which can then be resolved with a sequential ATPG. Experiments on the ISCAS89 circuits show that only 30% to 70% of flipflops require scanning in larger circuits and 96% to 100% fault coverage for almost all the circuits without resorting to a sequential ATPG.

Test case generation, selection and coverage from natural language

We introduce a class of sequential circuits with internally switched balanced structure which allows test generation with combinational test generation complexity. The proposed class includes any other known classes with this feature. This paper also considers faults in hold registers and switches regarded as macros, while any related work does not consider faults in such macros. Experimental results show the effectiveness of using combinational test generation for the circuits with internally switched balanced structure.

Fault characterization is an important part of combinatorial testing, enabling it to automatically narrow down failed test inputs to specific failure-inducing combinations. As most current fault characterization algorithms adaptively generate more test inputs based on previous test execution results, a framework that integrates modelling, generation, execution, and fault characterization is necessary. Until now, no such framework exists, resulting in much manual work needed to identify failureinducing combinations.We therefore introduce COFFEe, which is a framework for completely automatic combinatorial testing and fault characterization. In this paper, we derive an architecture for the framework and present coffee4j, a Java implementation of COFFEe that integrates the JUnit5 test framework.

A hybrid DFT method is proposed to reduce the hardware penalty of traditional DFT methods and test generation time. It takes advantages of both traditional DFT methods like scan and DFT methods which control clocks for testability. The hardware scheme and test generation algorithm are presented. Test generation results for ISCAS '89 circuits have been generated and showed an improvement in test generation time and fault coverage.