A New Approach to Generating High Quality Test Cases

A hybrid DFT method is proposed to reduce the hardware penalty of traditional DFT methods and test generation time. It takes advantages of both traditional DFT methods like scan and DFT methods which control clocks for testability. The hardware scheme and test generation algorithm are presented. Test generation results for ISCAS '89 circuits have been generated and showed an improvement in test generation time and fault coverage.

This paper adresses the generation of test cases for testing the conformance of a reactive black-box implementation with respect to its specification. We aim at extending the principles and algorithms of model-based testing for recursive interprocedural specifications that can be modeled by Push-Down Systems (PDS). Such specifications may be more compact than non-recursive ones and are more expressive.The generated test cases are selected according to a test purpose, a (set of) scenario of interest that one wants to observe during test execution. The test generation method we propose in this paper is based on program transformations and a coreachability analysis, which allows to decide whether and how the test purpose can still be satisfied. However, despite the possibility to perform an exact analysis, the inability of test cases to inspect their own stack prevents it from using fully the coreachability information. We discuss this partial observation problem, its consequences, and how to minimize its impact.KeywordsTest PurposeLabel Transition SystemProgram TransformationGenerate Test CaseTest ExecutionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

MLIR, a modular compiler framework, evolves quickly, with regular updates expanding its dialects and operations across LLVM versions and downstream projects. This fast development reduces the effectiveness of traditional fuzzing tools, which test only a small portion of dialects, require extensive manual work (e.g., nearly ten thousand lines of C++ code), and do not match the update speed of MLIR. To address these challenges, we propose NÜwa, the first LLM-based approach for MLIR fuzzing. Nüwa employs a two-phase strategy: first generating valid operations by encoding constraints into LLMs prompts, then synthesizing multi-operation test cases by learning inter-operation dependencies. To enhance operation coverage, it incorporates high-coverage cases from MLIR's test suite and uses LLM-driven mutations to boost diversity. A self-improvement mechanism enhances the prompts using feedback from highquality test cases, improving the LLMs' understanding of MLIR's complex semantics. Nüwa demonstrates that the generation and mutation process can be fully automated via the intrinsic capabilities of llMs (including in-context learning), while being applicable to MLIR's fast evolution. The experimental study shows that NÜwa outperforms the state-of-the-art tools MLIRSmith and MLIRod, detecting <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.9 \times$</tex> more unique bugs and achieving <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.6 \times$</tex> greater code coverage. To date, Nüwa has identified 55 bugs in the MLIR framework, with 18 confirmed or fixed.

In the context of automatic test generation, the use of propositional satisfiability (SAT) and Satisfiability Modulo Theories (SMT) solvers is becoming an attractive alternative to traditional algorithmic test generation methods, especially when testing Boolean expressions. The main advantages are the capability to deal with constraints over the inputs, the generation of compact test suites and the support for fault-detecting test generation methods. However, these solvers normally require more time and a greater amount of memory than classical test generation algorithms, making their applicability not always feasible in practice. In this paper, we propose several ways to optimize the SAT/SMT-based process of test generation for Boolean expressions and we compare several solving tools and propositional transformation rules. These optimizations promise to make SAT/SMT-based techniques as efficient as standard methods for testing purposes, especially when dealing with Boolean expressions, as proved by our experiments.

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

This paper presents a method of broadside transition test generation for partial scan circuits. The proposed method first transforms the kernel circuit of a given partial scan circuit into some combinational circuits. Then, by performing stuck-at test generation on the transformed circuits, broadside transition tests for the original circuit are obtained. This method allows us to use existing stuck-at test generation tools in order to generate broadside transition tests. It is shown that the proposed scheme is effective in area overhead and test generation time by experiments. In this paper, some variations of broadside transition testing of partial scan circuits are also discussed in terms of different test application strategies and fault sizes.

Automotive infotainment systems were getting more and more features in recent years. The usability of their HMIs (human-machine interfaces) has been improved considerably. However the complexity of the HMI software is growing. Testing the HMI became very demanding and time consuming. Because of multiplicity of HMI variants, a better code coverage is a goal for the development process of most manufacturers. Model-based testing is one way to achieve a better code coverage and keep the costs and complexity acceptable. However, the existing research approaches in the area of model-based HMI testing can not satisfy the needs for our testing purposes. In the work a model-based testing approach will be proposed for testing both the logical behavior and the graphical interface of the automotive infotainment HMI. As an important part of the testing approach a test-oriented HMI specification model will be designed. It is a model, which describes the required behavior of the HMI and contains the necessary information for the testing process. Test generation methods and the design of tests will also be proposed. These results can be generally used for testing advanced GUI-driven applications. Specific coverage criteria for infotainment HMIs, methods for automatic test generation and verification of the system behavior are also focuses of the work. The paper introduces the ideas and the goals of our model-based testing approach for infotainment HMIs.

Since scan testing is not based on the function of the circuit, but rather its structure, this method is considered to be a form of over testing or under testing. It is important to test VLSIs using the given function. Since the functional specifications are described explicitly in the FSMs, high test quality is expected by performing logical testing and timing testing. This paper proposes two test generation methods, a fault-independent test generation method and a fault-dependent test generation method, for state-observable FSMs. We give experimental results for MCNC'91 benchmark circuits. The quality and cost of the logic testing and timing testing for proposed test generation methods was evaluated.

This paper presents a novel method for functional test generation aiming at exploring control state space of the design. The steady-state probabilities (SP's) of the abstract design's control FSM are used to guide test generation. The SP's of the states can reflect how hard the states can be reached, and the hard-to-reach states are assigned with high priority to be exercised. Experimental results show that our method has better performance in test generation in comparison with constrained random simulation, and demonstrate that SP's provide good guidance on traversing hard-to-reach states of the design under validation.

In this paper a method for high-level test generation for systems described in VHDL is presented. First, two test generation algorithms for VHDL processes are presented. The first algorithm works on processes which represent combinational logic whereas the second works on processes which represent sequential logic. The goal of both algorithms is to test all portions of the process body by traversing all the feasible paths. Employing these two algorithms, two new algorithms for system-level test generation for both combinational and sequential systems are presented. The approach employs software metrics as well as signal coverage for interconnections and state and transition coverage for FSMs.

In this paper, we present methods for automatic test generation from formal specifications. These are used to create intelligent test benches that are able to cover corner case behaviors in much less time. We have developed a prototype tool for intelligent test generation within the layered test bench architecture proposed in RVM. We present results on verification IPs of standard bus protocols to show the effectiveness of our approach.

Capacitive crosstalk can slowdown transitions which can propagate to outputs and cause erroneous operation. Test generation methods such as XGEN and XGEN-E were proposed to generate tests for such failures. However, a drawback of these test generation methods is that a large proportion of faults are aborted. In this paper, we systematically derive a multi-valued algebra. We first show that a composite value system must be derived considering all operations performed by the algorithm that will use the value system. In particular, we identify that for our test generation algorithm it is imperative to consider its timing operations and their impact on what composite values can exist in our algebra. We also identify the fact that even some key procedures - namely the backtrace procedure as well as the search procedure - need to be modified to work with the composite value system. We derive a new 57-valued algebra and modify the key ATPG procedures to obtain a test generation methodology which we call XGEN-M ('M' for multi-valued). We present experimental results that demonstrate the superiority of XGEN-M compared to previous methods.

Hardware testing is a process aimed at detecting manufacturing faults in integrated circuits. To measure test quality, two main metrics are in use: fault detection abilities (fault coverage) and test application time (test length). Many algorithms have been suggested for test generation; however, no scalable solution exists. In this paper, we analyze applicability of functional tests generated from high-level models for low-level manufacturing testing. A particular test generation method is considered. The input information is an HDL description. The key steps of the method are system model construction and coverage model construction. Both models are automatically extracted from the given description. The system model is a representation of the design in the form of high-level decision diagrams. The coverage model is a set of LTL formulae defining reachability conditions for the transitions of the extended finite state machine. The models are translated into the input format of a model checker. For each coverage model formula the model checker generates a counterexample, i.e. an execution that violates the formula (makes the corresponding transition to fire). The approach is intended for covering of all possible execution paths of the input HDL description and detecting dead code. Experimental comparison with the existing analogues has shown that it produces shorter tests, but they achieve lower stuck-at fault coverage comparing with the dedicated approach. An improvement has been proposed to overcome the issue.

Test case generation, selection and coverage from natural language

Testing circuit vulnerability to probabilistic faults, like single-event upsets (SEUs), requires a reformulation of test-generation methods. Unlike in traditional testing, a probabilistic fault can be detected by different test vectors with different probabilities, i.e., they have different sensitivities to probabilistic faults. We develop an efficient PTM-based algorithm to compute test-vector sensitivity. We also propose several test generation and compaction methods for probabilistic faults, with the goals of estimating and bounding fault detection probabilities. These methods use integer linear programming (ILP) to optimize test sets. The results show that our methods can generate tests quickly and require only half as many (repeated) vectors as testing with random patterns.