Software test generation using refinement types

Test case generation, selection and coverage from natural language

In this paper, we present an automatic grammar-based test generation approach which takes a symbolic grammar as input, requires zero control input from users, and produces well-distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. The adjustment is based on a tabling strategy to keep track of the recursion of each grammar variable. We further present a test generation coverage tree illustrating the distribution of generated test cases and their detailed derivations, more importantly, it provides various implicit balance control mechanisms. We implemented this approach in a Java-based system, named Gena. Experimental results demonstrate the effectiveness of our test generation approach and show the balanced distribution of generated test cases over grammatical structures.KeywordsTest GenerationCoverage TreeProduction RuleSoftware Product LineGenerate Test CaseThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Automatic Software-Based Self Test Generation for Embedded Processors

A novel architectural level test generation methodology is proposed to solve both data flow path conflicts and data flow value conflicts. For each pattern to be justified at a high level, an instruction sequence and the underdetermined system of nonlinear equations are derived based on preprocessing information. The solution of the system of equations is calculated by a signal-driven discrete relaxation algorithm without making any high-level decisions. The test generation is performed by recursively assembling the instruction sequence and solving the system of equations. This test generation approach has been implemented, and the tests of several microprocessors have been generated successfully. The results show that this approach is effective and promising. >

<p><strong>Context and relevance.</strong><strong> </strong>Modern web application development requires continuous testing, but maintaining automated tests is becoming increasingly labor-intensive due to locator instability and growing interface complexity. The emergence of Large Language Models (LLM) opens new opportunities for test creation automation, but their practical application faces challenges in processing large HTML documents and the need to create maintainable code. <strong>Objective. </strong>To develop and evaluate the effectiveness of a method for automatic generation of maintainable web application tests using LLM based on HTML structure analysis and the Page Object Model(POM) pattern. <strong>Hypotheses. </strong>Primary hypothesis: combining LLM with a two-stage generation approach and the POM pattern will enable the creation of maintainable tests, reducing development time by at least one-third (to 67% or less) while preserving code readability. Secondary hypothesis: the success rate of automatic generation will be inversely proportional to the complexity of interface components. <strong>Methods and materials. </strong>The study employed an approach based on Playwright, LLM, and a two-stage generation procedure with intermediate validation. Testing was conducted on four components of an SPA application for virtual infrastructure management. Validation of results was performed by a team of three testers who assessed the correctness and readability of generated tests. <strong>Results. </strong>The proposed method achieved high success rates in automatic test generation and substantial reduction in time costs for test creation. The two-stage procedure with intermediate validation enabled localization of a significant portion of errors at the early stage of Page Object creation. Automatically generated tests provided coverage of most required functionality while maintaining code readability. An inverse relationship between generation success and interface component complexity was confirmed: standardized interfaces demonstrated significantly higher success rates. <strong>Conclusions.</strong> The proposed method provides substantial time savings in creating a baseline test suite while maintaining quality and maintainability. The approach is recommended for early stages of feature development with expert control retained for validating critical scenarios. The method is particularly effective for projects with frequent interface changes, large volumes of regression testing, and components with standardized interfaces.</p>

We present Ketchum, a tool that was developed to improve the productivity of simulation-based functional verification by providing two capabilities: (1) automatic test generation and (2) unreachability analysis. Given a set of interesting signals in the design under test (DUT), automatic test generation creates input stimuli that drive the DUT through as many different combinations (called coverage states) of these signals as possible to thoroughly exercise the DUT. Unreachability analysis identifies as many unreachable coverage states as possible.Ketchum differs from the previous published results for several reasons. First, Ketchum provides 10x higher capacity than previous published results. The higher capacity is achieved by carefully orchestrating simulation and multiple formal methods including symbolic simulation, SAT-based BMC, symbolic fixpoint computation and automatic abstraction. Second, Ketchum performs not only automatic test generation but also unreachability analysis, which enables the test generation effort to be focused on coverage states that are not unreachable. Third, the backbone of Ketchum is an off-the-shelf commercial simulator. It enables Ketchum to reach deep states of the design quickly and supports simulation monitors through the standard API of the simulator during test generation.We applied Ketchum to several industrial designs, including the picoJava microprocessor from SUN and the DW8051 microcontroller from Synopsys and obtained very promising results. The experiments show that Ketchum can (1) handle design blocks containing more than 4500 latches and 170K gates, (2) reach up to 6x more coverage states than random simulation and (3) identify a majority of the unreachable coverage states.

Adapting automated test generation to GUI testing of industry applications

An overview of test generation approaches to maximize power supply noise is provided and issues related to and research directions for generation of test vectors with representative power are discussed. In particular, to facilitate the early design and validation of the power delivery network (PDN), the need for automatic high-level test generation to create functionally representative vectors that cause worst-case power events, which stress the power grid, is described.

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

Automated test generation from SDL specifications

Model Based Testing (MBT) is a relatively an approach widely discussed in software testing. It extends test automation from test design to test execution using automatic test generation from models. The effective use of this approach requires new skills, technology and knowledge, such as test modelling skills, but also good programs and tool support. This paper explores a design of a Behaviour driven test automation framework using MBT and how it can be effectively used during Agile Development. The automation framework is experimented upon the integration of the Graphwalker, a Model Based Graphical User Interface test generator with behaviour driven development framework and Robot Framework.

Testing is an essential process for ensuring the quality of the software. Designing software with as few errors as possible in most embedded systems is often critical. Resource usage is a significant concern for proper behaviour because of the very nature of embedded systems. To design energy-efficient systems, approaches are needed to catch desirable consumption points and correct them before deployment. Model-based testing can reduce testing effort, one testing method that allows for automatic test generation. However, this technique has yet to be studied extensively for revealing resource usage anomalies in embedded systems development. UPPAAL SMC is a statistical model-checking tool that can model the system’s resource usage. This paper shows the tooling needed to achieve this and experimental results on automated test generation and selection using mutation analysis in UPPAAL SMC and how this is applied to a Brake by Wire industrial system. The evaluation shows that this approach, which we call MATS, is applicable and efficient for energy-based test generation.

Finite state machine based formal methods in protocol conformance testing: from theory to implementation

Unit testing is an established practice in professional software development. However, in high-performance computing (HPC) with its scientific applications, it is not widely applied. Besides general problems regarding testing of scientific software, for many HPC applications the effort of creating small test cases with a consistent set of test data is high.We have created a tool called FortranTestGenerator, that significantly reduces the effort of creating unit tests for subroutines of an existing Fortran application. It is based on Capture & Replay (C&R), that is, it extracts data while running the original application and uses the extracted data as test input data. The tool automatically generates code for capturing the input data and a basic test driver which can be extended by the developer to an appropriate unit test. A static source code analysis is conducted, to reduce the number of captured variables. Code is generated based on flexibly customizable templates. Thus, both the capturing process and the unit tests can easily be integrated into an existing software ecosystem.Since most HPC applications use message passing for parallel processing, we also present an approach to extend our C&R model to MPI communication. This allows extraction of unit tests from massively parallel applications that can be run with a single process.

We present an approach in automatic test generation that combines features of static analysis and bounded symbolic computation that is capable of producing a test suite that can be used to declare a program under test safe within bounds. We first use the results produced by static analysis which will identify a list of potential errors in the program. We restrict our search to the locations where errors can exist and aim to find exactly one test case per real bug. We have built a prototype tool (called Batg) that implements our approach. We report the results of running it on a number of benchmarks from well known benchmarking suites. We compare Batgto KLEE (an automatic test generation framework) and CBMC(a bounded model checker). This comparison is based on the time taken by the tools, the number of bugs found and the number of generated test cases. We analyse the results of our experiment, demonstrating the benefits of our approach.