Test generation games from formal specifications

Automotive infotainment systems were getting more and more features in recent years. The usability of their HMIs (human-machine interfaces) has been improved considerably. However the complexity of the HMI software is growing. Testing the HMI became very demanding and time consuming. Because of multiplicity of HMI variants, a better code coverage is a goal for the development process of most manufacturers. Model-based testing is one way to achieve a better code coverage and keep the costs and complexity acceptable. However, the existing research approaches in the area of model-based HMI testing can not satisfy the needs for our testing purposes. In the work a model-based testing approach will be proposed for testing both the logical behavior and the graphical interface of the automotive infotainment HMI. As an important part of the testing approach a test-oriented HMI specification model will be designed. It is a model, which describes the required behavior of the HMI and contains the necessary information for the testing process. Test generation methods and the design of tests will also be proposed. These results can be generally used for testing advanced GUI-driven applications. Specific coverage criteria for infotainment HMIs, methods for automatic test generation and verification of the system behavior are also focuses of the work. The paper introduces the ideas and the goals of our model-based testing approach for infotainment HMIs.

This paper presents a method of broadside transition test generation for partial scan circuits. The proposed method first transforms the kernel circuit of a given partial scan circuit into some combinational circuits. Then, by performing stuck-at test generation on the transformed circuits, broadside transition tests for the original circuit are obtained. This method allows us to use existing stuck-at test generation tools in order to generate broadside transition tests. It is shown that the proposed scheme is effective in area overhead and test generation time by experiments. In this paper, some variations of broadside transition testing of partial scan circuits are also discussed in terms of different test application strategies and fault sizes.

Adapting automated test generation to GUI testing of industry applications

In the context of automatic test generation, the use of propositional satisfiability (SAT) and Satisfiability Modulo Theories (SMT) solvers is becoming an attractive alternative to traditional algorithmic test generation methods, especially when testing Boolean expressions. The main advantages are the capability to deal with constraints over the inputs, the generation of compact test suites and the support for fault-detecting test generation methods. However, these solvers normally require more time and a greater amount of memory than classical test generation algorithms, making their applicability not always feasible in practice. In this paper, we propose several ways to optimize the SAT/SMT-based process of test generation for Boolean expressions and we compare several solving tools and propositional transformation rules. These optimizations promise to make SAT/SMT-based techniques as efficient as standard methods for testing purposes, especially when dealing with Boolean expressions, as proved by our experiments.

A hybrid DFT method is proposed to reduce the hardware penalty of traditional DFT methods and test generation time. It takes advantages of both traditional DFT methods like scan and DFT methods which control clocks for testability. The hardware scheme and test generation algorithm are presented. Test generation results for ISCAS '89 circuits have been generated and showed an improvement in test generation time and fault coverage.

Test case generation, selection and coverage from natural language

Activity theory is commonly used in designing Human-device interaction. Donald Norman in his paper Human-Centered Design Considered Harmful criticized traditional approach, which he called Human-Centered Design, and introduced a new approach, called Activity-Centered Design. In this paper his new approach is directly used and widened in order to be used with a tool for test generation. It is used as a point of view, a philosophy, on which graphical user interfaces are designed. Workflow paradigm is used for sequential flow of elements, called activities. Activity theory is directly implemented on Workflow elements by splitting the work on: Activities, Actions and Operations. Combination of Activity theory and Workflow is used as start point for broad range of Human computer interaction, and in this paper it is used as upgrade of test generation framework for TV and STB testing called BBT (Black Box Testing). Goal of this approach is to simplify test script creation process for user with modest or none programming experience.

Software development is concerned with more than just generation of code; the program must have the desired properties, and these must be demonstrated via suitable tests and correctness arguments. One way of viewing these aspects is to group them into programs, properties, and data. When represented as vertices in a triangle, the edges represent processes to produce one element from another. The research reported in this article aims at practical methods for automatic test generation by restricting the use of both predicate logic and programming constructs. It promises to eventually result in practical domain-specific programming languages, with a significant boost in both quality and productivity.

The development of in-vehicle software, often controlling safety-critical functions related to braking, steering and transmission systems, requires rigorous techniques to ensure high-integrity and reliability requirements. Formal models of requirements and design artifacts based on state-transition systems and other formalisms serve as a means to apply rigorous analysis and verification techniques at every stage in the development process. We present here one such formal analysis and verification flow, developed at General Motors R&D, provide an overview of methods for automatic test generation based on mathematical modeling and discuss the future directions for research.

This paper deals with a benchmark of automated test generation methods for software testing. The existing methods are usually demonstrated using quite different examples. This makes their mutual comparison difficult. Additionally, the quality of the methods is often evaluated using code coverage or other metrics, such as generated tests count, test generation time, or memory usage. The most important feature -the ability of the method to find realistic errors in realistic applications -is only rarely used. To enable mutual comparison of various methods and to investigate their ability to find realistic errors, we propose a benchmark consisting of several applications with wittingly introduced errors. These errors should be found by the investigated test generation methods during the benchmark. To enable an easy introduction of various errors of various types into the benchmark applications, we created the Testing Applications Generator (TAG) tool. The description of the TAG along with two applications, which we developed as a part of the intended benchmark, is the main contribution of this paper.

Aim: This study aims to introduce search-based intelligent test generation for synthetic testing as a means of enabling optimal cross-layer testing in distributed software architectures comprising front-end interfaces, API gateways, microservices, databases, and cloud-based runtime services. It addresses the limitation of legacy quality assurance automation, which remains layer-isolated and rule-based, and is therefore unable to effectively detect inter-service faults, asynchronous failures, or unreliable execution behaviors in complex workflows. Methods: The research adopts a comparative mixed-methods approach, analyzing an AI-based framework that incorporates behavioral trace modeling, dependency graph generation, historical defect learning, and synthetic scenario generation. The evaluation was conducted across three types of enterprise applications, involving 30 autonomous test runs and 12 expert interviews. Results: The findings demonstrate substantial improvements in testing performance, with cross-layer validation coverage increasing to 84% and 62%, defects detected rising to 87% and 68%, and flaky tests reduced to 7% and 18%. Additionally, failure reproducibility improved by 19%, while scenario generation expanded by a factor of 2.7, increasing from 45 to 120 scenarios. Conclusion: The study concludes that synthetic test generation represents a realistic, scalable, and statistically informed approach that significantly enhances validation under realistic operating conditions, thereby providing a more reliable baseline for release confidence and operational decision-making. Recommendation: The study recommends further research focusing on reinforcement learning, explainability, increased autonomy through CI/CD integration, and the development of cross-domain benchmarks to advance the effectiveness and applicability of intelligent testing frameworks.

This paper presents a novel method for functional test generation aiming at exploring control state space of the design. The steady-state probabilities (SP's) of the abstract design's control FSM are used to guide test generation. The SP's of the states can reflect how hard the states can be reached, and the hard-to-reach states are assigned with high priority to be exercised. Experimental results show that our method has better performance in test generation in comparison with constrained random simulation, and demonstrate that SP's provide good guidance on traversing hard-to-reach states of the design under validation.

In this paper a method for high-level test generation for systems described in VHDL is presented. First, two test generation algorithms for VHDL processes are presented. The first algorithm works on processes which represent combinational logic whereas the second works on processes which represent sequential logic. The goal of both algorithms is to test all portions of the process body by traversing all the feasible paths. Employing these two algorithms, two new algorithms for system-level test generation for both combinational and sequential systems are presented. The approach employs software metrics as well as signal coverage for interconnections and state and transition coverage for FSMs.

High quality test cases can effectively detect software errors and ensure software quality. However, except the regular expression-based test generation method, test cases generated from other model-based test generation methods have not contain the whole information of the model, resulting in test inadequacy. And test cases derived from regular expression have the prohibited lengths that cause the sustainable increase of test cost. To obtain high quality test cases, we suggest a new method for test generation by way of regular expression decomposition. Unlike the previous model decomposition techniques, our method lays emphasis on information completeness after regular expression is decomposed. Based on two empirical assumptions, we propose two processes of regular expression decomposition and three decomposition rules. Then we perform a case study to demonstrate our approach. The results show that our approach generates high quality test cases as well as avoids the problem of test complexity.

<p><strong>Context and relevance.</strong><strong> </strong>Modern web application development requires continuous testing, but maintaining automated tests is becoming increasingly labor-intensive due to locator instability and growing interface complexity. The emergence of Large Language Models (LLM) opens new opportunities for test creation automation, but their practical application faces challenges in processing large HTML documents and the need to create maintainable code. <strong>Objective. </strong>To develop and evaluate the effectiveness of a method for automatic generation of maintainable web application tests using LLM based on HTML structure analysis and the Page Object Model(POM) pattern. <strong>Hypotheses. </strong>Primary hypothesis: combining LLM with a two-stage generation approach and the POM pattern will enable the creation of maintainable tests, reducing development time by at least one-third (to 67% or less) while preserving code readability. Secondary hypothesis: the success rate of automatic generation will be inversely proportional to the complexity of interface components. <strong>Methods and materials. </strong>The study employed an approach based on Playwright, LLM, and a two-stage generation procedure with intermediate validation. Testing was conducted on four components of an SPA application for virtual infrastructure management. Validation of results was performed by a team of three testers who assessed the correctness and readability of generated tests. <strong>Results. </strong>The proposed method achieved high success rates in automatic test generation and substantial reduction in time costs for test creation. The two-stage procedure with intermediate validation enabled localization of a significant portion of errors at the early stage of Page Object creation. Automatically generated tests provided coverage of most required functionality while maintaining code readability. An inverse relationship between generation success and interface component complexity was confirmed: standardized interfaces demonstrated significantly higher success rates. <strong>Conclusions.</strong> The proposed method provides substantial time savings in creating a baseline test suite while maintaining quality and maintainability. The approach is recommended for early stages of feature development with expert control retained for validating critical scenarios. The method is particularly effective for projects with frequent interface changes, large volumes of regression testing, and components with standardized interfaces.</p>