Current and Future Directions in Automatic Test Pattern Generation for Power Delivery Network Validation

A novel approach for automatic software test generation is presented, which combines ideas from structural and functional testing as well as formal verification methods. It involves as an intermediate step, the construction of graphs and refinement types, which can be regarded as an automatically constructed semi-specification and used for formal verification. The technique is illustrated using a simple functional language, with algorithms for assigning refinement types and for test generation. Some desirable theoretical properties of the approach are briefly considered. It is also compared informally to other well-known as well as new techniques for automatic test generation.

Test case generation, selection and coverage from natural language

<p><strong>Context and relevance.</strong><strong> </strong>Modern web application development requires continuous testing, but maintaining automated tests is becoming increasingly labor-intensive due to locator instability and growing interface complexity. The emergence of Large Language Models (LLM) opens new opportunities for test creation automation, but their practical application faces challenges in processing large HTML documents and the need to create maintainable code. <strong>Objective. </strong>To develop and evaluate the effectiveness of a method for automatic generation of maintainable web application tests using LLM based on HTML structure analysis and the Page Object Model(POM) pattern. <strong>Hypotheses. </strong>Primary hypothesis: combining LLM with a two-stage generation approach and the POM pattern will enable the creation of maintainable tests, reducing development time by at least one-third (to 67% or less) while preserving code readability. Secondary hypothesis: the success rate of automatic generation will be inversely proportional to the complexity of interface components. <strong>Methods and materials. </strong>The study employed an approach based on Playwright, LLM, and a two-stage generation procedure with intermediate validation. Testing was conducted on four components of an SPA application for virtual infrastructure management. Validation of results was performed by a team of three testers who assessed the correctness and readability of generated tests. <strong>Results. </strong>The proposed method achieved high success rates in automatic test generation and substantial reduction in time costs for test creation. The two-stage procedure with intermediate validation enabled localization of a significant portion of errors at the early stage of Page Object creation. Automatically generated tests provided coverage of most required functionality while maintaining code readability. An inverse relationship between generation success and interface component complexity was confirmed: standardized interfaces demonstrated significantly higher success rates. <strong>Conclusions.</strong> The proposed method provides substantial time savings in creating a baseline test suite while maintaining quality and maintainability. The approach is recommended for early stages of feature development with expert control retained for validating critical scenarios. The method is particularly effective for projects with frequent interface changes, large volumes of regression testing, and components with standardized interfaces.</p>

We present Ketchum, a tool that was developed to improve the productivity of simulation-based functional verification by providing two capabilities: (1) automatic test generation and (2) unreachability analysis. Given a set of interesting signals in the design under test (DUT), automatic test generation creates input stimuli that drive the DUT through as many different combinations (called coverage states) of these signals as possible to thoroughly exercise the DUT. Unreachability analysis identifies as many unreachable coverage states as possible.Ketchum differs from the previous published results for several reasons. First, Ketchum provides 10x higher capacity than previous published results. The higher capacity is achieved by carefully orchestrating simulation and multiple formal methods including symbolic simulation, SAT-based BMC, symbolic fixpoint computation and automatic abstraction. Second, Ketchum performs not only automatic test generation but also unreachability analysis, which enables the test generation effort to be focused on coverage states that are not unreachable. Third, the backbone of Ketchum is an off-the-shelf commercial simulator. It enables Ketchum to reach deep states of the design quickly and supports simulation monitors through the standard API of the simulator during test generation.We applied Ketchum to several industrial designs, including the picoJava microprocessor from SUN and the DW8051 microcontroller from Synopsys and obtained very promising results. The experiments show that Ketchum can (1) handle design blocks containing more than 4500 latches and 170K gates, (2) reach up to 6x more coverage states than random simulation and (3) identify a majority of the unreachable coverage states.

Model Based Testing (MBT) is a relatively an approach widely discussed in software testing. It extends test automation from test design to test execution using automatic test generation from models. The effective use of this approach requires new skills, technology and knowledge, such as test modelling skills, but also good programs and tool support. This paper explores a design of a Behaviour driven test automation framework using MBT and how it can be effectively used during Agile Development. The automation framework is experimented upon the integration of the Graphwalker, a Model Based Graphical User Interface test generator with behaviour driven development framework and Robot Framework.

Boundary Scan is being added to the manufacturing test strategy. With no experience, where do you start? An organization venturing into this new territory must adopt changes in Test and Design processes to achieve a successful implementation as an enhancement to board test. Highlighting the benefits for the effort is key to getting buy-in from all groups involved from concept to product, especially the Design group. As experience is gained, less interaction between groups is required to implement Boundary Scan, but that relationship can never be severed without compromising the design or manufacturability. Addressing the issues of Design For Test (DFT) early in a product life-cycle will prevent costly oversights and mistakes. A methodical approach is necessary to design in Boundary Scan testability. Very complex designs can require significant design effort during schematic capture that hardware developers are not accustomed to providing. Often times, complex IC access points for developers share pins with the IEEE 1149.1 Test Access Port (TAP). Care must be taken to prevent a manufacturable Boundary Scan design from disabling a design debug port (and vice versa), requiring close coordination between Test and Development during this phase of the design. In addition, both groups must deal with the requirements for signal integrity so that test and programming time targets can be achieved. Solutions may include dedicated Boundary Scan components designed into the product and test fixture. Upon schematic completion, generation of test and programming vectors follows. (Vectors are the logic bits that are loaded into the Boundary Scan registers – taken from the term for a one-dimensional array.) The Automatic Test Pattern Generator (ATPG) tool must be capable of handling the specific requirements of your design such as programming of persistent memory and management of partitioning devices. Details of vector generation are beyond the scope of this discussion, however care and effort to ensure a match between the design netlist and physical manufacturing to achieve maximum test coverage may require changes to the development and production cycle of a new design and should be considered in scheduling resources. Bringing all of the prior efforts together to test a product doesn't complete the process. Deficient Boundary Scan implementations in ICs and oversights in the design can result in significant delays during the test of initial prototypes, delays that have never been experienced in a test cycle. These must be overcome through interaction with the part suppliers and the design group, and rectified to satisfy test coverage requirements. This method will be explained in generic terms to assist the reader in employing the technique in a product design with specific examples given from the implementation of boundary scan in a dense and complex Motorola Cellular Infrastructure board-set. The shift in test method for Motorola from bed-of-nails to Boundary Scan was intensified by the incorporation of a system-level 1149.1 architecture to re-use designed-in product features for postproduction testing.

In this paper, we present an automatic grammar-based test generation approach which takes a symbolic grammar as input, requires zero control input from users, and produces well-distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. The adjustment is based on a tabling strategy to keep track of the recursion of each grammar variable. We further present a test generation coverage tree illustrating the distribution of generated test cases and their detailed derivations, more importantly, it provides various implicit balance control mechanisms. We implemented this approach in a Java-based system, named Gena. Experimental results demonstrate the effectiveness of our test generation approach and show the balanced distribution of generated test cases over grammatical structures.KeywordsTest GenerationCoverage TreeProduction RuleSoftware Product LineGenerate Test CaseThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Today, VLSI design has progressed to a stage where it needs to incorporate methods of testing circuits. The Automatic Test Pattern Generation (ATPG) is a very attractive method and feasible on almost any combinational and sequential circuit. Currently available automatic test pattern generators (ATPGs) generate test sets that may be excessively long. Because a cost of testing depends on the test length, compaction techniques have been used to reduce that length. The motivation for studying test compaction is twofold. Firstly, by reducing the test sequence length, the memory requirements during the test application and the test application time are reduced. Secondly, the extent of test compaction possible for deterministic test sequences indicates that test pattern generators spend a significant amount of time generating test vectors that are not necessary. The compacted test sequences provide a target for more efficient deterministic test generators. Two types of compaction techniques exist: dynamic and static. The dynamic test sequence compaction performs compaction concurrently with the test generation process and often requires modification of the test generator. The static test sequence compaction is done in a post-processing step to the test generation and is independent of the test generation algorithm and process. In the thesis, a new idea for static compaction of test sequences for synchronous sequential circuits has been proposed. Our new method - SUSEM (Set Up Sequence Elimination Method) uses the circuit state information to eliminate some setup sequences for the target faults and consequently reduce the test sequence length. The technique has been used for the test sequences generated by HITEC test generator. ISCAS89 benchmark circuits were used in our experiments, for some circuits which have a large number of target faults and relatively small number of flip-flops, the very significant compactions have been obtained. The more important is that this method can be used to improve the test generation procedure unlike most static compaction methods which blindly or randomly remove parts of test vectors and cannot be used to improve the test generators.

Testing is an essential process for ensuring the quality of the software. Designing software with as few errors as possible in most embedded systems is often critical. Resource usage is a significant concern for proper behaviour because of the very nature of embedded systems. To design energy-efficient systems, approaches are needed to catch desirable consumption points and correct them before deployment. Model-based testing can reduce testing effort, one testing method that allows for automatic test generation. However, this technique has yet to be studied extensively for revealing resource usage anomalies in embedded systems development. UPPAAL SMC is a statistical model-checking tool that can model the system’s resource usage. This paper shows the tooling needed to achieve this and experimental results on automated test generation and selection using mutation analysis in UPPAAL SMC and how this is applied to a Brake by Wire industrial system. The evaluation shows that this approach, which we call MATS, is applicable and efficient for energy-based test generation.

Adapting automated test generation to GUI testing of industry applications

Unit testing is an established practice in professional software development. However, in high-performance computing (HPC) with its scientific applications, it is not widely applied. Besides general problems regarding testing of scientific software, for many HPC applications the effort of creating small test cases with a consistent set of test data is high.We have created a tool called FortranTestGenerator, that significantly reduces the effort of creating unit tests for subroutines of an existing Fortran application. It is based on Capture & Replay (C&R), that is, it extracts data while running the original application and uses the extracted data as test input data. The tool automatically generates code for capturing the input data and a basic test driver which can be extended by the developer to an appropriate unit test. A static source code analysis is conducted, to reduce the number of captured variables. Code is generated based on flexibly customizable templates. Thus, both the capturing process and the unit tests can easily be integrated into an existing software ecosystem.Since most HPC applications use message passing for parallel processing, we also present an approach to extend our C&R model to MPI communication. This allows extraction of unit tests from massively parallel applications that can be run with a single process.

We present an approach in automatic test generation that combines features of static analysis and bounded symbolic computation that is capable of producing a test suite that can be used to declare a program under test safe within bounds. We first use the results produced by static analysis which will identify a list of potential errors in the program. We restrict our search to the locations where errors can exist and aim to find exactly one test case per real bug. We have built a prototype tool (called Batg) that implements our approach. We report the results of running it on a number of benchmarks from well known benchmarking suites. We compare Batgto KLEE (an automatic test generation framework) and CBMC(a bounded model checker). This comparison is based on the time taken by the tools, the number of bugs found and the number of generated test cases. We analyse the results of our experiment, demonstrating the benefits of our approach.

Suboptimal design of power delivery networks (PDNs) may cause performance deterioration and severe functional failures on high-speed computer platforms. Voltage regulators (VRs) distribute controlled voltage in the PDN to the active devices, providing a steady power supply at a desired DC voltage level with an acceptable noise level or ripple. Unacceptable voltage drops can be caused by transient switching currents at the devices. Many decoupling capacitors are commonly used to lower the PDN impedance profile in order to reduce power supply noise and to supply fast transient current to switching devices. However, commercially available decoupling capacitors typically present large manufacturing variability. In this article, we first propose an optimization methodology that gradually finds the best compensation parameter values of a buck converter VR to meet suitable stability criteria. Simultaneously, the number of parallel decoupling capacitors in the PDN is minimized while meeting a frequency-domain impedance profile specification and a time-domain minimum voltage droop requirement under nominal parameter values. Finally, a statistical analysis, yield estimation, and yield optimization of the nominally optimized PDN subject to large decoupling capacitor tolerances is presented. We consider the impedance profile, transient voltage droop, and VR stability as the responses of interest for yield calculation.

Randoop is a well-known tool that proposes a feedback-directed algorithm for automatic and random generation of unit tests for a given Java class. It automatically generates two test suites for the class under test: (1) an error-revealing test suite, and (2) a regression test suite. Despite successful experiences with applying Randoop on real world projects like Java Development Kit (JDK) which have led to creation of error-revealing tests and identification of real bugs, it has not been investigated in the literature how useful are the regression test suites generated by Randoop. In this paper, we have investigated flakiness and fragility of Randoop’s regression tests during evolution of 5 open source Java projects with a total of 78 versions. The results demonstrate that the flakiness of the regression tests is not generally noticeable, since in our dataset, only 5% of the classes have at least one flaky regression tests. In addition, test fragility analysis reveals that in most versions of the projects under study, the regression tests generated by Randoop could be successfully executed on many of later versions. Actually, for 2 out of 5 projects in the experiments, the regression tests generated for each version could be successfully executed on all the later versions of the project.

Software industry produces high quality software, in that software testing plays a vital role. In software testing automatic generation can reduce the process time and cost. Automatic generation of object-oriented unit tests in software testing is a challenging task especially in white box testing. In software engineering Software testing becomes challenging issue with the development of software tools. Using Genetic Algorithm will provide best results in the software testing toolkit. This paper suggests an upgrading of automated search-based test generation of test cases in order to obtain high branch coverage in white box testing. The proposed technique outperforms providing greater efficiency in testing technique which combines Genetic Algorithm and Symbolic Execution named GASE. GASE automatically generates test data in order to obtain high branch coverage in white box testing aiming effective testing process which can be extended for software testing in cloud computing where it becomes necessity.