Substra

Adapting automated test generation to GUI testing of industry applications

We present Ketchum, a tool that was developed to improve the productivity of simulation-based functional verification by providing two capabilities: (1) automatic test generation and (2) unreachability analysis. Given a set of interesting signals in the design under test (DUT), automatic test generation creates input stimuli that drive the DUT through as many different combinations (called coverage states) of these signals as possible to thoroughly exercise the DUT. Unreachability analysis identifies as many unreachable coverage states as possible.Ketchum differs from the previous published results for several reasons. First, Ketchum provides 10x higher capacity than previous published results. The higher capacity is achieved by carefully orchestrating simulation and multiple formal methods including symbolic simulation, SAT-based BMC, symbolic fixpoint computation and automatic abstraction. Second, Ketchum performs not only automatic test generation but also unreachability analysis, which enables the test generation effort to be focused on coverage states that are not unreachable. Third, the backbone of Ketchum is an off-the-shelf commercial simulator. It enables Ketchum to reach deep states of the design quickly and supports simulation monitors through the standard API of the simulator during test generation.We applied Ketchum to several industrial designs, including the picoJava microprocessor from SUN and the DW8051 microcontroller from Synopsys and obtained very promising results. The experiments show that Ketchum can (1) handle design blocks containing more than 4500 latches and 170K gates, (2) reach up to 6x more coverage states than random simulation and (3) identify a majority of the unreachable coverage states.

Automated test generation from SDL specifications

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

Testing is an essential process for ensuring the quality of the software. Designing software with as few errors as possible in most embedded systems is often critical. Resource usage is a significant concern for proper behaviour because of the very nature of embedded systems. To design energy-efficient systems, approaches are needed to catch desirable consumption points and correct them before deployment. Model-based testing can reduce testing effort, one testing method that allows for automatic test generation. However, this technique has yet to be studied extensively for revealing resource usage anomalies in embedded systems development. UPPAAL SMC is a statistical model-checking tool that can model the system’s resource usage. This paper shows the tooling needed to achieve this and experimental results on automated test generation and selection using mutation analysis in UPPAAL SMC and how this is applied to a Brake by Wire industrial system. The evaluation shows that this approach, which we call MATS, is applicable and efficient for energy-based test generation.

Search-based testing can be applied to automatically generate unit tests that achieve high levels of code coverage on object-oriented classes. However, test generation takes time, in particular if projects consist of many classes, like in the case of the Guava library. To allow search-based test generation to scale up and to integrate it better into software development, continuous test generation applies test generation incrementally during continuous integration. In this paper, we report on the application of continuous test generation with EvoSuite at the SSBSE’15 challenge on the Guava library. Our results show that continuous test generation reduces the time spent on automated test generation by 96 %, while increasing code coverage by 13.9 % on average.

Component-based software systems (CBSSs) enable modular design and reuse, but their integration poses substantial testing challenges. Integration faults often stem from interactions between components and are difficult to detect with isolated unit testing. Existing integration testing practices require significant manual effort, particularly in configuring test environments and maintaining test suites as systems evolve.This dissertation introduces the Interaction-based Integration (IBI) Testing Approach, a novel approach to automate integration testing by reusing unit test cases. The IBI testing approach comprises the IBI Metamodel, which captures structural, behavioral, and interaction expectation aspects of CBSSs, and the IBI Testing Process, which operationalizes the IBI Metamodel. The approach leverages unit test cases to derive interaction expectations, and generate so called interaction test cases, by reusing the test outputs of one unit test case as inputs for another. It thereby simulates system behavior, enabling integration testing without the need for dedicated integration test cases or environments. In contrast to existing approaches, which require dedicated integration environments and handcrafted test cases, this work enables integration validation through existing unit test artifacts.The IBI testing approach is formalized using property graphs and regular path queries. It is implemented in the InterACt tool, evaluated through a demonstration case study on a microservice system and an exploratory case study in collaboration with an industry partner. The results confirm the feasibility of the IBI Testing Approach while identifying practical limitations that guide future improvements.This work contributes a reusable integration testing model, a formalized testing process, and a tool-supported implementation that provide a novel approach to integration testing of CBSSs. The findings highlight the potential of reusing unit test cases for integration testing, while also addressing the challenges and limitations of this approach in practice.

A novel approach for automatic software test generation is presented, which combines ideas from structural and functional testing as well as formal verification methods. It involves as an intermediate step, the construction of graphs and refinement types, which can be regarded as an automatically constructed semi-specification and used for formal verification. The technique is illustrated using a simple functional language, with algorithms for assigning refinement types and for test generation. Some desirable theoretical properties of the approach are briefly considered. It is also compared informally to other well-known as well as new techniques for automatic test generation.

<p><strong>Context and relevance.</strong><strong> </strong>Modern web application development requires continuous testing, but maintaining automated tests is becoming increasingly labor-intensive due to locator instability and growing interface complexity. The emergence of Large Language Models (LLM) opens new opportunities for test creation automation, but their practical application faces challenges in processing large HTML documents and the need to create maintainable code. <strong>Objective. </strong>To develop and evaluate the effectiveness of a method for automatic generation of maintainable web application tests using LLM based on HTML structure analysis and the Page Object Model(POM) pattern. <strong>Hypotheses. </strong>Primary hypothesis: combining LLM with a two-stage generation approach and the POM pattern will enable the creation of maintainable tests, reducing development time by at least one-third (to 67% or less) while preserving code readability. Secondary hypothesis: the success rate of automatic generation will be inversely proportional to the complexity of interface components. <strong>Methods and materials. </strong>The study employed an approach based on Playwright, LLM, and a two-stage generation procedure with intermediate validation. Testing was conducted on four components of an SPA application for virtual infrastructure management. Validation of results was performed by a team of three testers who assessed the correctness and readability of generated tests. <strong>Results. </strong>The proposed method achieved high success rates in automatic test generation and substantial reduction in time costs for test creation. The two-stage procedure with intermediate validation enabled localization of a significant portion of errors at the early stage of Page Object creation. Automatically generated tests provided coverage of most required functionality while maintaining code readability. An inverse relationship between generation success and interface component complexity was confirmed: standardized interfaces demonstrated significantly higher success rates. <strong>Conclusions.</strong> The proposed method provides substantial time savings in creating a baseline test suite while maintaining quality and maintainability. The approach is recommended for early stages of feature development with expert control retained for validating critical scenarios. The method is particularly effective for projects with frequent interface changes, large volumes of regression testing, and components with standardized interfaces.</p>

Massive Open Online Courses (MOOCs) have recently gained high popularity among various universities and even in global societies. A critical factor for their success in teaching and learning effectiveness is assignment grading. Traditional ways of assignment grading are not scalable and do not give timely or interactive feedback to students. To address these issues, we present an interactive-gaming-based teaching and learning platform called Pex4Fun. Pex4Fun is a browser-based teaching and learning environment targeting teachers and students for introductory to advanced programming or software engineering courses. At the core of the platform is an automated grading engine based on symbolic execution. In Pex4Fun, teachers can create virtual classrooms, customize existing courses, and publish new learning material including learning games. Pex4Fun was released to the public in June 2010 and since then the number of attempts made by users to solve games has reached over one million. Our work on Pex4Fun illustrates that a sophisticated software engineering technique -- automated test generation -- can be successfully used to underpin automatic grading in an online programming system that can scale to hundreds of thousands of users.

This paper adresses the generation of test cases for testing the conformance of a reactive black-box implementation with respect to its specification. We aim at extending the principles and algorithms of model-based testing for recursive interprocedural specifications that can be modeled by Push-Down Systems (PDS). Such specifications may be more compact than non-recursive ones and are more expressive.The generated test cases are selected according to a test purpose, a (set of) scenario of interest that one wants to observe during test execution. The test generation method we propose in this paper is based on program transformations and a coreachability analysis, which allows to decide whether and how the test purpose can still be satisfied. However, despite the possibility to perform an exact analysis, the inability of test cases to inspect their own stack prevents it from using fully the coreachability information. We discuss this partial observation problem, its consequences, and how to minimize its impact.KeywordsTest PurposeLabel Transition SystemProgram TransformationGenerate Test CaseTest ExecutionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Model Based Testing (MBT) is a relatively an approach widely discussed in software testing. It extends test automation from test design to test execution using automatic test generation from models. The effective use of this approach requires new skills, technology and knowledge, such as test modelling skills, but also good programs and tool support. This paper explores a design of a Behaviour driven test automation framework using MBT and how it can be effectively used during Agile Development. The automation framework is experimented upon the integration of the Graphwalker, a Model Based Graphical User Interface test generator with behaviour driven development framework and Robot Framework.

Software industry produces high quality software, in that software testing plays a vital role. In software testing automatic generation can reduce the process time and cost. Automatic generation of object-oriented unit tests in software testing is a challenging task especially in white box testing. In software engineering Software testing becomes challenging issue with the development of software tools. Using Genetic Algorithm will provide best results in the software testing toolkit. This paper suggests an upgrading of automated search-based test generation of test cases in order to obtain high branch coverage in white box testing. The proposed technique outperforms providing greater efficiency in testing technique which combines Genetic Algorithm and Symbolic Execution named GASE. GASE automatically generates test data in order to obtain high branch coverage in white box testing aiming effective testing process which can be extended for software testing in cloud computing where it becomes necessity.

The increasing popularity of dynamically-typed programming languages, such as JavaScript or Python, requires specific support methods for developers to avoid pitfalls arising from the dynamic nature of these languages. Static analyses are frequently used but the dynamic type systems limit their applicability. Dynamic analyses, in contrast, depend on the execution of the code under analysis, and thus depend on the quality of existing tests. This quality of the test suite can be improved by the use of automated test generation but automated test generation for dynamically-typed programming languages itself is hard due to the lack of type information in the programs. The limitations of each of these approaches will be overcome by iteratively combining test generation with static and dynamic analysis techniques for dynamically-typed programs.

There is extensive literature on automated test generation using reactive design models, where control is determined by events. In contrast, the (idealised) simulation paradigm defines control through cycles dictated by the passage of time. Within each cycle, inputs are read and processed, and outputs are provided, all instantaneously, and afterwards time progresses. To exercise a simulation using tests generated from a reactive design model requires changes to the tests to take into account this paradigm shift. This paper focuses on automation of the necessary changes and of the use of the resulting tests in a simulation campaign. Based on a notion of conformance that establishes whether a simulation is correct with respect to a reactive design, we (1) identify the reactive tests that are meaningful; (2) define a process to convert those tests; (3) provide an algorithm to execute those tests; and (4) prove soundness and completeness of our approach. Our work is described in the context of the RoboStar framework for model-based development of control software for robotics applications, and its process algebraic semantics. The testing approach we propose here represents a significant advancement in the current testing practices within the field of robotics, where simulations are widely used.