A Case Study in Branch Testing Automation

Coverage metrics, which evaluate the ability of a test sequence to detect design faults, are essential to the validation process. A key source of difficulty in determining fault detection is that the control flow path traversed in the presence of a fault cannot be determined. Fault detection can only be accurately determined by exploring the set of all control flow paths, which may be traversed as a result of a fault. We present a coverage metric that determines the propagation of fault effects along all possible faulty control flow paths. The complexity of exploring multiple control flow paths is greatly alleviated by heuristically pruning infeasible control flow paths using the algorithm that we present. The proposed coverage metric provides high accuracy in designs that contain complex control flow. The results obtained are promising.

sushi and tardis are two automatic test generators for Java programs. They generate test suites, aiming to satisfy the branch coverage adequacy criterion. Both tools exploit symbolic execution to analyze the control flow paths of the program under test and generate path conditions, i.e., constraints on the program inputs that determine the execution of a path, focusing in particular on control flow paths that lead to uncovered branches. Then, sushi and tardis exploit these path conditions with a search-based algorithm to generate unit tests (in JUnit) that execute those program paths. In this paper we report the results of sushi and tardis at the seventh edition of the unit testing tool competition at the 2019 edition of the International Workshop on Search-Based Software testing.

Impact of code context and prompting strategies on automated unit test generation with modern general-purpose large language models

Unit testing is crucial for software development and maintenance. Effective unit testing ensures and improves software quality, but writing unit tests is time-consuming and labor-intensive. Recent studies have proposed deep learning (DL) techniques or large language models (LLMs) to automate unit test generation. These models are usually trained or fine-tuned on large-scale datasets. Despite growing awareness of the importance of data quality, there has been limited research on the quality of datasets used for test generation. To bridge this gap, we systematically examine the impact of noise on the performance of learning-based test generation models. We first apply the open card sorting method to analyze the most popular and largest test generation dataset, Methods2Test, to categorize eight distinct types of noise. Further, we conduct detailed interviews with 17 domain experts to validate and assess the importance, reasonableness, and correctness of the noise taxonomy. Then, we propose CleanTest, an automated noise-cleaning framework designed to improve the quality of test generation datasets. CleanTest comprises three filters: a rule-based syntax filter, a rule-based relevance filter, and a model-based coverage filter. To evaluate its effectiveness, we apply CleanTest on two widely-used test generation datasets, i.e., Methods2Test and Atlas. Our findings indicate that 43.52% and 29.65% of datasets contain noise, highlighting its prevalence. Finally, we conduct comparative experiments using four LLMs (i.e., CodeBERT, AthenaTest, StarCoder, and CodeLlama7B) to assess the impact of noise on test generation performance. The results show that filtering noise positively influences the test generation ability of the models. Fine-tuning the four LLMs with the filtered Methods2Test dataset, on average, improves its performance by 67% in branch coverage, using the Defects4J benchmark. For the Atlas dataset, the four LLMs improve branch coverage by 39%. Additionally, filtering noise improves bug detection performance, resulting in a 21.42% increase in bugs detected by the generated tests.

Search-based techniques have been widely used for white-box test generation. Many of these approaches rely on the approach level and branch distance heuristics to guide the search process and generate test cases with high line and branch coverage. Despite the positive results achieved by these two heuristics, they only use the information related to the coverage of explicit branches (e.g., indicated by conditional and loop statements), but ignore potential implicit branchings within basic blocks of code. If such implicit branching happens at runtime (e.g., if an exception is thrown in a branchless-method), the existing fitness functions cannot guide the search process. To address this issue, we introduce a new secondary objective, called Basic Block Coverage (BBC), which takes into account the coverage level of relevant basic blocks in the control flow graph. We evaluated the impact of BBC on search-based unit test generation (using the DynaMOSA algorithm) and search-based crash reproduction (using the STDistance and WeightedSum fitness functions). Our results show that for unit test generation, BBC improves the branch coverage of the generated tests. Although small (sim 1.5%), this improvement in the branch coverage is systematic and leads to an increase of the output domain coverage and implicit runtime exception coverage, and of the diversity of runtime states. In terms of crash reproduction, in the combination of STDistance and WeightedSum, BBC helps in reproducing 3 new crashes for each fitness function. BBC significantly decreases the time required to reproduce 43.5% and 45.1% of the crashes using STDistance and WeightedSum, respectively. For these crashes, BBC reduces the consumed time by 71.7% (for STDistance) and 68.7% (for WeightedSum) on average.

Object-oriented programs cause a shift in focus from software units to the way software classes and components are connected. Thus, we are finding that we need less emphasis on unit testing and more on integration testing. The compositional relationships of inheritance and aggregation, especially when combined with polymorphism, introduce new kinds of integration faults, which can be covered using testing criteria that take the effects of inheritance and polymorphism into account. This paper demonstrates, via a set of experiments, the relative effectiveness of several coupling-based OO testing criteria and branch coverage. OO criteria are all more effective at detecting faults due to the use of inheritance and polymorphism than branch coverage.

In software testing process, test data generation represents an important step for high quality software, even for mobile devices. As proposed in previous works, a potential source for random data generation is represented by the UI layout files that are used for almost all mobile platforms (Android, iOS, Windows Phone/Mobile). This paper continues the previous work and presents a test data generation system based on Android layout files. The test data generator uses DSL files as input and generates test data that conform to several testing principles. The generated test data could be stored in XML files or any format required by the testing frameworks.

Various search-based test generation techniques have been proposed to automate the generation of unit tests fulfilling different criteria (e.g., line coverage, branch coverage, mutation score, etc.). Despite several advances made over the years, search-based unit test generation still suffers from a lack of guidance due to the limited amount of information available in the source code that, for instance, hampers the generation of complex objects. Previous studies introduced many strategies to address this issue, e.g., dynamic symbolic execution or seeding, but do not take the internal execution of the methods into account. In this paper, we introduce a novel secondary objective called commonality score, measuring how close the execution path of a test case is from reproducing a common or uncommon execution pattern observed during the operation of the software. To assess the commonality score, we implemented it in EvoSuite and evaluated its application on 150 classes from JabRef, an open-source software for managing bibliography references. Our results are mixed. Our approach leads to test cases that indeed follow common or uncommon execution patterns. However, if the commonality score can have a positive impact on the structural coverage and mutation score of the generated test suites, it can also be detrimental in some cases.

Despite the availability of a variety of program analysis tools, evaluation of these tools is difficult, as only few benchmark suites exist. Existing benchmark suites lack the uniformity needed for automation of experiments. We introduce the design of a uniform build/installation platform, which constitutes an important part of the solution. This platform is used to manage the build and test process, which is enhanced by a tool that analyzes the structure of unit tests. Benchmark applications lack detailed information about unit tests. Such knowledge is useful: For analysis algorithms that target specific program features, it is desirable to analyze only relevant tests. Using aspect-oriented programming, we wrap test execution and implement a tool providing coverage data of individual unit tests. Furthermore, the wrapper provides a front-end for the selection of subsets of a test suite. We successfully applied our tool to several large programs. This evaluation also gave us interesting insights about the quality of different test suites.

Component-based software systems (CBSSs) enable modular design and reuse, but their integration poses substantial testing challenges. Integration faults often stem from interactions between components and are difficult to detect with isolated unit testing. Existing integration testing practices require significant manual effort, particularly in configuring test environments and maintaining test suites as systems evolve.This dissertation introduces the Interaction-based Integration (IBI) Testing Approach, a novel approach to automate integration testing by reusing unit test cases. The IBI testing approach comprises the IBI Metamodel, which captures structural, behavioral, and interaction expectation aspects of CBSSs, and the IBI Testing Process, which operationalizes the IBI Metamodel. The approach leverages unit test cases to derive interaction expectations, and generate so called interaction test cases, by reusing the test outputs of one unit test case as inputs for another. It thereby simulates system behavior, enabling integration testing without the need for dedicated integration test cases or environments. In contrast to existing approaches, which require dedicated integration environments and handcrafted test cases, this work enables integration validation through existing unit test artifacts.The IBI testing approach is formalized using property graphs and regular path queries. It is implemented in the InterACt tool, evaluated through a demonstration case study on a microservice system and an exploratory case study in collaboration with an industry partner. The results confirm the feasibility of the IBI Testing Approach while identifying practical limitations that guide future improvements.This work contributes a reusable integration testing model, a formalized testing process, and a tool-supported implementation that provide a novel approach to integration testing of CBSSs. The findings highlight the potential of reusing unit test cases for integration testing, while also addressing the challenges and limitations of this approach in practice.

Software testing is an important stage of the software development life cycle. However, the test execution is time-consuming and costly. To accelerate the test execution, researchers have applied several methods to run the testing in parallel. One method of parallelizing the test execution is by using a GPU machine to distribute test case inputs among several threads running in parallel. However, when testing a program on a GPU, different test case inputs could have different control flow paths. This leads to branch divergence which may increase the execution time of testing. In addition, transferring data between host and device may increase the execution time of the test execution when data are allocated statically instead of dynamically. Also, different data sizes from different test case inputs may increase the number of memory transactions. To address these challenges, three studies were conducted. The first study investigated three programming models: CUDA Unified Memory, CUDA Non-Unified Memory, and OpenMP GPU Offloading to parallelize the test execution. This study utilized 11 benchmarks and parallelized their test suites by using these models. The evaluation was based on their performance in terms of execution time. The results showed that using CUDA Unified Memory to implement a test suite for GPU was the easiest and most productive programming model among the other two models. The second study discussed three challenges (control flow, memory access pattern, and input size of a program) when parallelizing test execution on two major computing platforms: multi-core CPU and GPU. It analyzed 14 programs from different domains to evaluate parallelizing test execution on the two platforms. The results showed that the cache size of a GPU and branch divergence significantly increase the execution time of many test suites of programs. The last study proposed a novel algorithm to minimize the branch divergence when testing an application on a GPU. Essentially, the algorithm arranges test inputs based on the warp size of a GPU. Test inputs that have similar control flow paths are grouped within the same warp to be executed in parallel. The validation and evaluation were conducted on six different domains of benchmarks (57 programs in total). The results showed that the algorithm speeds up the testing execution up to 3.8x and improves the warp execution efficiency up to 15x. Therefore, the algorithm reduces the branch divergence by minimizing the number of inactive threads per warp.

This paper presents the results of steady state compliance testing of phasor measurement units (PMUs) from three different vendors in a laboratory environment. Testing is performed by providing three phase voltage and current injections to the VT and CT inputs of the PMUs through Freja-300 stand-alone protection relay test set. Testing is performed according to the standard “IEEE C37.242-2013 - IEEE Guide for Synchronization, Calibration, Testing, and Installation of Phasor Measurement Units (PMUs) for Power System Protection and Control”. The paper discusses the test setup, testing process and overall test results of this project. The limitations of stand-alone testing equipment and recommendations for utilizing real-time hardware-in-the-loop simulation for thorough testing of PMUs are also presented.

Software industry produces high quality software, in that software testing plays a vital role. In software testing automatic generation can reduce the process time and cost. Automatic generation of object-oriented unit tests in software testing is a challenging task especially in white box testing. In software engineering Software testing becomes challenging issue with the development of software tools. Using Genetic Algorithm will provide best results in the software testing toolkit. This paper suggests an upgrading of automated search-based test generation of test cases in order to obtain high branch coverage in white box testing. The proposed technique outperforms providing greater efficiency in testing technique which combines Genetic Algorithm and Symbolic Execution named GASE. GASE automatically generates test data in order to obtain high branch coverage in white box testing aiming effective testing process which can be extended for software testing in cloud computing where it becomes necessity.

In this paper, we present the results of case studies conducted in three software companies in order to identify the improvement targets in their test processes. The test processes were modeled using the LAPPI (A Light-weight Technique to Practical Process Modeling and Improvement Target Identification) technique. Identifying improvement targets quickly and in a cost-effective way is important to maximize the value of SPI (Software Process Improvement) initiatives. The most significant problems identified in this study were the low level of unit testing, unknown repercussions of code changes, and missing exit criteria for testing. These findings support the results of earlier studies. Further, a less reported problem was identified. It appears that software companies find it difficult to integrate test automation with their manual test processes. Making the recurring test related problems known among the software industry helps to justify the need for test process improvement and standardization of test processes.

A scheme is presented for constructing prototype testing tools to experiment with issues in data-flow testing of arrays. Data-flow testing is a variant of path testing that falls between the extremes of simple run-time instrumentation, and full path testing. In data-flow testing, the paths selected for exercising are variants of def/use (DU) paths for program variables. Tool support for data-flow testing takes the form of static program analysis followed by run-time instrumentation. The output from a test analyzer is a list of untested, possible DU associations. With the novel approach to constructing testing-supporting tools, it is easy to experiment with new analysis methods. The authors describe the implementation of a DU-path analyzer as an example of this approach. Using the prototype tool, experiments and issues are considered raised by an analyzer that treats arrays differently from existing tools. Each array element is considered to be a distinct data-flow object. >