Can operational profile coverage explain post‐release bug detection?

ince programmers write programs there has always been a need to analyze the correctness of these programs, which is mostly done by testing. However, testing our programs does not give any direct quality guarantee on them, because it highly depends on the used test data set. Numerous code coverage metrics can be applied to measure the quality of our test set, but the majority of them were primarily designed for imperative programs, and they rely mostly on control structures like branching and looping. The problem is that expression-heavy programs and functional programming languages normally do not have these structures. Hence, the corresponding code coverage metrics are unsuitable at least, but mainly useless for these kinds of programs. In this paper we propose a new code coverage (domain coverage) metric, which is based on (arithmetic) expressions. The relations and effects among them are taken into account, such as some kind of semantics information about the programming language constructs. The paper also presents an automated test data generation method, which is related to domain coverage, and aims to reach the highest possible coverage ratio.

This paper discusses an MC/DC analysis tool built for the Erlang programming language. Code coverage metrics are one way to measure the adequacy of a test suite, however, despite widespread industrial use in business-critical software, the only coverage metric readily available for Erlang is statement coverage, provided by the Cover tool that is distributed by the OTP standard libraries. An alternative to statement cover is Multiple Condition/Decision Coverage (MC/DC) -- one of the most extensive coverage metrics, which is mandated by many software safety standards. Here we describe the application of MC/DC analysis to Erlang programs, and include an extension to traditional MC/DC analysis that applies the underlying philosophy to the pattern matching decision structures of functional languages. We have implemented the approach in the Smother tool that we also describe here as well as its use by our industrial project partners.

Previous works have shown that high-level decision diagrams (HLDD-s) are suitable for system representation for analyzing code coverage metrics. This is due to the fact that HLDD models implicitly represent classical code coverage items, such as statement and branch coverage. However, research on the properties of HLDD-s, which contribute to the accuracy of coverage assessment, is missing. Current paper proposes a set of HLDD manipulations in order to generate diagrams that would allow more stringent code coverage measurement without sacrificing performance, i.e., computation time and memory requirements. The techniques include generation of HLDD-trees from Hardware Description Language (HDL)descriptions and two types of HLDD collapsing methods, which are a generalization of the BDD reduction rules. Experiments on ITC99 benchmark circuits show that the code coverage assessment based on the proposed HLDD manipulation is more stringent than what can be achieved with classical methods. At the same time, the model is well scalable because HLDD generation is terminated in the HDL variables.

Coverage-guided fuzzing's aggressive, high-volume testing has helped reveal tens of thousands of software security flaws. While executing billions of test cases mandates fast code coverage tracing, the nature of binary-only targets leads to reduced tracing performance. A recent advancement in binary fuzzing performance is Coverage-guided Tracing (CGT), which brings orders-of-magnitude gains in throughput by restricting the expense of coverage tracing to only when new coverage is guaranteed. Unfortunately, CGT suits only a basic block coverage granularity -- yet most fuzzers require finer-grain coverage metrics: edge coverage and hit counts. It is this limitation which prohibits nearly all of today's state-of-the-art fuzzers from attaining the performance benefits of CGT. This paper tackles the challenges of adapting CGT to fuzzing's most ubiquitous coverage metrics. We introduce and implement a suite of enhancements that expand CGT's introspection to fuzzing's most common code coverage metrics, while maintaining its orders-of-magnitude speedup over conventional always-on coverage tracing. We evaluate their trade-offs with respect to fuzzing performance and effectiveness across 12 diverse real-world binaries (8 open- and 4 closed-source). On average, our coverage-preserving CGT attains near-identical speed to the present block-coverage-only CGT, UnTracer; and outperforms leading binary- and source-level coverage tracers QEMU, Dyninst, RetroWrite, and AFL-Clang by 2-24x, finding more bugs in less time.

The logical interrelationship between different code coverage types has been well studied, but less so their evolution through time or test. We study the dynamic relationship of four coverage types, namely, block, decision, c-use and p-use by comparing their growth using empirical coverage data generated from extensive testing of a software application with 35 KLOC of code. Our results indicate that as testing increases, the growth trends for each coverage type are surprisingly similar. Not only is each trend consistent with an underlying lognormal distribution of event rate, but also the parameters of the fitted lognormal distributions are closely related. Within the limits of the data, we find quantitative relations between the four coverage types. The paper thus takes a significant step in linking concepts from prior studies of software test sufficiency, test efficiency, and reliability in the context of software execution

Traditional test generation methodologies for peripheral cores resort heavily to low-level descriptions of the circuit, leading to long generation times. Methodologies based on high-level descriptions can only be used if a clear relationship exists between the measured high-level coverage and the gate-level fault coverage. Even in medium complexity circuits, however, a direct relationship between code coverage metrics and fault coverage is not guaranteed, while other RT level metrics require an effort comparable to the use of low level descriptions. To overcome this problem, in the case of peripheral cores, a new approach is proposed: FSMs embedded in the system are identified and dynamically extracted via simulation, while transition coverage is used as a measure of how much the system is exercised. Model extraction and coverage maximization are performed concurrently in a completely automated way. This new technique is exploited to drive an unsupervised methodology for generating tests for peripheral cores. Experimental analysis shows the effectiveness of the approach.

Creating and implementing fault-tolerant distributed algorithms is a challenging task in highly safety-critical industries. Using formal methods supports design and development of complex algorithms. However, formal methods are often perceived as an unjustifiable overhead. This paper presents the experience and insights when using TLA+ and PlusCal to model and develop fault-tolerant and safety-critical modules for TAS Control Platform, a platform for railway control applications up to safety integrity level (SIL) 4. We show how formal methods helped us improve the correctness of the algorithms, improved development efficiency and how part of the gap between model and implementation has been closed by translation to C code. Additionally, we describe how we gained trust in the formal model and tools by following a specific design process called property-driven design, which also implicitly addresses software quality metrics such as code coverage metrics.

Most Systems-on-a-Chips include a custom microprocessor core, and time and resource constraints make the design of such devices a challenging task. This paper presents a simulation-based methodology for the automatic completion and refinement of verification test sets. The approach extends the µGP, an evolutionary test program generator, with the possibility to enhance existing test sets. Already devised test programs are not merely included in the new set, but assimilated and used as a starting point for a new test-program cultivation task. Reusing existing material cuts down the time required to generate a verification test set during the microprocessor design. Experimental results are reported on a small pipelined microprocessor, and show the effectiveness of the approach. Additionally, the use of the proposed methodology enabled to experimentally analyze the relationship of the different code coverage metrics used in the test program generation.

Coverage-Guided fuzzing is the type of fuzzing which focuses on the code or branch coverage. It is mainly efficient in detecting buffer overflow. For the code coverage, it is an important parameter of performance evaluation of the Coverage-Guided fuzzing tools, since the higher coverage means higher possibility of bug detection. However, the timeout set for the fuzzing also affects the efficiency, due to the fact that the growth rate of code coverage will gradually become slower as the running time becoming longer. Setting a timeout that is too long cannot help improving the coverage nor the bug detected, and it will be a complete waste of time. Since selecting the fuzzing time might be quite confusing for testers, in this paper, the author decided to discover the relationship of the growth of the coverage and the running time of the. AFL and FuzzFactory were used for the evaluation and the timeout were set as 1 second, 1 minute, 1 hour, 6 hours and 12 hours respectively. The results showed the relationship between fuzzing time and the performance, for all fuzzers and the effect between different fuzzers.

In software testing, different testers focus on different aspects of the software such as functionality, performance, design, and other attributes. While many tools and coverage metrics exist to support testers at the code level, not much support is targeted for testers who want to inspect the output of a program such as a dynamic web application. To support this category of testers, we propose a family of output-coverage metrics (similar to statement, branch, and path coverage metrics on code) that measure how much of the possible output has been produced by a test suite and what parts of the output are still uncovered. To do that, we first approximate the output universe using our existing symbolic execution technique. Then, given a set of test cases, we map the produced outputs onto the output universe to identify the covered and uncovered parts and compute output-coverage metrics. In our empirical evaluation on seven real-world PHP web applications, we show that selecting test cases by output coverage is more effective at identifying presentation faults such as HTML validation errors and spelling errors than selecting test cases by traditional code coverage. In addition, to help testers understand output coverage and augment test cases, we also develop a tool called WebTest that displays the output universe in one single web page and allows testers to visually explore covered and uncovered parts of the output.

This paper deals with a benchmark of automated test generation methods for software testing. The existing methods are usually demonstrated using quite different examples. This makes their mutual comparison difficult. Additionally, the quality of the methods is often evaluated using code coverage or other metrics, such as generated tests count, test generation time, or memory usage. The most important feature -the ability of the method to find realistic errors in realistic applications -is only rarely used. To enable mutual comparison of various methods and to investigate their ability to find realistic errors, we propose a benchmark consisting of several applications with wittingly introduced errors. These errors should be found by the investigated test generation methods during the benchmark. To enable an easy introduction of various errors of various types into the benchmark applications, we created the Testing Applications Generator (TAG) tool. The description of the TAG along with two applications, which we developed as a part of the intended benchmark, is the main contribution of this paper.

Refasctoring is a technique used in software development to improve the quality of code without changing its functionality. One metric that is often used to measure code quality is Code Coverage. This study aims to examine refactoring techniques that can maximize Code Coverage Metric. Through the study, identification, evaluation, and summary of empirical evidence from various literature sources are carried out. The results of this study provide guidance on effective refactoring techniques to improve Code Coverage as well as other positive impacts for software development. There are ten refactoring techniques that can be used to improve Code Coverage Metrics in software testing.

We propose Specification-Based Test Generation (SBTG) which automatically generates functional tests directly from specification, rather than the HDL description of the design. The main benefit of generating tests from the specification is the ability to detect Specification-based Translation Errors (SBTEs) that occur due to a misunderstanding of the specification. Our results show that our test generation approach is more effective at detecting these errors than approaches that generate tests from the HDL code to maximize code coverage metrics.

In fast-paced software development environments, the efficiency of Continuous Integration (CI) pipelines is frequently hampered by the escalating time required to execute comprehensive test suites. This paper presents an AI-driven framework for Test Impact Prediction (TIP) designed to intelligently identify and prioritize only the most relevant test cases affected by a specific code commit, thereby significantly reducing test execution time without compromising quality. By training sophisticated AI models on a rich array of data—including detailed code coverage metrics, granular commit differences (diffs), historical mappings between code changes and affected tests, and intricate dependency graphs of both code and tests—our system accurately predicts which tests are most likely to be impacted. This intelligent prediction enables the dynamic selection and prioritization of tests within CI pipelines, leading to an anticipated reduction in test execution time by over 50% with minimal risk of regressions. This approach not only accelerates feedback cycles for developers but also optimizes computational resource utilization, fostering a more agile and efficient development workflow crucial for modern software delivery.

Coverage-based fuzzing has been actively studied and widely adopted for finding vulnerabilities in real-world software applications. With coverage information, such as statement coverage and transition coverage, as the guidance of input mutation, coverage-based fuzzing can generate inputs that cover more code and thus find more vulnerabilities without prerequisite information such as input format. Current coveragebased fuzzing tools treat covered code equally. All inputs that contribute to new statements or transitions are kept for future mutation no matter what the statements or transitions are and how much they impact security. Although this design is reasonable from the perspective of software testing that aims at full code coverage, it is inefficient for vulnerability discovery since that 1) current techniques are still inadequate to reach full coverage within a reasonable amount of time, and that 2) we always want to discover vulnerabilities early so that it can be fixed promptly. Even worse, due to the non-discriminative code coverage treatment, current fuzzing tools suffer from recent anti-fuzzing techniques and become much less effective in finding vulnerabilities from programs enabled with anti-fuzzing schemes.