Hybrid Concolic Testing

This work describes the working of white-box testing for the Service-Oriented Architecture (SOA) based Application. Now, a days it is very essential to perform code coverage testing to understand the quality of software. This paper deals with the measurement of branch coverage percentage for the BPEL architecture that orchestrates all the services, which are distributed geographically. Here, we are testing the code coverage of BPEL architecture, which actually shows the invocations of services when it is required. This work integrates the existing open source tools, to automate the computation of branch coverage for SOA based application. This paper shows a novel technique for generation of test cases and computing branch coverage through our proposed tool.

This paper presents the first concolic testing framework explicitly designed for quantum programs. The framework introduces quantum constraint generation methods for quantum control statements that quantify quantum states and offers a symbolization method for quantum variables. Based on this framework, we generate path constraints for each concrete execution path of a quantum program. These constraints guide the exploration of new paths, with a quantum constraint solver determining outcomes to create novel input samples, thereby enhancing branch coverage. Our framework has been implemented in Python and integrated with Qiskit for practical evaluation. Experimental results show that our concolic testing framework improves branch coverage, generates high-quality quantum input samples, and detects bugs, demonstrating its effectiveness and efficiency in quantum programming and bug detection. Regarding branch coverage, our framework achieves more than 74.27% on quantum programs with under 5 qubits.

Simulation is widely used for validation of Register-Transfer-Level (RTL) models. While simulating with millions of random or constrained-random tests can cover majority of the functional scenarios, the number of remaining scenarios can still be huge (hundreds or thousands) in case of today's industrial designs. Hard-to-activate branches are one of the major contributors for such remaining/untested scenarios. While directed test generation techniques using formal methods are promising in activating branches, it is infeasible to apply them on large designs due to state space explosion. In this article, we propose a fully automated and scalable approach to cover the hard-to-activate branches using concolic testing of RTL models. While application of concolic testing on hardware designs has shown some promising results in improving the overall coverage, they are not designed to activate specific targets such as uncovered corner cases and rare scenarios. In other words, existing concolic testing approaches address state space explosion problem but leads to path explosion problem while searching for the uncovered targets. Our proposed approach maps directed test generation problem to target search problem while avoiding overlapping searches involving multiple targets. This article makes two important contributions. (1) We propose a directed test generation technique to activate a target by effective utilization of concolic testing on RTL models. (2) We develop efficient learning and clustering techniques to minimize the overlapping searches across targets to drastically reduce the overall test generation effort. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art methods in terms of test generation time (up to 205X, 69X on average) as well as memory requirements (up to 31X, 7X on average).

Concolic testing for functional languages

Recently, symbolic execution has been witnessed great success in applying symbolic execution to automatically generate high coverage test inputs for programs testing. While path explosion still a primary challenge in scaling concolic testing to large programs, meanwhile constraint solving is fundamental to concolic testing as a constraint solver is continuously invoked during analysis, while the main roadblock to performance of concolic testing engines is the time spent in constraint solving. To address this issue and achieve the purpose of selecting the path with simplest constraints to solve, this paper presents a new heuristic search strategy which determines the path priority based on the complexity of path constraints. We have implemented the strategy in angr, a binary-oriented concolic testing tools implemented by Shellphish, and evaluated this strategy on CTF binaries provided by famous institution. The results show that our strategy achieves significantly greater testing efficiency in larger programs while no apparent effect in minor programs.

Dynamic symbolic execution or concolic testing has been proposed recently to effectively generate test inputs for real-world programs. Unfortunately, dynamic symbolic execution techniques do not scale well for large realistic programs, because often the number of feasible execution paths of a program increases exponentially with the increase in the length of an execution path. We believe that programmers usually have a good understanding of programs they write and could help concolic testing to guide path exploration effectively so that concolic testing can achieve the desired testing goal quickly. In this paper, we propose a simple annotation mechanism, called Guide SE, that enables a programmer to guide concolic testing by annotating the program under test. We have currently identified three mechanisms for providing these annotations: control annotations, equivalence annotations, and data annotations. Controls annotations allow a programmer to specify the paths of interest using regular expressions. Equivalence annotations enable a programmer to prune paths that result in user defined equivalent states. Data annotations allow a programmer to abstract portions of the program under test and to explore the reduced path space of the abstract program.

Automatic testing, in particular test input generation, has become increasingly popular in the research community over the past ten years. In this paper, we conduct a survey on existing concolic testing tools, discussing their strengths and limitations, and environments in which they can be applied. We also conduct a case study to determine the prevalence of the identified limitations in six large software systems (four from open-source and two from ABB), as well as the effectiveness and scalability of the publicly available tools. The results show that pointers and native calls are the most prevalent limitations, preventing tools from generating high branch coverage test cases, and variables of float type are the least prevalent. The scalability of the publically available tools is also a limitation for industrial use, due to the large overhead of creating a test harness. Finally, we propose suggestions on how practitioners can use these tools and how researchers can improve concolic testing.

This chapter presents efficient Virtual Prototype (VP) based approaches for Software (SW) verification. They improve the existing VP-based SW verification flow by integrating stronger coverage metrics and providing automated test-case generation techniques as well as leverage formal methods. Ensuring correct functional behavior is very important to avoid errors and security vulnerabilities (such as buffer overflows). The first approach integrates concolic testing into the VP. Essentially, concolic testing is an automated technique that successively explores new paths through the SW program by solving symbolic constraints, that are tracked alongside the concrete execution. However, an integration of concolic testing with VPs is challenging due to complex HW/SW interactions and peripherals. The proposed solution enables a high simulation performance with accurate results and comparatively little effort to integrate peripherals with concolic execution capabilities. The second approach leverages state-of-the-art CGF in combination with VPs to enable a scalable and efficient verification of embedded SW binaries. To guide the fuzzing process the coverage from the embedded SW is combined with the coverage of the SystemC-based peripherals of the VP.

Software testing techniques have advanced significantly over several decades; however, most of current techniques still test a target program as it is, and fail to utilize valuable information of diverse test executions on many variants of the original program in test generation. This paper proposes a new direction for software testing - Invasive Software Testing (IST). IST first generates a set of target program variants m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> from an original target program p by applying mutation operations <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, μ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> . Second, given a test suite T, IST executes m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> with T and records the test runs which increase test coverage compared to p with T. Based on the recorded information, IST generates guideposts for automated test generation on p toward high test coverage. Finally, IST generates test inputs on p with the guideposts and achieves higher test coverage. We developed DEMINER which implements IST for C programs through software mutation and concolic testing. Further, we showed the effectiveness of DEMINER on three real-world target programs Busybox-ls, Busybox-printf, and GNU-find. The experiment results show that the amount of the improved branch coverage by DEMINER is 24.7% relatively larger than those of the conventional concolic testing techniques on average.

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the fault detection effectiveness of metamorphic testing. Most of the previous studies on metamorphic testing have used either random test data or existing test cases as source test cases. There has been limited research done on systematic source test case generation for metamorphic testing. This paper provides a comprehensive evaluation on the impact of source test case generation techniques on the fault finding effectiveness of metamorphic testing. We evaluated the effectiveness of line coverage, branch coverage, weak mutation and random test generation strategies for source test case generation. The experiments are conducted with 77 methods from 4 open source code repositories. Our results show that by systematically creating source test cases, we can significantly increase the fault finding effectiveness of metamorphic testing. Further, in this paper we introduce a simple metamorphic testing tool called "METtester" that we use to conduct metamorphic testing on these methods.

JavaScript has become the most popular programming language for web front-end development. With such popularity, there is a great demand for thorough testing of client-side JavaScript web applications. In this paper, we present a novel approach to concolic testing of front-end JavaScript web applications. This approach leverages widely used JavaScript testing frameworks such as Jest and Puppeteer and conducts concolic execution on JavaScript functions in web applications for unit testing. The seamless integration of concolic testing with these testing frameworks allows injection of symbolic variables within the native execution context of a JavaScript web function and precise capture of concrete execution traces of the function under test. Such concise execution traces greatly improve the effectiveness and efficiency of the subsequent symbolic analysis for test generation. We have implemented our approach on Jest and Puppeteer. The application of our Jest implementation on Metamask, one of the most popular Crypto wallets, has uncovered 3 bugs and 1 test suite improvement, whose bug reports have all been accepted by Metamask developers on Github. We also applied our Puppeteer implementation to 21 Github projects and detected 4 bugs.

Techniques and tools for verification of multi-threaded programs must cope with the huge number of possible thread interleavings. Tools based on systematic exploration of a program state space employ partial order reduction to avoid redundant thread interleavings. The key idea is to make non-deterministic thread scheduling choices only at statements that read or modify the global state shared by multiple threads. We focus on the approach to partial order reduction used in tools such as Java Pathfinder (JPF), which construct the program state space on-the-fly, and therefore can use only information available in the current program state and execution history to identify statements that may be globally-relevant. In our previous work, we developed a field access analysis that provides information about fields that may be accessed during program execution, and used it in JPF for more precise identification of globally-relevant statements. We build upon that and propose a may-happen-before analysis that computes a sound approximation of the happens-before ordering. Partial order reduction techniques can use the happens-before ordering to detect pairs of globally-relevant field access statements that cannot be interleaved arbitrarily (due to thread synchronization), and based on that avoid making unnecessary thread scheduling choices. The may-happen-before analysis combines static analysis with knowledge of information available from the dynamic program state. Results of experiments with several Java programs show that usage of the may-happen-before analysis further improves the performance of JPF.

AUSTIN: An open source tool for search based software testing of C programs

Metamorphic testing uses domain-specific properties about a program’s intended behaviour to alleviate the oracle problem. From a given set of source test inputs, a set of follow- up test inputs are generated which have some relation to the source inputs, and their outputs are compared to outputs from the source tests, using metamorphic relations. We evaluate the use of an automated test input generation technique called dynamic symbolic execution (DSE) to generate the source test inputs for metamorphic testing. We investigate whether DSE increases source-code coverage and fault finding effectiveness of metamorphic testing compared to the use of random testing, and whether the use of metamorphic relations as a supportive technique improves the test inputs generated by DSE. Our results show that DSE improves the coverage and fault detection rate of metamorphic testing compared to random testing using significantly smaller test suites, and the use of metamorphic relations increases code coverage of both DSE and random tests considerably, but the improvement in the fault detection rate may be marginal and depends on the used metamorphic relations.

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.