Symbolic execution for software testing in practice

Dynamic test generation approach is becoming increasingly popular to find security vulnerabilities in software. More and more research institutes and organizations use this approach to find security vulnerabilities in binary code. However, the existing binary level dynamic test generation approaches and tools are not retargetable, and can only find vulnerabilities in binaries for a specific ISA. This paper presents a new binary-level dynamic test generation technique and a tool, ReTBLDTG, short for retargetable binary-level dynamic test generation, that implements this technique. Unlike other such techniques that can operate only on binaries in a specific ISA, ReTBLDTG takes binaries of any ISAs as input and dynamically generates new inputs that exercise different control paths in the program, which may lead to security vulnerabilities. ReTBLDTG defines a meta instruction set architecture (MetalSA); ReTBLDTG maps the execution information, which is collected during the binary source code execution, to MetalSA; and symbolic execution, constraint collection and constraint solver operates on MetalSA, thus making these processes ISA-independent. We have implemented our ReTBLDTG, retargeted it to 32-bit x86, PowerPC and Sparc ISAs, and used it to automatically find the six known bugs in the six benchmarks. Our results indicate that our ReTBLDTG can be easily retargeted to any ISA with only a few overheads; and ReTBLDTG can effectively find bugs located deep within large applications from their binaries for 32-bit x86, PowerPC or Sparc ISA.

Implementation of an effective dynamic concolic execution framework for analyzing binary programs

In software security testing, fuzz testing and symbolic execution are two main testing techniques. Fuzz testing finds program bugs by executing the target program with random inputs it generates while monitoring the execution for abnormal behaviors. Though fuzz testing is able to explore deep into a program's state space efficiently, it usually cannot guarantee the code coverage ratio in many situations. Symbolic execution treats a program's input as symbols and executes the program with such symbolic input. Symbolic execution tries to explore the whole state space of a program by analyzing all of the program's paths. Symbolic execution always guarantees the code coverage ratio in theory, but for real programs it has many problems such as path explosion. Our new method, hybrid fuzz testing, combines the benefits of fuzz testing and symbolic execution. Symbolic execution will start to work and to generate new program input when fuzz testing cannot increase the code coverage ratio. The experiment result implies that our hybrid fuzz testing method can obviously increase the code coverage ratio efficiently in reasonable resources using.

Symbolic execution is widely used in software test cases automatic generating, but itpsilas hard to use it in embedded software testing, for embedded software is related to hardware too close. And at present, embedded software testing tools can only generate flow chart and test cases' coverage information, can not generate test cases that satisfy designated coverage automatically. This paper brings forward an idea that realizes embedded software test cases automatic generating using symbolic execution. First translate the embedded program into an intermediate language program, which preserve all the necessary information from the origin embedded program and according hardware, then symbolic execute the intermediate language program to generate test cases automatically.

The problem of test data generation in software testing is well referenced. Of the various methods used in the literature, symbolic execution appears to be a promising approach. It can be used either for software verification or to facilitate the automated test data generation process. A number of symbolic systems that employ symbolic execution to generate test data have already been constructed. However, these systems analyze programs written in specific programming languages. Thus, although some of them use an internal representation for performing symbolic execution, each one can only deal with programs in a single language. In this paper a script language called SYMEXLAN is presented. It can be adapted in order to construct a general symbolic execution testing system independent of the language in which the software under test is written.

From the perspective of industry, formal methods over-promise and under-deliver. Theoretical computer scientists love the notion of proving programs correct, but have slowly come round to the realization that promises in grant proposals aren’t the same as delivering in the real world. Essentially we started seeing a slow erosion of the importance of the notion of soundness; completeness was dropped long before. The ideal of showing that programs behave according to their specification, became the reality of finding situations where they don’t. This maps perfectly onto an expensive activity well known to industry, namely software testing. This presentation looks at the happy marriage of techniques from formal methods and software testing. Software testing is expensive since it is time-consuming to derive tests to adequately cover the software’s behavior. Techniques from formal methods allow one to generate tests automatically (hence reducing costs) and systematically (hence increasing the likelihood of discovering errors). We look at the use of software model checking to find errors in complex code, and specifically, consider the evolution of one of the world’s most popular model checkers, JavaPathFinder (JPF). One of the core techniques in JPF is symbolic execution that, although introduced in the early seventies, has recently made a big comeback in the testing world. We discuss the reasons why it took this long for such a powerful technique to become popular (again) and how it is used within JPF. In addition we discuss some of the new advances in symbolic execution and how it is used for bug finding and test generation. Finally, we consider some of the new challenges facing the automated testing field and how formal techniques can be applied to address them.

Forward symbolic execution is a program analysis technique that allows using symbolic inputs to explore program executions. The traditional applications of this technique have focused on programs that manipulate primitive data types, such as integer or boolean. Recent extensions have shown how to handle reference types at their representation level. The extensions have favorably been backed by advances in constraint solving technology, and together they have made symbolic execution applicable, at least in theory, to a large class of programs. In practice, however, the increased potential for applications has created significant issues with scalability of symbolic execution to programs of non-trivial size---the ensuing path conditions rapidly become unfeasibly complex.We present Dianju, a new technique that aims to address the scalability of symbolic execution. The fundamental idea in Dianju is to perform symbolic execution of commonly used library classes (such as strings, sets and maps) at the abstract level rather than the representation level. Dianju defines semantics of operations on symbolic objects of these classes, which allows Dianju to abstract away from the complexity that is normally inherent in library implementations, thus promising scalable analyses based on symbolic execution.

Symbolic execution based test-patterns generation algorithm for hardware Trojan detection

In a software system, most of the runtime failures may come to light only during test execution, and this may have a very high cost. To help address this problem, a symbolic execution engine called RTEHunter, which has been developed at the Department of Software Engineering at the University of Szeged, is able to detect runtime errors (such as null pointer dereference, bad array indexing, division by zero) in Java programs without actually running the program in a real-life environment. Applying the theory of symbolic execution, RTEHunter builds a tree, called a symbolic execution tree, composed of all the possible execution paths of the program. RTEHunter detects runtime issues by traversing the symbolic execution tree and if a certain condition is fulfilled the engine reports an issue. However, as the number of execution paths increases exponentially with the number of branching points, the exploration of the whole symbolic execution tree becomes impossible in practice. To overcome this problem, different kinds of constraints can be set up over the tree. E.g. the number of symbolic states, the depth of the execution tree, or the time consumption could be restricted. Our goal in this study is to find the optimal parametrization of RTEHunter in terms of the maximum number of states, maximum depth of the symbolic execution tree and search strategy in order to find more runtime issues in a shorter time. Results on three open-source Java systems demonstrate that more runtime issues can be detected in the 0 to 60 basic block-depth levels than in deeper ones within the same time frame. We also developed two novel search strategies for traversing the tree based on the number of null pointer references in the program and on linear regression that performs better than the default depth-first search strategy.

Software industry produces high quality software, in that software testing plays a vital role. In software testing automatic generation can reduce the process time and cost. Automatic generation of object-oriented unit tests in software testing is a challenging task especially in white box testing. In software engineering Software testing becomes challenging issue with the development of software tools. Using Genetic Algorithm will provide best results in the software testing toolkit. This paper suggests an upgrading of automated search-based test generation of test cases in order to obtain high branch coverage in white box testing. The proposed technique outperforms providing greater efficiency in testing technique which combines Genetic Algorithm and Symbolic Execution named GASE. GASE automatically generates test data in order to obtain high branch coverage in white box testing aiming effective testing process which can be extended for software testing in cloud computing where it becomes necessity.

This paper describes the symbolic execution of programs. Instead of supplying the normal inputs to a program (e.g. numbers) one supplies symbols representing arbitrary values. The execution proceeds as in a normal execution except that values may be symbolic formulas over the input symbols. The difficult, yet interesting issues arise during the symbolic execution of conditional branch type statements. A particular system called EFFIGY which provides symbolic execution for program testing and debugging is also described. It interpretively executes programs written in a simple PL/I style programming language. It includes many standard debugging features, the ability to manage and to prove things about symbolic expressions, a simple program testing manager, and a program verifier. A brief discussion of the relationship between symbolic execution and program proving is also included.

This paper introduces a novel approach to scale symbolic execution --- a program analysis technique for systematic exploration of bounded execution paths---for test input generation. While the foundations of symbolic execution were developed over three decades ago, recent years have seen a real resurgence of the technique, specifically for systematic bug finding. However, scaling symbolic execution remains a primary technical challenge due to the inherent complexity of the path-based exploration that lies at core of the technique. Our key insight is that the state of the analysis can be represented highly compactly: a test input is all that is needed to effectively encode the state of a symbolic execution run. We present ranged symbolic execution, which embodies this insight and uses two test inputs to define a range, i.e., the beginning and end, for a symbolic execution run. As an application of our approach, we show how it enables scalability by distributing the path exploration---both in a sequential setting with a single worker node and in a parallel setting with multiple workers. As an enabling technology, we leverage the open-source, state-of-the-art symbolic execution tool KLEE. Experimental results using 71 programs chosen from the widely deployed GNU Coreutils set of Unix utilities show that our approach provides a significant speedup over KLEE. For example, using 10 worker cores, we achieve an average speed-up of 6.6X for the 71 programs.

It is our great pleasure to welcome you to Trento for ISSTA 2010, the 19th International Symposium on Software Testing and Analysis. ISSTA is the leading research conference in software testing and analysis and brings together academics, industrial researchers, and practitioners to exchange new ideas, problems, and experience on how to analyze and test software systems. The ISSTA 2010 program includes keynotes, technical, tool, and doctoral symposium papers, tutorials and workshops. ISSTA 2010 attracted over 100 submissions, which demonstrates the ongoing interest in the field. Each submission was evaluated by at least three members of the Program Committee. The Program Committee met to discuss the submissions and accepted 23 papers that cover a variety of topics, including formal verification, symbolic execution, test input generation, debugging, and concurrency testing and analysis. In addition, the program includes two distinguished keynote speakers: Christian Ferdinand (AbsInt) and Patrice Godefroid (Microsoft Research). The ISSTA week includes five workshops and two tutorials: PADTAD: Workshop on Parallel and Distributed Systems: Testing, Analysis, and DebuggingSSEAT: Workshop on State-space Exploration for Automated TestingSTOV/MIIT: Joint Workshop on Software Test Output Validation & Model Inference In TestingWODA: Workshop on Dynamic AnalysisSBT: Tutorial on Search-based Testing, given by Phil McMinnJPF: Tutorial on Automated Testing with Java PathFinder, given by Corina Pasareanu and Willem Visser

Software testing is an expensive and time consuming process, often involving the manual creation of comprehensive regression test suites. However, current testing methodologies do not take full advantage of these tests. In this paper, we present a technique for amplifying the effect of existing test suites using a lightweight symbolic execution mechanism, which thoroughly checks all sensitive operations (e.g., pointer dereferences) executed by the test suite for errors, and explores additional paths around sensitive operations. We implemented this technique in a prototype system called ZESTI (Zero-Effort Symbolic Test Improvement), and applied it to three open-source code bases — GNU Coreutils, libdwarf and readelf — where it found 52 previously unknown bugs, many of which are out of reach of standard symbolic execution. Our technique works transparently to the tester, requiring no additional human effort or changes to source code or tests.

Preface: Special issue on software verification and testing