Can You Trust Your Compiler? Recent Developments in Automated Testing of Compilation and Verification Tools

The ever-growing complexity of software and its target hardware makes it increasingly challenging to develop reliable compilers that preserve the semantics of source code during compilation. Traditional testing methods often lack sufficient test coverage and fail to identify subtle errors and undefined behaviors that can lead to compiler malfunctions. Furthermore, while formal compiler certification ensures semantic preservation through theorem proving, the inherent complexity of this process makes re-certification after each compiler revision substantially labor-intensive. This often significantly hinders the improvement of compilers. This research aims to bridge the gap between increasing compiler complexity and the limited scalability of formal verification techniques. We present a comprehensive approach for automatic verification of compiler transformations that are the core components of any compiler design. Our approach particularly targets the foundations of state-of-the-art compilers: their intermediate representations (IRs). It provides automated verification algorithms that formally compare the IRs before and after a compiler transformation and systematically enumerate the execution paths of such transformation. First, we have developed an automated equivalence checking framework for hardware designs represented in HalideIR, the foundation of several well-known hardware design compilers. This framework conducts scalable equivalence checking by analyzing structural similarities in HalideIR across various design iterations. It identifies minimal design constructs essential for verification and performs efficient localized checks. Its application in deep learning accelerator designs has proven its capability to detect inconsistencies that manual testing fails to identify, thereby bolstering hardware design reliability. Second, we have developed an automated verification framework for HalideIR-based compiler transformations using symbolic execution. This method systematically enumerates and verifies execution paths of compiler transformations to ensure that the output program for each path is equivalent to its source. The application of this framework in common HalideIR-based compilers has uncovered previously undetected transformation bugs, highlighting its effectiveness. Third, we have pioneered a novel translation validation framework for MLIR-based compilers. This framework overcomes the complexity arising from the numerous MLIR dialects and employs the Z3 SMT solver to validate the refinement of target programs with respect to source programs in MLIR. Its evaluations on diverse MLIR-based compilers have not only showcased its versatility but also its crucial role in identifying bugs and undefined behaviors. Finally, we have also explored integrating Large Language Models (LLMs) into translation validation to target compiler transformations where formal verification tools fall short. Our framework utilizes existing formal verification tools for translation validation (in this work, we use Alive2, a

Formal program verification tools check that a program correctly implements its specification. Existing specification languages for well-known programming languages (Ada, C, Java, C#) have been developed independently from the programming language to which they apply. As a result, specifications are expressed separately from the code, typically as stylized comments, and the verification tools often bear no direct relation to the production compiler. We argue that this approach is problematic, and that the compiler and the verification tools should be integrated seamlessly. Based on our current work on the Hi-Lite project to develop a formal verification tool for Ada2012, we show that in an integrated setting, the compiler becomes the centerpiece of the verification architecture, and supports both static proofs and run-time assertion checking. Such an environment does much to simplify software certification.

The area of formal verification of protocols has gained substantial importance in the recent years. The research results and subsequent applications have amply demonstrated that the formal verification tools have indeed helped correct the protocols even after being standardized. However, the standard protocol verification tools and techniques do not verify the security properties of a cryptographic protocol. This has resulted in the emergence of the security protocol verifiers to fill the need. In this paper, taking the two popular security verification tools namely Scyther and ProVerif as the basis, we identify a few security protocols and implement them in both Scyther and ProVerif, to aptly evaluate the tools, in terms of the security properties of the selected protocols. In the process, we not only characteristically present a comparative evaluation of the two tools, but also reveal interesting security properties of the protocols selected, showing their strengths and weaknesses. To the best of our knowledge, this is a unique attempt to juxtapose and evaluate the two verification tools using the selected security protocols.

The Accellera EDA standards body has recently approved the PSL a standard property specification language for use in assertion-based verification via simulation and formal verification tools. This language, which is based on the Sugar language from IBM, is now supported by many EDA vendors. More than 40 individuals representing over 20 companies participated in the efforts to form the PSL standard from its Sugar basis.The tutorial comprises 2 parts. In the first part, we describe the basic principles of PSL/Sugar, focusing on the ease with which complex design behaviors may be described with concise, readable PSL/Sugar assertions that crisply capture design intent. We summarize the temporal constructs of the language, including parameterized sequences and properties, directives, and modeling capabilities. We cover the general timing model of PSL/Sugar, which transparently supports both (single- or multi-clock) synchronous and asynchronous design, and, time permitting, we explain how PSL/Sugar has been defined to ensure consistent semantics for both simulation and formal verification applications.In the second part of the tutorial, we present several applications of PSL/Sugar, ranging from simple to advanced assertion-based verification solutions. These include use of PSL/Sugar for dynamic assertion checking and formal model checking, including support for environment modeling and assume/guarantee reasoning. Examples of commercial verification tools which support the PSL/Sugar languages will also be presented.Participants in the tutorial will have an excellent opportunity to learn about both the language and its applications directly from the speaker, Dr. Danny Geist, who heads a research group in the IBM Haifa lab where Sugar was conceived.

A myriad of proprietary (i.e., commercial tool specific) verification assertion languages exist today. These languages emerged due to the lack of formal language constructs supporting assertion specification in today’s hardware description languages (HDL). The method described in this paper presents a unique process for unifying (or neutralizing the effects of) the many proprietary languages by using a set of predefined specification modules instantiated as assertions within the designer’s HDL. This methodology creates numerous advantages and precludes the need to introduce new HDL constructs (i.e., extensions to Verilog are not required). It enables the design engineer to “specify once,” then leverage the same HDL assertion specification over multiple verification processes -- such as traditional simulation, semi-formal, and formal verification tools. It also eliminates the need for the design engineer to master tool specific and proprietary language details. Furthermore, this methodology enables evaluating new verification processes and tools (containing their own proprietary languages) seamlessly for the duration of the project, without the need to modify the original HDL text.

The complexity of designing concurrent and highly-evolving interactive systems has grown to a point where system verification has become a hurdle. Fortunately, formal verification methods have arrived at the right time. They detect errors, inconsistencies and incompleteness at early development stages of a system formally modeled using a formal specification language. -calculus (Milner, 1999) is one such formal language which provides strong mathematical base that can be used for verifying system specifications. But manually verifying the specifications of concurrent systems is a very tedious and error-prone work, especially if the specifications are large. Consequently, an automated verification tool would be essential for efficient system design and development. In addition, formal verification tools are vital ingredient to fully harness the potential of component-based software composition. The authors developed such an automated verification tool which is highly portable and seamlessly integrates with the visualization, reduction and performance evaluation tools introduced (Ahmad & Rahimi, 2008; Rahimi, 2006; Rahimi et al., 2001, 2008) to provide a comprehensive tool for designing and analyzing multi process/agent systems. Open-Bisimulation (Sangiorgi, 1996) concept is utilized as the theoretical base for the design and implementation of the tool which incorporates an expert system implemented in Java Expert System Shell (JESS) (Friedman-Hill, 2003).

Defects in software cost money and sometimes lives. Even with rigorous testing, there are countless ways for programs to go wrong. Testing does not guarantee that a given program is correct for every input. Concurrent program testing does not guarantee that a program is correct for the example inputs. Formal program verification has been used as a technique to ensure program correctness for several years. It analyses the properties of the code and ensures that nothing goes wrong. In this thesis, a formal verification tool is designed and implemented based on Boogie IVL (Intermediate Verification Language) for a multi-threaded and object-oriented language named HARPO (HARdware Parallel Objects). We have designed the specific annotations intended only for use in verification. These annotations help the HARPO verifier to translate the program into an equivalent Boogie code. The resulting Boogie code is checked with theorem provers to verify the correctness of the HARPO program. Boogie code generation is tested using unit testing, and some of the case studies are reported. Consequently, programmers can use the HARPO verifier for verification of their concurrent programs.

With the increasing complexity of protocol and circuit designs, formal verification has become an important research area and binary decision diagrams (BDDs) have been shown to be a powerful tool in formal verification. This paper presents a parallel algorithm for BDD construction targeted at shared memory multiprocessors and distributed shared memory systems. This algorithm focuses on improving memory access locality through specialized memory managers and partial breadth-first expansion, and on improving processor utilization through dynamic load balancing. The results on a shared memory system show speedups of over two on four processors and speedups of up to four on eight processors. The measured results clearly identify the main source of bottlenecks and point out some intereeting directions for further improvements.

All formal hardware verification tools in the market today interpret hardware description languages (HDLs) based on their synthesis semantics. This limits formal verification to synthesizable designs. The result, either a proof or a counterexample, produced by a formal tool can be inconsistent with simulation due to synthesis and simulation mismatches. And finally, conversion from a synthesized gate-level circuit to a formal model such as a Kripke structure or a Mealy machine is complex for designs containing gated clocks or latches. Existing solutions are often based on heuristics rather than language semantics. In this paper, we propose a new approach that constructs formal models based on simulation semantics. We symbolically simulate HDL designs using non-canonical word-level expressions to represent the values of design signals. We show that the formal model is consistent with simulation at specified sample points, which can be chosen to represent a clock cycle or a transaction. Our approach has been implemented in a tool called Simon. Experimental results show that Simon can efficiently construct formal models for large industrial designs.

This paper proposes an automated test method for detecting performance bugs in compilers. It is based on differential random testing, in which randomly generated programs are compiled by two different compilers and resulting pairs of assembly codes are compared. Our method attempts to achieve efficient and accurate detection of performance difference, by combining dynamic measurement of execution time with static assembly-level comparison and test program minimization. In the first step, discrepant pairs of code sections in the assembly codes are extracted, and then the sums of the weights of discrepant instructions in the sections are computed. If significant differences are detected, the test program is reduced to a small program that still exhibits the static difference and then the actual execution time of the codes are compared. A test system has been implemented on top of the random test system Orange4, which has successfully detected a regression in the optimizer of a development version of GCC-8.0.0 (latest as of May, 2017).

<span lang="EN-US">Network Function Virtualization (NFV) architectures are emerging to increase networks flexibility. However, this renewed scenario poses new challenges, because virtualized networks, need to be carefully verified before being actually deployed in production environments in order to preserve network coherency (e.g., absence of forwarding loops, preservation of security on network traffic, etc.). Nowadays, model checking tools, SAT solvers, and Theorem Provers are available for formal verification of such properties in virtualized networks. Unfortunately, most of those verification tools accept input descriptions written in specification languages that are difficult to use for people not experienced in formal methods. Also, in order to enable the use of formal verification tools in real scenarios, vendors of Virtual Network Functions (VNFs) should provide abstract mathematical models of their functions, coded in the specific input languages of the verification tools. This process is error-prone, time-consuming, and often outside the VNF developers’ expertise. This paper presents a framework that we designed for automatically extracting verification models starting from a Java-based representation of a given VNF. It comprises a Java library of classes to define VNFs in a more developer-friendly way, and a tool to translate VNF definitions into formal verification models of different verification tools.</span>

Network Function Virtualization (NFV) architectures are emerging to increase networks flexibility. However, this renewed scenario poses new challenges, because virtualized networks, need to be carefully verified before being actually deployed in production environments in order to preserve network coherency (e.g., absence of forwarding loops, preservation of security on network traffic, etc.). Nowadays, model checking tools, SAT solvers, and Theorem Provers are available for formal verification of such properties in virtualized networks. Unfortunately, most of those verification tools accept input descriptions written in specification languages that are difficult to use for people not experienced in formal methods. Also, in order to enable the use of formal verification tools in real scenarios, vendors of Virtual Network Functions (VNFs) should provide abstract mathematical models of their functions, coded in the specific input languages of the verification tools. This process is error-prone, time-consuming, and often outside the VNF developers' expertise. This paper presents a framework that we designed for automatically extracting verification models starting from a Java based representation of a given VNF. It comprises a Java library of classes to define VNFs in a more developer-friendly way, and a tool to translate VNF definitions into formal verification models of different verification tools.

Craig interpolation has become a versatile tool in formal verification, used for instance to generate program assertions that serve as candidates for loop invariants. In this paper, we consider Craig interpolation for quantifier-free Presburger arithmetic (QFPA). Until recently, quantifier elimination was the only available interpolation method for this theory, which is, however, known to be potentially costly and inflexible. We introduce an interpolation approach based on a sequent calculus for QFPA that determines interpolants by annotating the steps of an unsatisfiability proof with partial interpolants. We prove our calculus to be sound and complete. We have extended the Princess theorem prover to generate interpolating proofs, and applied it to a large number of publicly available Presburger arithmetic benchmarks. The results document the robustness and efficiency of our interpolation procedure. Finally, we compare the procedure against alternative interpolation methods, both for QFPA and linear rational arithmetic.

Equivalence checking is an integral part of the formal verification of ASIC to ensure that the constructed design flow meets the specifications. State-of-the-art techniques and heuristics are available for combinational equivalence checking restricted to binary netlists and do not work for netlists containing and propagating the “X” value. To accommodate the X-value, a 3-valued formal analysis has to be done which is computationally exponentially harder than its binary counterpart and can be a capacity challenge for the verification tool. X-value propagation becomes crucial for applications including low power equivalence checking under power shutoff and comparing a Register Transfer Level (RTL) design to a synthesized netlist. This paper introduces an algorithm for X-value equivalence checking using Binary Decision Diagrams (BDD) for combinational circuits. It extends the functionality of pre-existing formal verification tools available for binary netlists, to X-value netlists by introducing new variables to the BDD. The Colorado University Decision Diagram (CUDD) package has been used for BDD manipulation.

Chisel is an open-source hardware description language embedded in Scala to facilitate parameterized and reusable digital circuit design. Chisel is becoming increasingly popular and has been used to design RISC-V CPUs, e.g. RocketChip and XiangShan. While Chisel features high-level hardware designs, its verification is still low-level: Low-level (e.g. Verilog) programs are first generated from Chisel programs, then the verification tools are applied to these low-level programs. In this work, we focus on formal verification of arithmetic units. Efficient low-level formal verification of arithmetic units has always been a challenge and remains an active research area, attributed to the state explosion problem brought on by bit widths. To circumvent this problem for arithmetic Chisel designs, we propose an approach to their high-level formal verification so that their correctness is verified for all bit widths at once, instead of for each bit width separately. The key idea is to transform arithmetic Chisel designs into Scala software programs that simulate their behaviors, where the high-level features are preserved, then resort to Stainless, a deductive formal verification tool for Scala. We validate the effectiveness of this approach by formally verifying the correctness of dividers and multipliers in two representative open source RISC-V processors, namely, RocketChip and XiangShan. Compared to the existing proof-assistant-based parameterized verification approaches for arithmetic designs (e.g. Kami), the verification cost in our approach is much lower on average.