Research about Software Fault Injection Technology Based on Distributed System

Software fault injection (SFI) has been used to intentionally cause "failures" in software components and assess their impacts on the entire software system. A key property that SFI should satisfy is the representative ness of injected failures, the failures caused by SFI should be as close as possible to failures in the wild. If injected failures do not represent realistic failures, the measured resilience or tolerance against failures of the investigated system is not trustworthy. To the best of the authors' knowledge, the representative ness of "faults" has been investigated. However, it is an open problem whether the failures caused by injected faults represent realistic failures. In this paper, we report the preliminary results of the investigation on the representative ness of injected failures. To compare injected failures with real failures, we have collected 43,742 real crash logs of Linux from the Red Hat repository, and conducted a fault injection campaign on Linux, using SAFE, a state-of-theatre injector of software faults. In the fault injection campaign, 50,000 faults are injected to the Linux file system and 71,470 runs of a workload are executed. The crash logs generated by SFI are compared with the real Red Hat logs with respect to crash causes, crashed system calls, and crashed modules. Our preliminary results suggest that failures caused by injected faults do not represent real failures, probably because injected faults are not representative enough or because the selected workload is not realistic.

Software fault injection is a process that discovers how badly software can behave after its state gets corrupted. Fault injection results are sometimes viewed suspiciously since state corruption is hypothetically-based. In response to this suspicion, this paper explores the potential return-on-investment when artificial state corruptions are used. We will primarily focus on fault injection's unique ability to reveal hazards that were inadvertently overlooked during software requirements and design. To our knowledge, this application of fault injection has never been exploited.

This paper presents an experimental study on the emulation of software faults by fault injection. In a first experiment, a set of real software faults has been compared with faults injected by a SWIFI tool (Xception) to evaluate the accuracy of the injected faults. Results revealed the limitations of Xception (and other SWIFI tools) in the emulation of different classes of software faults (about 44% of the software faults cannot be emulated). The use of field data about real faults was discussed and software metrics were suggested as an alternative to guide the injection process when field data is nor available. In a second experiment, a set of rules for the injection of errors meant to emulate classes of software faults was evaluated. The fault triggers used seem to be the cause for the observed strong impact of the faults in the target system and in the program results. The results also show the influence in the fault emulation of aspects such as code size, complexity of data structures, and recursive versus sequential execution.

Fault injection is an accepted method for emulating the effect of ionizing radiation on digital electronic circuits. It can be oriented either to ASIC designs or to SRAM-FPGA designs. When the target device is an SRAM-FPGA the injection has to be assessed both in the functional plane and in the configuration plane. It has been demonstrated that the classical protections oriented to the functional structure are not enough, so the configuration plane has to be analyzed, in the same way. This paper describes the adaption of the FT-UNSHADES2 platform as a fault injection system that tests faults in the configuration plane. The mechanism that assesses the effect of faults in the configuration is read-modify-write, in cycles of inject and repair, based on partial reconfiguration.

Real-world programs require error handling code to handle various kinds of possible errors. However, these errors just infrequently occur due to special conditions, so error handling code is difficult to test. Coverage-guided fuzzing and software fault injection (SFI) are two common techniques that can test error handling code, but they still have major limitations. Specifically, existing fuzzing approaches generate program inputs guided by code coverage, but many occasional errors (such as insufficient memory) are unrelated to inputs, and code coverage cannot effectively reflect the execution contexts of these errors; existing SFI approaches often inject single or random faults, without exploring fault space or using program feedback. In this paper, we propose a new fuzzing framework named EH-Fuzz, to effectively test error handling code. EH-Fuzz uses a context-sensitive SFI-based fuzzing approach to explore fault space and perform fault injection, guided by a new metric named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">error coverage</i> . We evaluate EH-Fuzz on 9 user-level programs and 6 kernel-level modules, and find 45 new real bugs, 31 of which have been confirmed and fixed. We compare EH-Fuzz to existing fuzzing approaches (including AFL, AFL++, Syzkaller, FIZZER and FIFUZZ), and EH-Fuzz finds many real bugs missed by these approaches with higher testing coverage.

KITO tool: A fault injection environment in Linux kernel data structures

CAFI: A Configurable location-Aware Fault Injection technique for software reliability assessment against soft errors

Modern software is plagued by elusive corner-case bugs (e.g., security bugs). Because there are no scalable, automated ways of finding them, such bugs can remain hidden until software is deployed in production. This thesis proposes approaches to solve this problem. First, we present black-box and white-box fault injection mechanisms, which allow developers to test the behavior of their own code in the presence of failures in external components, e.g., in libraries, in the kernel, or in remote nodes of a distributed system. We describe how to make black-box fault injection more efficient, by prioritizing tests based on their estimated impact. For white-box testing, we proposed and implemented a technique to find Trojan messages in distributed systems, i.e., messages that are accepted as valid by receiver nodes, yet cannot be sent by any correct sender node. We show that Trojan messages can lead to subtle semantic bugs. We used fault injection techniques to find new bugs in systems such as the MySQL database, the Apache HTTP server, the FSP file service protocol suite, and the PBFT Byzantine-fault-tolerant replication library. Testing can find bugs and build confidence in the correctness of a system. However, exhaustive testing is often unfeasible, and therefore testing may not discover all bugs before a system is deployed. In the second part of this thesis, we describe how to automatically harden production systems, reducing the impact of any corner-case bugs missed by testing. We present a framework that reduces the overhead cost of instrumentation tools such as memory error detectors. Lowering the cost enables system developers to use such tools in production to harden their systems, reducing the impact of any remaining corner-case bugs. We used our framework to generate a version of the Linux kernel hardened with Address Sanitizer. Our hardened kernel has most of the benefit of full instrumentation: it detects the same vulnerabilities as full instrumentation (7 out of 11 privilege escalation exploits from 2013-2014 can be detected using instrumentation tools). Yet, it obtains these benefits at only a quarter of the overhead.

Reliability is the most important feature in this more and more complex computer system era. Fault injection is dependability validation technique to evaluating the system. Hardware and Software implementations of fault injection have a long history and are much more mature than simulated fault injection. In this paper, we compare the differences between these three types of fault injections at first. Then, we identify and understand the types of fault. We design a low- cost, simulation-based fault injection system and design experiments to verify the correctness. Keywords-simulation fault injection; reliability;

As software for distributed systems becomes more complex, ensuring that a system meets its prescribed specification is a growing challenge that confronts software developers. This is particularly important for distributed applications with strict dependability and timeliness constraints. This paper reports on ORCHESTRA, a portable fault injection environment for testing implementations of distributed protocols. This tool is based on a simple yet powerful framework called script-driven probing and fault injection, for the evaluation and validation of the fault-tolerance and timing characteristics of distributed protocols. The tool, which was initially developed on the Real-Time Mach operating system and later ported to other platforms including Solaris and SunOS, has been used to conduct extensive experiments on several protocol implementations. This paper describes the design and implementation of the fault injection tool focusing on architectural features to support portability, minimizing intrusiveness on target protocols, and explicit support for testing real-time systems. The paper also describes the experimental evaluation of two protocol implementations: a real-time audio-conferencing application on Real-Time Mach, and a distributed group membership service on the Sun Solaris operating system.

For accurately assessing the reliability of space application toward soft errors, this paper proposed a method of Single Event Effect simulation method for Large Scale Integrated Circuits (LSI). This simulation method divided into three levels: fault model, fault injection and target system simulation, which consists of a variety of simulation tools, such as TCAD, SPICE circuit simulator and analyzing tools of gate-level netlist. A 4-stage in-order RISC-V soft processor is used as the simulation target system. The simulation method has the following advantages: 1. It is a universal method of testing any large-scale integrated circuit including its peripheral components. 2. By input the gate-level net-list of the circuit and add the simulation stimulus, the method can get the simulation report automatically. 3. The soft error can be traced: The faulty node and the voltage waveform of the node can be reported intuitively.

Because of SRAM sensitivity to radiation, SRAM-based FPGA systems deployed in harsh environments require error mitigation methods to reduce their overall downtime. This paper presents a fault-tolerant reconfigurable imaging system that relies on the DPR feature for correcting errors in the configuration memory and loading camera system IPs. The system reliability is evaluated by injecting faults in the FPGA configuration memory at runtime using the Xilinx SEM IP. The faults are injected internally using the Internal Configuration Access Port (ICAP), which is shared between the fault injection core and system parts. The results showed that 95% of the errors can by corrected automatically. This paper also proposes a fast-Built-in-Self-Test (BIST) mitigation technique to reduce the overall downtime in case of errors. This technique can reduce the recovery time by 80%. Moreover, Triple Modular Redundancy (TMR) is used to increase the overall reliability without significantly increasing the resource overhead.

We investigate the implementation of triple modular redundancy (TMR) with different number of voters in a softcore RISC-V microprocessor to mitigate radiation-induced faults in the configuration memory (CRAM) of SRAM-based FPGA. The case-study microprocessor is the Steel Core implemented in AMD Xilinx 7-Series/Zynq-7000 device, executing benchmark software on the FreeRTOS operating system. We implemented coarse-grained TMR (CGTMR) and fine-grained distributed TMR (FGDTMR) using an automated redundancy and voter insertion tool. TMR is used in combination with memory scrubber featured by the FPGA hardware, which uses error correction codes (ECC) to revert radiation-induced faults in CRAM. Fault injection in CRAM emulating radiation-induced faults is used to evaluate the benefits of both TMR and ECC. Under fault injection, combining the fault masking capability of TMR with the self-healing by ECC, improved mean time between failures (MTBF) up to 5×. ECC alone did not improve significantly the fault tolerance. Without ECC, FGDTMR improves MTBF in 2×, but at a higher cost in circuit area and resources usage. When combined with ECC, the CGTMR provides fault tolerance comparable to FGDTMR but with 30% less resources usage.

This paper presents an FPGA fault injection system, a methodology for soft processor fault injection, and fault injection experimental results for MicroBlaze and LEON3 soft processor designs. The Xilinx Radiation Test Consortium—Virtex 5 Fault Injector (XRTC-V5FI) was built to evaluate the configuration memory sensitivity of soft processor designs. To overcome some of the challenges of soft processor fault injection, we designed the XRTC-V5FI to be fast, flexible, and to fully cover all configuration memory bits. The minimum time to inject a full bitstream is 28 minutes and the individual fault injection can be as fast as 49 μS. The LEON3 has 81.3 % more sensitive bits than the MicroBlaze, yet when normalized by the number of used slices, the MicroBlaze is 26.2 % more sensitive than the LEON3.

This paper introduces fault injection technology. This paper discusses the application of software fault injection technology to the testing of airborne aviation software with high security application requirements. The characteristics, classification and application of software fault injection technology are analyzed. This paper focuses on middle-ware based fault injection technology and its application in high security airborne software testing. System level middle-ware fault injection method and module level middle-ware fault injection method are proposed.