Fault Injection for Linux Device Drivers on x86, MIPS and ARM Architectures

To ensure resilient neural network processing on even unreliable hardware, comprehensive reliability analysis against various hardware faults is generally required before the neural network models are deployed, and efficient fault injection tools are highly demanded. However, many existing fault injection tools remain limited to basic fault injection and fail to provide fine-grained vulnerability analysis capability. In addition, many of the fault injection tools also need to change the neural network models and make the fault injection closely coupled with normal neural network processing, which complicates the use of these tools and slows down the fault simulation. The various fault injection implementations and error metrics make the comparison between different fault-tolerant studies difficult. To this end, we propose MRFI, a highly configurable multiresolution fault injection tool for deep neural networks. It enables users to modify an independent fault configuration file rather than neural network models for fault injection and vulnerability analysis. Particularly, it integrates extensive fault analysis functionalities from different perspectives and enables multiresolution investigation of the vulnerability of neural networks. In addition, it does not modify the major neural network computing framework of PyTorch. Hence, it allows parallel processing on GPUs naturally and exhibits fast fault simulation according to our experiments. Moreover, we also have the fault injection calibrated with fault simulation with architectural details and validate the accuracy of the proposed fault injection. Finally, MRFI is also open-sourced on GitHub (MRFI https://github.com/fffasttime/MRFI).

Transient and permanent faults in complex digital systems used for safety-critical applications may result in catastrophic effects. It becomes therefore extremely important to adopt techniques such as fault injection to observe the behavior of the system in the presence of faults. Several tools have been proposed in the literature that support fault injection. However, few of them allow observing complex computer-based systems. This paper presents current advances in this field, by focusing on Low Level Virtual Machine (LLVM) based fault injectors and FAUMachine. We give an overview of the LLVM environment, and two based fault injection tools: LLFI and KULFI. Moreover, we introduce FAUmachine as virtual machine that supports fault injection in different components of the system (memory, disk and network). We present limitations and difficulties of the tool, and we propose a new implementation that allows injecting faults into the register of the target processor. The paper concludes with a comparison between the fault injection tools based on virtual machine in a first level, and between the LLVM-based fault injection tools in a second level.

The mechanism involved by fault injection tools for virtual machine system such as failure model,injection mode,trigging mechanism,monitor and trace was studied.The typical failure model(including fault type,fault location,occurrence mechanism and duration) and its implementation were proposed by statistically analyzing the fault symptoms occurring in various virtual machine systems.By investigating the architecture of virtual machine system and the principle of fault injection,a fault injection tool for Xen system using software simulation method was designed—software fault injector for virtualized environments(SFIVE),and its injection mode,trigging mechanism,monitor and trace were introduced.Experimental results show that SFIVE has higher fault type coverage of virtual machine system,and can activate and monitor most of fault types related to virtual machine.

This paper presents the prototype of an automatic and model-independent fault injection tool, to use on an IBM-PC (or compatible) platform. The tool has been built around a commercial VHDL simulator. With this tool, both transient and permanent faults, of a wide range of types, can be injected into medium-complexity models. Another remarkable aspect of the tool is the fact that it is able to analyse the results obtained from the injection campaigns, in order to study the error syndrome of the system model and/or validate its fault-tolerance mechanisms. Some results of a fault injection campaign carried out to validate the dependability of a fault tolerant microcomputer system are shown. We have analysed the pathology of the propagated errors, measured their latencies, and calculated both error detection and recovery latencies and coverages.

Availability was considered a critical factor of Network Function Virtualization(NFV) platforms. And many existing testing tools with fault injection features for NFV, only aim at availability, while now more and more attention have been paid to the performance degradation when failure happens. Moreover, performance data on hardware/software resources, like CPU consumption, memory usage, are required to collect in performance and fault-tolerant testing. To fulfill the requirements above, this paper proposes a service-based testing framework for NFV platform performance evaluation, under workloads and fault loads. By taking advantage of the Microservice Architecture, the proposed framework will integrate some existing open-source testing frameworks, including performance testing tools, fault injection tools, and monitoring tools, and build a more comprehensive testing scenario. A case study is conducted on Clearwater, a widely used open-source NFV application, to validate the efficiency and usability of the proposed framework.

With the emergence of big data and remarkable improvement of computational power, deep neural network (DNN) based intelligent systems, with the superb performance on computer vision, nature language processing, and optimization processing, etc, has been acceleratingly replacing traditional software in various aspects. However, due to the uncertainty of DNN modules learned from data, the intelligent systems are more likely to exhibit incorrect behaviors. Faults in software and hardware are also inevitably in practice, where the hidden defects can easily cause model failure. These will lead to severe accidents and losses in safety- and reliability-critical scenarios, such as autonomous driving. Techniques to test the differences between actual and desired behaviors and evaluate the reliability of DNN applications at faulty conditions is therefore significant for building a trustworthy DNN system. A popular method is fault injection and various fault injection tools have been developed for ML frameworks, such as Tensorflow, PyTorch. In this paper, we present a tool, MindFI, which targets to cover a variety of faults in ML programs written in Mindspore. Data, software and hardware faults can be easily injected in general Mindspore programs. We also use MindFI to evaluate the resilience of several commonly used ML programs against a assessment metrics.

This paper presents an approach for building SoftWareImplemented Fault Injection (SWIFI) tools for real-time systems starting from temporal logic specifications. Temporal logic formulas are provided describing the behavior of the main SWIFI mechanisms, both for the injection of faults and for the observation of the target system. Besides conventional fault injectors (i.e., that flip bits in memory and in system calls’ parameters), we have also considered fault injectors based on the notion of saboteur. Concerning the observation mechanisms, we have defined on-line software probes aimed at observing not only the failure modes of the system under test, but also the real-time behavior of both the application and the underlying operating system. The specifications of all such mechanisms are then translated into programs that are executed on-line by a virtual machine. This approach was applied to the development of MAFALDA-RT, a SWIFI tool for real-time systems. A case study exemplifies how the proposed approach allows for the SWIFI mechanisms to be defined and applied.

Fault injection is needed for different purposes such as analyzing the reaction of a system in a faulty environment or validating fault-detection and/or fault-correction techniques. In this paper we propose a simulation-based fault injection tool able to work at different abstraction levels and with user-defined fault models. By exploiting the facilities provided by a functional verification environment it allows to speed up the entire fault injection process: from the creation of the workload to the analysis of the results of injection campaigns. Moreover, the adoption of techniques to optimize the fault list significantly reduces the simulation time. Being the tool targeted to the validation of dependable systems, it includes a way to extract information from the Failure Mode and Effect Analysis and to correlate fault injection results with estimates.

Code execution in modern superscalar processors is inherently redundant. Many instructions execute repeatedly with the same inputs, producing the same outputs, thus wasting resources in the process. Value reuse techniques memorize previous executions of instructions, blocks or traces which may be reused if they appear again with the same input contexts. Although trace reuse techniques show great potential for both performance and energy consumption improvement, they have not been studied yet in one of the most widely available computer architectures - the ARM architecture. In this paper, the main issues with reusing traces in instruction sets with conditional execution are revisited. Afterwards, the reuse potential in the benchmark suite MiBench is analyzed varying (i) how traces are generated, and (ii) the size of reuse tables. Our results show that a memoization table of 32 KiB allows to reuse 18.36% of the total instructions on average.

CAFI: A Configurable location-Aware Fault Injection technique for software reliability assessment against soft errors

Fault-injection experiments on the instruction-set architecture level are commonly used to analyze embedded software's susceptibility to hardware faults, typically involving a vast number of experiments with systematically varying fault locations and times. Determinism and high performance are the predominant requirements on fault-injection platforms. Injecting faults into a real embedded hardware platform instead of a simulator is favorable for both workload execution speed and result accuracy. The most performance-critical part of such a fault-injection platform is the “fast forward” operation, which executes the target machine code without faults until the exact dynamic instruction is reached at which the execution must be stopped to inject the next fault. Unfortunately, most embedded CPUs do not support this operation efficiently. In this paper we present an approach that speeds up fast-forwarding significantly for most workloads with minimal requirements on hardware support. Based on a previously recorded instruction trace - which is needed for systematic fault-injection experiment planning anyways - we use standard debugging hardware to advance to a chosen point in program execution with a minimal number of steps. We evaluate our FAIL* tool platform with two MiBench benchmark categories, and improve experiment throughput by up to several magnitudes compared to similar fault-injection tools in the field.

GPUs have found wide acceptance in domains such as high-performance computing and autonomous vehicles, which require fast processing of large amounts of data along with provisions for reliability, availability, and safety. A key component of these dependability characteristics is the propagation of errors and their eventual effect on system outputs. In addition to analytical and simulation models, fault injection is an important technique that can evaluate the effect of errors on a complete computing system running the full software stack. However, the complexity of modern GPU systems and workloads challenges existing fault injection tools. Some tools require the recompilation of source code that may not be available, struggle to handle dynamic libraries, lack support for modern GPUs, or add unacceptable performance overheads. We introduce the NVBitFI tool for fault injection into GPU programs. In contrast with existing tools, NVBitFI performs instrumentation of code dynamically and selectively to instrument the minimal set of target dynamic kernels; as it requires no access to source code, NVBitFI provides improvements in performance and usability. The NVBitFI tool is publicly available for download and use at https://github.com/NVlabs/nvbitfi.

In this paper, we study device driver architectures on two different operating systems, Fuchsia and Linux. Fuchsia is a relatively new operating system developed by Google and it is based on a microkernel named Zircon, while Linux-based operating system is based on a monolithic kernel. This paper examines technical details of device driver on Fuchsia and Linux operating systems with the focus on different kernel designs. We also quantitatively evaluate the performance of device drivers on both operating systems by measuring I/O throughput in a real device.

Zero-copy means on any network node, during message transmission, there is no data copy among memory segments and all messages transmission are operated directly between user application space and outside of network through network interfaces. By DMA technology, network interface devices can store data packets into the address where it can be accessed directly by user application, the data packets copy operation is removed in kernel and reducing the data transmission path. By working together with TOE technology, it can drop the communication delay effectively and save a lot of CPU cost. In this paper, we based on Linux (kernel version 2.6.11), by modifying its network device driver snull.c (attached in Linux Device Driver version 3), to implement the memory mapping between Linux kernel space to user application space. The main method we used is the combination of MMAP and PROC procedures, and at the end, we also introduce the implementation of test program and the test strategies. The test results show us that the applications can access the applied memory of kernel by mapping

As software for distributed systems becomes more complex, ensuring that a system meets its prescribed specification is a growing challenge that confronts software developers. This is particularly important for distributed applications with strict dependability and timeliness constraints. This paper reports on ORCHESTRA, a portable fault injection environment for testing implementations of distributed protocols. This tool is based on a simple yet powerful framework called script-driven probing and fault injection, for the evaluation and validation of the fault-tolerance and timing characteristics of distributed protocols. The tool, which was initially developed on the Real-Time Mach operating system and later ported to other platforms including Solaris and SunOS, has been used to conduct extensive experiments on several protocol implementations. This paper describes the design and implementation of the fault injection tool focusing on architectural features to support portability, minimizing intrusiveness on target protocols, and explicit support for testing real-time systems. The paper also describes the experimental evaluation of two protocol implementations: a real-time audio-conferencing application on Real-Time Mach, and a distributed group membership service on the Sun Solaris operating system.