KITO tool: A fault injection environment in Linux kernel data structures

Evaluating and possibly improving the fault tolerance and error detecting mechanisms is becoming a key issue when designing safety-critical electronic systems. The proposed approach is based on simulation-based fault injection and allows the analysis of the system behavior when faults occur. The paper describes how a microprocessor board employed in an automated light-metro control system has been modeled in VHDL and a Fault Injection Environment has been set up using a commercial simulator. Preliminary results about the effectiveness of the hardware fault-detection mechanisms are also reported. Such results will address the activity of experimental evaluation in subsequent phases of the validation process.

CAFI: A Configurable location-Aware Fault Injection technique for software reliability assessment against soft errors

Transient and permanent faults in complex digital systems used for safety-critical applications may result in catastrophic effects. It becomes therefore extremely important to adopt techniques such as fault injection to observe the behavior of the system in the presence of faults. Several tools have been proposed in the literature that support fault injection. However, few of them allow observing complex computer-based systems. This paper presents current advances in this field, by focusing on Low Level Virtual Machine (LLVM) based fault injectors and FAUMachine. We give an overview of the LLVM environment, and two based fault injection tools: LLFI and KULFI. Moreover, we introduce FAUmachine as virtual machine that supports fault injection in different components of the system (memory, disk and network). We present limitations and difficulties of the tool, and we propose a new implementation that allows injecting faults into the register of the target processor. The paper concludes with a comparison between the fault injection tools based on virtual machine in a first level, and between the LLVM-based fault injection tools in a second level.

As software for distributed systems becomes more complex, ensuring that a system meets its prescribed specification is a growing challenge that confronts software developers. This is particularly important for distributed applications with strict dependability and timeliness constraints. This paper reports on ORCHESTRA, a portable fault injection environment for testing implementations of distributed protocols. This tool is based on a simple yet powerful framework called script-driven probing and fault injection, for the evaluation and validation of the fault-tolerance and timing characteristics of distributed protocols. The tool, which was initially developed on the Real-Time Mach operating system and later ported to other platforms including Solaris and SunOS, has been used to conduct extensive experiments on several protocol implementations. This paper describes the design and implementation of the fault injection tool focusing on architectural features to support portability, minimizing intrusiveness on target protocols, and explicit support for testing real-time systems. The paper also describes the experimental evaluation of two protocol implementations: a real-time audio-conferencing application on Real-Time Mach, and a distributed group membership service on the Sun Solaris operating system.

Firstly, the paper made a contrast between the current domestic and international research condition, and introduced the basic concept of fault injection and distributed system. secondly, it discussed the classification and requirements of fault injection. There are mainly three types of distributed fault, namely the memory fault, CPU fault and correspondence fault. Besides, it discussed the method of distributed software fault injection about DOCTOR and illustrated the comprehensive structure and its respective parts of DOCTOR in detail. Thirdly, it reached a conclusion about the fault model of distributed system of fault injection and its realization method.

We investigate the implementation of triple modular redundancy (TMR) with different number of voters in a softcore RISC-V microprocessor to mitigate radiation-induced faults in the configuration memory (CRAM) of SRAM-based FPGA. The case-study microprocessor is the Steel Core implemented in AMD Xilinx 7-Series/Zynq-7000 device, executing benchmark software on the FreeRTOS operating system. We implemented coarse-grained TMR (CGTMR) and fine-grained distributed TMR (FGDTMR) using an automated redundancy and voter insertion tool. TMR is used in combination with memory scrubber featured by the FPGA hardware, which uses error correction codes (ECC) to revert radiation-induced faults in CRAM. Fault injection in CRAM emulating radiation-induced faults is used to evaluate the benefits of both TMR and ECC. Under fault injection, combining the fault masking capability of TMR with the self-healing by ECC, improved mean time between failures (MTBF) up to 5×. ECC alone did not improve significantly the fault tolerance. Without ECC, FGDTMR improves MTBF in 2×, but at a higher cost in circuit area and resources usage. When combined with ECC, the CGTMR provides fault tolerance comparable to FGDTMR but with 30% less resources usage.

Designers of safety-critical VLSI systems are asking for effective tools for evaluating and validating their designs. Fault Injection is commonly adopted for this task, and its effectiveness is therefore a key factor. In this paper we propose to exploit FPGAs to speed-up Fault Injection for fault tolerance evaluation of VLSI circuits. A complete Fault Injection environment is described, relying on FPGA-based emulation of the circuit for fault effect analysis. The proposed approach allows combining the efficiency of hardware-based techniques, and the flexibility of simulation-based techniques. Experimental results are provided to support the feasibility and effectiveness of the approach.

Dynamic thread duplication is a known redundancy technique for multi-cores. Recent research applied this concept to hybrid multi-cores for error detection and introduced thread shadowing that runs hardware threads in the reconfigurable cores and compares their outputs for deviation at configurable signature levels. Previously published work evaluated this concept in terms of performance, error detection latency and resource consumption. In this paper we report on the error detection capabilities of thread shadowing by presenting an extensive fault injection campaign. We employ the Xilinx Soft Error Mitigation Controller for fault injection and the Xilinx Essential Bit facility to limit the fault injections to relevant bits in the configuration bitstream. Our findings from fault injection experiments with a sorting benchmark are threefold: First, up to 98% of all errors are detected by the operating system of the hybrid multi-core supported by thread shadowing. Second, thread shadowing's signature levels provide a useful trade-off between detected errors and effort needed, with around 5% of all errors detected in calls to operating system functions and around 52% of errors detected in memory accesses of the hardware thread. Third, essential bit testing is effective and cuts down the amount of bits to be tested by a factor of 14.48 compared to the total amount of bits available in the configuration address space.

For fine-grained synchronization of application and kernel threads, the Linux kernel provides a multitude of different locking mechanisms that are being used on various individually locked data structures. Understanding which locks are required in which order for a particular member variable of a kernel data structure has become truly difficult, even for Linux-kernel experts themselves. In this paper we introduce LockDoc -- an approach that, based on the analysis of execution traces of an instrumented Linux kernel, automatically deduces the most likely locking rule for all members of arbitrary kernel data structures. From these locking rules, LockDoc generates documentation that supports kernel developers and helps avoiding concurrency bugs. Additionally, the (very limited) existing documentation can be verified, and locking-rule violations -- potential bugs in the kernel code -- can be found. Our results include generated locking rules for previously predominantly undocumented member variables of 11 different Linux-kernel data structures. Manually inspecting the scarce source-code documentation for five of these data structures reveals that only 53 percent of the variables with a documented locking rule are actually consistently accessed with the required locks held. This indicates possible documentation or synchronization bugs in the Linux kernel, of which one has already been confirmed by kernel experts.

The dependability assessment of critical systems must consider the emulation of transient faults, as they pose an important dependability threat for modern VLSI designs. FPGA-based systems are dominated by bit-flips in configuration memory (CM), which are relatively easy to emulate using partial runtime reconfiguration (RTR) FPGA fault injection (FFI) approaches. However, when FPGA is used as an ASIC prototyping platform, transient fault models representative of ASIC designs must be considered, such as bit-flips in sequential logic cells (Flip-Flops and on-chip RAM blocks). Existing RTR-FFI approaches do not adequately cover these faults, as they require the orchestrated manipulation of multiple CM bits for each target logic cell, and the location of these CM bits is unknown (not documented) for modern FPGA generations. This work experimentally formalises the mapping of the necessary CM bits, proposes an enhanced RTR-FFI methodology to emulate bit-flips in registers and on-chip RAMs of current-generation AMD Ultrascale+ FPGAs, and provides an upgraded publicly available open-source FFI tool (BAFFI 1 ) supporting the proposed methodology. The validity of the proposed FFI approach is demonstrated by comparison with gate-level simulation-based fault injection in a case study of two soft-core processors (MC8051 and NOEL-V).

Electronic computing systems are integrating modern multicore processors and GPUs aiming to perform complex software stacks in different life-critical systems, including health devices and emerging self-driving cars. Such systems are expected to experience at least one soft error per day in the near future, which may lead to life-threatening failures. To prevent these failures, critical system must be tested and verified while under realistic workloads. This paper presents four novel non-intrusive fault injection techniques that enable full fault injection control and inspection of multicore systems behavior in the presence of faults. Proposed techniques were integrated into a fault injection framework and verified through a real automotive case study with up to 43 billions instructions. Results show that compared to traditional methods, the new techniques can increase the efficiency of fault injection campaigns during early development phase by 32.28%.

Fault injection is needed for different purposes such as analyzing the reaction of a system in a faulty environment or validating fault-detection and/or fault-correction techniques. In this paper we propose a simulation-based fault injection tool able to work at different abstraction levels and with user-defined fault models. By exploiting the facilities provided by a functional verification environment it allows to speed up the entire fault injection process: from the creation of the workload to the analysis of the results of injection campaigns. Moreover, the adoption of techniques to optimize the fault list significantly reduces the simulation time. Being the tool targeted to the validation of dependable systems, it includes a way to extract information from the Failure Mode and Effect Analysis and to correlate fault injection results with estimates.

This paper presents a new phenomenon called error correlation prediction for lockstep processors. Lockstep processors run the same copy of a program, and their outputs are compared at every cycle to detect divergence, and have been popular in safety-critical systems. When the lockstep error checker detects an error, it alerts the safety-critical system by putting the lockstep processor in a safe state in order to prevent hazards. This is done by running the online diagnostics to identify the cause of the error because the lockstep processor has no knowledge of whether the error is caused by a transient or permanent fault. The online diagnostics can be avoided if the error is caused by a transient fault, and the lockstep processor can recover from it. If, however, it is caused by a permanent fault, having prior knowledge about error's likely location(s) within the CPU speeds up the diagnostics process. We discover that the error's type and likely location(s) inside CPUs from which the fault may have originated can be predicted by analyzing the output signals of the CPU(s) when the error is detected. We design a simple static predictor exploiting this phenomenon and show that system availability can be increased by 42--64% with an overhead of less than 2% in silicon area and power.

Technology trends are increasing the frequency of serious transient (soft) faults in digital systems. For example, ICs are becoming more susceptible to cosmic radiation, and are being embedded in applications with dynamic noisy environments. We propose a generic framework for representing such faults and characterizing them on-line. We formally define the impact of a transient fault in terms of three basic parameters: frequency, observability and severity. We distinguish fault modes in systems whose noise environment changes dynamically. Based on these ideas, the problem of designing on-line architectures for transient fault characterization is formulated and analyzed for several optimization goals. Finally, experiments are described that determine transient fault impact and the corresponding tests for various simulated fault modes of the ISCAS-89 benchmark circuits

ABSTRACTAn operating system is the essence of software, serving as the foundation for the operation of various application software. The security of the operating system is crucial for national informatization construction. Data indicate that many cybersecurity incidents result from exploiting security vulnerabilities in the operating system. Linux is currently the most widely used open‐source operating system, with thousands of Common Vulnerabilities and Exposures (CVEs) related to Linux systems reported each year. Therefore, research and prevention of vulnerabilities in the Linux system are particularly important. To gain a better understanding of the characteristics of Linux system vulnerabilities, this paper leverages knowledge in the field of software security to analyze nearly 10,000 historical vulnerability data in two core systems of Linux: Linux Kernel and Debian Linux. The study explores the evolutionary patterns of vulnerability characteristics. Specific research contents include the following: (1) data collection and cleaning of vulnerability data in Linux Kernel and Debian Linux systems; (2) cross‐statistical analysis of structured data features in vulnerability reports; (3) unstructured data characteristics mining in vulnerability reports based on domain knowledge; (4) analysis of the evolution of vulnerability characteristics. This paper provides empirical lessons and guidance for Linux system vulnerabilities to assist practitioners and researchers in better preventing and detecting vulnerabilities in Linux and Linux‐based systems.