On Architecting Software Fault Tolerance using Abstractions

While typical solutions focus on fault tolerance (and specifically, exception handling) during the design and implementation phases of the software life-cycle (e.g., Java and Windows NT exception handling), more recently the need for explicit exception handling solutions during the entire life cycle has been advocated by some researchers. Several solutions have been proposed for fault tolerance via exception handling at the software architecture and component levels. This paper describes how the two concepts of fault tolerance and software architectures have been integrated so far. It is structured in two parts (overview on fault tolerance and exception handling, and integrating fault tolerance into software architecture) and is based on a survey study on architecting fault tolerant systems where more than fifteen approaches have been analyzed and classified. This paper concludes by identifying those issues that remain still open and require deeper investigation.

In fault tolerant software systems, the Level of Decomposition (LoD) where design diversity is applied has a major impact on software system reliability. By disregarding this impact, current fault tolerance techniques are prone to reliability decrease due to the inappropriate application level of design diversity. In this paper, we quantify the effect of the LoD on system reliability during software recomposition when the functionalities of the system are redistributed across its components. We discuss the LoD in fault tolerant software architectures according to three component failure transitions: component failure occurrence, component failure propagation, and component failure impact. We illustrate the component aspects that relate the LoD to each of these failure transitions. Finally, we quantify the effect of the LoD on system reliability according to a series of decomposition and/or merge operations that may occur during software recomposition.

Fault tolerance in microprocessor systems has become a popular topic of architecture research. Much work has been done at different levels to accomplish reliability against soft errors, and some fault tolerance architectures have been proposed. But little attention is paid to the thread level superscalar fault tolerance. This letter introduces microthread concept into superscalar processor fault tolerance domain, and puts forward a novel fault tolerance architecture, namely, MicroThread Based (MTB) coarse grained transient fault tolerance superscalar processor architecture, then discusses some detailed implementations.

Currently, using commercial off-the-shelf (COTS) components and constructing parallel system becomes the main way to improve satellite system performance remarkably. System reliability relying on its structure rather than devices makes system fault-tolerant architecture and strategy increasingly important. As centralized fault-tolerant solution has the inherent drawback of single point of failure (SPOF), it can not guarantee system reliability well. Distributed fault- tolerant method, due to its complexity and large overhead, has not been maturely used. In this paper, we propose a new distributed fault-tolerant scheme with high reliability and flexibility, in which an equal symmetric distributed fault- tolerant (ESDFt) architecture is constructed. Based on the structure, functional module framework of system nodes and autonomous fault-tolerant strategy are designed. The ESDFt scheme creates favorable conditions for flexible implementation of system fault tolerance. It utilizes the master election mechanism, which eliminates the risks of SPOF and chooses master node intelligently based on user's configuration, making fault-tolerant complexity and processing overhead flexibly adjustable. The verification and evaluation of ESDFt scheme has been performed on a prototype system. The experiment results prove that ESDFt scheme not only ensures system reliability well, but also enhances system flexibility and practicality at the same time.

Fault tolerance engineering has been advocated as one of the main approaches to ensuring the overall system dependability. The 2nd International Workshop on Engineering Fault Tolerant Systems (EFTS 2007) aims to investigate how fault tolerance mechanisms can be taken into account when engineering complex software systems and to improve our understanding of where and how fault-tolerance should be integrated in the software life-cycle. The focus of the workshop is on developing novel models to be applied at different abstraction levels (requirements, architecture and design models for fault tolerance, together with new implementation schemes), innovative technologies (tools and frameworks for implementing distributed fault tolerant systems) and advanced verification environments (to assess the achieved level of fault tolerance and to evaluate the dependability properties of the systems). Recently there has been a growing interest in the areas directly related and overlapping with fault tolerance, such as self-healing, resilience, self-adaptation and self-management. The topics related to engineering of systems with such properties are in the scope of the workshop as the intention is to improve the current understanding of how fault tolerance engineering can benefit from research on these areas.

The concept, design, and features of a fault-tolerant intelligent robotic control system being developed for Space and commercial applications that require high dependability are described. The comprehensive strategy integrates system level hardware/software fault tolerance with task level handling of uncertainties and unexpected events for robotic control. The underlying architecture for system level fault tolerance is the distributed recovery block which protects against application software, system software, hardware, and network failures. Task-level fault tolerance provisions are implemented in a knowledge-based system which utilizes advanced automation techniques such as rule-based and model-based reasoning to monitor, diagnose, and recover from unexpected events. The two-level design provides tolerance of two or more faults occurring serially at any level of command, control, sensing, or actuation. >

We have reached the end of our discussion about application-level fault-tolerance protocols, which were defined as the methods, architectures, and tools that allow the expression of fault-tolerance in the application software of our computers. Several “messages” have been given: • First of all, fault-tolerance is a “pervasive” concern, spanning the whole of the system layers. Neglecting one layer, for instance the application, means leaving a backdoor open for problems. • Next, fault-tolerance is not abstract: It is a function of the target platform, the target environment, and the target quality of service. The tools to deal with this are the system model and the fault model, plus the awareness that (1) all assumptions have coverage and (2) coverage means that, sooner or later, maybe quite later but “as sure as eggs is eggs,” cases will show up where each coverage assumptions will fail. • This means that there is a (even ethical) need to design our systems thinking of the consequences of coverage failures at mission time, especially considering safety critical missions. I coined a word for those supposed fault-tolerant software engineers that do not take this need into account: Endangeneers. Three well-known accidents have been presented and interpreted in view of coverage failures in the fault and system models. • Next, the critical role of the system structure for the expression of fault-tolerance in computer applications was put forth: From this stemmed the three properties characterizing any application-level fault-tolerance protocol: Separation of concerns, adequacy to host different solutions, and support for adaptability. Those properties address the following question: Given a certain fault-tolerance provision, is it able to guarantee an adequate separation of the functional and non-functional design concerns? Does it tolerate a fixed set of faulty scenarios, or does it dynamically change that set? And, is it flexible enough as to host a large number of different strategies? • Then it has been shown that there exist a large number of techniques and hence of system structures able to enhance the fault-tolerance of the application. Each of these techniques has its pros and cons, which we tried to point out as best as we could. We also attempted to qualify each technique with respect to the above mentioned properties1. A summary of the results of this process is depicted in Fig. 1. • Another key message is that complexity is a threat to dependability, and we must make sure that the extra complexity to manage fault-tolerance does not become another source of potential failures. In other words, simplicity must be a key ingredient of our fault-tolerance protocols, and a faulty fault-tolerant software may produce the same consequence of a faulty non fault-tolerant software—or maybe direr. • Finally, we showed with some examples that adaptive behavior is the only way to match the ever mutating and unstable environments characterizing mobile systems. As an example, static designs would make bad use of the available redundancy.

Dependability modeling and evaluation (encompassing reliability and safety issues) of the two major fault tolerance software approaches-recovery blocks (RBs) and N version programming (NVP)-are presented. The study is based on the detailed analysis of software fault-tolerance architectures able to tolerate a single fault (RB: two alternates and an acceptance test; NVP: three versions and a decider). For each approach a detailed model based on the software production process is established and then simplified by assuming that only a single fault type may manifest during execution of the fault-tolerant software and that no error compensation may take place within the software. The analytical results obtained make it possible to identify the improvement, compared to a non-fault-tolerant software, that could result from the use of RB (the acceptance test has to be more reliable from the alternates) and NVP (related faults among the versions and the decider have to be minimized) and to determine the most critical types of related faults. Nested RBs are studied, showing that the proposed analysis approach can be applied to such realistic software structures and that when an alternate is itself an RB, the results are analogous to the case of the addition of a third alternate. The reliability analysis shows that only a small improvement can be expected.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Describes an approach for enabling the synergistic coordination between two fault-tolerance protocols to simultaneously tolerate software and hardware faults in a distributed computing environment. Specifically, our approach is based on a message-driven confidence-driven (MDCD) protocol that we have devised for tolerating software design faults, and a time-based (TB) checkpointing protocol that was developed by N. Neves and W.K. Fuchs (1996) for tolerating hardware faults. By carrying out algorithm modifications that are conducive to synergistic coordination between volatile-storage and stable-storage checkpoint establishments, we are able to circumvent the potential interference between the MDCD and TB protocols, and to allow them to effectively complement each other to extend a system's fault tolerance capability. Moreover, the protocol coordination approach preserves and enhances the features and advantages of the individual protocols that participate in the coordination, keeping the performance cost low.

This paper discusses the modeling and analysis of three major fault-tolerant software system architectures: DRB (Distributed Recovery Blocks), NVP (N-Version Programming) and NSCP (N Self-Checking Programming). In the system-level reliability modeling domain, fault tree analysis techniques and Markov modeling techniques are combined to incorporate transient and permanent hardware faults as well as unrelated and related software faults. These models are parameterized by a real-world fault-tolerant flight control computer application for evaluations and comparisons. In particular, a series of sensitivity analysis is performed to explore the critical components in each fault-tolerant architecture and display their quantitative impacts to the overall system reliability.

As computing capabilities continue to advance, there will be a concurrent rise in the number of both hardware and software faults. These will be caused by the greater volume of more complex software, by the increased number of untested software states, and by more incidents of hardware/software interaction faults as a result of increased hardware speed and density. The traditional software implemented fault tolerance approaches have been successfully utilized in life-critical systems, such as digital flight controls, where their additional costs can be easily justified. Examples include N-Version Programming and Recovery Block approaches. However, there is still a need for dependable computing for mission-critical applications as well. Often, these traditional techniques are avoided for mission-critical systems due to the difficulty in justifying their extra upfront development cost. We provide an alternative for the high sunk cost of traditional software fault tolerance techniques. The methodology, called Data Fusion Integrity Processes (DFIPs), is a simple, yet effective technique for mission critical systems. In addition, the approach establishes a framework from which other costlier, more extensive traditional techniques can be added. We present details of the DFIP methodology and a DFIP framework for Ada programs. We also briefly discuss development of a DFIP code generation system which exploits Java that will enable users to quickly build a DFIP framework in Ada, and select reusable DFIP component methods.

: This report provides a synopsis of research performed in fault- tolerant computing, for the first year of grant AFOSR-84-0052. Also included is a list of publications that have resulted from the research supported by this grant. Additionally, this report reviews the future direction for the continuing research under this grant. In the past year, this effort has focussed on the following problems: (1) Investigation of novel fault-tolerant processor array architectures with the potential of a high degree of defect tolerance, but having low processor and interconnect overhead associated with the fault tolerance mechanisms; (2) Development of realistic models to evaluate the yield, redundancy and performance tradeoffs for the designs. Such models would help establish the viability of these architectures, also enabling them to be compared with other designs in the literature; (3) Development of new and efficient testing strategies, and reconfiguration schemes for their structures; (4) Testable design of large size VLSI memory; and (5) Development of novel sorting networks that can be implemented on a single chip or wafer. Three journal articles were supported during the grant period: 'Synthesis of Directed Multicommodity Flow Networks,' Vol. 14, pp. 213-224 (with A. Stai); 'Fault- Tolerant Multiprocessor Link and Bus Network Architectures, IEEE Transactions on Computers, Vol. C-34, No. 1, January 1985, pp. 33-46; and 'Dynamically Restructurable Fault-Tolerant Processor Network Architectures,' IEEE transactions on Computers (to appear).

Fault tolerance is the most imperious issue in the cloud to provide reliable services. Inherent vulnerability to failure hampers the performance and reliability of cloud services. Hence, to achieve reliability, fault tolerance becomes a mandatory feature which is hard to implement due to the dynamic infrastructure and complex interdependencies. Numerous fault tolerance techniques have been developed in the literature to address the challenges of cloud reliability. A recent research survey presented in this paper attempts to integrate the different fault tolerance architecture. This study presents a critical research review on various existing fault tolerance techniques to improve services reliability, availability, and applications execution in the cloud. A comparative analysis, based on different critical metrics like failure prediction, detection strategy, failure history, VM placement, and limitations, of the reviewed framework systems is also included in the paper. This review intends to facilitate the development of the new fault tolerance technique for the cloud environment.

This paper addresses the practical implementation of means of tolerating residual software faults in complex software systems, especially concurrent and distributed ones. There are several inherent difficulties in implementing such fault-tolerant software systems, including the controlled use of extra redundancy and the mixture of different design concerns. In an attempt to minimise these difficulties, we present a generic implementation approach, composed of a multi-layered reference architecture, a configuration method and an architectural pattern. We evaluate our implementation approach using an industrial control application whose control software we equip with the ability to tolerate a variety of software faults. The preliminary evidence shows that our approach can simplify the implementation process, reduce repetitive development effort and provide high flexibility through a generic interface for a wide range of fault tolerance schemes.

A comparative study of exception handling mechanisms for building dependable object-oriented software