Analysis of Fault Tolerance in MPI Programs

Network bandwidth and hardware technology are developing rapidly, resulting in the vigorous development of the Internet. A new concept, cloud computing, uses lowpower hosts to achieve high reliability. Cloud computing increases the number of user applications on the Internet. However, the fault-tolerance is an important research topic in the study of distributed systems. To cope with the influence from faulty components, reaching a common agreement at the presence of faults before performing some user applications is essential. Moreover, the agreement problem is fundamental to fault tolerant distributed systems. To enhance fault tolerance, the agreement problem in a cloud-computing environment is revisited in this study. The proposed protocol, Fast Generalized Cloud Agreement (FGCA), achieves agreement on a common value among all nodes in a minimal number of message exchange rounds, and can tolerate a maximal number of allowable faulty components in a generalized cloud-computing environment.

In this paper we describe initial ideas about modeling and analyzing fault tolerance mechanisms in self managed/self healing systems. Specifications are component based, with coordination mechanisms for building systems from components. A modal action logic is augmented with deontic operators to describe normal vs abnormal behaviours. Fault tolerance mechanisms can be specified in terms of the kind of abnormality encountered and the desired recovery route. Abstract programming models can be systematically constructed from the specifications in LTSA, a finite state, process algebra based modeling tool. LTSA then enables us to check that various properties do or do not hold for the specified fault tolerance mechanisms.

The importance of reliability and fault tolerance is paramount in quantum computation. This paper proposes a Fault Tolerance Algorithms and Methodologies (FTAM) assessment technique for quantum circuits, by adopting the saboteur-based Simulated Fault Injection methodology from classical computation. By drawing the inspiration from classical computation, the HDLs were employed for performing fault injection, due to their capacity of behavioral and structural circuit description, including hierarchical features. The cornerstone of this approach is the adaptation of the available quantum computation error models (with the quantum computing features and constraints) to the classical, HDL framework of the simulated fault injection techniques. The experimental simulated fault injection campaign results are consistent with the analytical assessments -from a qualitative point of view - but at the same time they provide a much realistic description.

Software fault tolerance makes system complicated by means of diverse fault-tolerant mechanisms. Aspect-oriented programming has been confirmed as a well suited candidate to ease the problem. However, most of the existing works only focus on implementation level. This paper proposes an AOFTM model for aspect-oriented fault tolerance modeling with Coloured Petri nets at design level. Moreover, the weaving mechanism formally defined conducts the way to derive a woven net from AOFTM. A case study presents how AOFTM works, and its properties are analyzed using the CPN Tools.

Cluster computing has recently gone through an evolution from single processor systems to multicore/multi-socket systems. This has resulted in lowering the cost/performance ratio of the compute machines. Compute farms that host these machines tend to become heterogeneous over time due to incremental extensions, hardware upgrades and/or nodes being purchased for users with particular needs. This heterogeneity is not surprising given the wide range of processor, memory and network technologies that become available and the relatively small price difference between these various options. Different CPU architectures, memory capacities, communication and I/O interfaces of the participating compute nodes present many challenges to job scheduling and often result in under or over utilization of the compute resources. In general, it is not feasible for the application programmers to specifically optimize their programs for such a set of differing compute nodes, due to the difficulty and time-intensiveness of such a task. The trend of heterogeneous compute farms has coincided with resurgence in the virtualization technology. Virtualization technology is receiving widespread adoption, mainly due to the benefits of server consolidation and isolation, load balancing, security and fault tolerance. Virtualization has also generated considerable interest in the High Performance Computing (HPC) community, due to the resulting high availability, fault tolerance, cluster partitioning and accommodation of conflicting user requirements. However, the HPC community is still wary of the potential overheads associated with virtualization, as it results in slower network communications and disk I/O, which need to be addressed. The live migration feature, available to most virtualization technologies, can be leveraged to improve the throughput of a heterogeneous compute farm (HC) used for HPC applications. For this we mitigated the slow network communication in Xen; an open source virtual machine monitor. We present a detailed analysis of the communication framework of Xen and propose communication configurations that give 50% improvement over the conventional Xen network configuration. From a detailed study of the migration facility in Xen, we propose an improvement in the live migration facility specifically targeting HPC applications. This optimization gives around 50% improvement over the default migration facility of Xen. In this thesis, we also investigate resource scheduling in heterogeneous compute farm with the perspective of dynamic resource re-mapping. Our approach is to profile each job in the compute farm at runtime, and propose a better resource mapping compared to the initial allocation. We then migrate the job(s) to the best-suited homogeneous sub-cluster to improve overall throughput of the HC. For this, we develop a novel heterogeneity and virtualization-aware profiling framework, which is able to predict the CPU and communication characteristics of high performance scientific applications. Our experimental setup consists of a 32 node virtualized cluster with machines having CPU frequency between 2 GHz to 3 GHz. The framework was tested with a subset of NAS parallel benchmarks with the workload generated from Dror Feitelson’s workload archives. The prediction accuracy of our performance estimation model is over 80%. The framework implementation is lightweight, with an overhead of 3%. Our experiments show that we are able to improve the throughput of the compute farm by 25% and the time saved by the HC with our framework is over 30%. The framework can be readily extended to HCs supporting a cloud computing environment.

State-machine replication (SMR) is a software technique for tolerating failures and for providing high availability in large-scale systems, through the use of commodity hardware. A replicated state-machine comprises a number of replicas, each of which runs an agreement protocol, with the goal of ensuring a consistent state across all of the replicas. In hostile environments, such as the Internet, Byzantine fault tolerant state-machine replication (BFT) is an important technique for providing robust services. During the past decade, we have seen an emergence of various BFT protocols. In order to be adopted, besides providing correctness, a BFT must provide good performance as well. Consequently, all of the new protocols focus on improving performance under various conditions. However, a closer look at the performance of state-of-the-art BFT protocols reveals that even in best-case execution scenarios, they still remain far behind their theoretical maximum. Based on exhaustive evaluation and monitoring of existing BFT protocols, we highlight a few impediments to their scalability. These obstructions include the use of IP multicast, the presence of bottlenecks due to asymmetric replica processing, and an unbalanced network bandwidth utilization. The goal of this thesis is to evaluate the actual impact of these scalability impediments, and to offer a solution for a high-throughput BFT protocol in the case in which the network itself is the bottleneck. To that end, we have developed Ring, a new BFT protocol which circumvents the aforementioned impediments. As its name suggests, Ring uses the ring communication topology, in the fault-free case. In the ring topology, each replica only performs point-to-point communications with two other replicas, namely its neighbors on the ring. Moreover, all of the replicas equally accept requests from clients and perform symmetric processing. Our performance evaluation shows that, with the network as the bottleneck, Ring outperforms all other, state-of-the-art BFT protocols. Ring achieves 118 Mbps on the Fast Ethernet – a 24% improvement in throughput over previous protocols. Finally, we conducted an extensive practical and analytic evaluation of Ring. In order to analyse benefits (and drawbacks) of Ring (and other protocols) under different settings, without resorting to costly experimentation, we developed an analytical performance model. Our performance model is based on queueing theory, and relies only on a handful of protocolagnostic measurements of the environment.

Interconnection networks play a key role in the fault tolerance of massively parallel computers, since faults may isolate a large fraction of the machine containing many healthy nodes In this paper, we present a methodology to design fully adaptive fault-tolerant routing algorithms for direct interconnection networks that can be applied to different regular topologies The methodology is mainly based on the selection of an intermediate node (if needed) for each source-destination pair Packets are adaptively routed to the intermediate node and, from this node, they are adaptively forwarded to their destination This methodology requires only one additional virtual channel, even for tori Evaluation results show that the methodology is 7-fault tolerant, and for up to 14 faults, more than 99% of the combinations are tolerated, also without significantly degrading performance in the presence of faults.

i»? A. A. Helal, A. A. Heddaya, and B. B. Bhargrava Kluwer Academic Publishers, Boston, 1996, 176 pp. ISBN 0-7923-9800-9, $118.50 Computing in the 1990s has reached the state of distributed computing. A basis of this form of computing is a distributed computing system, which is built on the following three components: (a) personal computers, (b) local and fast wide area networks, and (c) system and application software. By amalgamating computers and networks into one single computing system and providing an appropriate system software, a distributed computing system has created the possibility of sharing information and peripheral resources. Furthermore, these systems improved performance of a computing system and individual users through parallel execution of programs, load balancing and sharing, and replication of programs and data. Distributed computing systems are also characterized by enhanced availability, and increased reliability. Replication is the key to providing high availability, fault tolerance, and enhanced performance in a distributed computing system. As companies move toward systems that are more open and distributed, replication is becoming increasingly important in the ability to provide data and services that are current, correct and available, which is a key factor in maintaining a competitive advantage over rivals. However, replication has also generated some serious challenges. For example, when a replica is updated, how do we propagate such an update to other replicas? If multiple replicas are updated simultaneously and some of the updates conflict one another, how do we resolve the conflict? How do we deal with the situation where a replica is down and then subsequently recovered? How do we deal with network partitions? Not surprisingly, considerable research efforts have been directed towards the solution of these challenges. However, most of the research results are scattered among many journals, conference proceedings, theses, and technical reports. The book by Helal, Heddaya and Bhargava has gathered the best material of these research results and formed a coherent collection that includes definitions, theoretical background, algorithms, annotated bibliographies of commercial and experimental prototype systems, and more than 200 references. This book classifies the entities that can be replicated on a distributed computing system into the following categories: (a) data, (b) process, (c) objects, and (d) messages. After the introduction on the goals and main approaches of replication techniques, the book uses individual chapters to describe major methods for each of the above replication categories. The main focus of this part, however, is on the replication of data. The book also contains two chapters on the replication issues in heterogeneous, mobile, and large-scale systems, and on the future of replication techniques. Apart from the normal chapters, the book contains a rich set of appendices that are very useful for further studies of replication techniques. These appendices include brief descriptions of two dozen commercial and experimental systems that use replication techniques, annotated bibliographies of selected literature on various topics of replication, and an introduction on serializability theory. Each annotated bibliography contains an overview of the specific topic and is written by an invited expert in that field. The combination of the normal chapters and appendices cover the entire spectrum of replication, from definitions, concepts, theories, algorithms, techniques, to systems. The book serves as an excellent introduction and roadmap to the issues of replication. Although it does not provide a detailed description of every replication techniques and systems, the book covers most of the fundamental work that allows the reader to understand the roots of these techniques and systems, and to find the ways to search for the details. This book is a valuable reference book for anyone studying replication techniques or implementing replication systems, specifically, it can be very useful to post-graduate students, practitioners, and beginning researchers of this exciting field. Wanlei Zhou, School of Computing and Mathematics, Deakin University

Until now, the analysis of fault tolerance of peer-to-peer (p2p) systems usually only covers random faults of some kind. Contrary to traditional algorithmic research, faults as well as joins and leaves occurring in a worst-case manner in p2p systems are hardly considered. Most fault tolerance analyzes are static in the sense that it is shown that a system tolerates a certain number of simultaneous faults. The much more realistic dynamic case where faults steadily occur has not found much attention. The goal of this thesis is to work towards a general understanding of fault tolerance especially for the case of dynamic and worst-case failures. The thesis introduces a simple dynamic model where a malicious adversary — controlling the arrivals and departures of the peers — and a repairing algorithm take turns. The insights gained from this model are then used to study the more realistic scenario where a repairing algorithm maintains certain properties of the topology against a concurrent adversary. Besides the comparison of different fault-tolerance models, the thesis presents a distributed hash table which maintains an efficient search structure as well as a low peer degree in spite of the worst-case failures. By a dynamic analysis we prove that no data item is lost by the system.

Diminutive devices and high clock frequency of future microprocessor generations are causing increased concerns for transient soft failures in hardware, necessitating fault detection and recovery mechanisms even in commodity processors. In this paper, we propose a fault-tolerant extension for modern superscalar out-of-order datapath that can be supported by only modest additional hardware. In the proposed extensions, error-detection is achieved by verifying the redundant results of dynamically replicated threads of executions, while the error-recovery scheme employs the instruction-rewind mechanism to restart at a failed instruction. We study the performance impact of augmenting superscalar microarchitectures with this fault tolerance mechanism. An analytical performance model is used in conjunction with a performance simulator. The simulation results of 11 SPEC95 and SPEC2000 benchmarks show that in the absence of faults, error detection causes a 2% to 45% reduction in throughput, which is in line with other proposed detection schemes. In the presence of transient faults, the fast error recovery scheme contributes very little additional slowdown.

Track switches are essential in order to enable railway vehicles to change routes however they are also the largest single cause of failure on the railway network and remain a single point of failure. This thesis focusses on modelling and fault tolerant control of a new type of electromechanical railway track switch. Unlike conventional track switch machines, this switch needs closed loop feedback control. Firstly, a detailed simulation model of the switch is developed and validated against experimental results. Two model-based control designs are then developed and tested: a classical cascaded P/PI controller and a modern state feedback LQR controller. The two controllers are compared and it is found that, whilst there are some performance differences, both meet the requirements for use in a redundantly actuated track switch. Following this, a fault tolerant control scheme for sensor faults is developed and validated using fault detection, identification and accommodation methods applied to a detailed non-linear simulation model and then validated on the experimental test rig. The results demonstrate that using a reliable fault tolerant control configuration could increase the availability and reliability of the switch in the presence of sensor faults. Finally, a fault tolerant control approach to actuator faults on the switch with triplex redundancy is developed and validated. In order to achieve this, a dynamic model of a railway track is developed in simulation as the basis for load application to the system. Three parallel actuators with their associated controllers are used to actuate the switch showing good performance in meeting the position control requirements. Different actuator faults are applied and their control performance results are shown. A summary of the control performance is also presented highlighting the differences in performance with different permutations of actuator faults. Overall, it is shown that redundancy in actuation with a suitable control scheme provides benefits in achieving post-fault availability of a redundantly actuated electromechanical railway track switch in the presence of actuator faults.

Fault-tolerance has been the focus of attention recently in the research of parallel processing and computer networking. A fault-tolerant routing algorithm should guarantee the delivery of messages in the presence of faulty components. In this paper, we present a comparative performance study of seven prominent fault-tolerant routings in 2-D wormhole-switched tori. The routing algorithms here are coupled with the Software-Based routing scheme which supports dynamic fault-model, enabling the network to remain fully operational at all times. The performance measures studied here are the throughput and average message latency. Results obtained through simulation suggest two classes of presented routing schemes as high performance candidates in most faulty networks.

Current space applications are pushing for improved on-board processing abilities, in terms of higher computing power, flexibility and fault-resistance, in order to keep up with the huge amount of collected scientific data. Multiprocessor systems seem a viable solution to match these requisites. In particular, systems employing Multistage Interconnection Networks (MINs) offer the advantage of an effective resource allocation, depending on variable workloads and occurrence of faults. The paper presents a fault-tolerant interconnection mechanism, based on redundant MIN, for multi-sensor systems. The proposed system is implemented by means of Field Programmable Gate Arrays (FPGAs) and allows a flexible re-organization of computing resources in dependence of varying operating conditions. Fault-tolerance is achieved by using an efficient FPGA re-configuration technique. Upon detection of a fault in a MIN switching element, the developed technique allows to reconfigure the corresponding FPGA.

The concept of the minimum spanning tree (MST) plays an important role in topological network design, because it models a cheapest connected network. In a tree, however, the failure of a vertex can disconnect the network. In order to tolerate such a failure, we generalize the MST to the concept of a cheapest biconnected network. For a set of points in the Euclidean plane, we show that it is NP-hard to find a cheapest biconnected spanning graph, where edge costs are the Euclidean distances of the respective points. We propose a different type of subgraph, based on forbidding (due to failure) the use of a vertex. A minimum spanning multi-tree is a spanning graph that contains for each possible forbidden vertex a spanning tree that is minimum among the spanning trees that do not use the forbidden vertex. We propose a worst-case time optimal algorithm for computing a minimum spanning multi-tree for a planar Euclidean point set. A minimum spanning multi-tree is cheap, even though it embeds a linear number of MSTs: Its cost is more than the MST cost only by a constant factor. Furthermore, we propose a linear time algorithm for computing a cheap vertex failure tolerant graph, given the Delaunay triangulation. This graph bounds the cost of the minimum spanning multi-tree from above.

Sun, J., 2020. Fault tolerance mechanism of distributed system for marine communication network. In: Gong, D.; Zhang, M., and Liu, R. (eds.), Advances in Coastal Research: Engineering, Industry, Economy, and Sustainable Development. Journal of Coastal Research, Special Issue No. 106, pp. 605–608. Coconut Creek (Florida), ISSN 0749-0208.When the communication network fails or the signal is interrupted, the distributed fault-tolerant system can continue to complete the communication task, which is needed to improve the fault-tolerant performance of the distributed system. Therefore, the key technologies of fault tolerance need to be studied, including the distributed fault-tolerant communication network simulation platform, the communication-network information-storage data structure, the communication protocol between communication network nodes, and the fault-tolerant algorithm of distributed communication network system. In the marine environment, the fleet exchanges data through wireless networks to understand the situation of the fleet-detection area, which can share the processing and calculation tasks. However, the marine environment is complex and changeable, which can cause great interference to the wireless signal, such as sea and cloud-rain clutter. These marine environments will cause the network communication signals to be greatly disturbed, which makes the distributed system unable to guarantee the communication interconnection between each node at any time. Therefore, a data-sharing and distributed-system platform needs to be built, which will design a reasonable system fault-tolerance mechanism. Through the fault-tolerance mechanism of the communication network system, the problem of frequent-node exits can be solved. By optimizing the fault-tolerant strategy of distributed architecture, the system's disaster resistance can be improved. In this research, a fault-tolerance mechanism for a distributed system was designed, which is more suitable for node election.