DOORS: towards high-performance fault tolerant CORBA

As multiprocessor systems become more complex, their reliability will need to increase as well. In this paper we propose a novel technique which is applicable to a wide variety of distributed real-time systems, especially those exhibiting data parallelism. System-level fault tolerance involves reliability techniques incorporated within the system hardware and software whereas application-level fault tolerance involves reliability techniques incorporated within the application software. We assert that, for high reliability, a combination of system-level fault tolerance and application-level fault tolerance works best. In many systems, application-level fault tolerance can be used to bridge the gap when system-level fault tolerance alone does not provide the required reliability. We exemplify this with the RTHT target tracking benchmark and the ABF beamforming benchmark.

A growing class of real-time systems require end-to-end support for various quality-of-service (QoS) aspects, including bandwidth, latency, jitter and dependability. Applications include command and control, manufacturing process control, videoconferencing, large-scale distributed interactive simulation, and testbeam data acquisition. These systems require support for stringent QoS requirements. To meet this challenge, developers are turning to distributed object computing middleware, such as the Common Object Request Broker Architecture, an Object Management Group (OMG) industry standard. In complex real-time systems, DOC middleware resides between applications and the underlying operating systems, protocol stacks and hardware. CORBA helps decrease the cycle time and effort required to develop high-quality systems by composing applications using reusable software component services rather than building them entirely from scratch. The Real-Time CORBA specification includes features to manage CPU, network and memory resources. The authors describe the key Real-Time CORBA features that they feel are the most relevant to researchers and developers of distributed real-time and embedded systems.

Fault tolerance (FT) is a topic of major concern in achieving dependable systems, for both real time as well as non real time systems. The paper provides a model of achieving fault tolerance, based on the ISO/ITU Reference Model for Open Distributed Processing (RM-ODP). This reference model provides a system software engineering methodology for fault tolerance, an object based model of fault tolerance, system requirements for achieving fault tolerance in an open manner, modeling constructs and rules to enable a proper system specification of fault tolerance, and business rules in terms of policies to achieve a well formed system specification. All these aspects are discussed at some depth, but the author primarily focuses on how certain behavior can be specified and achieved in an object based system, the constructs of the Unified Modeling Language (UML) and the Object Constraint Language (OCL).

Failures are normal rather than exceptional in cloud computing environments, high fault tolerance issue is one of the major obstacles for opening up a new era of high serviceability cloud computing as fault tolerance plays a key role in ensuring cloud serviceability. Fault tolerant service is an essential part of Service Level Objectives (SLOs) in clouds. To achieve high level of cloud serviceability and to meet high level of cloud SLOs, a foolproof fault tolerance strategy is needed. In this paper, the definitions of fault, error, and failure in a cloud are given, and the principles for high fault tolerance objectives are systematically analyzed by referring to the fault tolerance theories suitable for large-scale distributed computing environments. Based on the principles and semantics of cloud fault tolerance, a dynamic adaptive fault tolerance strategy DAFT is put forward. It includes: (i) analyzing the mathematical relationship between different failure rates and two different fault tolerance strategies, which are checkpointing fault tolerance strategy and data replication fault tolerance strategy; (ii) building a dynamic adaptive checkpointing fault tolerance model and a dynamic adaptive replication fault tolerance model by combining the two fault tolerance models together to maximize the serviceability and meet the SLOs; and (iii) evaluating the dynamic adaptive fault tolerance strategy under various conditions in large-scale cloud data centers and consider different system centric parameters, such as fault tolerance degree, fault tolerance overhead, response time, etc. Theoretical as well as experimental results conclusively demonstrate that the dynamic adaptive fault tolerance strategy DAFT has high potential as it provides efficient fault tolerance enhancements, significant cloud serviceability improvement, and great SLOs satisfaction. It efficiently and effectively achieves a trade-off for fault tolerance objectives in cloud computing environments.

Business processes show unpredictable and unstable behavior. This requires workflow systems to support monitoring functions with the ability to flexibly and robustly adapt to the changing activities of the workflow at runtime. In this paper flexible distributed realtime workflow monitoring is realized by a layered architecture. The architecture consists of: (1) middleware components supporting real-time and fault-tolerant monitoring and (2) different controllers for controlling workflows and performance data, view models for the specification of the graphical user interface and quality of service (QoS) requirements. A software system was implemented as a client-server application in Java using real-time and fault-tolerant CORBA middleware components. The prototype shows that the separation of process models, data models and view models can lead to flexible monitoring of workflows in real-time and thus guarantees robust and valuable information about business processes at runtime.

In this paper we present the design and implementation of a Pluggable Fault Tolerant CORBA Infrastructure that provides fault tolerance for CORBA applications by utilizing the pluggable protocols framework that is available for most CORBA ORBS. Our approach does not require modification to the CORBA ORB, and requires only minimal modifications to the application. Moreover; it avoids the difficulty of retrieving and assigning the ORB state, by incorporating the fault tolerance mechanisms into the ORB. The Pluggable Fault Tolerant CORBA Infrastructure achieves performance that is similar to, or better than, that of other Fault Tolerant CORBA systems, while providing strong replica consistency.

The Real-Time CORBA and minimumCORBA specifications are important steps towards defining standard-based middleware which can satisfy real-time requirements in an embedded system. These requirements can only be fulfilled if the middleware utilizes the features of a real-time network. The controller area network (CAN) is one of the most important networks in the field of real-time embedded systems. Consequently, this paper presents a CAN-based connection-oriented point-to-point communication model and its integration into Real-Time CORBA. In order to make efficient use of the advantages of CAN, we present an inter-ORB protocol, which uses smaller message headers for CAN and maps the CAN priorities to a band of CORBA priorities. We also present design and implementation details with some preliminary performance results.

To be an effective platform for performance-sensitive real-time and embedded applications, off-the-shelf CORBA middleware must preserve communication-layer quality of service (QoS) properties to applications end-to-end. However, the standard CORBA’s GIOP/IIOP interoperability protocols are not well suited for applications that cannot tolerate the message footprint size, latency, and jitter associated with general-purpose messaging and transport protocols. It is essential, therefore, to develop standard pluggable protocols frameworks that allow custom messaging and transport protocols to be configured flexibly and used transparently by applications. This paper provides three contributions to research on pluggable protocols frameworks for performance-sensitive communication middleware. First, we outline the key design challenges faced by pluggable protocols developers. Second, we describe how TAO, our high-performance, real-time CORBAcompliant ORB, addresses these challenges in its pluggable protocols framework. Third, we present the results of benchmarks that pinpoint the impact of TAO’s OO design on its endto-end efficiency, predictability, and scalability. Our results demonstrate how applying optimizations to communication middleware can yield highly flexible/reusable designs and highly efficient/predictable implementations. In particular, the overall round-trip latency of a TAO two-way method invocation using the standard inter-ORB protocol and using a commercial, off-the-self Pentium II Xeon 400 MHz workstation running in loopback mode is 125 secs. The ORB middleware accounts for approximately 48% or 60 secs of the total round-trip latency. These results illustrate that (1) communication middleware performance is largely This work was supported in part by Boeing, DARPA contract 9701516, GDIS, NSF grant NCR-9628218, Nortel, Siemens, and Sprint. an implementation detail and (2) the next-generation of optimized, standards-based CORBA middleware can replace ad hoc and proprietary solutions. Subject areas: Frameworks; Design Patterns; Distributed and Real-Time Systems

Process replication is provided as the central mechanism for application level software fault tolerance in SwiFT and DOORS. These technologies, implemented as reusable software modules, support cold and warm schemes of passive replication. The choice of a scheme for a particular application is based on its availability and performance requirements. In this paper we analyze the performability of a server software which may potentially use these technologies. We derive closed form formulae for availability throughput and probability of loss of a job. Six scenarios of loss are modeled and for each, these expressions are derived. The formulae can be used either of time or online to determine the optimal replication scheme.

In this paper we describe a software infrastructure that unifies replication and transaction processing in three-tier architectures and, thus, provides high availability and fault tolerance for enterprise applications. The infrastructure is based on the Fault Tolerant CORBA and CORBA Object Transaction Service standards, and works with commercial-off-the-shelf application servers and database systems. The infrastructure replicates the application servers to protect the business logic processing. In addition, it replicates the transaction coordinator which renders the two-phase commit protocol non-blocking and, thus, avoids potentially long service disruptions caused by coordinator failure. The infrastructure handles the interactions between the application servers and the database servers through replicated gateways that prevent duplicate requests from reaching the database servers. The infrastructure implements client-side automatic failover mechanisms, which guarantees that clients know the outcome of the requests that they have made. The infrastructure starts the transactions at the application servers, and retries aborted transactions, caused by process or communication failures, automatically on the behalf of the clients.

Many sensor-driven systems, such as those for avionics mission computing and for manufacturing process control, have stringent timing requirements for processing sensor data. Furthermore, many of these systems manage multiple sources of sensor simultaneously. Our previous work has shown that sensor-driven systems can be implemented efficiently and predictably using a real-time CORBA Event Service. This approach allows designers of real-time systems to leverage the benefits of flexible and open distributed computing architectures, such as those defined in the CORBA specification, while still meeting real-time requirements for efficiency, scalability, and predictability. To build and manage these types of systems, application developers and test engineers must be able to monitor and visualize the systems' real-time behavior. This paper describes how we have extended our distributed object visualization environment (DOVE) framework to monitor the timing behavior of a real-time application that generates and processes two separate streams of simulated sensor data events. The principal contributions of this paper are: (1) applying the DOVE framework to a simulated sensor-driven application, (2) extending the DOVE framework to support new application requirements, and (3) demonstrating and visualizing quality of service (QoS) control for multiple event streams within a real-time CORBA Event Service.

A wireless sensor network (WSN) is a collection of various tiny devices known as sensor nodes, which are also called motes. Due to high-energy consumption, the possibility of hardware, link or node failure, and some malicious attacks, sensor networks are considered error-prone networks. Hence, fault tolerance (FT) in WSN is one of the prominent issues. This article presents a novel FT approach named node-link failure fault tolerance model (NLFFT Model) in WSN, to handle the faults that occur either by link or node failure during data transmission from the sensor to the sink or base station. The NLFFT model consists of an improved quadratic minimum spanning tree (Imp-QMST) approach. This approach helps in finding the alternate link whenever it fails due to various situations and also an improved-handoff (Imp-Handoff) algorithm to support the node failure to the fault tolerance. Improved QMST presents a novel mechanism to find an alternate edge in place of the broken or failed edge in the spanning tree, to improve the fault tolerance in WSN. Imp-Handoff suggests a novel way to find the faulty node owing to less battery power and replaces a defective node by an appropriate neighbor to shift the tasks performed by a faulty node in WSN. Simulation results clearly state that as compared to the basic techniques i.e. Q-MST and Handoff algorithm, the proposed NLFFT model improvises the performance of WSN around by 7%. The results prove that the Imp-QMST gives about 6% improved throughput, 5% less end-to-end delay, and 6% less power consumption than the QMST algorithm. Similarly, Imp-Handoff improves about 4% throughput, 6% less end-to-end delay, and utilizes 7% less power consumption.

The Eternal system provides transparent fault tolerance for CORBA applications, without requiring modifications to the application or to the object request broker (ORB), and without requiring special skills of the CORBA application programmers. Eternal maintains strong replica consistency as replicas of objects perform operations, and even as they fail and are recovered. Eternal implements the new fault-tolerant CORBA standard.

Until now, the analysis of fault tolerance of peer-to-peer (p2p) systems usually only covers random faults of some kind. Contrary to traditional algorithmic research, faults as well as joins and leaves occurring in a worst-case manner in p2p systems are hardly considered. Most fault tolerance analyzes are static in the sense that it is shown that a system tolerates a certain number of simultaneous faults. The much more realistic dynamic case where faults steadily occur has not found much attention. The goal of this thesis is to work towards a general understanding of fault tolerance especially for the case of dynamic and worst-case failures. The thesis introduces a simple dynamic model where a malicious adversary — controlling the arrivals and departures of the peers — and a repairing algorithm take turns. The insights gained from this model are then used to study the more realistic scenario where a repairing algorithm maintains certain properties of the topology against a concurrent adversary. Besides the comparison of different fault-tolerance models, the thesis presents a distributed hash table which maintains an efficient search structure as well as a low peer degree in spite of the worst-case failures. By a dynamic analysis we prove that no data item is lost by the system.

In real-time systems, fault-tolerant techniques play an important role in guaranteeing the service level agreement (SLA). Primary-backup scheduling is widely used in providing fault tolerance while reducing hardware redundancy. Most existing fault tolerance scheduling algorithms only take fault tolerance performance into account while ignore other metrics, such as quality of service (QoS), load balancing and energy efficiency. Motivated by this fact, this paper proposes a QoS-aware fault-tolerant rate-monotonic first-fit scheduling algorithm, QFTRMFF, consisting of two steps. The first step is to assign tasks to processors using existing real-time fault-tolerant scheduling algorithm. In the second step, a new strategy is put forward to improve the QoS levels of tasks. Experimental results show that compared with other baseline algorithms, the proposed algorithm can substantially improve both quality of security and resource utilization under different task load.