Replica Divergence in Data-Centric Consistency Models

Eventual consistency is demanded nowadays in geo-replicated services that need to be highly scalable and available. According to the CAP constraints, when network partitions may arise, a distributed service should choose between being strongly consistent or being highly available. Since scalable services should be available, a relaxed consistency (while the network is partitioned) is the preferred choice. Eventual consistency is not a common data-centric consistency model, but only a state convergence condition to be added to a relaxed consistency model. There are still several aspects of eventual consistency that have not been analysed in depth in previous works: 1. which are the oldest replication proposals providing eventual consistency, 2. which replica consistency models provide the best basis for building eventually consistent services, 3. which mechanisms should be considered for implementing an eventually consistent service, and 4. which are the best combinations of those mechanisms for achieving different concrete goals. This paper provides some notes on these important topics.

International audience

CRDTs are distributed data types that make eventual consistency of a distributed object possible and non ad-hoc. Geo-distributed systems are spread across multiple data centers at different geographic locations to ensure availability and performance despite network partitions. These systems must accept updates at any replica and propagate these updates asynchronously to every other replica. Conflict-Free Replicated Data Types (CRDTs) ensures eventual consistency in the replicas despite asynchronous delivery of updates. Extending this idea to fog computing servers where connection reliability is low, eventual consistency amongst the servers is required. We configure Kubernetes, an open-source container orchestration system used for automating deployment, scaling, and management of containerized applications, and use it for cluster deployment of CRDT based low resource intensive AntidoteDB can be used for deployment on fog servers to ensure eventual consistency amongst these servers. We have developed an automated benchmarking tool for benchmarking edge computing applications.

Information systems based on Command Query Responsibility Segregation utilize the separation of read and write operations to the database. This separation enables the use of multiple databases and their configurations to further separate read and write operations. In the case of multiple databases, it is essential to implement synchronization between databases. Regardless of the chosen database configuration, the bottleneck remains. Bottleneck are write operations to the database. To minimize the impact of the bottleneck, developers must choose the appropriate solution. A strong consistency database model guarantees that all changes will immediately be written in the database. Another database consistency model is eventual consistency, which guarantees that all changes will eventually be written in the database with a delay. This delay in the information systems can result in inconsistent information in the user interface if more users simultaneously work with the same data. This paper aims to describe the use of strong and eventual consistency in the database for write operations in the information system based on Command Query Responsibility Segregation. The presented solution enables the use of two database consistency models for write operations and improves system performance. This paper presents a practical application of the described database consistency models.

Online program monitoring is an effective technique for detecting bugs and security attacks in running applications. Extending these tools to monitor parallel programs is challenging because the tools must account for inter-thread dependences and relaxed memory consistency models. Existing tools assume sequential consistency and often slow down the monitored program by orders of magnitude. In this paper, we present a novel approach that avoids these pitfalls by not relying on strong consistency models or detailed inter-thread dependence tracking. Instead, we only assume that events in the distant past on all threads have become visible; we make no assumptions on (and avoid the overheads of tracking) the relative ordering of more recent events on other threads. To overcome the potential state explosion of considering all the possible orderings among recent events, we adapt two techniques from static dataflow analysis, reaching definitions and reaching expressions, to this new domain of dynamic parallel monitoring. Significant modifications to these techniques are proposed to ensure the correctness and efficiency of our approach. We show how our adapted analysis can be used in two popular memory and security tools. We prove that our approach does not miss errors, and sacrifices precision only due to the lack of a relative ordering among recent events. Moreover, our simulation study on a collection of Splash-2 and Parsec 2.0 benchmarks running a memory-checking tool on a hardware-assisted logging platform demonstrates the potential benefits in trading off a very low false positive rate for (i) reduced overhead and (ii) the ability to run on relaxed consistency models.

Coherence protocols and memory consistency models are two improtant issues in hardware coherent shared memory multiprocessors and softare distributed shared memory(DSM) systems. Over the years, many researchers have made extensive study on these two issues repectively. However, the interaction between them has not been studied in the literature. In this paper, we study the coherence protocols and memory consistency models used by hardware and software DSM systems in detail. Based on our analysis, we draw a general definition for memory consistency model, i.e., memory consistency model is the logical sum of the ordering of events in each processor and coherence protocol. We also point that in hardware DSM system the emphasis of memory consistency model is relaxing the restriction of event ordering, while in software DSM system, memory consistency model focuses mainly on relaxing coherence protocol. Taking Lazy Release Consistency(LRC) as an example, we analyze the relationship between coherence protocols and memory consistency models in software DSM systems, and find that whether the advantages of LRC can be exploited or not depends greatly on it's corresponding protocol. We draw the conclusion that the more relaxed consistency model is, the more relaxed coherence protocol needed to support it. This conclusion is very useful when we design a new consistency model. Furthermore, we make some improvements on traditional multiple writer protocol, and as far as we aware, we describe the complex state transition for multiple writer protocol for the first time. In the end, we list the main research directions for memory consistency models in hardware and software DSM systems.

Systems that maintain coherence at large granularity, such as shared virtual memory systems, suffer from false sharing and extra communication. Relaxed memory consistency models have been used to alleviate these problems, but at a cost in programming complexity. Release Consistency (RC) and Lazy Release Consistency (LRC) are accepted to offer a reasonable tradeoff between performance and programming complexity. Entry Consistency (EC) offers a more relaxed consistency model, but it requires explicit association of shared data objects with synchronization variables. The programming burden of providing such associations can be substantial. This paper proposes a new consistency model for such systems, called Scope Consistency (ScC), which offers most of the performance advantages of the EC model without requiring explicit bindings between data and synchronization variables. Instead, ScC dynamically detects the associations implied by the programmer, using a programming interface similar to that of RC or LRC. We propose two ScC protocols: one that uses hardware support for fine-grained remote writes (automatic updates or AU) and the other, an all-software protocol. We compare the AU-based ScC protocol with Automatic Update Release Consistency (AURC), a modified LRC protocol that also takes advantage of automatic update support. AURC already improves performance substantially over an all-software LRC protocol. For three of the five applications we used, ScC further improves the speedups achieved by AURC by about 10%.

Over the last few years, Cloud storage systems and so-called NoSQL datastores have found widespread adoption. In contrast to traditional databases, these storage systems typically sacrifice consistency in favor of latency and availability as mandated by the CAP theorem, so that they only guarantee eventual consistency. Existing approaches to benchmark these storage systems typically omit the consistency dimension or did not investigate eventuality of consistency guarantees. In this work we present a novel approach to benchmark staleness in distributed datastores and use the approach to evaluate Amazon's Simple Storage Service (S3). We report on our unexpected findings.

Developers of modern distributed systems continuously face the impossibility result proved by the CAP theorem. In a nutshell, the theorem states that a partition-tolerant system can either guarantee consistency or availability.\n\n Most distributed programming languages implicitly make the choice between consistency or availability in their designs and implementations. Concretely, distributed programming languages can be roughly divided into two categories. A first category of languages provide abstractions to implement the consistent parts of a distributed system. A second category of languages provide abstractions to implement the available parts of a distributed system.\n\n However, real-world distributed systems often require consistency for some parts while requiring availability for others. Programmers are therefore forced to implement the abstractions missing from their chosen distributed programming language themselves or rely on external libraries.\n\n In this paper we present a novel distributed programming model. This model introduces two object-oriented abstractions: consistents and availables. The former guarantees strong consistency by sacrificing availability. The latter guarantees availability, but only provides eventual consistency. Through these constructs programmers are able to implement the entirety of their distributed system within the same language.\n\n We present a prototypical implementation of the model as a TypeScript library called CAPtain.js. To showcase the usefulness of our approach we implement a non-trivial example application. Moreover, we highlight both the functional as well as the performance characteristics of both language abstractions.

Existing group communication systems like ISIS, Spread, Jgroups etc., provide group communication in a synchronous environment. They are built on top of TCP/IP or UDP and guarantee virtual synchrony and consistency. However, wide area distributed systems are inherently asynchronous. Existing group communication systems are not suitable for wide area deployment. They do not provide persistent communication; i.e., if a node gets temporarily disconnected, all messages directed to that node during that period are lost. Hence such systems are not suitable for deployment in disadvantaged networks. While, according to Brewer’s CAP theorem, it is impossible for a distributed computer system to achieve the three objectives of consistency, availability and partition-tolerance simultaneously, especially in an asynchronous environment, we present the design and development a reliable group communication system over an asynchronous substrate, where we achieve the objectives of eventual consistency, availability and partition-tolerance. We say that distributed system is eventually consistent, over a long period of time, if no updates are sent and all updates will eventually propagate through the system and all the nodes will be consistent. By availability we mean that if a node failure does not prevent the system to operate continuously. We say that distributed system is partition-tolerant if in the event of network failure that splits the processing nodes which are communicating, then the system should allow the processing to continue in both subgroups.

Data replication is a well-known strategy to achieve the availability, scalability and performance improvement goals in the data management world. However, the cost of maintaining several database replicas always strongly consistent is very high. The CAP theorem shows that a shared-data system can choose at most two out of three properties: consistency, availability, and tolerance to partitions. In practice, most of the cloud-based data management systems tend to overcome the difficulties of distributed replication by relaxing the consistency guarantees of the system. In particular, they implement various forms of weaker consistency models such as eventual consistency. This solution is accepted by many new Web 2.0 applications (e.g. social networks) which could be more tolerant with a wider window of data staleness (replication delay).However, unfortunately, there are no generic application-independent and consumer-centric mechanisms by which software applications can specify and manage to what extent inconsistencies can be tolerated. We introduce an adaptive framework for database replication at the middleware layer of cloud environments. The framework provides flexible mechanisms to enable software applications of keeping several database replicas (that can be hosted in different data centers) with different levels of service level agreements (SLA) for their data freshness. The experimental evaluation demonstrates the effectiveness of our framework in providing the software applications with the required flexibility to achieve and optimize their requirements in terms of overall system throughput, data freshness and invested monetary cost.

Developers of distributed data-stores must trade consistency for performance and availability. Such systems may in fact implement weak consistency models, e.g., causal consistency or eventual consistency, corresponding to different costs and guarantees to the clients. We consider the case of distributed systems that offer not just one level of consistency but multiple levels of consistency to the clients. This corresponds to many practical situations. For instance, popular data-stores such as Amazon DynamoDB and Apache’s Cassandra allow applications to tag each query within the same session with a separate consistency level. In this paper, we provide a formal framework for the specification of multilevel consistency, and we address the problem of checking the conformance of a computation to such a specification. We provide a principled algorithmic approach to this problem and apply it to several instances of models with multilevel consistency.

Distributed financial systems must reconcile scalability with strict correctness guarantees. In payroll automation platforms, even minor consistency violations can produce cumulative monetary errors, regulatory exposure, and operational instability. Traditional consistency models—such as strong consistency or eventual consistency—are insufficient when applied uniformly across complex financial architectures. Instead, systems must define explicit consistency boundaries that align with financial invariants. This paper introduces the concept of consistency boundaries in distributed payroll backends and proposes architectural strategies for defining, enforcing, and validating these boundaries. By structuring identity-scoped mutation domains, separating read and write consistency zones, and coordinating cross-service financial workflows through controlled interfaces, backend systems can achieve deterministic payroll automation without sacrificing scalability. The study emphasizes that consistency is not a global property but a deliberately engineered boundary condition embedded within system design.

In this paper we develop a novel technique, called MULTIVIEW, which enables implementation of page-based fine-grain DSMs. We show how the traditional techniques for implementing page-based DSMs can be extended to control the sharing granularity in a flexible way, even when the size of the sharing unit varies, and is smaller than the operating system's page size. The run-time overhead imposed in the proposed technique is negligible. We present a DSM system, called MILLIPAGE, which builds upon MULTIVIEW in order to support sharing in variable-size units. MILLIPAGE efficiently implements Sequential Consistency and shows comparable (sometimes superior) performance to related systems which use relaxed consistency models. It uses standard user-level operating system API and requires no compiler intervention, page twinning, diffs, code instrumentation, or sophisticated protocols. The resulting system is a thin software layer consisting mainly of a simple, clean protocol that handles page-faults.

Nowadays clusters are one of the most used platforms in High Performance Computing and most programmers use the Message Passing Interface (MPI) library to program their applications in these distributed platforms getting their maximum performance, although it is a complex task. On the other side, OpenMP has been established as the de facto standard to program applications on shared memory platforms because it is easy to use and obtains good performance without too much effort. So, could it be possible to join both worlds? Could programmers use the easiness of OpenMP in distributed platforms? A lot of researchers think so. And one of the developed ideas is the distributed shared memory (DSM), a software layer on top of a distributed platform giving an abstract shared memory view to the applications. Even though it seems a good solution it also has some inconveniences. The memory coherence between the nodes in the platform is difficult to maintain (complex management, scalability issues, high overhead and others) and the latency of the remote-memory accesses which can be orders of magnitude greater than on a shared bus due to the interconnection network. Therefore this research improves the performance of OpenMP applications being executed on distributed memory platforms using a DSM with sequential consistency evaluating thoroughly the results from the NAS parallel benchmarks. The vast majority of designed DSMs use a relaxed consistency model because it avoids some major problems in the area. In contrast, we use a sequential consistency model because we think that showing these potential problems that otherwise are hidden may allow the finding of some solutions and, therefore, apply them to both models. The main idea behind this work is that both runtimes, the OpenMP and the DSM layer, should cooperate to achieve good performance, otherwise they interfere one each other trashing the final performance of applications. We develop three different contributions to improve the performance of these applications: (a) a technique to avoid false sharing at runtime, (b) a technique to mimic the MPI behaviour, where produced data is forwarded to their consumers and, finally, (c) a mechanism to avoid the network congestion due to the DSM coherence messages. The NAS Parallel Benchmarks are used to test the contributions. The results of this work shows that the false-sharing problem is a relative problem depending on each application. Another result is the importance to move the data flow outside of the critical path and to use techniques that forwards data as early as possible, similar to MPI, benefits the final application performance. Additionally, this data movement is usually concentrated at single points and affects the application performance due to the limited bandwidth of the network. Therefore it is necessary to provide mechanisms that allows the distribution of this data through the computation time using an otherwise idle network. Finally, results shows that the proposed contributions improve the performance of OpenMP applications on this kind of environments.