DARM: A Deduplication-Aware Redundancy Management Approach for Reliable-Enhanced Storage Systems

Large-scale distributed systems are a collection of loosely coupled computers interconnected by a communication network. They are now an integral part of everyday life with the development of large web applications, social networks, peer-to-peer systems, wireless sensor networks and many more. At such a scale, hardware components by themselves are prone to failure. Therefore, one key challenge in designing distributed storage systems is how to tolerate faults. To this end, fault tolerance mechanisms such as replication have been widely used to provide high availability for decades. More recently, many systems start supporting erasure coding for fault tolerance, which is expected to achieve high reliability at a lower storage cost compared to replication. However, the reduced storage overhead comes at the cost of more complicated recovery which hurts performance. In this paper, we study the fault tolerance mechanisms of two representative distributed file systems: HDFS and Ceph. In addition to the traditional replication, both HDFS and Ceph support erasure coding in their latest version. We evaluate the replication and erasure coding implementations in both systems using standard benchmarks and fault injection, and quantitatively measure the performance and storage overhead. Our results demonstrate the trade-offs between replication and erasure coding techniques, and serve as a foundation for building optimal storage systems with high availability as well as high performance.

In the era of big data, ensuring both data confidentiality and reliability has become a critical concern for data owners. While encryption algorithms and erasure coding techniques can independently guarantee confidentiality and reliability, respectively, traditional methods that apply encryption before or after erasure coding often lead to significant performance degradation and increased storage overhead. To address these limitations, this paper proposes DERAID, a novel RAID-based technology that integrates encryption directly into the coding process. DERAID is designed to improve the efficiency of encoding, decoding, updating and reconstruction operations, while simultaneously ensuring data confidentiality and reliability and minimizing the data expansion rate. The scheme employs a unified coding structure based on the ECB encryption mode, where conventional parity blocks are replaced by a combination of cryptographic keys and verification blocks. Experimental results demonstrate that, compared to traditional methods and QS-code, DERAID improves encoding performance by 44.7–95.3%, updating performance by 48.2–99.6% and reconstruction performance by 63.0–99.9%, while reducing the data expansion rate by 3.3–36.9%. These results indicate that DERAID achieves a practical balance between security, reliability and performance.

Distributed storage systems use erasure codes to reliably store data with a small storage overhead. To further improve system performance, some novel erasure codes introduce new features such as the regenerating property or symbol locality, enabling these codes to have optimal repair times and optimal degraded read performance. Unfortunately, the introduction of these new features often exacerbates the performance of other system metrics such as encoding throughput, data reliability, and storage overhead, among others. In this paper we describe the intricate relationships between erasure code properties and system-level performance metrics, showing the different tradeoffs distributed storage designers need to face. We also present Spider Codes, a new erasure code achieving a practical trade-off between the different system-level performance metrics.

Distributed storage systems are increasingly transitioning to the use of erasure codes since they offer higher reliability at significantly lower storage costs than data replication. However, these codes tradeoff recovery performance as they require multiple disk reads and network transfers for reconstructing an unavailable data block. As a result, most existing systems use an erasure code either optimized for storage overhead or recovery performance.In this paper, we present HACFS, a new erasure-coded storage system that instead uses two different erasure codes and dynamically adapts to workload changes. It uses a fast code to optimize for recovery performance and a compact code to reduce the storage overhead. A novel conversion mechanism is used to efficiently upcode and downcode data blocks between fast and compact codes. We show that HACFS design techniques are generic and successfully apply it to two different code families: Product and LRC codes.We have implemented HACFS as an extension to the Hadoop Distributed File System (HDFS) and experimentally evaluate it with five different workloads from production clusters. The HACFS system always maintains a low storage overhead and significantly improves the recovery performance as compared to three popular single-code storage systems. It reduces the degraded read latency by up to 46%, and the reconstruction time and disk/network traffic by up to 45%.

Distributed storage systems typically need data to be stored redundantly to guarantee data durability and reliability. While the conventional approach towards this objective is to store multiple replicas, today's unprecedented data growth rates encourage modern distributed storage systems to employ Erasure Coding (EC) techniques, which can achieve better storage efficiency. Various hardware-based EC schemes have been proposed in the community to leverage the advanced compute capabilities on modern data center and cloud environments. Currently, there is no unified and easy way for distributed storage systems to fully exploit multiple devices such as CPUs, GPUs, and network devices (i.e., multi-rail support) to perform EC operations in parallel; thus, leading to the under-utilization of the available compute power. In this paper, we first introduce an analytical model to analyze the design scope of efficient EC schemes in distributed storage systems. Guided by the performance model, we propose UMR-EC, a Unified and Multi-Rail Erasure Coding library that can fully exploit heterogeneous EC coders. Our proposed interface is complemented by asynchronous semantics with optimized metadata-free scheme and EC rate-aware task scheduling that can enable a highly-efficient I/O pipeline. To show the benefits and effectiveness of UMR-EC, we re-design HDFS 3.x write/read pipelines based on the guidelines observed in the proposed performance model. Our performance evaluations show that our proposed designs can outperform the write performance of replication schemes and the default HDFS EC coder by 3.7x - 6.1x and 2.4x - 3.3x, respectively, and can improve the performance of read with failure recoveries up to 5.1x compared with the default HDFS EC coder. Compared with the fastest available CPU coder (i.e., ISA-L), our proposed designs have an improvement of up to 66.0% and 19.4% for write and read with failure recoveries, respectively.

Most of the data in large-scale storage clusters is erasure coded. At exascale, optimizing erasure codes for low storage overhead, efficient reconstruction, and easy deployment is of critical importance. Locally recoverable codes (LRCs) have deservedly gained central importance in this field, because they can balance many of these requirements. In our work, we study wide LRCs; LRCs with large number of blocks per stripe and low storage overhead. These codes are a natural next step for practitioners to unlock higher storage savings, but they come with their own challenges. Of particular interest is their reliability , since wider stripes are prone to more simultaneous failures. We conduct a practically minded analysis of several popular and novel LRCs. We find that wide LRC reliability is a subtle phenomenon that is sensitive to several design choices, some of which are overlooked by theoreticians, and others by practitioners. Based on these insights, we construct novel LRCs called Uniform Cauchy LRCs , which show excellent performance in simulations and a 33% improvement in reliability on unavailability events observed by a wide LRC deployed in a Google storage cluster. We also show that these codes are easy to deploy in a manner that improves their robustness to common maintenance events. Along the way, we also give a remarkably simple and novel construction of distance-optimal LRCs (other constructions are also known), which may be of interest to theory-minded readers.

Non-volatile memories (NVM) including flash memories and resistive memories have attracted significant interest as data storage media. Flash memories are widely employed in mobile devices and solid-state drives (SSD). Resistive memories are promising as storage class memory and embedded memory applications. Data reliability is the fundamental requirement of NVM as data storage media. However, modern nano-scale NVM suffers from challenges of inter-cell interference (ICI), charge leakage, and write endurance, which threaten the reliability of stored data. In order to cope with these adverse effects, advanced coding techniques including soft decision decoding have been investigated actively. However, current coding techniques do not capture the physical properties of NVM well, so the improvement of data reliability is limited. Although soft decision decoding improves the data reliability by using soft decision values, it degrades read speed performance due to multiple read operations needed to obtain soft decision values. In this dissertation, we explore coding schemes that use side information corresponding to the physical phenomena to improve the data reliability significantly. The side information is obtained before writing data into memory and incorporated during the encoding stage. Hence, the proposed coding schemes maintain the read speed whereas the write speed performance would be degraded. It is a big advantage from the perspective of speed performance since the read speed is more critical than the write speed in many memory applications. First, this dissertation investigates the coding techniques for memory with stuckat defects. The idea of coding techniques for memory with stuck-at defects is employed to handle critical problems of flash memories and resistive memories. For 2D planar flash memories, we propose a coding scheme that combats the ICI, which is a primary challenge of 2D planar flash memories. Also, we propose a coding scheme that reduces the effect of fast detrapping, a degradation factor in 3D vertical flash memories. Finally, we investigate the coding techniques that improve write endurance and power consumption of resistive memories.

In modern energy-saving replication storage systems, a primary group of disks is always powered up to serve incoming requests while other disks are often spun down to save energy during slack periods. However, since new writes cannot be immediately synchronized into all disks, system reliability is degraded. In this paper, we develop a high-reliability and energy-efficient replication storage system, named RERAID, based on RAID10. RERAID employs part of the free space in the primary disk group and uses erasure coding to construct a code cache at the front end to absorb new writes. Since code cache supports failure recovery of two or more disks by using erasure coding, RERAID guarantees a reliability comparable with that of the RAID10 storage system. In addition, we develop an algorithm, called erasure coding write (ECW), to buffer many small random writes into a few large writes, which are then written to the code cache in a parallel fashion sequentially to improve the write performance. Experimental results show that RERAID significantly improves write performance and saves more energy than existing solutions.

Erasure coding has been increasingly replacing replication in distributed storage systems, thanks to its lower storage overhead with the same level of failure tolerance. However, with lower storage overhead, the reconstruction overhead of erasure codes can increase significantly as well. Under the ever-changing workload, in which the data access can be highly skewed, it is difficult to achieve a well trade-off between the storage overhead and the reconstruction overhead. In this paper, we propose Zebra, a framework that encodes data into multiple tiers by their demand. Given the overall storage overhead and the number of failures to tolerate, Zebra determines the parameters of erasure coding in each tier by solving a geometric programming problem. Based on the demand of data, Zebra can dynamically assign data into the corresponding tiers to minimize the overall reconstruction overhead, and achieve a flexible tradeoff between the storage overhead and the reconstruction overhead in multiple tiers, such that hot data can enjoy less overhead of reconstruction and cold data can be stored with lower storage overhead. When demand changes, Zebra can adjust itself accordingly with a marginal amount of network transfer.

This paper surveys previous distributed storage systems and related data redundancy and fault-tolerance schemes which are introduced to overcome the impact of host churn on data reliability. Furthermore, a hybrid storage system model is proposed which offers a reliable data storage service by integrating idle storage contributed by volatile peer nodes and stable and durable storage utilities. In order to ensure high availability and durability for this hybrid storage system, we explore four reliability improvement strategies, including File Replica Strategy, File Encoding Strategy, Replica Repair Strategy, and Stable-Volatile Strategy, as well as the combination of these four strategies. Extensive simulations based on real traces are performed, in which data availability, data durability, and storage overhead are evaluated. Simulation results show that compared with previous peer-to-peer storage systems, the proposed hybrid storage system could achieve a higher availability and durability with less storage consumption, due to proposed new strategies. Finally, taking into account storage and traffic cost, the tradeoffs between storage efficiency and reliability are discussed.

Coping with failures in modern distributed storage systems that handle massive volumes of heterogeneous and potentially rapidly changing data, has become a very important challenge. A common practice is to utilize fault tolerance methods like Replication and Erasure Coding for maximizing data availability. However, while erasure codes provide better fault tolerance compared to replication with a more affordable storage overhead, they frequently suffer from high reconstruction cost as they require to access all available nodes when a data block needs to be repaired, and also can repair up to a limited number of unavailable data blocks, depending on the number of the code’s parity block capabilities. Furthermore, storing and placing the encoded data in the federated storage system also remains a challenge. In this paper we present Fed-DIC, a framework which combines Diagonally Interleaved Coding on client devices at the edge of the network with organized storage of encoded data in a federated cloud system comprised of multiple independent storage clusters. The erasure coding operations are performed on client devices at the edge while they interact with the federated cloud to store the encoded data. We describe how our solution integrates the functionality of federated clouds alongside erasure coding implemented on edge devices for maximizing data availability and we evaluate the working and benefits of our approach in terms of read access cost, data availability, storage overhead, load balancing and network bandwidth rate compared to popular Replication and Erasure Coding schemes.

Erasure coding is a technique for achieving high availability and reliability in storage and communication systems. We revisit the analysis of erasure code replication and point out some situations when whole-file replication is preferred. The switchover point (from preferring whole-file replication to erasure code replication) is studied, and characterized using asymptotic analysis. We also discuss the additional considerations in building erasure code replication systems.

Data centers store a massive amount of data in a large number of servers built by commodity hardware. To maintain data integrity against server failures, erasure codes have been extensively deployed in modern data centers to provide a higher level of failure tolerance with less storage overhead than replication. Yet, compared to replication, disseminating erasure-coded data from a source server into multiple servers will also take significantly more time. In this paper, we design and implement Mist, a new mechanism for disseminating erasure-coded data efficiently to multiple receiving servers (receivers) in data centers. Mist speeds up the dissemination process by building an efficient topology among the receivers with heterogeneous performance, so that coded data can be received from other receivers in a pipelined fashion, rather than directly from the source. Mist flexibly supports a wide range of erasure codes, without imposing constraints to the range of system parameters, and can be extended for specific erasure codes with better performance by taking advantage of the corresponding erasure code. We have implemented Mist in Python, and our experimental results in Amazon EC2 have demonstrated that the dissemination time can be reduced by up to 96.3 percent with different kinds of erasure codes. © 2013 IEEE.

International audience

Erasure codes, such as Reed-Solomon (RS) codes, are increasingly being deployed as an alternative to data-replication for fault tolerance in distributed storage systems. While RS codes provide significant savings in storage space, they can impose a huge burden on the I/O and network resources when reconstructing failed or otherwise unavailable data. A recent class of erasure codes, called minimum-storage-regeneration (MSR) codes, has emerged as a superior alternative to the popular RS codes, in that it minimizes network transfers during reconstruction while also being optimal with respect to storage and reliability. However, existing practical MSR codes do not address the increasingly important problem of I/O overhead incurred during reconstructions, and are, in general, inferior to RS codes in this regard. In this paper, we design erasure codes that are simultaneously optimal in terms of I/O, storage, and network bandwidth. Our design builds on top of a class of powerful practical codes, called the product-matrix-MSR codes. Evaluations show that our proposed design results in a significant reduction the number of I/Os consumed during reconstructions (a 5× reduction for typical parameters), while retaining optimality with respect to storage, reliability, and network bandwidth.