Comparison of Redundancy Schemes for Distributed Storage Systems

The exponentially growing demand for storage puts a huge stress on traditional distributed storage systems. Historically, I/Ops (Inputs/Outputs per second) of hard drives have been the main limitation of storage systems. With the rapid deployment of solid state drives (SSDs) and the expected evolutions of their capacities, price and performance, we claim that CPU and network capacities will become bottlenecks in the future. In this context, we introduce 6Stor, an innovative, software-defined distributed storage system fully integrated with the networking layer. This storage system departs from traditional approaches in two manners: it leverages IPv6 new capabilities to increase the efficiency of its data plane—notably by using directly UDP and TCP rather than HTTP—and thus its performance; and it circumvents scalability limitations of other distributed systems by using a fully distributed metadata layer of indirection to offer flexibility. In this paper, we introduce and describe in details the architecture of 6Stor, with an emphasis on dynamic scalability and robustness to failure. We also present a testbed that we use to evaluate our novel approach by using Ceph—another well known distributed storage system—as baseline. Results obtained on an extensive testbed are presented and some initial conclusions are drawn.

This article has no abstract.

We have developed a distributed parallel storage system that employs the aggregate bandwidth of multiple data servers connected by a high-speed wide-area network to achieve scalability and high data throughput. This paper studies different schemes to enhance the reliability and availability of such network-based distributed storage systems. The general approach of this paper employs erasure error-correcting codes that can be used to reconstruct missing information caused by hardware, software, or human faults. The paper describes the approach and develops optimized algorithms for the encoding and decoding operations. Moreover, the paper presents techniques for reducing the communication and computation overhead incurred while reconstructing missing data from the redundant information. These techniques include clustering, multidimensional coding, and the full two-dimensional parity schemes. The paper considers trade-offs between redundancy, fault tolerance, and complexity of error recovery.

This paper surveys previous distributed storage systems and related data redundancy and fault-tolerance schemes which are introduced to overcome the impact of host churn on data reliability. Furthermore, a hybrid storage system model is proposed which offers a reliable data storage service by integrating idle storage contributed by volatile peer nodes and stable and durable storage utilities. In order to ensure high availability and durability for this hybrid storage system, we explore four reliability improvement strategies, including File Replica Strategy, File Encoding Strategy, Replica Repair Strategy, and Stable-Volatile Strategy, as well as the combination of these four strategies. Extensive simulations based on real traces are performed, in which data availability, data durability, and storage overhead are evaluated. Simulation results show that compared with previous peer-to-peer storage systems, the proposed hybrid storage system could achieve a higher availability and durability with less storage consumption, due to proposed new strategies. Finally, taking into account storage and traffic cost, the tradeoffs between storage efficiency and reliability are discussed.

Data deduplication has become a crucial technique for reducing data in modern storage systems. We present SEP-D, a practical scale-out distributed storage system to incorporate data deduplication for primary storage. SEP-D introduces a novel metadata handling mechanism which combines contentbased hashing with built-in distributed data placement strategies such as CRUSH. This enables SEP-D to eliminate the needs for remote metadata lookups, thus incorporating deduplication without affecting scalability. SEP-D integrates smoothly with the existing storage system, allowing the re-use of storage policies across different pools of storage. We implemented SEP-D in Ceph, a popular distributed storage system widely adopted in the industry, and demonstrated that SEP-D has minimal impact on I/O performance in data while maintaining existing storage policies implemented in underlying distributed storage systems.

In applications of distributed storage systems to distributed computing and implementation of key- value stores, the following property, usually referred to as consistency in computer science and engineering, is an important requirement: as the data stored changes, the latest version of the data must be accessible to a client that connects to the storage system. An information theoretic formulation called multi-version coding is introduced in the paper, in order to study storage costs of consistent distributed storage systems. Multi-version coding is characterized by {\nu} totally ordered versions of a message, and a storage system with n servers. At each server, values corresponding to an arbitrary subset of the {\nu} versions are received and encoded. For any subset of c servers in the storage system, the value corresponding to the latest common version, or a later version as per the total ordering, among the c servers is required to be decodable. An achievable multi-version code construction via linear coding and a converse result that shows that the construction is approximately tight, are provided. An implication of the converse is that there is an inevitable price, in terms of storage cost, to ensure consistency in distributed storage systems.

This study develops a methodology for coordinated operation of distributed energy storage systems in distribution networks. The developed methodology considers that energy storage resources can contribute to their owners’ inherent activities and to a more flexible and efficient distribution network operation. The optimisation tool based on mixed‐integer linear programming is developed to maximise the technical and economic value of distributed storage taking into account their multifunctional potential and the presence of intermittent renewable sources. The methodological developments are validated in a case study of a real medium‐voltage distribution network with two storage systems coupled with a wind park and an industrial ‘prosumer’ (i.e. energy consumers who are producing their own energy). Results make evident the robustness of the methodology and enable the assessment of the technical and economic impacts of the integration of distributed storage. Moreover, opportunity costs for distributed storage to perform services to the distribution system operator are demonstrated.

Erasure codes are recently used in real-world distributed storage systems (DSSs) such as Google File System,Microsoft Azure Storage, and Facebook HDFS-RAID for data reliability. When designing erasure codes for DSSs, special attention is given to the associated costs of data handling operations such as repair or update. For example, locally repairable codes (LRC) are designed and used in DSSs to allow for low-cost repair of failed blocks. Update complexity (defined as the number of blocks that need to be updated when an information block is changed) is yet another design parameter. This parameter can be seen as a measure of the computation, I/O and networking costs associated with updating an information block in a DSS. Since information is frequently updated by many applications, lowering update complexity can result in lower power consumptions in DSSs. In this work, we study the update complexity of LRCs. Based on our study, we propose an improvement over the LRC used by Facebook HDFS-RAID. Keeping the same code parameters including length, storage overhead, minimum distance and cost of repair (locality), we improve the update complexity by more than 16%. Moreover, we show that with these parameters achieving a lower update complexity is impossible.

With the development of smart devices and social media, massive unstructured data is uploaded to distributed storage systems. Since the characteristics of multi-users and high concurrency the unstructured data accesses have, it brings new challenges to traditional distributed storage systems designed for large files. We propose a grouping strategy to analyze relevant data in access according to disk access logs in the real distributed storage systems environment. When any data in the group is accessed, the whole group is prefetched from disk to the cache. Firstly, we conduct statistical analysis on the access logs and propose a preliminary classification method to classify files in spatiotemporal locality. Secondly, a strength-priority tree structure relation model (SP-TSRM) is proposed to mine file group efficiently. Finally, experiments show that the proposed model can improve the cache hit rate significantly, thereby improving the read efficiency of distributed storage systems.

Regenerating codes (RGC) are a class of distributed storage codes that can provide efficient repair of failure nodes in distributed storage systems. In general, the reduction of repair bandwidth of RGC is at the expense of a small increase in storage cost and computational cost. The high computational complexity of data coding over a finite field of large size makes it unsuitable for practical distributed storage systems. BASIC codes, which stands for Binary Addition and Shift Implementable Convolutional codes, is introduced in [1] with the aim of reducing computational complexity, while retaining the benefits of RGC. In this paper, we present a construction of exact-repair BASIC codes at the minimum-storage point (MSR). A helper node needs no coding to repair a failure node for the minimum-storage BASIC codes. The results of simulation show minimum-storage BASIC codes outperform Cauchy Reed-Solomon codes in both repairing cost and coding cost.

The failure of a single unreliable commodity component is very common in large-scale distributed storage systems. In order to ensure the reliability of data in large-scale distributed storage systems, many studies have emerged one after another. Among them, Erasure Codes are widely used in actual storage systems, such as Hadoop Distributed File System (HDFS), which provides high fault-tolerance with lower storage overhead. However, usually the recovery of erasure-coded storage system when encountering node failure results in severe cross-node and cross-rack bandwidth consumption, which affects the efficiency of failure recovery and wastes additional resources. In this paper, we improve the erasure coding storage strategy in distributed storage systems, and propose a low-overhead data recovery method based on cross-checking, namely HV-SNSP. In HV-SNSP, horizontal and vertical cross parity checking is realized by adding RS parity inside the data node, that is, <inline-formula> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, k)}}$ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', \textrm {}k')}}$ </tex-math></inline-formula> storage architecture. Based on <inline-formula> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, \textrm {}k)}}$ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', k')}}$ </tex-math></inline-formula>, a low-cost supply node selection strategy, namely SNSP, is designed, and nodes with shorter network distance and lower load are selected to participate in recovery. This strategy can effectively reduce the amount of data transmission, shorten the recovery time, and improve the recovery efficiency. The experimental results show that compared with traditional RS, HV-SNSP can reduce the amount of cross-rack data transmission by 62.5&#x0025; during data recovery, and can shorten the recovery time by up to 42.41&#x0025;; Compared with D3, HV-SNSP can reduce the occupation of cross-rack bandwidth by 25&#x0025; and shorten the recovery time by 36.58&#x0025;.

The limitation of scheduling modules and the gradual addition of disk pools in distributed storage systems often result in imbalances among their disk pools in terms of both disk usage and file count. This can cause various problems to the storage system such as single point of failure, low system throughput and imbalanced resource utilization and system loads. An algorithm named Fuzzy Pool Balance (FPB) is proposed here to solve this problem. The input of FPB is the current file distribution among disk pools and the output is a file migration plan indicating what files are to be migrated to which pools. FPB uses an array to classify the files by their sizes. The file classification array is dynamically calculated with a defined threshold named Tmax that defines the allowed pool disk usage deviations. File classification is the basis of file migration. FPB also defines the Immigration Pool (IP) and Emigration Pool (EP) according to the pool disk usage and File Quantity Ratio (FQR) that indicates the percentage of each category of files in each disk pool, so files with higher FQR in an EP will be migrated to IP(s) with a lower FQR of this file category. To verify this algorithm, we implemented FPB on an ATLAS Tier2 dCache production system. The results show that FPB can achieve a very good balance in both free space and file counts, and adjusting the threshold value Tmax and the correction factor to the average FQR can achieve a tradeoff between free space and file count.

With high penetration of renewables-based distributed generation sources (RDGs) and distributed storage systems (DSSs), such as electric vehicles (EVs), in distribution systems, several issues such as maintaining energy-neutrality and computation/communication bottlenecks arise. These can be efficiently handled with coordination of DSSs that act as buffer and absorb/inject necessary active power, and by using distributed computing approaches. Therefore, in this paper, a holistic strategy is proposed to optimize the costs and revenues of owners in a community while maintaining net energy neutrality using a fully distributed consensus-based algorithm. Simulation studies performed on modified IEEE 30-bus test system show that the proposed algorithm effectively optimizes the community's devices under dynamic conditions.

To satisfy more and more complicated requirements on information recording, processing, exchange etc. in newly developed intelligent satellites, a distributed storage system is designed based on SpaceWire network in satellite platform. In this distributed storage system, a SpaceWire router unit serving as the core device of star network connects several nodes, including onboard computers, namely, satellite management unit (SMU), data interface units (DIU) and other nodes like attitude and orbit sensors, space environment monitor instruments and payload controllers, etc. A certain number of standard flash memory modules are separately incorporated into SMU and DIUs to provide comprehensive data recording and exchange service for all nodes in the network. All memory modules are centrally controlled by SMU software to store authorized data in specific type files and retrieve data according to policies set by remote memory access protocol commands. Memory modules that can run in parallel in the distributed storage system produce very high data throughput, which boosts satellite platform capability to a great extent. Backup mechanism among memory modules is deployed and any module can be replaced by others when it meets failure, this makes the storage system robust and reliable. Testing and verification work performed on the distributed storage system shows that the system works quite well and also reveals its attractive effect on improving satellite platform efficiency.

Among different setups of cloud storage systems, fountain-codes based distributed cloud storage system provides reliable online storage solution through placing coded content fragments into multiple storage nodes. Luby Transform (LT) code is one of the popular fountain codes for storage systems due to its efficient recovery. However, to ensure high success decoding of fountain codes based storage, retrieval of additional fragments is required, and this requirement introduces additional delay, which is critical for content retrieval or downloading applications. In this paper, we show that multiple-stage retrieval of fragments is effective to reduce the content-retrieval delay. We first develop a delay model for various multiple-stage retrieval schemes applicable to our considered system. With the developed model, we study optimal retrieval schemes given the success decodability requirement. Our numerical results demonstrate that the content-retrieval delay can be significantly reduced by optimally scheduling packet requests in a multi-stage fashion.