Efficiency Comparison of Row-Diagonal Parity and EVENODD Encoded Check Disk Repair Algorithms

Redundant array of independent disks (RAID) has been widely used to address the reliability and performance issues of storage systems. As the scale of modern storage systems continues growing, disk failure becomes the norm. With the ever-increasing disk capacity, RAID recovery based on disk rebuild becomes more and more costly, which causes significant performance degradation and even unavailability of storage systems. Declustered data layout enables parallel RAID reconstruction by shuffling data and parity blocks among all drives (including spares) in a RAID group. However, the reliability and performance of declustered RAID in real-world storage environments have not been thoroughly studied. With the popularity of ZFS file system and software RAID used in production data centers, in this paper, we extensively evaluate declustered RAID with regard to the RAID recovery time and I/O performance on a high-performance storage platform at Los Alamos National Laboratory. Our empirical study reveals the advantages and disadvantages of declustered RAID technology. We qualitatively characterize the recovery performance of declustered RAID and compare with that of ZFS RAIDZ under various I/O workloads and access patterns. The experimental results show that the speedup of declustered RAID over traditional RAID is sub-linear to the parallelism of recovery I/O. Furthermore, we formally model and analyze the reliability of declustered RAID in terms of the mean-time-to-data-loss (MTTDL) and discover that the improved recovery performance leads to higher storage reliability compared with the traditional RAID.

This paper studies how RAID (redundant array of independent disks) could take full advantage of modern SSDs (solid-state drives) with built-in transparent compression. In current practice, RAID users are forced to choose a specific RAID level (e.g., RAID 10 or RAID 5) with a fixed storage cost vs. speed performance trade-off. The commercial market is witnessing the emergence of a new family of SSDs that can internally perform hardware-based lossless compression on each 4KB LBA (logical block address) block, transparent to host OS and user applications. Beyond straightforwardly reducing the RAID storage cost, such modern SSDs make it possible to relieve RAID users from being locked into a fixed storage cost vs. speed performance trade-off. In particular, RAID systems could opportunistically leverage higher-than-expected runtime user data compressibility to enable dynamic RAID level conversion to improve the speed performance without compromising the effective storage capacity. This paper presents techniques to enable and optimize the practical implementation of such elastic RAID systems. We implemented a Linux software-based elastic RAID prototype that supports dynamic conversion between RAID 5 and RAID 10. Compared with a baseline software-based RAID 5, under sufficient runtime data compressibility that enables the conversion from RAID 5 to RAID 10 over 60% of user data, the elastic RAID could improve the 4KB random write IOPS (I/O per second) by 42% and 4KB random read IOPS in degraded mode by 46%, while maintaining the same effective storage capacity.

Redundant Array of Independent Disks (RAID) storage architectures provide protection of digital infrastructure against potential disks failures. For example, RAID-5 and RAID-6 architectures provide protection against one and two disk failures, respectively. Recently, the data generation has significantly increased due to the emergence of new technologies. Thus, the size of storage systems is also growing rapidly to accommodate such large data sizes, which increases the probability for disks failures. This necessitates a new RAID architecture that can tolerate up to three disk failures. RAID architectures implement coding techniques. The code specifies how data is stored among multiple disks and how lost data can be recovered from surviving disks. This abstract introduces a novel coding scheme for new RAID-7 architectures that can tolerate up to three disks failures. The code is an improved version of the existing BP-XOR code and is called “Almost BP-XOR”.There are multiple codes that can be used for RAID-7 architectures. However, [5,2] BP-XOR codes have significantly lower encoding and decoding complexities than most common codes [1]. Regardless of this fact, this code does not achieve the fastest data decoding and reconstruction speeds due to its relatively low efficiency of 0.4. Furthermore, the existence of MDS [6,3] bx6 BP-XOR codes, b>2 (which achieves efficiency of 0.5) is still an open research question. This work proposes [6,3] 2 x 6 Almost BP-XOR codes. These codes largely utilize the simple and fast BP-XOR decoder while achieving an efficiency of 0.5, leading to the fastest recovery from disk failures among other state-of-the-art codes. An algorithm to generate a [6, 3] 2 x 6 Almost BP-XOR code has been developed and an example code is provided in Table 1. The [6, 3] 2 x 6 Almost BP-XOR codes are constructed in a way that any three-column-erasure pattern will result in one of the following two main scenarios. First: At least one of the surviving degree-three encoding symbols contains two known information symbols. This scenario occurs in 70% of three-column erasure cases (i.e. 14 out of the 20 possible cases). The recovery process in such scenario is identical to that of the BP-XOR codes; Knowing any two information symbols in a degree-three encoding symbol is sufficient to know the third information symbol through performing a simple XOR operation. Second: None of the surviving degree-three encoding symbols contains two known information symbols. This scenario occurs in the remaining 30% of three-column erasure cases (i.e., 6 out if the possible 20). The BP-XOR decoder fails in such a scenario. However, due to the construction of the codes, at least one surviving degree-three encoding symbol contains a known information symbol. Thus, knowing one of the reaming two information symbols in such a degree-three encoding symbol will initiate the BP-XOR decoder again.Table 2 shows these erasure patterns along with an expression for one of the missing information symbols. these expressions can be stored in buffers and used whenever the corresponding erasure pattern occurs. Solutions in Table 2 are derived from the inverse of a 6x6 submatrix that results from a generator matrix G by deleting columns from G corresponding to erased code columns. The read complexity of almost BP-XOR codes is 1. On the other hand. The decoding of almost BP-XOR codes require just 6 XOR operations when for a given three-column-erasure pattern BP-XOR decoding succeeds. However, when the BP- XOR decoder fails, it will require up to 15 XOR operations in total. The normalized repairing complexity is 15/6 = 2.5.Experimentally, Fig.1 shows that the proposed Almost BP-XOR codes require the least amount of time to decode and reconstruct erased columns. Thus, it is concluded that the [6, 3] 2x6 almost BP-XOR codes are best suited for RAID-7 system that requires storage efficiency of 0.5.

Because of the ever increasing amount of information needing to be managed, the development of storage technology has become very important. RAID (redundant array of independent disks) is usually used for administering large storage, although traditional RAIDs have only one or two fault tolerances, which is not sufficient for constructing large storage. Although hierarchical RAID (HiRAID) has three fault tolerances, efficiency in capacity is low. In the previous work, we proposed N-ary RAID, which is flat (not hierarchical) RAID with multiple parities based on a block unit. N-ary RAID is a 3 fault tolerant RAID. In this paper, we propose N-ary RAID MP, which is combined N-ary RAID to RAID MP (multiple parity). It drastically increases the reliability of RAID based storage.

This paper is a summary and evaluation of redundant array of independent disks (also called redundant array of inexpensive disks before) that was proposed years ago. Recently more information is produced every year, and the demand of data storage is increasing exponentially due to Moore's Law. So, it's much more common to group a dozen of hard disk drives together, to achieve better performance or to achieve better data safety. Firstly, the redundant array of independent disks aims to solve the data safety/data in-out performance problems, and multiple ways are proposed including Level 0 to solve the performance problem sacrificing system stability and data safety and Level 1 aims to achieve better stability but using only 50% (or less) of the data capacity. The development of redundant array of independent disks tends to be merged and have more functions in one single system of redundant array of inexpensive disks.

RAIDs (Redundant Arrays of Independent Disks) are a set of disks organized to achieve parallel I/O to multiple disks and to provide tolerance of disk failures. RAIDs offer these advantages at the cost of additional space and additional disk I/O for writes. Previous methods of reducing this I/O overhead suffered from such problems as requiring periods during which data is reorganized and not available, destroying the physical locality of data, or weakening the RAID's fault-tolerance properties. We propose a new method called data logging which reduces the I/O overhead without requiring periodic downtime for reorganization. Instead, incremental maintenance can be performed concurrently with routine processing. This is particularly advantageous in applications requiring 24/spl times/7 uptime. Data logging preserves both physical locality of data and RAID fault tolerance. The major cost of our method is a moderate amount of nonvolatile RAM. This paper describes our method, as well as two schemes for efficient encoding of the information that must be stored in nonvolatile RAM.

The current parallel storage systems use thousands of inexpensive disks to meet the storage requirement of applications. Data redundancy and/or coding are used to enhance data availability, for instance, Row-diagonal parity (RDP) and EVENODD codes, which are widely used in RAID-6 storage systems, provide data availability with up to two disk failures . To reduce the probability of data unavailability, whenever a single disk fails, disk recovery will be carried out. We find that the conventional recovery schemes of RDP and EVENODD codes for a single failed disk only use one parity disk. However, there are two parity disks in the system, and both can be used for single disk failure recovery. In this article, we propose a hybrid recovery approach that uses both parities for single disk failure recovery, and we design efficient recovery schemes for RDP code (RDOR-RDP) and EVENODD code (RDOR-EVENODD). Our recovery scheme has the following attractive properties: (1) “ read optimality ” in the sense that our scheme issues the smallest number of disk reads to recover a single failed disk and it reduces approximately 1/4 of disk reads compared with conventional schemes; (2) “ load balancing property ” in that all surviving disks will be subjected to the same (or almost the same) amount of additional workload in rebuilding the failed disk. We carry out performance evaluation to quantify the merits of RDOR-RDP and RDOR-EVENODD on some widely used disks with DiskSim. The offline experimental results show that RDOR-RDP and RDOR-EVENODD outperform the conventional recovery schemes of RDP and EVENODD codes in terms of total recovery time and recovery workload on individual surviving disk. However, the improvements are less than the theoretical value (approximately 25%), as RDOR-RDP and RDOR-EVENODD change the disk access pattern from purely sequential to a more random one compared with their conventional schemes.

A family of codes with a natural 2-D structure is presented, inspired by an application of redundant arrays of independent disks (RAID) type of architectures whose units are solid-state drives (SSDs). Arrays of SSDs behave differently from arrays of hard disk drives, since hard errors in sectors are common and traditional RAID approaches (like RAID 5 or RAID 6) may be either insufficient or excessive. An efficient solution to this problem is given by the new codes presented, called partial maximum distance separable (PMDS) codes.

To satisfy the explosive growth of data in large-scale data centers, where redundant arrays of independent disks (RAIDs), especially RAID-5, are widely deployed, effective storage scaling and disk expansion methods are desired. However, a way to reduce the data migration overhead and maintain the reliability of the original RAID are major concerns of storage scaling. To address these problems, we propose a new RAID scaling scheme, H-Scale, to achieve fast RAID scaling via hybrid stripe layouts. H-Scale takes advantage of the loose restriction of stripe structures to choose migrated data and to create hybrid stripe structures. The main advantages of our scheme include: (1) dramatically reducing the data migration overhead and thus speeding up the scaling process, (2) maintaining the original RAID’s reliability, (3) balancing the workload among disks after scaling, and (4) providing a general scaling approach for different RAID levels. Our theoretical analysis show that H-Scale outperforms existing scaling solutions in terms of data migration, I/O overheads, and parity update operations. Evaluation results on a prototype implementation demonstrate that H-Scale speeds up the online scaling process by up to 60% under SPC traces, and similar improvements on scaling time and user response time are also achieved by evaluations using standard benchmarks.

본 논문에서는 고비용의 엔트프라이즈급 RAID 저장장치를 저비용의 RAID 저장장치로 대체하고 인공위성 연구센터에서 개발한 데이터 수신카드를 이용하여 고속 데이터 수신저장 시스템을 구축하였다. 350Mbps로 수신되는 데이터를 처리하기 위해서 S-ATA2 방식의 하드디스크 3개와 자체 제작한 성능 시험 소프트웨어를 이용하였다. 성능 시험 결과 79MB의 속도로 RAID에 저장할 수 있음을 확인하였다. This paper describes the establishment result of high speed data receiving and archiving system(HDRAS) using data receiving card(DRC) developped by SaTReC and low cost RAID(Redundant Array of Independent Disks) storage instead of expensive enterprise RAID storage. We used three S-ATA2 type hard disks and performance test program developped by SaTReC to receive and save data at 350Mbps. Finally, we verified that the HDRAS using low cost RAID storage has a capacity to process received data at 79MB/s(632Mbps).

정전 방지 장치가 없는 중소형 레이드 (RAID: redundant arrays of independent disks)는 갑작스런 정전 또는 오류로 인한 종료 이후에 수시간의 긴 재동기화 시간을 요구한다. 레이드에서는 데이터 블록과 패리티 블록이 일관성 있게 갱신되어야 하는데, 데이터를 기록하다가 정전이 되면 데이터 블록은 갱신되었는데 패리티 블록은 갱신되지 않거나 반대인 경우가 발생할 수 있다. 이렇게 부분적으로 갱신된 스트라이프를 반드시 올바른 패리티로 갱신해야 하나 어떤 스트라이프에 이런 문제가 발생하였는지 찾기가 매우 어려웠다. 기존에는 전 저장공간을 검색하고 오류 있는 스트라이프를 수정하는, 수 시간을 요구하는, 패리티 재동기화 방법이 사용되어 왔다. 본 논문은 중소형 레이드에서 낮은 오버헤드를 갖는 고속의 재동기화 기술을 제안한다. 제안하는 기술은 리눅스의 소프트웨어 레이드에서 구현되었다. 성능 실험 결과에 따르면, 제안하는 기법은 재동기화 과정을 200분에서 5초로 단축시키고, 일반 입출력에서 22%의 오버헤드를 2%로 낮추었다. Redundant arrays of independent disks (RAID) without a power-fail-safe component in small and mid-sized business suffers from intolerably long resynchronization time after a unclean power-failure. Data blocks and a parity block in a stripe must be updated in a consistent manner, however a data block may be updated but the corresponding parity block may not be updated when a power goes off. Such a partially modified stripe must be updated with a correct parity block. However, it is difficult to find which stripe is partially updated (inconsistent). The widely-used traditional parity resynchronization manner is a intolerably long process that scans the entire volume to find and fix inconsistent stripes. This paper presents a fast resynchronization scheme with a negligible overhead for small and mid-sized RAIDs. The proposed scheme is integrated into a software RAID driver in a Linux system. According to the performance evaluation, the proposed scheme shortens the resynchronization process from 200 minutes to 5 seconds with 2% overhead for normal I/Os.

Disk reliability is a serious problem in the big data foundation environment. Although the reliability of disk drives has greatly improved over the past few years, they are still the most vulnerable core components in the server. If they fail, the result can be catastrophic: it can take some days to recover data, sometimes data lost forever. These are unacceptable for some important data. XOR parity is a typical method to generate reliability syndrome, thus improving the reliability of the data. In practice, we find that the data is still likely to be lost. In most storage systems reliability improvements are achieved through the allocation of additional disks in Redundant Arrays of Independent Disks (RAID), which will increase the hardware costs, thus it will be very difficult for cost-constrained environments. Therefore, how to improve the data integrity without raising the hardware cost has aroused much interest of big data researchers. This challenge is when creating non-traditional RAID geometries, care must be taken to respect data dependence relationships to ensure that the new RAID strategy improves reliability, which is a NP-hard problem. In this paper, we present an approach for characterizing these challenges using high-dimension variants of the n-queens problem that enables performable solutions via the SAT solver MiniSAT, and use the greedy algorithm to analyze the queen's attack domain, as a basis for reliability syndrome generation. A large number of experiments show that the approach proposed in this paper is feasible in software-defined data centers and the performance of the algorithm can meet the current requirements of the big data environment.

The complexity of programmable logic design has increased ever since its conception because of the increased focus on seamless integration of design creation. Interconnections, which occupy 60 to 90% of the chip area, play a vital role in deciding the power and delay in such designs. Hence there is an increased focus on Multi-Valued Logic, because of its inherent ability to reduce the number of interconnections. In this paper, we present the design of (n=15, k=13) Reed Solomon code algorithm for advanced Redundant Array of Independent Disks (RAID) system in quaternary domain, to tolerate multiple disk failures. We present two designs to implement the algorithm on a FPGA- a heterogeneous design consisting of quaternary and binary circuits, a complete quaternary design. The first design performs all computations using only binary EX-OR gates. This design requires more EX-OR operations than the binary counterpart. This number is a function of the generator polynomial used for encoding. The second design based on quaternary look-up tables is efficient and can be easily implemented. The look-up tables are in turn based on quaternary multiplexers that can be further optimized by reducing the feature size.

Analysis of NGS (Next Generation Sequencing) data is a computationally demanding task requiring large amounts of CPU, memory, and disk space. There is also a requirement for high performance data storage systems, resilient to hardware failure, to be connected directly to the computing infrastructure (typically a multi-node cluster) to store large quantities of NGS data reliably. Traditional shared file systems such as NFS (Network File System) do not offer the performance, scalability or cache coherence required by modern NGS data analysis, so alternatives including GlusterFS, Ceph, and Lustre have been developed. However, there is a trade-off between data safety on replicated local storage and degradation of performance across distributed storage. Resilience to hardware failure is typically provided by RAID (Redundant Array of Independent Disks) and redundant storage nodes. Here we describe the evaluation of an alternative file system, RozoFS ( https://github.com/rozofs/rozofs ) for use with demanding NGS data analysis workloads. We used a synthetic data set (DREAM-TCGA data set 3) to run a complete tumor-normal analysis pipeline (“bcbio”, https://github.com/chapmanb/bcbio-nextgen ), including base quality recalibration, local indel realignment, somatic variant calling, and structural variants as a benchmark to compare RozoFS with a traditional shared file system (NFS) on two different HPC (High Performance Computing; Cloud4CaRE project) clusters. Our results show high reliability and good performance of RozoFS compared to NFS, in particular, during heavy I/O workloads. These findings indicate that the reliability and robustness of RozoFS's make it a good candidate for demanding NGS analysis workloads.

Providing data availability in a high performance computing environment is very important, especially in this data-intensive world. Most clusters either equip with RAID (Redundant Array of Independent Disks) devices or use redundant nodes to protect data from loss. However, neither of these can really solve the reliability problem incurred in a striped file system. Striping provides an efficient way to increase I/O throughput both in the distributed and parallel paradigms. But it also reduces the overall reliability of a disk system by N fold, where N is the number of independent disks in the system. Parallel Virtual File System (PVFS) is an open source parallel file system which has been widely used in the Linux environment. Its striping structure is good for performance but provides no fault tolerance. We implement Reliable Parallel File System (RPFS) based on PVFS but with reliability support. Our quantitative analysis shows that MTTF (Mean Time To Failure) of our RPFS is better than that of PVFS. Besides, we propose a parity cache table (PCT) to alleviate the penalty of parity updating. The evaluation of our RPFS shows that its read performance is almost the same as that of PVFS (2% to 13% degradation). As to the write performance, 28% to 45% improvement can be achieved depending on the behavior of the operations.