The adaptive research of data layout in large-scale meteorological data storage system

In this paper, we consider two rack-aware storage systems. First, large-scale rack-aware storage system, which is very common in large-scale storage system, is a rack-aware storage system where all sizes of racks are at least the number of redundant nodes. For such storage system, we prove that any Maximum Distance Separable (MDS) codes can have optimal inter-rack repair bandwidth and give a closed-form representation of all repair schemes with optimal inter-rack repair bandwidth. Furthermore, we show that the optimal repair access and optimal inter-rack repair bandwidth can be attained simultaneously for such storage system. Second, we investigate the rack-aware storage system of all racks with the same size, which is called uniform rack-aware storage system. We prove that, except the trivial cases, we cannot attain optimal inter-rack repair bandwidth and optimal repair access for such storage system at the same time. Specifically, we establish the lower bound of repair access for a repair scheme with optimal interrack repair bandwidth, which is tight for some parameters, and also the tight lower bound of inter-rack repair bandwidth for a repair scheme with optimal repair access.

Large-scale storage systems are crucial components in data-intensive applications such as search engine clusters, video-on-demand servers, sensor networks and grid computing. A storage server typically consists of a set of storage devices. In such systems, data layouts may need to be reconfigured over time for load balancing or in the event of system failure/upgrades. It is critical to migrate data to their target locations as quickly as possible to obtain the best performance. Most of the previous results on data migration assume that each storage node can perform only one data transfer at a time. A storage node, however, can typically handle multiple transfers simultaneously and this can reduce the total migration time significantly. Moreover, storage devices tend to have heterogeneous capabilities as devices may be added over time due to storage demand increase. In this paper, we consider the heterogeneous data migration problem, where we assume that each storage node v has different transfer constraint cv, which represents how many simultaneous transfers v can handle. We develop algorithms to minimize the data migration time. We show that it is possible to find an optimal migration schedule when all cvs are even. Furthermore, though the problem is NP-hard in general, we give an efficient algorithm that offers a rigorous (1 + o(1))-approximation guarantee.

Hydrogen produced by electrolysis is one of the promising means by which to store electricity coming from renewables and to address its intermittency. With the growing global electricity share from renewable energy sources, large-scale storage systems are required. Alkaline electrolysis can be a sustainable solution for the large-scale energy storage system. However, high overpotential and slow kinetics particularly of the oxygen evolution reaction (OER) still remains as a hurdle to limit efficiency of the system. To enhance efficiency of alkaline electrolysis cells, catalysts which have high electrocatalytic activity for OER should be developed. Platinum-like behavior of Tungsten carbide in surface catalysis was first reported by Levy and Boudart in 1973[1]. In addition, tungsten carbide was reported to have Pt-like electronic structure by Colton in 1975[2]. After that, tungsten carbide has been studied widely and expected to substitute noble metal catalysts which are used in many electrochemical reactions. Here, we studied tungsten carbide as a support for 3d-transition metals and make bi-metallic surfaces for developing a high activity OER catalyst. First, WC-Co was synthesized by heat treatment. Carbon nitride (g-C3N4) and tungsten chloride were used as a carbide precursor and cobalt acetate was also used for helping carbonization of tungsten. After heat treatment, acidic leaching process was followed to eliminate cobalt. Through additional mixing with Nickel precursor and heat treatment, Ni/WC was synthesized and to compare electrocatalytic activity Ni/C was also prepared. In addition, many different Ni/WC which have different Ni to WC ratios were synthesized and measured in electrochemical way. Finally, the optimal ratio was determined and it show better electrocatalytic activity than precious catalysts and any other non-noble catalysts. Cyclic voltammetry was measured to evaluate OER activity in 1 M KOH in potential range from 1 to 1.8 V. Figure 1 shows that pure WC and Ni/C have 394 mV and 413 mV overpotentials at 10 mA∙cm-2 respectively. In Metal/WC cases, Cobalt/WC shows only 417 mV. But Nickel/WC shows 340 mV overpotential at 10 mA∙cm-2 and higher electrocatalytic activity for OER than RuO2 in alkaline media. η @ 10 mA cm-2 (mV) Ni/C 394 WC 413 Co/WC 417 Ni/WC 340 RuO2 355 Table1. Activities for OER at 10 mA cm-2 References [1] R. B. Levy, M. Boudart, Science, New series, Vol. 181, No. 4099 (1973) 547-579 [2] Richard J. Colton, Jan-Tsyu j. Huang, J.Wayne Rabalais, Chemical Physics Letters, Vol. 34, No. 2 (1975) 337-339 Figure1. Cyclic voltammetry for the oxygen evolution reaction in 1 M KOH of Ni/C, WC, Co/WC, Ni/WC, RuO2 Figure 1

The need for reliable and fast storage systems is increasingly critical in various fields including artificial intelligence and data analytics. This paper proposes a new architecture for large-scale data storage systems, focusing on comparing performance of software and hardware storage technologies that effectively reduce the computational latency and improve performance. The main contributions include: (1) the combination of Singled Magnetic Recording (SMR) for storing data and Solid State Devices (SSD) for storing metadata is a viable solution for implementing large data storage systems; and (2) the combination of Conventional Magnetic Recording (CMR) for storing data and SSD for storing metadata shows the highest performance for high performance computing. Our experiments are carried out in multiple settings, demonstrating that the proposed architecture successfully improves performance for sequential and random reads/writes. The prototypes are evaluated with a set of workloads, showing the superiority of the proposed data storage configurations. This work provides new opportunities for efficiently processing and storing data and metadata in large-scale data analysis systems.

Distributed storage systems provide large-scale data storage services, yet they are confronted with frequent node failures. To ensure data availability, a storage system often introduces data redundancy via replication or erasure coding. As erasure coding incurs significantly less redundancy overhead than replication under the same fault tolerance, it has been increasingly adopted in large-scale storage systems. In erasure-coded storage systems, degraded reads to temporarily unavailable data are very common, and hence boosting the performance of degraded reads becomes important. One challenge is that storage nodes tend to be heterogeneous with different storage capacities and I/O bandwidths. To this end, we propose FastDR, a system that addresses node heterogeneity and exploits I/O parallelism, so as to boost the performance of degraded reads to temporarily unavailable data. FastDR incorporates a greedy algorithm that seeks to reduce the data transfer cost of reading surviving data for degraded reads, while allowing the search of the efficient degraded read solution to be completed in a timely manner. We implement a FastDR prototype, and conduct extensive evaluation through simulation studies as well as testbed experiments on a Hadoop cluster with 10 storage nodes. We demonstrate that our FastDR achieves efficient degraded reads compared to existing approaches.

An effective decentralized metadata management schema is critical to a reliable and scalable large-scale distributed storage system.With regard to the defects of huge expansion cost and high sensitivity to cluster alteration borne in both the hash-based and subtree-based metadata management schemas, a metadata server (MDS) clustering schema based on consistent hash structure, named DSVL, is proposed.By introducing virtual MDS into the consistent MDS cluster, it can effectively balance MDS cluster load.Besides, the standby mechanism and the lazy update policy are merged and applied to the MDS cluster to accomplish rapid MDS failover as well as zero data migration in case of cluster alteration.The results of prototype system and simulation test prove that DSVL, with the features of balanced metadata distribution, rapid failover, flexible scalability and zero data migration amount in case of node alteration, can meet the demands for flexible and effective management of large-scale data storage systems with ever increasing data amount.

The ever-growing creation of data requires large-scale network storage systems. One of the key issues related such systems is how to place several petabytes of data among large number of devices. Itpsilas necessary to design a reliable, fair, adaptable data placement algorithm. All proposed approaches are oblivious to the reliability-based requirements of data in such systems. In this paper, we present a reliability-aware data placement algorithm for large-scale network storage systems. With considering the reliability-aware differences of storage devices and the reliability-based requirement of data, we formulate the problem as an integer programming to minimize the reliability cost and propose a polynomial-time algorithm to solve the problem. For each reliability level, we use a fair and adaptive data placement to distribute data. It can support arbitrary heterogeneous storage systems, distribute data in a fair way and allow an efficient adaptation to a changing set of devices. The theoretical analysis as well as the experimental study show that the approach can meet the reliability-based requirements of data, distribute data evenly among devices, and adapt well to the changes of devices.

Large-scale storage systems may service millions of clients. Effectively identifying and authenticating these clients can not only prevent illegal accesses but also eliminate unnecessary redundancies of access control. However, existing security solutions have largely ignored the identification and authentication issues or separately consider them from access control mechanisms. As a result, these solutions either have incurred a high cost of access control or led to new insurmountable problems. In this paper, we discuss the important consideration in designing an appropriate authentication scheme for large-scale storage systems. First, we discuss potential authentication techniques for large-scale storage systems. The looming crisis of PKI, a widely used technology for authentication in today's information security area is discussed and potential alternative technologies to PKI are suggested. Second, by merging authentication into access control we develop a decentralized security architecture for large-scale storage systems, designed to improve the scalability of authentication and reduce the overhead of access control. Finally, we experimentally evaluate how much overhead can be caused when various cryptographic algorithms used in an authentication protocol are applied to a large-scale storage system.

In this paper, we propose Elastic-RAID, a new RAID architecture to achieve high performance and high reliability for large-scale distributed and parallel storage systems. The key idea behind Elastic-RAID is to smartly utilize the free space existing in parity-based disk arrays to store additional mirroring data. This additional mirroring data redundancy, when strategically and judiciously activated and exploited in a RAID system, enables improved system I/O performance, fault tolerance and recovery. Depending on the amount of free space available and whether the emphasis is on performance or reliability, the elasticity in Elastic-RAID is manifested in how each design objective is achieved. For the performance objective, Elastic-RAID improves small-write performance by writing original and mirroring data synchronously and leaving the costly parity update in the background at a later idle/lightly-loaded time. For the reliability objective, at least two concurrent disk failures can be tolerated when Elastic-RAID is employed in a RAID5 system that has 50 percent or more free space. Higher reliability is provided for important data when free space is less than 50 percent. To achieve the design goal of elasticity, we introduce a novel data layout and addressing scheme. Our extensive trace-driven evaluations on an Elastic-RAID prototype in the typical configurations of RAID5 show that Elastic-RAID boosts the small-write performance in the normal operational state by at least 40 percent, improves the user I/O performance in the reconstruction state by at least 30 percent and shortens the recovery time by at least 40 percent.

Secure administration of cryptographic role-based access control for large-scale cloud storage systems

The network technologies and storage devices are developing quickly and the technical devices are inexpensive. Therefore, users have a large amount of storage space, Data Grid can collect distributed storage device to share to other users. In this paper, we discuss several file systems that consist of NFS, SRB, iRODS, Lustre, Gfarm and GPFS-WAN. In Data Grid, we select the specific file system to administrate the storage devices. We must consider the file system that obtains scalibilty, security and stability. Finally, we choose two file systems that consist of Gfarm and GPFS-WAN to be our evaluative target. Gfarm and GPFS-WAN are used to build up large-scale dataset storage system that consists of PRAGMA and TeraGrid, respectively. In this paper, we use Gfarm and GPFS-WAN to create a Data Grid environment and evaluate the performance of data transmission, respectively. As results, the GPFS-WAN's performance of data transmission is better than Gfarm. The data transmission is one of the factors in Data Grid environment. For instance, the system obtains replication, open source software, easy to install and maintain that is important to be a great Data Grid environment, these important information will be discussed in this paper.

As various types of data grow explosively, large-scale data storage, backup, and transmission become challenging, which motivates many researchers to propose efficient universal compression algorithms for multi-source data. In recent years, due to the emergence of hardware acceleration devices such as GPUs, TPUs, DPUs, and FPGAs, the performance bottleneck of neural networks (NN) has been overcome, making NN-based compression algorithms increasingly practical and popular. However, the research survey for the NN-based universal lossless compressors has not been conducted yet, and there is also a lack of unified evaluation metrics. To address the above problems, in this paper, we present a holistic survey as well as benchmark evaluations. Specifically, i) we thoroughly investigate NN-based lossless universal compression algorithms toward multi-source data and classify them into 3 types: static pre-training, adaptive, and semi-adaptive. ii) We unify 19 evaluation metrics to comprehensively assess the compression effect, resource consumption, and model performance of compressors. iii) We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text, images, videos, audio, etc. iv) We also summarize the strengths and drawbacks of NN-based lossless data compressors and discuss promising research directions. We summarize the results as the NN-based Lossless Compressors Benchmark (NNLCB, See fahaihi.github.io/NNLCB website), which will be updated and maintained continuously in the future.

Disk failure is one of the most important reliability problems in large-scale network storage systems. Disk failure may lead to serious data loss and even disastrous consequences if the missing data cannot be recovered. Therefore, predicting disk failures is an important means of ensuring storage security in network storage systems. However, since the fault data in the fast degradation stage is smaller than the healthy data in the normal state, the mixture of healthy data and faulty data leads to extremely unbalanced data, which brings great challenges to finding hidden fault information, thus making fault prediction more accurate. Aiming at the above problems, a convolutional transformer model ConvTrans-TPS model for disk failure prediction in large-scale network storage systems is proposed. The ConvTrans-TPS model acquires dependencies between long-term sequence data through transformers and uses convolutional projections for attention computation to enhance attention to local contextual information. Data augmentation to predict failures in the next 7 days. Validated by the analysis on the Backblaze dataset, the F1 is 0.96 and the Matthews correlation coefficient (MCC) is 0.92. Compared with the popular CNN-LSTM model in recent years, our proposed method improves F1 and MCC by 4% and 5%, respectively, improving the prediction accuracy.

The failure of a single unreliable commodity component is very common in large-scale distributed storage systems. In order to ensure the reliability of data in large-scale distributed storage systems, many studies have emerged one after another. Among them, Erasure Codes are widely used in actual storage systems, such as Hadoop Distributed File System (HDFS), which provides high fault-tolerance with lower storage overhead. However, usually the recovery of erasure-coded storage system when encountering node failure results in severe cross-node and cross-rack bandwidth consumption, which affects the efficiency of failure recovery and wastes additional resources. In this paper, we improve the erasure coding storage strategy in distributed storage systems, and propose a low-overhead data recovery method based on cross-checking, namely HV-SNSP. In HV-SNSP, horizontal and vertical cross parity checking is realized by adding RS parity inside the data node, that is, <inline-formula> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, k)}}$ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', \textrm {}k')}}$ </tex-math></inline-formula> storage architecture. Based on <inline-formula> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, \textrm {}k)}}$ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', k')}}$ </tex-math></inline-formula>, a low-cost supply node selection strategy, namely SNSP, is designed, and nodes with shorter network distance and lower load are selected to participate in recovery. This strategy can effectively reduce the amount of data transmission, shorten the recovery time, and improve the recovery efficiency. The experimental results show that compared with traditional RS, HV-SNSP can reduce the amount of cross-rack data transmission by 62.5&#x0025; during data recovery, and can shorten the recovery time by up to 42.41&#x0025;; Compared with D3, HV-SNSP can reduce the occupation of cross-rack bandwidth by 25&#x0025; and shorten the recovery time by 36.58&#x0025;.

The paper describes the formulation and application of an Optimal Power Flow (OPF)-driven Under Frequency Load Shedding (UFLS) scheme in low-inertia power grids hosting large-scale battery energy storage systems. Thanks to the accurate prediction of the system dynamics subsequent to a large contingency, the adopted OPF-UFLS is capable of minimizing the amount of load to be shed while ensuring a safe trajectory of the system frequency and preventing nodal voltages and branch currents to violate their feasible limits. The performance of the method is assessed by using numerical simulations of a fully-modeled IEEE 39-bus low-inertia power grid with two configurations. Furthermore, the obtained results are compared with those of the UFLS strategy recommended by European Network of Transmission System Operators (ENTSO-E).