A Prediction-Based Parallel Replication Algorithm in Distributed Storage System

Reliable distributed data storage systems have to employ redundancy codes to tolerate the loss of storages. Many appropriate codes and algorithms can be found in the literature, but efficient schemes for tolerating several storage failures and their embedding in a distributed system are still research issues. In this paper, a variety of redundancy schemes are compared that got implemented in a distributed storage system. All schemes are based on parity and Reed/Solomon and are integrated in the storage system NetRAID. This system allows to configure several user-specified layouts. A performance and reliability analysis of several data and redundancy layouts is presented that combines analytical and experimental results. In a detail, we present performance results for an optimized Reed/Solomon implementation and give an outline for speeding up encoding and recovery by reconfigurable hardware employed in the distributed storage system

The exponentially growing demand for storage puts a huge stress on traditional distributed storage systems. Historically, I/Ops (Inputs/Outputs per second) of hard drives have been the main limitation of storage systems. With the rapid deployment of solid state drives (SSDs) and the expected evolutions of their capacities, price and performance, we claim that CPU and network capacities will become bottlenecks in the future. In this context, we introduce 6Stor, an innovative, software-defined distributed storage system fully integrated with the networking layer. This storage system departs from traditional approaches in two manners: it leverages IPv6 new capabilities to increase the efficiency of its data plane—notably by using directly UDP and TCP rather than HTTP—and thus its performance; and it circumvents scalability limitations of other distributed systems by using a fully distributed metadata layer of indirection to offer flexibility. In this paper, we introduce and describe in details the architecture of 6Stor, with an emphasis on dynamic scalability and robustness to failure. We also present a testbed that we use to evaluate our novel approach by using Ceph—another well known distributed storage system—as baseline. Results obtained on an extensive testbed are presented and some initial conclusions are drawn.

Data replication is a common method to reduce the bandwidth consumption and the access latency in distributed systems. The existing replication methods mostly deal with the read-only data, thus the replica management mechanism is relatively simple. Most replication methods replicate files in all nodes or two endpoints on a query path. Naturally, the larger number of replicas, the better performance it can obtain. However, these methods lead to high overhead for unnecessary file replications and consistency maintenance. This paper presents a dynamic minimum access cost based replication strategy (MAC replication strategy). MAC replication strategy takes into account the access frequency, the status of the network connection and average response time to perform optimal replication. It first selects the popular files. To each popular file, the average response time is calculated to determine which logical resource should be replicated. Then MAC replication strategy calculates an appropriate site to replicate for better shortening the response time of the data resource. Simulation results demonstrate the efficiency and effectiveness of MAC replication strategy in comparison with other approaches.

The data grid is one of the infrastructures for data and storage resource management. Data replication can improve data access performance, fault tolerance, and reduce network transmission bandwidth. As the site storage space is limited in the grid, how to effectively use grid resources become an important challenge. This paper introduces a dynamic grid replication algorithm based on popularity support and confidence (BPSC). Through the algorithm, data and its associated copy can be placed on a suitable site, with reduced access latency. Through Optorsim, simulation results show that the algorithm can provide better performance compared with other algorithms Grid computing is an important branch of distributed computing. Grid computing constructs a virtual computer system through using a different location of the large number of heterogeneous resources (CPU and storage), provides the model to solve the problem of large scale computing and storage. The data grid is mainly the grid for data storage management. The data grid is defined as an effective combination of data and computing resources of distributed systems by Data Grid Group. The large-scale data- intensive applications, such as bioinformatics, high energy physics, space information generated vast amounts of data, these data may be TB scale, even up to the scale of PB. It is not practical to maintain large scale data at each site, and provide computing resources for visitors. At the same time, it will lead to increased access latency through a centralized way to storing and accessing these data from each site. Data replication technology is an effective method to reduce access latency in the data grid system. Data replication is one of the basic methods of distributed systems to improve the efficiency of system access, maintenance, system high availability. Data replication is to store data in the appropriate site for users to effective access to massive data. Data replication can reduce data access latency, reduce network bandwidth consumption, avoid congestion, adjust the load balancing in server-side, improve system reliability. Many data replication algorithm can reduce data access latency, improve system reliability, but which data will be copied, when the data will be copied at, where the data will be copied are the basic means of determining data replication algorithm. In this paper, we introduced a real-time dynamic data replication algorithm. The algorithm is based on the popularity of file dynamic to replicate data, and consider the storage space limitations of the site. Through optorsim simulation verified the algorithm. This paper is organized as follows: Section 2 is about related works on replication in data grids. In Section 3, system model and component in system are presented. The file

This article has no abstract.

In data grids, data replication on variant nodes can change some problems such as response time and availability. Also, in data replication, there are some challenges to finding the best replica efficiently in relation to performance and location of physical storage systems. In this paper, various replica placement strategies are discussed. These replica placement strategies are available in the works. Replica placement contains recognizing the best possible node to duplicate data based on network latency and user request. These strategies measure and analyze different parameters such as access cost, bandwidth consumption, scalability, execution time and storage consumption. This paper also analyses the performance of various strategies with respect to the parameters mentioned above in data grid.

With the development of smart devices and social media, massive unstructured data is uploaded to distributed storage systems. Since the characteristics of multi-users and high concurrency the unstructured data accesses have, it brings new challenges to traditional distributed storage systems designed for large files. We propose a grouping strategy to analyze relevant data in access according to disk access logs in the real distributed storage systems environment. When any data in the group is accessed, the whole group is prefetched from disk to the cache. Firstly, we conduct statistical analysis on the access logs and propose a preliminary classification method to classify files in spatiotemporal locality. Secondly, a strength-priority tree structure relation model (SP-TSRM) is proposed to mine file group efficiently. Finally, experiments show that the proposed model can improve the cache hit rate significantly, thereby improving the read efficiency of distributed storage systems.

Erasure codes are recently used in real-world distributed storage systems (DSSs) such as Google File System,Microsoft Azure Storage, and Facebook HDFS-RAID for data reliability. When designing erasure codes for DSSs, special attention is given to the associated costs of data handling operations such as repair or update. For example, locally repairable codes (LRC) are designed and used in DSSs to allow for low-cost repair of failed blocks. Update complexity (defined as the number of blocks that need to be updated when an information block is changed) is yet another design parameter. This parameter can be seen as a measure of the computation, I/O and networking costs associated with updating an information block in a DSS. Since information is frequently updated by many applications, lowering update complexity can result in lower power consumptions in DSSs. In this work, we study the update complexity of LRCs. Based on our study, we propose an improvement over the LRC used by Facebook HDFS-RAID. Keeping the same code parameters including length, storage overhead, minimum distance and cost of repair (locality), we improve the update complexity by more than 16%. Moreover, we show that with these parameters achieving a lower update complexity is impossible.

i»? A. A. Helal, A. A. Heddaya, and B. B. Bhargrava Kluwer Academic Publishers, Boston, 1996, 176 pp. ISBN 0-7923-9800-9, $118.50 Computing in the 1990s has reached the state of distributed computing. A basis of this form of computing is a distributed computing system, which is built on the following three components: (a) personal computers, (b) local and fast wide area networks, and (c) system and application software. By amalgamating computers and networks into one single computing system and providing an appropriate system software, a distributed computing system has created the possibility of sharing information and peripheral resources. Furthermore, these systems improved performance of a computing system and individual users through parallel execution of programs, load balancing and sharing, and replication of programs and data. Distributed computing systems are also characterized by enhanced availability, and increased reliability. Replication is the key to providing high availability, fault tolerance, and enhanced performance in a distributed computing system. As companies move toward systems that are more open and distributed, replication is becoming increasingly important in the ability to provide data and services that are current, correct and available, which is a key factor in maintaining a competitive advantage over rivals. However, replication has also generated some serious challenges. For example, when a replica is updated, how do we propagate such an update to other replicas? If multiple replicas are updated simultaneously and some of the updates conflict one another, how do we resolve the conflict? How do we deal with the situation where a replica is down and then subsequently recovered? How do we deal with network partitions? Not surprisingly, considerable research efforts have been directed towards the solution of these challenges. However, most of the research results are scattered among many journals, conference proceedings, theses, and technical reports. The book by Helal, Heddaya and Bhargava has gathered the best material of these research results and formed a coherent collection that includes definitions, theoretical background, algorithms, annotated bibliographies of commercial and experimental prototype systems, and more than 200 references. This book classifies the entities that can be replicated on a distributed computing system into the following categories: (a) data, (b) process, (c) objects, and (d) messages. After the introduction on the goals and main approaches of replication techniques, the book uses individual chapters to describe major methods for each of the above replication categories. The main focus of this part, however, is on the replication of data. The book also contains two chapters on the replication issues in heterogeneous, mobile, and large-scale systems, and on the future of replication techniques. Apart from the normal chapters, the book contains a rich set of appendices that are very useful for further studies of replication techniques. These appendices include brief descriptions of two dozen commercial and experimental systems that use replication techniques, annotated bibliographies of selected literature on various topics of replication, and an introduction on serializability theory. Each annotated bibliography contains an overview of the specific topic and is written by an invited expert in that field. The combination of the normal chapters and appendices cover the entire spectrum of replication, from definitions, concepts, theories, algorithms, techniques, to systems. The book serves as an excellent introduction and roadmap to the issues of replication. Although it does not provide a detailed description of every replication techniques and systems, the book covers most of the fundamental work that allows the reader to understand the roots of these techniques and systems, and to find the ways to search for the details. This book is a valuable reference book for anyone studying replication techniques or implementing replication systems, specifically, it can be very useful to post-graduate students, practitioners, and beginning researchers of this exciting field. Wanlei Zhou, School of Computing and Mathematics, Deakin University

A new approach to performing resource allocation in autonomous distributed computer systems is explored. As opposed to previous work which is based on interprocessor cooperation, the distributed system is modeled as a competitive society of microeconomic agents. The model is applied to the problem of managing distributed, replicated data objects. In this economy, jobs and transactions submitting read/write operations on data objects pay the processors to perform the operations. The processors use the revenue they earn to lease copies of data objects on which they can make a profit. The data objects themselves are active agents and set the prices for leasing copies. The economy is evaluated by a simulation study. The experiments show that the economy can substantially improve performance by varying the placement and number of copies of each data object. >

Data deduplication has become a crucial technique for reducing data in modern storage systems. We present SEP-D, a practical scale-out distributed storage system to incorporate data deduplication for primary storage. SEP-D introduces a novel metadata handling mechanism which combines contentbased hashing with built-in distributed data placement strategies such as CRUSH. This enables SEP-D to eliminate the needs for remote metadata lookups, thus incorporating deduplication without affecting scalability. SEP-D integrates smoothly with the existing storage system, allowing the re-use of storage policies across different pools of storage. We implemented SEP-D in Ceph, a popular distributed storage system widely adopted in the industry, and demonstrated that SEP-D has minimal impact on I/O performance in data while maintaining existing storage policies implemented in underlying distributed storage systems.

This research work focus on distributed system based network issues. The networked servers and computers organize their actions only by passing the proper messages. This networked system having especially significant characteristics like concurrency components and independent failures components. While we share the files through by passing message, there will be security problems because of connecting the entire computers in network. For High performance in data accessing, data surviving and also secure storage designing, the data partitioning and replication techniques can be combined. In this work, two heuristic algorithms developed for two sub problem in which the Authentication of the data share can be done using network manager to improve security. The replicated data are authenticated and send dynamically in the distributed system. The newly developed algorithms provide maximum performances in data sharing through distributed system and in achieving maximum security.

With high penetration of renewables-based distributed generation sources (RDGs) and distributed storage systems (DSSs), such as electric vehicles (EVs), in distribution systems, several issues such as maintaining energy-neutrality and computation/communication bottlenecks arise. These can be efficiently handled with coordination of DSSs that act as buffer and absorb/inject necessary active power, and by using distributed computing approaches. Therefore, in this paper, a holistic strategy is proposed to optimize the costs and revenues of owners in a community while maintaining net energy neutrality using a fully distributed consensus-based algorithm. Simulation studies performed on modified IEEE 30-bus test system show that the proposed algorithm effectively optimizes the community's devices under dynamic conditions.

The digital era has witnessed an unprecedented surge in data generation, driven by diverse sources such as social media platforms, e-commerce, Internet of Things (IoT) devices, mobile applications, and enterprise systems. This explosive growth has led to the emergence of big data, characterized by the five Vs: volume, velocity, variety, veracity, and value. Managing such massive and complex datasets necessitates the use of scalable storage systems that can efficiently store, control, and retrieve data. Traditional storage infrastructures—such as centralized databases and file systems—struggle to meet the demands of big data applications due to limitations in scalability, fault tolerance, data redundancy, and performance bottlenecks. There are lot of problems with old system. In response, modern storage architectures have evolved to embrace distributed and cloud-based storage systems that offer horizontal scalability and high availability. The exponential increase in data generation across industries has intensified the demand for storage architectures that are both scalable and efficient. Conventional systems often face challenges in sustaining high ingestion rates, minimizing retrieval latency, and adapting to the dynamic requirements of large-scale data analytics. This research focuses on optimizing scalable storage architectures to enhance the performance of big data processing frameworks. It examines distributed file systems, object-based storage, and cloud-native approaches with particular attention to data distribution, replication mechanisms, metadata handling, and resource utilization. The study also analyzes the balance between scalability, fault tolerance, and consistency, while considering integration with platforms such as Hadoop and Spark. Through systematic benchmarking and performance evaluation, the proposed models aim to improve throughput, reliability, and cost-effectiveness. The outcomes of this research are intended to guide the development of next-generation storage solutions capable of supporting the continuous growth of big data ecosystems.

Data security aims to prevent the use, modification, and spread of data by unauthorized people. In this study, our purpose was to provide data privacy and confidentiality with information flow control in distributed databases. In particular, a decentralized label model was developed that maintained confidentiality, including privacy, with data flow control. This model consists of an actor, an object, and a label. The owners of the objects are actors, and they need to share their data objects with others. Actors label the data objects and then send them out. A label contains the policy statements of data security issued by each of the owners. Each owner sets its own security and privacy policy independently of the other owners. The confidentiality of data in unsecured transport channels is ensured for all the actors in the system by means of labels while the data are in flow. Data objects are spread and shared securely among actors within unsecured environments. In addition, with the path compression, the long node chain that is formed while the data objects are passing between the source node and the destination is broken, so that the objects are retrieved fast, and the cost of access is reduced. This result was shown experimentally by modeling the distributed environment.