The Design and Implementation of an Intelligent Agent-Based File System

Reliable distributed data storage systems have to employ redundancy codes to tolerate the loss of storages. Many appropriate codes and algorithms can be found in the literature, but efficient schemes for tolerating several storage failures and their embedding in a distributed system are still research issues. In this paper, a variety of redundancy schemes are compared that got implemented in a distributed storage system. All schemes are based on parity and Reed/Solomon and are integrated in the storage system NetRAID. This system allows to configure several user-specified layouts. A performance and reliability analysis of several data and redundancy layouts is presented that combines analytical and experimental results. In a detail, we present performance results for an optimized Reed/Solomon implementation and give an outline for speeding up encoding and recovery by reconfigurable hardware employed in the distributed storage system

The exponentially growing demand for storage puts a huge stress on traditional distributed storage systems. Historically, I/Ops (Inputs/Outputs per second) of hard drives have been the main limitation of storage systems. With the rapid deployment of solid state drives (SSDs) and the expected evolutions of their capacities, price and performance, we claim that CPU and network capacities will become bottlenecks in the future. In this context, we introduce 6Stor, an innovative, software-defined distributed storage system fully integrated with the networking layer. This storage system departs from traditional approaches in two manners: it leverages IPv6 new capabilities to increase the efficiency of its data plane—notably by using directly UDP and TCP rather than HTTP—and thus its performance; and it circumvents scalability limitations of other distributed systems by using a fully distributed metadata layer of indirection to offer flexibility. In this paper, we introduce and describe in details the architecture of 6Stor, with an emphasis on dynamic scalability and robustness to failure. We also present a testbed that we use to evaluate our novel approach by using Ceph—another well known distributed storage system—as baseline. Results obtained on an extensive testbed are presented and some initial conclusions are drawn.

Pangaea is a wide-area file system that supports data sharing among a community of widely distributed users. It is built on a symmetrically decentralized infrastructure that consists of commodity computers provided by the end users. Computers act autonomously to serve data to their local users. When possible, they exchange data with nearby peers to improve the system's overall performance, availability, and network economy. This approach is realized by aggressively creating a replica of a file whenever and wherever it is accessed.This paper presents the design, implementation, and evaluation of the Pangaea file system. Pangaea offers efficient, randomized algorithms to manage highly dynamic and potentially large groups of file replicas. It applies optimistic consistency semantics to replica contents, but it also offers stronger guarantees when required by the users. The evaluation demonstrates that Pangaea outperforms existing distributed file systems in large heterogeneous environments, typical of the Internet and of large corporate intranets.

Intelligent mobile agents are sent between routers to carry out a set of required tasks. Mobile agents can carry algorithms and information to various routers they visit. Critical information that all mobile agents should carry includes: timer, navigation algorithm, network information, stability of the network, and load traffic of the network. By clustering intelligent mobile agents, It is possible to minimize the cost of time for each agent to travel to its required routers and the number of intelligent mobile agents. Each router has certain capability and hence an only process certain requests of each intelligent agent. The problem is to group the intelligent mobile agent into a number of clusters such 1h.t that each cluster has similarity in responding to a group of routers. This can significantly decrease the demand imposed upon a network for a set of required task be he performed. In this paper, we present the problem of multiple criteria clustering of intelligent mobile agents where the decision is to cluster mobile agents such that a group of similar intelligent mobile agents will visit a group of similar routers. This grouping will decrease the total number of visits and the total number of agents, decreasing demand on the network.

Despite continual improvements in the performance and reliability of large scale file systems, the management of user-defined file system metadata has changed little in the past decade. The mismatch between the size and complexity of large scale data stores and their ability to organize and query their metadata has led to a de facto standard in which raw data is stored in traditional file systems, while related, application-specific metadata is stored in relational databases. This separation of data and semantic metadata requires considerable effort to maintain consistency and can result in complex, slow, and inflexible system operation. To address these problems, we have developed the Quasar File System (QFS), a metadata-rich file system in which files, user-defined attributes, and file relationships are all first class objects. In contrast to hierarchical file systems and relational databases, QFS defines a graph data model composed of files and their relationships. QFS incorporates Quasar, an XPATH-extended query language for searching the file system. Results from our QFS prototype show the effectiveness of this approach. Compared to the de facto standard, the QFS prototype shows superior ingest performance and comparable query performance on user metadata-intensive operations and superior performance on normal file metadata operations.

This article has no abstract.

Modern file systems associate the deletion of a file with the immediate release of storage, and file writes with the irrevocable change of file contents. We argue that this behavior is a relic of the past, when disk storage was a scarce resource. Today, large cheap disks make it possible for the file system to protect valuable data from accidental delete or overwrite.This paper describes the design, implementation, and performance of the Elephant file system, which automatically retains all important versions of user files. Users name previous file versions by combining a traditional pathname with a time when the desired version of a file or directory existed. Storage in Elephant is managed by the system using file-grain user-specified retention policies. This approach contrasts with checkpointing file systems such as Plan-9, AFS, and WAFL that periodically generate efficient checkpoints of entire file systems and thus restrict retention to be guided by a single policy for all files within that file system.Elephant is implemented as a new Virtual File System in the FreeBSD kernel.

The digital era has witnessed an unprecedented surge in data generation, driven by diverse sources such as social media platforms, e-commerce, Internet of Things (IoT) devices, mobile applications, and enterprise systems. This explosive growth has led to the emergence of big data, characterized by the five Vs: volume, velocity, variety, veracity, and value. Managing such massive and complex datasets necessitates the use of scalable storage systems that can efficiently store, control, and retrieve data. Traditional storage infrastructures—such as centralized databases and file systems—struggle to meet the demands of big data applications due to limitations in scalability, fault tolerance, data redundancy, and performance bottlenecks. There are lot of problems with old system. In response, modern storage architectures have evolved to embrace distributed and cloud-based storage systems that offer horizontal scalability and high availability. The exponential increase in data generation across industries has intensified the demand for storage architectures that are both scalable and efficient. Conventional systems often face challenges in sustaining high ingestion rates, minimizing retrieval latency, and adapting to the dynamic requirements of large-scale data analytics. This research focuses on optimizing scalable storage architectures to enhance the performance of big data processing frameworks. It examines distributed file systems, object-based storage, and cloud-native approaches with particular attention to data distribution, replication mechanisms, metadata handling, and resource utilization. The study also analyzes the balance between scalability, fault tolerance, and consistency, while considering integration with platforms such as Hadoop and Spark. Through systematic benchmarking and performance evaluation, the proposed models aim to improve throughput, reliability, and cost-effectiveness. The outcomes of this research are intended to guide the development of next-generation storage solutions capable of supporting the continuous growth of big data ecosystems.

Distributed file systems (DFSs) play important roles in datacenters. Recent advances in persistent memory (PM) and remote direct memory access (RDMA) technologies provide opportunities in enhancing distributed file systems. However, state-of-the-art distributed PM file systems (DPMFSs) still suffer from a duplication problem and a fixed transmission problem, leading to high network latency and low transmission throughput. To tackle these two problems, we propose Zebra, an efficient RDMA-enabled distributed PM file system—Zebra uses a replication group design for alleviating the heavy replication overhead, and leverages a novel transmission protocol for adaptively transmitting file replications among nodes, eliminating the fixed transmission problem. We implement Zebra and evaluate its performance against state-of-the-art distributed file systems on an Intel Optane DC PM platform. The evaluation results show that Zebra outperforms CephFS, GlusterFS, and NFS by 4.38\(\times \), 5.61\(\times \), and 2.71\(\times \) on average in throughput, respectively. KeywordsRDMAAdaptive transmissionPersistent memoryDistributed file system

For a decade, the Ceph distributed file system followed the conventional wisdom of building its storage backend on top of local file systems. This is a preferred choice for most distributed file systems today, because it allows them to benefit from the convenience and maturity of battle-tested code. Ceph’s experience, however, shows that this comes at a high price. First, developing a zero-overhead transaction mechanism is challenging. Second, metadata performance at the local level can significantly affect performance at the distributed level. Third, supporting emerging storage hardware is painstakingly slow. Ceph addressed these issues with BlueStore, a new backend designed to run directly on raw storage devices. In only two years since its inception, BlueStore outperformed previous established backends and is adopted by 70% of users in production. By running in user space and fully controlling the I/O stack, it has enabled space-efficient metadata and data checksums, fast overwrites of erasure-coded data, inline compression, decreased performance variability, and avoided a series of performance pitfalls of local file systems. Finally, it makes the adoption of backward-incompatible storage hardware possible, an important trait in a changing storage landscape that is learning to embrace hardware diversity.

Careful simulation‐based evaluation plays an important role in the design of file and disk systems. We describe here a particular approach to such evaluations that combines techniques in workload synthesis, file system modeling, and detailed disk behavior modeling. Together, these make feasible the detailed simulation of I/O hardware and file system software. In particular, using the techniques described here is likely to make comparative file system studies more accurate.In addition to these specific contributions, the paper makes two broader points. First, it argues that detailed models are appropriate and necessary in many cases. Second, it demonstrates that detailed models need not be difficult or time consuming to construct or execute.

With the vigorous rise of the Internet and the Internet of Things, as well as the rapid development of computer technology and artificial intelligence technology, the storage of massive data, has gradually become an urgent problem in the development of these technologies [1], and most of these data are various small files. Take Facebook as an example, it has stored more than 65 billion pictures, each picture has 4 different sizes, and the peak period of picture visits is 1 million per second [2]. In the process of image-related deep learning model training, a large number of labeled pictures are required, and the pictures are mostly small files of several hundred KB. Taking ResNet as an example, it only accepts picture input with a pixel of 224⁎224 [3]. In order to cope with the growth of massive small files, traditional stand-alone storage simply cannot meet the demand. The paper designs and implements a micro-service style high-availability distributed storage system, which has the following 4 advantages: 1) The easy-to-use service interface provides external file upload, download, and delete APIs. 2) Support the storage of small files in different data formats, and adopt a unified data description method for each format. 3) Data redundancy backup to make file storage persistently available. 4) Multi-point distributed storage, supporting the elastic expansion of storage capacity.

As one of the workload characteristics, the anomaly behaviorsin real workload have been recognized as a critical requirement for file system design. In this paper, a set of traces collected from typically realistic file system have been analyzed. Studying the correlations of I/O request inter-arrival times shows that it is necessary to examine the self-similarity in file system workload. And then the phenomenon of self-similarity which had been initially observed in network and disk I/O workload has also been visually and statistically observed in file system workload used in this paper. Furthermore, this property is further explored and we find that the self-similarity of I/O accesses in those sets of traces studied in this paper is mainly due to reads. This study is extremely important to provide useful insight into the design and analysis of file system for performance gains, as well as to model file system I/O workloads.

S.492-501

Distributed file systems have become popular in recent years. However, they still lack functions for doing fast arbitrary data insertion and truncation. To solve the problem, we present Wofs, an object-based distributed network file system which supports fast arbitrary data insertion and truncation. Wofs splits a file into many small objects, stores these objects in remote file servers, and uses a special B+tree to manage the metadata of these objects. Besides, Wofs uses the object-range locking policy to avoid data incoherence and improve performance.