Achieving Load Balance for Parallel Data Access on Distributed File Systems

Distributed file systems (DFSs) play important roles in datacenters. Recent advances in persistent memory (PM) and remote direct memory access (RDMA) technologies provide opportunities in enhancing distributed file systems. However, state-of-the-art distributed PM file systems (DPMFSs) still suffer from a duplication problem and a fixed transmission problem, leading to high network latency and low transmission throughput. To tackle these two problems, we propose Zebra, an efficient RDMA-enabled distributed PM file system—Zebra uses a replication group design for alleviating the heavy replication overhead, and leverages a novel transmission protocol for adaptively transmitting file replications among nodes, eliminating the fixed transmission problem. We implement Zebra and evaluate its performance against state-of-the-art distributed file systems on an Intel Optane DC PM platform. The evaluation results show that Zebra outperforms CephFS, GlusterFS, and NFS by 4.38\(\times \), 5.61\(\times \), and 2.71\(\times \) on average in throughput, respectively. KeywordsRDMAAdaptive transmissionPersistent memoryDistributed file system

This research paper is a survey of the present state of design and implementation of distributed file systems. In a distributed file system, one or more central servers store files that can be accessed, with proper authorization rights. It consists of four parts: an Introduction of Distributed file system, implementation, architecture and component of distributed file system and their design goals for good distributed file system i.e. transparency – the main goal of good DFS. In this paper we also discuss some distributed file systems like NFS, CIFS, OwFS and HDFS. We explain features, architecture of above file system. This paper also include the criteria for choosing right Distributed file system of our need and recommendation foe right Distributed File System .The above file systems are also compared on the basis of File size, Analytic functionality, TCO and their selection criteria. This research also included the future aspect of distributed file systems like the problem we based in current DFS like transparency and at last the conclusion of the whole paper on the basis of above survey.

Handling of Big Data and cloud computing are the two important prime concerns which have become more and more popular in recent years. Due to tremendous hike in data production, the need for the efficient processing, transaction storage or retrieval, and management of the structured and unstructured data have become one of the important issues of the IT industry. In this paper, we examine the concept of evolution of various distributed file systems, advantages, and limitations with respect to the cloud computing paradigm. Distributed File System is a client-based application which permits its users to access, process and modify the data which is stored on a remote server as if it existed on their local systems. The Google file system (GFS), a proprietary distributed file system was developed by Google to meet its own growing big data need, which involved the utilization of commodity hardware in managing the massive generation of raw data. Inspired by GFS, Hadoop Distributed File System (HDFS) was developed, which is an open-source community project. It is designed to store large chunks of data sets consistently as well as providing high bandwidth sets for streaming data on client based applications. In this paper, we describe the GFS and HDFS models in detail and relate these two distributed file systems on the basis of their properties and characteristic behavior in different environments. We also discuss several techniques and modifications to prevent and mitigate the limitations of these file systems.

152. Pain plan implementation decreases postoperative opioid use, hospital length of stay and clinic resource utilization for patients undergoing elective spine surgery

In recent years, there are various cloud storage services have been proposed. The users can acquire the enormous cloud storage spaces through Internet even if the storage space of portable devices and handsets are limited. In this paper, we implement a distributed file system based on Apache Hadoop, and the original Hadoop distributed file system is replaced with the Tahoe least-authority file system. Furthermore, we investigate the total transmission rate and download time with two different file size. The experiment result shows that the established file system provides a secure and reliable cloud storage service even if parts of storage nodes are broken.

Distributed file systems have become popular in recent years. However, they still lack functions for doing fast arbitrary data insertion and truncation. To solve the problem, we present Wofs, an object-based distributed network file system which supports fast arbitrary data insertion and truncation. Wofs splits a file into many small objects, stores these objects in remote file servers, and uses a special B+tree to manage the metadata of these objects. Besides, Wofs uses the object-range locking policy to avoid data incoherence and improve performance.

Massively parallel data analysis is an emerging research topic that is motivated by the continuous growth of data sets and the rising complexity of data analysis tasks. To facilitate the analysis of big data, several parallel data processing frameworks, such as MapReduce and parallel data flow processors, have emerged. However, the implementation and tuning of parallel data analysis tasks requires expert knowledge and is very time-consuming and costly. Higher-level abstraction frameworks have been designed to ease the definition of analysis tasks. Optimizers can automatically generate efficient parallel execution plans from higher-level task definitions. Therefore, optimization is a crucial technology for massively parallel data analysis. This chapter presents the state of the art in optimization of parallel data flows. It covers higher-level languages for MapReduce, approaches to optimize plain MapReduce jobs, and optimization for parallel data flow systems. The optimization capabilities of those approaches are discussed and compared with each other. The chapter concludes with directions for future research on parallel data flow optimization.

A large class of modern distributed file systems treat metadata services as an independent system component, separately from data servers. The availability of the metadata service is key to the availability of the overall system. Given the high rates of failures observed in large-scale data centers, distributed file systems usually incorporate high-availability (HA) features. A typical approach in the development of distributed file systems is to design and develop metadata services from the ground up, at significant cost in terms of complexity and time, often leading to functional shortcomings. Our motivation in this paper was to improve on this state of things by defining a general-purpose architecture for HA metadata services (which we call RMS) that can be easily incorporated and reused in new or existing file systems, reducing development time. Taking two prominent distributed file systems as case studies, PVFS and HDFS, we developed RMS variants that improve on functional shortcomings of the original HA solutions, while being easy to build and test. Our extensive evaluation of the RMS variant of HDFS shows that it does not incur an overall performance or availability penalty compared to the original implementation.

This paper presents a novel distributed file system TLDFS, which consists of two layers. One is the block device layer, which makes the distributed file system manage the consistency of data easily by choosing an appropriate granularity of lock. We implement the distributed interfaces and the new data organization at the block level, which is different from most distributed file systems. The other is the file system layer that consists of three parts: interconnecting module, locking module and distributed file system module. Our prototype system proves the feasibility of the two-layered structure and the experiments show several advantages of TLDFS.

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

Delayed write in a multilevel file system cache hierarchy offers a way to improve performance in diverse client/server scenarios, such as integrating mass store into a distributed file system or providing distributed file system access over low-speed links. Using file system traces and cache simulations, we explore extensions and modifications to the traditional client-caching model employed in such file systems as AFS, Sprite, and DFS. High cache hit rates at an intermediate cache server-a machine logically interposed between clients and servers that provides cached file service to the clients-combined with high client cache hit rates lend practicality to an integrated mass storage file system. In such a system, magnetic tape or optical-based mass storage devices may be used as a first-class data repository, fronted by disk and RAM caches to offer acceptable access times to the large, but slow, mass storage system. Similarly, a high cache hit rate is necessary for users accessing file systems via low-speed links, where a delayed write intermediate caching server can mediate traffic to make better use of available bandwidth. In an example taken from mobile computing, an intermediate server might be used as a docking station at a user's home. This arrangement would be convenient for users of mobile computers who upload large amounts of data generated while operating in disconnected mode. Simulations of delayed write caching strategies are applicable to both the mass storage and low-speed link scenarios.

This paper discusses the concepts and facilities of the Shared File System (SFS) support for Virtual Machine/Enterprise Systems Architecture™ (VM/ESA™) Coordinated Resource Recovery (CRR). It includes background information on limitations that lead to SFS support for coordination of file recovery functions. The level of support provided by the Virtual Machine/System Product (VM/SP) Release 6 SFS support is identified and contrasted with the support provided in VM/ESA. The paper contains an overview of the system structure and the rationale for the support and is a discussion from the overall perspective of the total system environment and system processing for resource recovery. After the concepts and structure of VM/ESA SFS support are introduced, the paper discusses the specific technology involved in providing SFS support for Coordinated Resource Recovery. This includes a discussion of specific facilities used by SFS and how SFS deals with certain conditions that can arise. In addition, this paper discusses the Conversational Monitor System (CMS) compatibility considerations that contributed to the design of SFS support for Coordinated Resource Recovery. This includes compatibility with prior releases and compatibility with the CMS file system support for minidisks. Finally, some of the future directions for file system support of resource recovery are identified along with some of the challenges that remain to be solved.

The world keeps contributing to the increase in data everyday drastically. Scientific applications, weather forecasting, researches, hospitals, military services are few such major contributors. As the amount of data increases, the need to provide efficient, easy to use solutions has become one of the main issues for these type of computations. The best solution to this issue is the use of Distributed File Systems(DFS). Some existing Distributed File Systems are too complex to deploy and maintain, although they are extremely scalable and cheap since they can be entirely built out of commodity Operating System(OS) and hardware. GlusterFS solves this problem. Gluster File System(GFS) is open source and is capable of scaling to several petabytes(upto 72 brontobytes) and handling thousands of clients. It is based on a stackable user space design and can deliver exceptional performance for diverse workloads. GlusterFS is an easy to use clustered file system that meets enterprise-level requirements. It is written in user space which uses FUSE(Filesystem in user space) to hook itself with the VFS layer. It takes a layered approach to the file system, where features are added/removed as per the requirement. Enforcer is a major component in Gluster which finds its extensive use in production. Enforcer is concerned more about limits. It helps to restrict the usage(both in terms of size and count) at directory or volume level in the file system. In this paper, we discuss the internal working of the translators and key architectural components of GlusterFS. Along with that, one can gain a deep insight about the enforcement techniques, few challenges, undergoing researches and projects of GlusterFS

Peer-to-peer (P2P) systems are, in contrast to client-server (C/S) systems, fault-tolerant, robust, and scalable. While C/S distributed file systems, such as NFS (network file system) or SMB (server message block), do not scale with respect to the number of clients and exhibit a single point of failure, P2P file systems have the potential to cope with an increasing number of participants. Thus, this paper presents DRFS (distributed reliable file system), a P2P file system for cooperative environments. DRFS uses random, content-independent identifiers for data storage, while maintaining high performance and low overhead with many concurrent reads and writes. A dynamic replication mechanism ensures data availability, even under high churn. The application scenario considers an office environment, where DRFS is installed on employees' machines, who store and request files. DRFS has been implemented using the Filesystem in Userspace (FUSE) interface, in order to provide users with transparent read and write operations. Experiments show the benefits of such a peer-to-peer architecture, when a small number of peers reads or writes in parallel: DRFS performs better than NFS, as soon as 6 peers read or write in parallel a 32 MB file. For unpopular files, it is also more reliable than IgorFS.

The Direct Access File System (DAFS) is a distributed file system built on top of direct-access transports (DAT). Direct-access transports are characterized by using remote direct memory access (RDMA) for data transfer and user-level networking. The motivation behind the DAT-enabled distributed file system architecture is the reduction of the CPU overhead on the I/O data path.We have created an implementation of DAFS for the FreeBSD platform. In this paper we describe the performance evaluation study of DAFS that we have performed using this software. The goal of this study is to determine whether the architecture of DAFS brings any fundamental performance benefits to applications compared to traditional distributed file systems, such as NFS. We perform comparison of DAFS to a version of NFS optimized to reduce the I/O overhead. In order to thoroughly understand the impact of DAFS on application performance, we consider a diverse range of applications workloads.We conclude that DAFS can accomplish superior performance for latency-sensitive applications, outperforming NFS by up to a factor of 2. Bandwidth-sensitive applications do equally well on both systems, unless they are CPU-intensive, in which case they perform better on DAFS. We also found that RDMA is a less restrictive mechanism to achieve copy avoidance than that used by the optimized NFS.