Serverless network file systems

This paper describes a simple distributed mechanism for caching files among a networked collection of workstations. We have implemented it as part of Sprite, a new operating system being implemented at the University of California at Berkeley. A preliminary version of Sprite is currently running on Sun-2 and Sun-3 workstations, which have about 1-2 MIPS processing power and 4-16 Mbytes of main memory. The system is targeted for workstations like these and newer models likely to become available in the near future; we expect the future machines to have at least five to ten times the processing power and main memory of our current machines, as well as small degrees of multiprocessing. We hope that Sprite will be suitable for networks of up to a few hundred of these workstations. Because of economic and environmental factors, most workstations will not have local disks; instead, large fast disks will be concentrated on a few server machines.

Users rarely consider running network file systems over slow or wide-area networks, as the performance would be unacceptable and the bandwidth consumption too high. Nonetheless, efficient remote file access would often be desirable over such networks---particularly when high latency makes remote login sessions unresponsive. Rather than run interactive programs such as editors remotely, users could run the programs locally and manipulate remote files through the file system. To do so, however, would require a network file system that consumes less bandwidth than most current file systems.This paper presents LBFS, a network file system designed for low-bandwidth networks. LBFS exploits similarities between files or versions of the same file to save bandwidth. It avoids sending data over the network when the same data can already be found in the server's file system or the client's cache. Using this technique in conjunction with conventional compression and caching, LBFS consumes over an order of magnitude less bandwidth than traditional network file systems on common workloads.

This paper presents Kosha, a peer-to-peer (p2p) enhancement for the widely-used Network File System (NFS). Kosha harvests redundant storage space on cluster nodes and user desktops to provide a reliable, shared file system that acts as a large storage with normal NFS semantics. P2p storage systems provide location transparency, mobility transparency, load balancing, and file replication - features that are not available in NFS. On the other hand, NFS provides hierarchical file organization, directory listings, and file permissions, which are missing from p2p storage systems. By blending the strengths of NFS and p2p storage systems, Kosha provides a low overhead storage solution. Our experiments show that compared to unmodified NFS, Kosha introduces a 4.1% fixed overhead and 1.5% additional overhead as nodes are increased from one to eight. For larger number of nodes, the additional overhead increases slowly. Kosha achieves load balancing in distributed directories, and guarantees 99.99% or better file availability.

Peer-to-peer (P2P) systems are, in contrast to client-server (C/S) systems, fault-tolerant, robust, and scalable. While C/S distributed file systems, such as NFS (network file system) or SMB (server message block), do not scale with respect to the number of clients and exhibit a single point of failure, P2P file systems have the potential to cope with an increasing number of participants. Thus, this paper presents DRFS (distributed reliable file system), a P2P file system for cooperative environments. DRFS uses random, content-independent identifiers for data storage, while maintaining high performance and low overhead with many concurrent reads and writes. A dynamic replication mechanism ensures data availability, even under high churn. The application scenario considers an office environment, where DRFS is installed on employees' machines, who store and request files. DRFS has been implemented using the Filesystem in Userspace (FUSE) interface, in order to provide users with transparent read and write operations. Experiments show the benefits of such a peer-to-peer architecture, when a small number of peers reads or writes in parallel: DRFS performs better than NFS, as soon as 6 peers read or write in parallel a 32 MB file. For unpopular files, it is also more reliable than IgorFS.

Server is a computer system that provides the kind of service in a computer network. One of the functions of the server is as data storage. Computers servers with high availability values are generally referred to as High Availability (HA) Server. HA enables a server can be operational in a sustainable manner for long periods of time. Now, HA has been combined with a cluster technique known as High Availability Clusters or failover cluster. In this research the author intends to apply, analyze, and implement a network file server that has a high availability of service. Researchers used a pacemaker as cluster resource management, Corosync as cluster messaging service and DRBD for the synchronization of data if a failover occurs and measured network performer cluster file system that does have a high availability service. The result is a network file system or the availability of the cluster has a value of 99.998% availability, which means it has a High Availability. The server network file system failover cluster meets the conditions, which, if the primary server (master) dies it will be replaced by a slave server.

Nowadays production flows are modular, each module in the enterprise being specialized and used to achieve a particular task. In many cases the modules are interconnected and materials are sequentially processed in each module resulting a final, unique product or assembly. One typical such production module is a flexible cell/system using multiple robots. In such complex enterprise environments, providing continuous services for applications is a key component of a successful implementing of robotized manufacturing. High availability (HA) is one of the components contributing to continuous service provision for applications, by masking or eliminating both planned and unplanned downtime of systems and applications. This is achieved by eliminating hardware and software single points of failure (SPOF). The systems configured for high availability are a combination of hardware and software components configured to work together to ensure automated recovery in case of failure with a minimal acceptable downtime. A high availability solution will ensure that the failure of any component of the solution either hardware, software or system management, will not cause the application and its data to become permanently unavailable. High availability solutions should eliminate single points of failure through appropriate design, planning, hardware selection, software configuring, application control, carefully environment control and change management discipline. In short, one can define high availability as the process of ensuring an application is available for use by duplicating and/or sharing hardware resources managed by a specialized software component. A high availability solution in robotized manufacturing provides automated failure detection, diagnosis, application recovery, and node (robot controller) re integration. The chapter discusses the implementing of a high availability solution in a robotized manufacturing structure (cell, line). The solution is based on a High Availability Linux Cluster which is responsible for data availability and preservation on a NFS (Network File System) file system, and a second cluster – Fabrication Cluster (FC) which runs on Adept robot controllers, developed under the V+ programming language.

Analysis of NGS (Next Generation Sequencing) data is a computationally demanding task requiring large amounts of CPU, memory, and disk space. There is also a requirement for high performance data storage systems, resilient to hardware failure, to be connected directly to the computing infrastructure (typically a multi-node cluster) to store large quantities of NGS data reliably. Traditional shared file systems such as NFS (Network File System) do not offer the performance, scalability or cache coherence required by modern NGS data analysis, so alternatives including GlusterFS, Ceph, and Lustre have been developed. However, there is a trade-off between data safety on replicated local storage and degradation of performance across distributed storage. Resilience to hardware failure is typically provided by RAID (Redundant Array of Independent Disks) and redundant storage nodes. Here we describe the evaluation of an alternative file system, RozoFS ( https://github.com/rozofs/rozofs ) for use with demanding NGS data analysis workloads. We used a synthetic data set (DREAM-TCGA data set 3) to run a complete tumor-normal analysis pipeline (“bcbio”, https://github.com/chapmanb/bcbio-nextgen ), including base quality recalibration, local indel realignment, somatic variant calling, and structural variants as a benchmark to compare RozoFS with a traditional shared file system (NFS) on two different HPC (High Performance Computing; Cloud4CaRE project) clusters. Our results show high reliability and good performance of RozoFS compared to NFS, in particular, during heavy I/O workloads. These findings indicate that the reliability and robustness of RozoFS's make it a good candidate for demanding NGS analysis workloads.

Network File System is a widely used distributed file system that allows the user to access and manipulate storage on remote computers, as if they were a part of the local machine. Network File System is notoriously slow in its default configuration and if more clients connects to the NFS environment, it merely accentuates the delay. When configured to deliver faster speeds, the system suffers from higher risk of data corruption and loss. This study proposes a number of modifications to the Network File System, enabling it to provide elevated system performance, while containing the risk of data loss and corruption. Further, the proposed system behaves better in congested networks by consuming less bandwidth, ensuring decent speeds, even during periods of heavy network traffic.

The ubiquitous wireless networks make it become possible for people to access a variety of services anytime and anywhere through mobile devices such as smartphone and tablet. However, due to the diversity and complexity of services, the requirement for storage and computing capacity for mobile devices is becoming higher and higher. Mobile transparent computing is a fit for this scenario. In mobile transparent computing, the client terminal is rather simple and light-weighted, and the operating systems, applications and user data are all regarded as software resources and stored in the server. The resources required in the user services are dynamically scheduled from the server to run on the client terminal in a streaming-block way via wireless network. In mobile transparent computing, we use the NFS (Network File System) protocol to realize the dynamic scheduling of resources. However, the wireless network environment is complex and unstable, and the wireless packet loss rate is high and the retransmission interval of lost packet is long in mobile transparent computing. In order to improve the scheduling efficiency of users' resources and the NFS performance, we analyze the factors that affect the NFS performance, including NFS read/write block size, wireless bandwidth and so on. We study the effect of bandwidth for the NFS performance when several mobile terminals schedule their resources at the same time. Finally, we improve the NFS performance by setting the optimal block size for read/write, and use TCP as our NFS transmission protocol in mobile transparent computing.

New non-volatile memory (NVM) technologies, e.g., Phase Change Memory (PCM), are emerging with main features such as byte addressability, cheapness and large size, and will be deployed to serve as main memory. For fast processing of big data, block device drivers have been developed for NVM such that traditional file systems (e.g., EXT4) operated on non-volatile main memory. However, the existing block driver blindly caches pages, which incurs unnecessary access delays. To overcome this drawback, this paper develops a novel block device driver, called NVMBD, for byte-addressable non-volatile main memory. It adopts two methods of processing block I/O requests, i.e., traditional block I/O and BMAP. On the one hand, traditional block I/O reads/writes 512B sectors with multiple load/store instructions. On the other hand, BMAP inserts NVM pages into the page cache tree when processing block I/O requests, which removes the page movement between the page cache and the NVM. BMAP involves in the management of page cache, but it is transparent to file systems. BMAP provides file systems with the DAX-like feature, improving the file system performance. Through comprehensive experiments, we demonstrate that the developed block device driver significantly improves the file access performance.

Despite continual improvements in the performance and reliability of large scale file systems, the management of user-defined file system metadata has changed little in the past decade. The mismatch between the size and complexity of large scale data stores and their ability to organize and query their metadata has led to a de facto standard in which raw data is stored in traditional file systems, while related, application-specific metadata is stored in relational databases. This separation of data and semantic metadata requires considerable effort to maintain consistency and can result in complex, slow, and inflexible system operation. To address these problems, we have developed the Quasar File System (QFS), a metadata-rich file system in which files, user-defined attributes, and file relationships are all first class objects. In contrast to hierarchical file systems and relational databases, QFS defines a graph data model composed of files and their relationships. QFS incorporates Quasar, an XPATH-extended query language for searching the file system. Results from our QFS prototype show the effectiveness of this approach. Compared to the de facto standard, the QFS prototype shows superior ingest performance and comparable query performance on user metadata-intensive operations and superior performance on normal file metadata operations.

Building a Portable File System for Heterogeneous Clusters

Traditionally, network storage systems have mainly been dominated by two IP-based storage technologies, i.e., Network Attached Storage (NAS) and Storage Area Network (SAN). In recent years, cloud based storage (e.g., Amazon S3) has gained growing popularity for its high flexibility and cross-platform compatibility. Many enterprises are considering to replace traditional storage systems with cloud-based systems. Evaluating such a transition demands a systematic study on understanding the performance behaviours of the emerging cloud storage. To fill in this gap, in this paper, we conduct a comprehensive study on the three storage systems with realistic network conditions. Specifically, we select one representative from each category for comparison, i.e., Network File System (NFS) from NAS, Internet Small Computer System Interface (iSCSI) from SAN, and OpenStack Swift from cloud storage. We build a testbed and develop a suite of micro-benchmarks to study the impact of network complexities. Through a set of experiments and detailed analysis, we make several key observations: (1) iSCSI excels under good network conditions, e.g., in local area networks (LANs) where network delay and packet loss are trivial, (2) NFS and Swift are more suitable for complex networks such as wireless networks and Internet environment, (3) Swift is a viable replacement for NFS in all scenarios we investigate, and (4) system configuration on the client side impacts storage performance significantly and deserves adequate attention. We hope our findings can not only shed light on storage service design and optimizations, but also encourage more research on emerging storage technologies.

The performance of a network based computing system that consists of a number of clients and a server, interconnected by a local area network, is studied. Each client represents a diskless or swapful workstation, or an X terminal, and generates application workload using either a conventional UNIX file system or the Network File System (NFS). To characterize the application workload and the effect of the protocol (login, rlogin, NFS), the user interface (workstation, X terminal) and its configuration (memory size, disk storage), a series of measurement experiments was performed in a controlled environment. During each experiment the traffic on the network and the activity on the various system resources were monitored. The measurements obtained are used to parameterize a queuing network model of the system, which is then used to project the performance of the system under various load conditions, to identify system bottlenecks, and to compare design alternatives. >

CHAPTER 2 - Managing Filesystems