SBaaS: Safe Box as a Service

Hadoop Distributed File System (HDFS) is the underlying storage engine of many Big Data processing frameworks such as Hadoop MapReduce, HBase, Hive, and Spark. Even though HDFS is well-known for its scalability and reliability, the requirement of large amount of local storage space makes HDFS deployment challenging on HPC clusters. Moreover, HPC clusters usually have large installation of parallel file system like Lustre. In this study, we propose a novel design to integrate HDFS with Lustre through a high performance key-value store. We design a burst buffer system using RDMA-based Mem cached and present three schemes to integrate HDFS with Lustre through this buffer layer, considering different aspects of I/O, data-locality, and fault-tolerance. Our proposed schemes can ensure performance improvement for Big Data applications on HPC clusters. At the same time, they lead to reduced local storage requirement. Performance evaluations show that, our design can improve the write performance of Test DFSIO by up to 2.6x over HDFS and 1.5x over Lustre. The gain in read throughput is up to 8x. Sort execution time is reduced by up to 28% over Lustre and 19% over HDFS. Our design can also significantly benefit I/O-intensive workloads compared to both HDFS and Lustre.

In last half decade, there is a tremendous growth in the network applications; we are experiencing an information explosion era and for that large amount of distributed data being stored and managed. Distributed file system is designed to handle these types of data. Major design issues in DFS are scalability, fault tolerance, flexibility and availability. The most prevalent DFS to deal with these challenges is the Hadoop Distributed File System (HDFS) which is a variant of the Google File System (GFS). Apache Hadoop able to solve current issues of Big Data by simplifying the implementation of data intensive and exceptionally parallel distributed applications. HDFS handles fault tolerance using Data Replication. HDFS replicates each data block on different datanode for reliability and availability. The existing implementation of HDFS in Hadoop performs replication in a pipelined manner which takes much time for replication. Here proposed system is an alternative parallel approach for efficient replica placement in HDFS to improve throughput. The experimentation has been performed to compare its performance with existing pipelined replication approach, which improve HDFS write throughput up to 10% testified by the TestDFSIO benchmark. This paper also depicts the analysis on the basis of different HDFS configuration parameter like file block size and replication factor which affects HDFS write performance in both approaches.

Working in tandem, the Hadoop Distributed File System (HDFS) plays a crucial role in managing large-scale data across distributed clusters. The basic structure of HDFS comprises the NameNode, which manages the file system namespace and client interactions, and the Data Nodes, which handle storage and data arrangement duties. Although HDFS faces constraints, particularly regarding scaling the NameNode, which is responsible for managing metadata, these constraints impact namespace scalability and performance. Therefore, this paper discusses the two architectures, the Quorum Journal Manager (QJM) components and the active and standby NameNodes, which enable the sharing of edit logs between them. To mitigate the risk of metadata corruption from split-brain scenarios, the system ensures that only a single NameNode writes to the JournalNodes. Through the utilization of the QJM, HDFS attains High Availability (HA), enabling swift failover in the event of machine crashes or scheduled maintenance. Dynamic Federated Metadata Management (DFMM) addresses the limitations of the current Hadoop architecture by distributing metadata management across multiple federated components. It entails dispersing metadata management among various federated components. HDFS can overcome the problems of connected block storage, namespace scalability, and performance bottlenecks by dynamically managing metadata on an exabyte scale. This work summarizes the key findings and presents a comparative analysis of both architectures. QJM performs a vital function that ensures the high availability, reliability, and fault tolerance of the NameNode in Hadoop’s HDFS. DFMM enhances the fault tolerance, scalability, and performance of distributed file systems by distributing metadata across multiple servers or nodes.

HDFS (Hadoop Distributed File System) is the primary storage of Hadoop. Even though data locality offered by HDFS is important for Big Data applications, HDFS suffers from huge I/O bottlenecks due to the tri-replicated data blocks and cannot efficiently utilize the available storage devices in an HPC (High Performance Computing) cluster. Moreover, due to the limitation of local storage space, it is challenging to deploy HDFS in HPC environments. In this paper, we present a hybrid design (Triple-H) that can minimize the I/O bottlenecks in HDFS and ensure efficient utilization of the heterogeneous storage devices (e.g. RAM, SSD, and HDD) available on HPC clusters. We also propose effective data placement policies to speed up Triple-H. Our design integrated with parallel file system (e.g. Lustre) can lead to significant storage space savings and guarantee fault-tolerance. Performance evaluations show that Triple-H can improve the write and read throughputs of HDFS by up to 7x and 2x, respectively. The execution times of data generation benchmarks are reduced by up to 3x. Our design also improves the execution time of the Sort benchmark by up to 40% over default HDFS and 54% over Lustre. The alignment phase of the Cloudburst application is accelerated by 19%. Triple-H also benefits the performance of SequenceCount and Grep in PUMA over both default HDFS and Lustre.

The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system (HDFS) as a case study and proposes a forensic method for efficient file extraction based on three-level (3L) mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.

With the increasing prominence of integrating highperformance computing (HPC) with big-data (BIGDATA) processing, running MPI over the Hadoop Distributed File System (HDFS) offers a promising approach for delivering better scalability and fault tolerance to traditional HPC applications. However, it comes with challenges that discourage such an approach: (1) two-sided MPI communication to support intermediate data processing, (2) a focus on enabling N-1 writes that is subject to the default HDFS block-placement policy, and (3) a pipelined writing mode in HDFS that cannot fully utilize the underlying HPC hardware. So, while directly integrating MPI with HDFS may deliver better scalability and fault tolerance to MPI applications, it will fall short of delivering competitive performance. Consequently, we present a performance study to evaluate the feasibility of integrating MPI applications to run over HDFS. Specifically, we show that by aggregating and reordering intermediate data and coordinating computation and 110 when running MPI over HDFS, we can deliver up to 1.92x and 1.78x speedup over MPI I/O and HDFS pipelined-write implementations, respectively. Consequently, we present a performance study to evaluate the feasibility of integrating MPI applications to run over HDFS. Specifically, we show that by aggregating and reordering intermediate data and coordinating computation and 110 when running MPI over HDFS, we can deliver up to 1.92x and 1.78x speedup over MPI I/O and HDFS pipelined-write implementations, respectively.

The usage of cloud systems is at an all-time high, and with more organizations reaching for Big Data the forensic implications must be analyzed. The Hadoop Distributed File System is widely used both as a cloud service and with organizations implementing it themselves. This paper analyzed the forensic viability of a RAM analysis method for Hadoop based investigations and compared it against targeted process data dumping through the Java heap information. The RAM analysis was done through string searching and the use of the RAM analysis tool Volatility. This work found that RAM analysis can be a valuable tool for discovering artefacts of deleted resources from a Hadoop cluster but was unable to discover further information such as the block to node mapping. The targeted process analysis managed to provide some partial information about deleted resources and produce important metadata on the current state of the file system.

Through analyzing the storage system requirements of supercomputer this paper designs a near-line storage system called NLSS based on the combination of HDFS (Hadoop distributed file system) and ZFS (Zettabyte file system). NLSS uses fat storage nodes (large storage servers) to build near-line storage clusters based on HDFS, and uses the ZFS file system to further enhance HDFS. NLSS effectively reduces the burden of supercomputer online storage system. Experiment results show that NLSS can acquire better storage utilization, reliability and scalability while ensuring appropriate performance.

The Hadoop Distributed File System (HDFS) is designed to store, analysis, transfer larg datasets reliably, and stream it at high bandwidth to the user applications. HDFS is a variant of the Google File System (GFS). It handles fault tolerance by using data replication, where each data block is replicated and stored on multiple DataNodes. Therefore, the HDFS supports availability and reliability. The existed implementation of the HDFS in Hadoop performs replication in a pipelined manner that takes much time for replication. In this paper, an alternative technique for efficient replica placement, called Enhancement Lazy replication technique, has been proposed. The main principle of this technique is that, the client allows to write a block to two DataNodes in parallel, which store the packet. which will send acknowledgement directly to the client without waiting of receiving acknowledgement from other DataNodes. The experiment has been performed to evaluate the performance of the proposed HDFS replication technique with the default pipelined replication technique and the existed replication techniques; using TestDFSIO benchmark. According to the experimental results (i.e., the execution time and throughput), it is found that the HDFS availability has been improved in the proposed replication technique.

Nowadays more and more data have been generated every day in some enterprises such as facebook and google. These data need to be collected and analyzed in time. So the speed of transmitting data must be very high and the latency must be very low. Hadoop is applied in these enterprises and they use several data centers to store and process these data. But if the amount of data is growing fast or we will use only one data center then the bandwidth of the Ethernet Hadoop Distributed File System (HDFS) using cannot meet the need. The bandwidth of the Ethernet is going to become the performance bottleneck of HDFS. In order to solve this problem we will introduce a relatively new switched fabric communication link–—Infiniband in this paper. Based on Infiniband we have designed a new communication mechanism of HDFS and implemented it by modifying the code of HDFS. We use remote direct memory access (RDMA) to send and receive data rather than socket. The new HDFS will not use original stream mode to transmit data. Instead it will dynamically expand buffer and use changeable threshold. In this way the new HDFS will make CPU idle and improve performance. Unlike IPoIB which only uses Infiniband hardware device, our optimized HDFS is not only based on Infiniband hardware but also changes the code of HDFS to use RDMA. Our HDFS uses socket to transmit control message and RDMA to transmit data to make full use of the bandwidth of Infiniband. So applying the Infiniband with RDMA network bandwidth has not been the performance bottleneck of HDFS any more. According to the experiment results we have found that the network bandwidth of HDFS over Infiniband is 60 percent higher than the Ethernet and our optimized HDFS has much better performance than the HDFS over the Ethernet. On the other hand, the performance of our HDFS is also higher than the one which only use IPoIB.KeywordsInfinibandRDMADFSNetwork bandwidth

The Hadoop Distributed File System (HDFS) is a cornerstone of modern big data ecosystems, offering robust and scalable storage solutions. Optimizing data loading processes into HDFS is crucial for enhancing the overall performance of data - intensive applications. This paper explores various strategies and best practices for efficient data loading, focusing on key factors such as data locality, parallelism, and network configurations. Firstly, we delve into the architecture of HDFS, highlighting the roles of NameNode, DataNodes, and the block structure, which are pivotal for understanding data distribution and management within the system. We then evaluate different methods for loading data, including direct HDFS commands, WebHDFS, HttpFS, and tools like DistCp, Apache Flume, and Sqoop, discussing their relative efficiencies and use - case applicability. Further, we present detailed optimization techniques starting with data preprocessing, which involves data cleaning and the adoption of suitable serialization formats such as Avro and Parquet to minimize I/O operations. The impact of data compression on storage and performance is also examined, alongside methods for balancing data across DataNodes to prevent data skewness. Advanced strategies such as enhancing data locality to reduce latency and configuring high - bandwidth networks to expedite data transfer are discussed comprehensively. Additionally, the use of parallel data loading techniques is explored to maximize throughput. Monitoring and tuning data loading performance are addressed in the latter part of the paper, where key performance metrics and the tools necessary for performance assessment are outlined. Recommendations on tuning HDFS configurations to optimize data loading are provided based on empirical data and industry practices. This paper aims to serve as a comprehensive guide for practitioners looking to enhance their HDFS implementations, ensuring efficient data handling and optimal operational performance.

MapReduce is increasingly becoming a popular framework, and a potent programming model. The most popular open source implementation of MapReduce, Hadoop, is based on the Hadoop Distributed File System (HDFS). However, as HDFS is not POSIX compliant, it cannot be fully leveraged by applications running on a majority of existing HPC environments such as Teragrid and NERSC. These HPC environments typically support globally shared file systems such as NFS and GPFS. On such resourceful HPC infrastructures, the use of Hadoop not only creates compatibility issues, but also affects overall performance due to the added overhead of the HDFS. This paper not only presents a MapReduce implementation directly suitable for HPC environments, but also exposes the design choices for better performance gains in those settings. By leveraging inherent distributed file systems' functions, and abstracting them away from its MapReduce framework, MARIANE (MApReduce Implementation Adapted for HPC Environments) not only allows for the use of the model in an expanding number of HPC environments, but also allows for better performance in such settings. This paper shows the applicability and high performance of the MapReduce paradigm through MARIANE, an implementation designed for clustered and shared-disk file systems and as such not dedicated to a specific MapReduce solution. The paper identifies the components and trade-offs necessary for this model, and quantifies the performance gains exhibited by our approach in distributed environments over Apache Hadoop in a data intensive setting, on the Magellan test bed at the National Energy Research Scientific Computing Center (NERSC).

Hadoop Distributed File System (HDFS) is the de facto file system in vanilla Hadoop and most of open source Hadoop distributions. However HDFS suffers from the so called Small File Problem. Storm is a parallelized stream processing framework. In this paper we evaluate buffering strategies to mitigate the small file problem. In our evaluations Storm will be the small file source.

Incorporating MPI programming model into data-intensive file system for big data application is significant in performance research for optimization purpose. In this paper we ported an MPI-SVM solver, originally developed for HPC environment to the Hadoop distributed file system (HDFS). We analyzed the performance bottlenecks with which the SVM solver will be confronted on the HDFS. It is known the storage expansion on HDFS comes with a skewed data distribution. As a result, we found out that some hot nodes always receive condensed I/O requests while other nodes always send remote requests. These remote requests make the I/O delays elongate on hot nodes, which leads to performance bottleneck for our solver. Thus we specifically improved the data preprocessing part that requires large amount of I/O operations by a deterministic scheduling method. Our improvement showed a balanced read pattern on each node. The time ratio between the longest process and the shortest process has been reduced by 60%. Also the average read time has significantly reduced by 78%. The data served on each node also showed a small variance in comparison with the originally ported SVM algorithm. We believe our design avoids the overhead introduced by remote I/O operations, which will be beneficial to many algorithms when coping with large scale of data.

Handling of Big Data and cloud computing are the two important prime concerns which have become more and more popular in recent years. Due to tremendous hike in data production, the need for the efficient processing, transaction storage or retrieval, and management of the structured and unstructured data have become one of the important issues of the IT industry. In this paper, we examine the concept of evolution of various distributed file systems, advantages, and limitations with respect to the cloud computing paradigm. Distributed File System is a client-based application which permits its users to access, process and modify the data which is stored on a remote server as if it existed on their local systems. The Google file system (GFS), a proprietary distributed file system was developed by Google to meet its own growing big data need, which involved the utilization of commodity hardware in managing the massive generation of raw data. Inspired by GFS, Hadoop Distributed File System (HDFS) was developed, which is an open-source community project. It is designed to store large chunks of data sets consistently as well as providing high bandwidth sets for streaming data on client based applications. In this paper, we describe the GFS and HDFS models in detail and relate these two distributed file systems on the basis of their properties and characteristic behavior in different environments. We also discuss several techniques and modifications to prevent and mitigate the limitations of these file systems.