Analysis of Data Reliability Tradeoffs in Hybrid Distributed Storage Systems

This paper is motivated by a global online satellite images distribution system operated at the Earth Resources Observation and Science (EROS) center of the U.S Geological Survey. Fundamental objectives of EROS include, but are not limited to, building high-speed and cost-effective massive data processing and storage systems to support online satellite images distribution. Hybrid storage systems -- containing solid-state drives (SSD), hard disks (HDD), and tapes -- can provide an ideal data storage solution for a wide variety of data processing centers like EROS. Large-scale hybrid storage systems will become increasingly popular in the next few years for the following two reasons. First, highly accessed storage objects in a hybrid storage system can be prefetched and cached to high-speed storage components such as solid-state drives. SSD-based hybrid storage system can provide large storage capacity, high I/O performance and data reliability. Second, hybrid storage systems are cost-effective, because inexpensive tapes help in increasing storage capacities at very low cost. Transferring data back and forth among SSDs, HDDs, and tapes plays a critical role in achieving high I/O performance. Thus, we proposed data mining algorithms that can judiciously prefetch data. Our analytical model and the experimental results reveals that our data mining prefetching algorithm increase the performance of the hybrid storage systems.

Flash memory has lately been used as a cache located between the system main memory and the magnetic hard drives in order to create robust and cost effective hybrid storage systems. The reason comes from the growing density of Solid State Devices (SSDs) at lower prices with main advantage of high random read efficiency compared to magnetic hard drives. When predicting the performance of such hybrid storage systems, it is inevitable to study the trade-off in selecting the different storage elements such as the main memory and the SSD cache relative to the capacity of the magnetic hard drive. The parameters of such prediction model are determined based on the application that the storage system would serve. In this paper, a prediction model that uses experimental evaluation of a hybrid storage system and real applications/benchmarks is used. The model utilizes parameters of both the storage system and applications in order to predict system performance based on metrics that are commonly used in storage system evaluation. The model allows the designer to select the best hybrid storage system parameters that satisfy certain application performance requirements. The model is highly accurate with a minimal error and a high prediction confidence level (95%) in reference to the experimental data collected from real applications using the proposed SSD-aware hybrid storage system.

Reliable distributed data storage systems have to employ redundancy codes to tolerate the loss of storages. Many appropriate codes and algorithms can be found in the literature, but efficient schemes for tolerating several storage failures and their embedding in a distributed system are still research issues. In this paper, a variety of redundancy schemes are compared that got implemented in a distributed storage system. All schemes are based on parity and Reed/Solomon and are integrated in the storage system NetRAID. This system allows to configure several user-specified layouts. A performance and reliability analysis of several data and redundancy layouts is presented that combines analytical and experimental results. In a detail, we present performance results for an optimized Reed/Solomon implementation and give an outline for speeding up encoding and recovery by reconfigurable hardware employed in the distributed storage system

This paper presents an experimental evaluation phase carried out on ZEBRA batteries and Electrical Double Layer Capacitors (EDLCs) based storage system, which is able to power electric urban mini-busses. The particular analyzed case study is focused on a power train, specifically designed for electric urban road applications, composed by a fixed ratio gear box connected to an asynchronous 70 kW electric drive, which is supplied by the electric parallel of two 20 kWh Zebra batteries and a 63 F super-capacitors bank. The power train is studied by means of a laboratory 100 kW regenerative electric brake, provided with speed and torque controls, in order to experimentally measure the performance of the vehicle under study, on both steady state and dynamic operative conditions. The energy flux management of the hybrid storage system is preliminarily evaluated and tested in a simulation environment and then directly embedded in the power converters, interposed between storage systems and traction electric motor, to run experiments on the whole vehicle power train. In this way the combination of the two technologies of storage systems are tested in laboratory under various working conditions and with a different contribution of each storage device, in terms of electric energy and power. The obtained experimental results shows, on one side the effectiveness of the control strategies, evaluated to balance energy and power coming from the storage system to supply the traction electric drive, and on the other side the identification of a methodology to evaluate the performance of innovative hybrid storage systems, when powering road urban transportation vehicles.

The exponentially growing demand for storage puts a huge stress on traditional distributed storage systems. Historically, I/Ops (Inputs/Outputs per second) of hard drives have been the main limitation of storage systems. With the rapid deployment of solid state drives (SSDs) and the expected evolutions of their capacities, price and performance, we claim that CPU and network capacities will become bottlenecks in the future. In this context, we introduce 6Stor, an innovative, software-defined distributed storage system fully integrated with the networking layer. This storage system departs from traditional approaches in two manners: it leverages IPv6 new capabilities to increase the efficiency of its data plane—notably by using directly UDP and TCP rather than HTTP—and thus its performance; and it circumvents scalability limitations of other distributed systems by using a fully distributed metadata layer of indirection to offer flexibility. In this paper, we introduce and describe in details the architecture of 6Stor, with an emphasis on dynamic scalability and robustness to failure. We also present a testbed that we use to evaluate our novel approach by using Ceph—another well known distributed storage system—as baseline. Results obtained on an extensive testbed are presented and some initial conclusions are drawn.

SummaryHybrid storage systems that consist of flash‐based solid state drives (SSDs) and traditional disks are now widely used. In hybrid storage systems, there exists a two‐level cache hierarchy that regard dynamic random access memory (DRAM) as the first level cache and SSD as the second level cache for disk storage. However, this two‐level cache hierarchy typically uses independent cache replacement policies for each level, which makes cache resource management inefficient and reduces system performance. In this paper, we propose a novel adaptive multi‐level cache (AMC) replacement algorithm in hybrid storage systems. The AMC algorithm adaptively adjusts cache blocks between DRAM and SSD cache levels using an integrated solution. AMC uses combined selective promote and demote operations to dynamically determine the level in which the blocks are to be cached. In this manner, the AMC algorithm achieves multi‐level cache exclusiveness and makes cache resource management more efficient. By using real‐life storage traces, our evaluation shows the proposed algorithm improves hybrid multi‐level cache performance and also increases the SSD lifetime compared with traditional multi‐level cache replacement algorithms. Copyright © 2015 John Wiley & Sons, Ltd.

In recent years, hybrid storage has gradually become a hotspot in the research of data storage owing to its high-performance and low cost. An OpenStack-based hybrid storage system is presented in this paper. According to the characteristics, the data is divided into small data, big data and temporary data in this hybrid storage system; meanwhile a storage strategy, combining database storage system, the virtual file system and servers file system, is designed. In the application of iCampus project, this proposed hybrid storage system shows better performance and higher efficiency than the traditional single storage systems.

Optimal hybrid pumped hydro-battery storage scheme for off-grid renewable energy systems

An increasing recourse to renewable energies is one of the key solutions to address the current resource and environmental concerns related to the world energy supply. Because of the distributed and intermittent nature of several of them (Solar, Wind), an efficient and economically viable exploitation of renewable energies relies on the use of energy storage means. Fuel-free compressed air energy storage technologies are highly compatible with renewable energies because of their inherent environmental advantages. However their low energy performances have been the main barrier to their widespread utilization. Pneumatic storage is considered in this thesis with the goal of improving its energetic and power performances so as to make it more efficient and suited for renewable sources support. Storing/generating electrical energy into/from compressed air requires a multiple-step conversion process through an intermediary mechanical energy. Pneumatic-to-mechanical energy conversion is studied first. The suppression of the pressure regulation is proposed to avoid the important energy losses related to this operation. Consequently the volumetric machine must operate at higher and variable pressure. The analysis of the efficiency characteristics of these machines shows the existence of a pressure dependent optimal speed that corresponds to the maximum efficiency. A Maximum Efficiency Point Tracking (MEPT) strategy, based on efficiency-controlled variable speed operation, is proposed for the real time optimization of the conversion efficiency. Experimental results confirm the effectiveness of the proposed strategy both with air machines and oil-hydraulic machines. Oil-hydraulic machines offer higher conversion efficiencies compared to air machines, but require an air-to-oil interface. Two possible ways of realizing such an interface have led to the two hydro-pneumatic storage systems presented. The proposed efficiency-controlled variable speed operation has allowed improving the cycle efficiency of the experimental hydro-pneumatic conversion system by about 4% compared to that of a constant speed operation. In order to provide good power quality and flexibility to these storage systems, a hybrid topology that associates the main, hydro-pneumatic storage subsystem with an auxiliary, supercapacitive storage subsystem is proposed. The power variation is achieved by an intermittent operation of the main storage subsystem and the use of the auxiliary storage subsystem to smooth the resulting power, through the regulation of the common DC bus voltage. The hybrid storage system is thus compatible with a wide range of load and source powers, thanks to the obtained power flexibility. An efficiency analysis shows that the performances of the auxiliary storage greatly affect that of the global storage system. The auxiliary storage should therefore exhibit very high conversion efficiencies so that an acceptable overall efficiency can be expected. A formal method for optimally sizing the supercapacitive auxiliary storage system is proposed, that allows meeting both the voltage and energy requirements while minimizing the cost. A control strategy to optimize the standby efficiency of the interfacing multi-phase DC-DC converter is also proposed, which is based on power-controlled variation of the number of active phases. Many other application-dependent topologies for the hybrid storage systems are proposed, that help meeting each application's particular requirements while optimizing its performances and cost. A comparative cost evaluation, realized in the context of a stand-alone photovoltaic home application, shows that in addition to its inherent environmental advantages, hydro-pneumatic storage is cost-effective compared to lead acid battery storage.

The combination of two or more different energy storage technologies in a hybrid energy storage system appears to be advantageous if the combination of technologies fulfils the storage purpose better than a single storage technology. Herein, requirements regarding the performance of the storage system are determined for a hybrid compensation system for provision of system services in medium‐voltage grids. In hybrid storage systems, a common control system ensures that the present storage technologies are utilized in such a way that total storage efficiency is highest and that usability of the application is maximized. The control system of a hybrid storage system consisting of a redox flow battery and high‐speed flywheel storage is introduced and evaluated.

Various energy harvesting technologies have been employed to extend the lifetime of wireless sensor networks. However, the network lifetime is still limited by the cycle life of rechargeable batteries (RBs) for the traditional RB-only storage system. Alternatively, supercapacitors (SCs) have extremely long cycle life-on the order of millions of cycles. A hybrid storage system combining RB and SC can leverage the complementary strengths of RB and SC and therefore significantly extend the lifetime of wireless sensors. This paper presents a generalized model of energy harvesting and hybrid storage system in order to evaluate the performance of hybrid energy storage systems (HESS) with different energy source and consumer profiles. Since high energy conversion efficiency is desired for most energy harvesting systems in addition to the lifetime, the system performance is analyzed in terms of two metrics: wireless sensor lifetime and energy conversion efficiency. A tradeoff is usually observed between the two performance metrics. However, under certain circumstances some specific HESS configurations can outperform the RB-only storage system in terms of both metrics. The results also suggest that an adaptive HESS that dynamically configures the storage devices based on the energy source and consumer profiles may have better performance comparing with a fixed HESS configuration.

Peer-to-peer (P2P) storage is a promising technology to provide users with cheap and online persistence. However, due the instability of these infrastructures, P2P storage systems must introduce redundancy in order to guarantee a reliable storage service. Besides, they need data repair algorithms to maintain this redundancy in front of permanent node departures. To ensure that such repairs can always be run, existing P2P storage systems aim to maintain 100% data availability. Unfortunately, this solution seems to overkill in preventing data loses, introducing network and data overheads.

Abstract-Systems adopting renewable energy, wind-solar microgrids are gaining popularity. They are clean and renewable but volatile too, creating an issue to maintain the system efficient and stable. This research comes up with a smart control framework for regulating wind-solar microgrids using hybrid storage systems-battery and supercapacitors. The aim is to balance power demand and supply, enhance the quality of power, and better utilize available energy. With active control over when and how the storage systems are deployed, pressure is relieved on the grid, and power spikes are minimized. Tests under varied operating conditions reveal that this method can provide voltage and frequency stability and enhance utilization of both energy sources and storage devices. Keywords: Microgrid, Wind power, Solar power, Hybrid storage system, Energy management, Power quality, Control strategy, Integration of renewable power

Erasure codes are recently used in real-world distributed storage systems (DSSs) such as Google File System,Microsoft Azure Storage, and Facebook HDFS-RAID for data reliability. When designing erasure codes for DSSs, special attention is given to the associated costs of data handling operations such as repair or update. For example, locally repairable codes (LRC) are designed and used in DSSs to allow for low-cost repair of failed blocks. Update complexity (defined as the number of blocks that need to be updated when an information block is changed) is yet another design parameter. This parameter can be seen as a measure of the computation, I/O and networking costs associated with updating an information block in a DSS. Since information is frequently updated by many applications, lowering update complexity can result in lower power consumptions in DSSs. In this work, we study the update complexity of LRCs. Based on our study, we propose an improvement over the LRC used by Facebook HDFS-RAID. Keeping the same code parameters including length, storage overhead, minimum distance and cost of repair (locality), we improve the update complexity by more than 16%. Moreover, we show that with these parameters achieving a lower update complexity is impossible.

The most important factors that need to be taken into consideration when designing a distributed storage system are data availability, reliability and access performance. This paper introduces an architecture to meet such goals by following the framework of cloud-of-clouds, where a complex system of clouds intercommunicate to create a robust and reliable cloud system. The optimal design of this architecture encompasses a general framework for achieving effective data reliability and availability, as well as fault tolerance. A simulation tool was developed to demonstrate the concept and functionality of the proposed architecture, testing its effectiveness in providing a reliable, available, and fault-tolerant cloud storage system. Simulation results proves that such architecture met the set goals to achieve the previously mentioned factors.