Hybrid system analysis and design for scale-out storage environments

This paper is motivated by a global online satellite images distribution system operated at the Earth Resources Observation and Science (EROS) center of the U.S Geological Survey. Fundamental objectives of EROS include, but are not limited to, building high-speed and cost-effective massive data processing and storage systems to support online satellite images distribution. Hybrid storage systems -- containing solid-state drives (SSD), hard disks (HDD), and tapes -- can provide an ideal data storage solution for a wide variety of data processing centers like EROS. Large-scale hybrid storage systems will become increasingly popular in the next few years for the following two reasons. First, highly accessed storage objects in a hybrid storage system can be prefetched and cached to high-speed storage components such as solid-state drives. SSD-based hybrid storage system can provide large storage capacity, high I/O performance and data reliability. Second, hybrid storage systems are cost-effective, because inexpensive tapes help in increasing storage capacities at very low cost. Transferring data back and forth among SSDs, HDDs, and tapes plays a critical role in achieving high I/O performance. Thus, we proposed data mining algorithms that can judiciously prefetch data. Our analytical model and the experimental results reveals that our data mining prefetching algorithm increase the performance of the hybrid storage systems.

SummaryHybrid storage systems that consist of flash‐based solid state drives (SSDs) and traditional disks are now widely used. In hybrid storage systems, there exists a two‐level cache hierarchy that regard dynamic random access memory (DRAM) as the first level cache and SSD as the second level cache for disk storage. However, this two‐level cache hierarchy typically uses independent cache replacement policies for each level, which makes cache resource management inefficient and reduces system performance. In this paper, we propose a novel adaptive multi‐level cache (AMC) replacement algorithm in hybrid storage systems. The AMC algorithm adaptively adjusts cache blocks between DRAM and SSD cache levels using an integrated solution. AMC uses combined selective promote and demote operations to dynamically determine the level in which the blocks are to be cached. In this manner, the AMC algorithm achieves multi‐level cache exclusiveness and makes cache resource management more efficient. By using real‐life storage traces, our evaluation shows the proposed algorithm improves hybrid multi‐level cache performance and also increases the SSD lifetime compared with traditional multi‐level cache replacement algorithms. Copyright © 2015 John Wiley & Sons, Ltd.

The need for performance has motivated the increasing adoption of solid-state drives (SSD) both in personal computers and data centers. However, the high cost per gigabyte and reduced lifetime hinder the thorough replacement of hard disk drives (HDD) with SSDs. To mitigate these issues, several architectures have been conceived based on hybrid storage systems, but performance and dependability models have not been proposed to better assess such different architectures. This paper presents an approach based on stochastic models for performance and availability modeling of hybrid storage systems, using stochastic Petri nets (SPN) and reliability block diagrams (RBD). A case study based on OpenStack Swift platform is adopted to demonstrate the feasibility of the proposed approach.

Solid State Drives (SSD) are integrated together with Hard Disk Drives (HDD) in Hybrid Storage Systems (HSS) for Cloud environment. When it comes to storing data, some placement strategies are used to find the best location (SSD or HDD). These strategies should minimize the cost of data placement while satisfying Service Level Objectives (SLO). This paper presents two Cost based Object Placement Strategies (COPS) for DBaaS objects in HSS: a Genetic based approach (G-COPS) and an ad-hoc Heuristic approach (H-COPS) based on incremental optimization. While G-COPS proved to be closer to the optimal solution in case of small instances, H-COPS showed a better scalability as it approached the exact solution even for large instances (by 10% in average). In addition, H-COPS showed small execution times (few seconds) even for large instances which makes it a good candidate to be used in runtime. Both H-COPS and G-COPS performed better than state-of-the-art solutions as they satisfied SLOs while reducing the overall cost by more than 40% for problems of small and large instances.

Flash-based Solid State Drive (SSD) has been productively shipped and deployed in large scale storage systems. However, a single flash-based SSD cannot satisfy the capacity, performance and reliability requirements of the modern storage systems that support increasingly demanding data-intensive computing applications. Applying RAID schemes to SSDs to meet these requirements, while a logical and viable solution, faces many challenges. In this article, we propose a Hybrid Parity-based Disk Array architecture (short for HPDA), which combines a group of SSDs and two hard disk drives (HDDs) to improve the performance and reliability of SSD-based storage systems. In HPDA, the SSDs (data disks) and part of one HDD (parity disk) compose a RAID4 disk array. Meanwhile, a second HDD and the free space of the parity disk are mirrored to form a RAID1-style write buffer that temporarily absorbs the small write requests and acts as a surrogate set during recovery when a disk fails. The write data is reclaimed to the data disks during the lightly loaded or idle periods of the system. Reliability analysis shows that the reliability of HPDA, in terms of MTTDL (Mean Time To Data Loss), is better than that of either pure HDD-based or SSD-based disk array. Our prototype implementation of HPDA and the performance evaluations show that HPDA significantly outperforms either HDD-based or SSD-based disk array.

Optimal hybrid pumped hydro-battery storage scheme for off-grid renewable energy systems

The performance gap between processor and storage device has continuously increased during the past few decades. The gap is further exacerbated recently because applications are becoming more data-intensive in both industry and academia. Traditional storage devices, such as hard disk drives (HDD), fail to keep up with the paces of this growth. A known solution is to use solid state drives (SSD) as fast storage. Due to high cost of SSD, data and supercomputing centers usually adopt a hybrid storage system, which consists of a combination of HDD and SSD I/O servers. However, hybrid I/O and storage systems have increased the complexity, making SSD often underutilized. The configuration and utilization of HDD/SSD hybrid systems is a lasting phenomenon. In this study, we propose a high performance hybrid parallel I/O and storage simulator, HPIS3. As a co-design tool, HPIS3 is capable of simulating a variety of parallel storage systems, especially under hybrid scenarios. The experimental results show that the lowest error rate is 2%, and the average is 11.98%.

Following a conventional design principle that pays more fast-CPU-cycles for fewer slow-I/Os, popular software storage architecture Linux Multiple-Disk (MD) for parity-based RAID (e.g., RAID5 and RAID6) assigns one or more centralized worker threads to efficiently process all user requests based on multi-stage asynchronous control and global data structures, successfully exploiting characteristics of slow devices, e.g., Hard Disk Drives (HDDs). However, we observe that, with high-performance NVMe-based Solid State Drives (SSDs), even the recently added multi-worker processing mode in MD achieves only limited performance gain because of the severe lock contentions under intensive write workloads. In this paper, we propose a novel stripe-threaded RAID architecture, StRAID, assigning a dedicated worker thread for each stripe-write (one-for-one model) to sufficiently exploit high parallelism inherent among RAID stripes, multi-core processors, and SSDs. For the notoriously performance-punishing partial-stripe writes that induce extra read and write I/Os, StRAID presents a two-stage stripe write mechanism and a two-dimensional multi-log SSD buffer. All writes first are opportunistically batched in memory, and then are written into the primary RAID for aggregated full-stripe writes or conditionally redirected to the buffer for partial-stripe writes. These buffered data are strategically reclaimed to the primary RAID. We evaluate a StRAID prototype with a variety of benchmarks and real-world traces. StRAID is demonstrated to outperform MD by up to 5.8 times in write throughput.

Flash memory has lately been used as a cache located between the system main memory and the magnetic hard drives in order to create robust and cost effective hybrid storage systems. The reason comes from the growing density of Solid State Devices (SSDs) at lower prices with main advantage of high random read efficiency compared to magnetic hard drives. When predicting the performance of such hybrid storage systems, it is inevitable to study the trade-off in selecting the different storage elements such as the main memory and the SSD cache relative to the capacity of the magnetic hard drive. The parameters of such prediction model are determined based on the application that the storage system would serve. In this paper, a prediction model that uses experimental evaluation of a hybrid storage system and real applications/benchmarks is used. The model utilizes parameters of both the storage system and applications in order to predict system performance based on metrics that are commonly used in storage system evaluation. The model allows the designer to select the best hybrid storage system parameters that satisfy certain application performance requirements. The model is highly accurate with a minimal error and a high prediction confidence level (95%) in reference to the experimental data collected from real applications using the proposed SSD-aware hybrid storage system.

The complementarity between energy resources can influence the performance of hybrid generation and storage systems, and can also decisively influence their design. Renewable resources may have intermittent characteristics that make the study of the influence of complementarity on the performance of hybrid systems quite difficult. The establishment of a performance limit of hybrid systems based on renewable resources and the study of the effects of complementarity considering this limit can provide interesting results. This performance limit can be established with an idealization of the mathematical functions describing the energy availability of the explored renewable resources. This article presents a method for analyzing the performance of hybrid systems based on complementary resources. The method allows to evaluate the influence of different levels of complementarity between the exploited resources on the cost of energy and capacity shortage. Utilizing idealized energy availability, the result sets a performance limit.•A method to evaluate the impact of complementarity on the performance and reliability of hybrid systems.•The energy availabilities of the renewable sources are idealized and allow the characterization of a limit of performance.•Different levels of complementarity can be related with design parameters of hybrid energy and storage systems.

It is critical to reduce the electric energy consumption of information systems to realize green societies. Applications like database and web applications take usage of data in storages systems of servers. In this paper, we consider RAID storage systems which are composed of multiple drives like hard disk drives (HDDs) and solid state drives (SSDs). Types of RAID storage systems, RAID0, RAID10(1+0), and RAID5 are considered in this paper. The performance and reliability of RAID storage systems are so far studied by many researchers. The more number of storage drives are possibly in parallel accessed in the RAID storage systems, the more amount of electric energy is consumed while the higher reliability and availability are supported. The electric energy consumption of the RAID storage systems to read and write data is so far not discussed. In this paper, we measure the power consumption of RAID storage systems and time to read and write data in the storage systems in order to make a power consumption model of a storage system. We make clear how much energy each type of RAID storage system consumes to sequentially and randomly read and write data through experiment in this paper.

Automated tiered storage with fast memory and slow flash storage (ATSMF) is a hybrid storage system located between non-volatile memories (NVMs) and solid state drives (SSDs). ATSMF reduces the average response time for input-output (IO) accesses by migrating concentrated IO-access areas from an SSD to an NVM. However, the current ATSMF implementation cannot sufficiently reduce the average response time because of the bottleneck caused by the Linux brd driver, which is used for the NVM access The response time of the brd driver is more than ten times longer than memory-access speed. To sufficiently reduce the average response time, we developed a block-level driver for an NVM called a two-mode (2M) memory driver. This driver has a .map-access mode and direct-access mode to reduce the response time while maintaining compatibility with the Linux device-mapper framework. The direct-access mode has a drastically lower response time than the Linux brd driver because the ATSMF driver can directly execute the IO-access function of the 2M memory Experimental results indicate that ATSMF using the 2M memory driver reduced the IO access response time to less than that of ATSMF using the Linux brd driver in most cases.

This paper presents an experimental evaluation phase carried out on ZEBRA batteries and Electrical Double Layer Capacitors (EDLCs) based storage system, which is able to power electric urban mini-busses. The particular analyzed case study is focused on a power train, specifically designed for electric urban road applications, composed by a fixed ratio gear box connected to an asynchronous 70 kW electric drive, which is supplied by the electric parallel of two 20 kWh Zebra batteries and a 63 F super-capacitors bank. The power train is studied by means of a laboratory 100 kW regenerative electric brake, provided with speed and torque controls, in order to experimentally measure the performance of the vehicle under study, on both steady state and dynamic operative conditions. The energy flux management of the hybrid storage system is preliminarily evaluated and tested in a simulation environment and then directly embedded in the power converters, interposed between storage systems and traction electric motor, to run experiments on the whole vehicle power train. In this way the combination of the two technologies of storage systems are tested in laboratory under various working conditions and with a different contribution of each storage device, in terms of electric energy and power. The obtained experimental results shows, on one side the effectiveness of the control strategies, evaluated to balance energy and power coming from the storage system to supply the traction electric drive, and on the other side the identification of a methodology to evaluate the performance of innovative hybrid storage systems, when powering road urban transportation vehicles.

In recent years, hybrid storage has gradually become a hotspot in the research of data storage owing to its high-performance and low cost. An OpenStack-based hybrid storage system is presented in this paper. According to the characteristics, the data is divided into small data, big data and temporary data in this hybrid storage system; meanwhile a storage strategy, combining database storage system, the virtual file system and servers file system, is designed. In the application of iCampus project, this proposed hybrid storage system shows better performance and higher efficiency than the traditional single storage systems.

Abstract-Systems adopting renewable energy, wind-solar microgrids are gaining popularity. They are clean and renewable but volatile too, creating an issue to maintain the system efficient and stable. This research comes up with a smart control framework for regulating wind-solar microgrids using hybrid storage systems-battery and supercapacitors. The aim is to balance power demand and supply, enhance the quality of power, and better utilize available energy. With active control over when and how the storage systems are deployed, pressure is relieved on the grid, and power spikes are minimized. Tests under varied operating conditions reveal that this method can provide voltage and frequency stability and enhance utilization of both energy sources and storage devices. Keywords: Microgrid, Wind power, Solar power, Hybrid storage system, Energy management, Power quality, Control strategy, Integration of renewable power