Data Migration in Heterogeneous Storage Systems

We have proposed a distributed storage system which aims to utilize storage fragments distributed in network nodes without administrators' maintenance cost. Even the performance of storage devices are not uniform, autonomously data block migration helps to achieve enhancing the whole I/O performance of the storage system. However, we have considered all the storage devices to be reliable equally. On this assumption, migration of an important data block to unstable storage might cause degradation of the availability of the data block. In this paper, we propose a scheme to choose the destination of data block migration by evaluating the nodes' reliability along with the performance of storage devices. This scheme aims to improve system stability by preventing to locate a critical data onto an unstable storage device. The storage nodes which lack stability are also utilized for storing a data block which can be easily regenerated or duplicated. The results of the preliminary investigation show that a large number of warning logs from Machine Check Architecture may suggest a sign of unexpected system shutdown.

In the existing large-scale meteorological data storage systems, the real-time of meteorological data applications is poor; transmission of data process is opaque; meteorological data sharing is complex and self-adaptability is bad; the capability of data quality assurance is not good. For these problems, this paper presents a solution of large-scale meteorological data storage system. This paper also presents a consistent hierarchical Hash data layout algorithm. It used to solve transmission and re-layout problem in the system among meteorological data, storage devices, content transmission network and key data buffer devices. And we test this algorithm's fairness, efficiency and adaptability by experiment.

Distributed storage systems provide large-scale data storage services, yet they are confronted with frequent node failures. To ensure data availability, a storage system often introduces data redundancy via replication or erasure coding. As erasure coding incurs significantly less redundancy overhead than replication under the same fault tolerance, it has been increasingly adopted in large-scale storage systems. In erasure-coded storage systems, degraded reads to temporarily unavailable data are very common, and hence boosting the performance of degraded reads becomes important. One challenge is that storage nodes tend to be heterogeneous with different storage capacities and I/O bandwidths. To this end, we propose FastDR, a system that addresses node heterogeneity and exploits I/O parallelism, so as to boost the performance of degraded reads to temporarily unavailable data. FastDR incorporates a greedy algorithm that seeks to reduce the data transfer cost of reading surviving data for degraded reads, while allowing the search of the efficient degraded read solution to be completed in a timely manner. We implement a FastDR prototype, and conduct extensive evaluation through simulation studies as well as testbed experiments on a Hadoop cluster with 10 storage nodes. We demonstrate that our FastDR achieves efficient degraded reads compared to existing approaches.

The data migration problem is to compute an efficient plan for moving data stored on devices in a network from one configuration to another. It is modeled by a transfer graph, where vertices represent the storage devices, and the edges represent the data transfers required between pairs of devices. Each vertex has a non-negative weight, and each edge has unit processing time. A vertex completes when all the edges incident on it complete; the constraint is that two edges incident on the same vertex cannot be processed simultaneously. The objective is to minimize the sum of weighted completion times of all vertices. Kim (Journal of Algorithms, 55:42-57, 2005) gave an LP-rounding 3-approximation algorithm. We give a more efficient primal-dual algorithm that achieves the same approximation guarantee, which can be extended to yield a 5.83-approximation for arbitrary processing times. We also study a variant of the open shop scheduling problem. This is a special case of the data migration problem in which the transfer graph is bipartite and the objective is to minimize the completion times of edges. We present a simple algorithm that achieves an approximation ratio of ${\sqrt{2}}$ ≈ 1.414, thus improving the 1.796-approximation given by Gandhi et al. (ACM Transaction on Algorithms, 2(1):116-129, 2006). We show that the analysis of our algorithm is almost tight.

In 1998, Ian Foster and Carl Kesselman, together with thirty distinguished experts in high-performance computing and networking, laid the foundations of a new computing model called GRID. Their vision was introduced in the book, The GRID: Blueprint for a New Computing Infrastructure (see review, SCPE Vol. 3, No. 3 ). The editors wrote: … A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities… allowing new classes of applications to emerge… as yet, we have only a preliminary understanding of what these new applications will look like. Over the next five years, Grid computing became the hottest R&D field. Academia and national labs built experimental infrastructures, new startups were established to develop Grid technologies, and the HPC giant vendors joined the party. Each one pushed in a different direction based on what and how he believed Grid computing should look. It was only a matter of time until people started to ask what the Grid is all about. A survey of ten Grid experts produced ten different answers. The question remains without a clear answer, leaving many people confused. Ian Foster picked up the gloves. In his article, What is the GRID? A three point checklist (GRIDtoday, July 22, 2002: Vol. 1, No. 6), he redefines the Grid model to reflect the trades and changes in the field during the past five years. According to Foster, a Grid is a system that: Coordinates resources that are not subject to centralized control. Uses standard, open, general-purpose protocols and interfaces. Delivers a nontrivial quality of service. But Foster's new definition could not encompass all of the Grid's approaches. Moreover, Foster explained by example what the Grid is not: A cluster management system such as Sun's Sun Grid Engine, Platform's Load Sharing Facility, or Viridian's Portable Batch System can, when installed on a parallel computer or local area network, deliver quality of service guarantees and thus constitutes a powerful Grid resource. However, such a system is not a Grid itself, due to its centralized control of the hosts that it manages Within a week the responses started to come in. The first one was from Wolfgang Gentzsch, Director of Grid computing at Sun Microsystems (GRIDtoday, August 5, 2002: Vol. 1, No. 8). Gentzsch wrote: I very much like Ian's two earlier definitions. Combined, they may provide a very generic Grid definition. The new checklist, however, reduces the potential of a wide and rich variety of Grids, with different architectures, operational models, and applications, and may miss the chance to becoming widely accepted as a standard definition of the Grid. Why do we need a definition for the Grid? Can the Grid be defined? Is a definition is necessary to the Grid's development? The Grid is an open vision and thus cannot be defined. An attempt to define the Grid can only reach the foreseen technology horizon, and this horizon is only one to two years ahead. Grid computing is a paradigm in shift. New architectures and standards are changing the shape and direction of the Grid's development. In the past year one such prominent architecture was Open Grid Services Architecture (OGSA). The OGSA is the first attempt to bridge the gap between two cyberspace worlds: the Internet and the Grid. The Internet and the Grid are two different computing models. There are people who believe that the Grid is the next generation of the Internet and the World Wide Web, saying that the Great Global Grid is still many years away. Others state that the Grid will complement, not replace, the Internet as we know it. In 1998, when the Grid blueprint was published, a new Internet language appeared called eXtensible Markup Language (XML). XML is a development tool for a new Internet computing model known as Web Services. At that time, nobody understood that there is a relationship between Web Services and the Grid. Three years past before Steve Tuecke from Argonne National Laboratory started to write a spec for Grid Services. Later, people from IBM joined the mission and together they created OGSA. The OGSA specification outlines interfaces to grid computing software that comply with Web Services standards. If adopted, Grid services such as job scheduling, authentication, failure-detection, staging of applications and data, and migration of applications and data, will all be accessed through standard Web Services architecture. OGSA is basically where Grid meets Web Services and maybe the first step towards full integration of the Internet and the Grid. The Internet appeared a decade ago. In the beginning, the Net offered primitive applications: a mailing system, ftp services, and a web of home pages. Nobody asked, What is the Internet? The years passed and new emerging Internet technologies were developed. Today we can talk on the phone over the Net, chat with friends, and video conference with colleagues on the other side of the world. New computing and business models were developed: e-commerce, e-learning, e-science, and e-you-name-it. And nobody asks what the Internet is. The Grid is a super-model, a bag of networking models associated with many types of architectures, software packages, applications, and standards. Today, when people talk about the Grid they mention Data Grids, Science Grids, and Campus Grids. And at the same time they mention Farm computing, Peer-to-peer computing, and Utility computing, among others. Ten years from now many of today's technologies will disappear and new ones will be developed. I don't know how Grid computing will look a decade from now, but I know that we will have g-commerce, g-learning, g-science, and g-chat. Nobody will ask what the Grid is. It will be obvious, just as the Internet is today. Ami Marowka The Computer Aided Design Laboratory School of Engineering and Computer Science Hebrew University of Jerusalem

New business models, constant technological progress, as well as ever-changing legal regulations require that companies replace their business applications from time to time. As a side effect, this demands for migrating the data from the existing source application to a target application. Since the success of the application replacement as a form of IT maintenance is contingent on the underlying data migration project, it is crucial to accomplish the migration in time and on budget. This however, calls for a stringent data migration process model combined with well-defined quality assurance measures. The paper presents, first, a field-tested process model for data migration projects. Secondly, it points out the typical risks we have frequently observed in the course of this type of project. Thirdly, the paper provides practice-based testing and quality assurance techniques to reduce or even eliminate these data migration risks.

I/O performance for very large-scale storage systems has become more and more important as the need for storage has grown dramatically. Replication is one of the most important approaches to improve the I/O performance of massive distributed storage system. The storage node to which the Read/Write request is directed is the critical issue when multiple replicas exist in some different storage nodes simultaneously. This paper proposes and evaluates a novel master OSD model, called Dynamic Selectable Master OSD (DS- MOSD), which allows client to direct the Read/Write request to the optimized OSD. This approach has the salient advantage of simultaneously improving I/O performance and balancing the loads of all the OSDs. This paper first describes the architecture of Master OSD and characterizes the advantages of this model contrasting to the pseudo random hash function. Then an improved model, which is considered as DS-MOSD, is presented and evaluated. The experimental results show that the load unbalance degree and network overheads of DS-MOSD model are smaller than that of Fixed MOSD model, while the success probability of Write/Update transactions of DS-MOSD model is larger than that of Fixed MOSD model.

As one database offloading strategy, elastic key-value stores are often introduced to speed up the application performance with dynamic scalability. Since the workload is varied, efficient data migration with minimal impact in service is critical for the issue of elasticity and scalability. However, due to the new virtualization technology, real-time and low-latency requirements, data migration within cloud-based key-value stores has to face new challenges: effects of VM interference, and the need to trade off between the two ingredients of migration cost, namely migration time and performance impact. To fulfill these challenges, in this paper we explore a new approach to optimize the data migration. Explicitly, we build two interference-aware models to predict the migration time and performance impact for each migration action using statistical machine learning, and then create a cost model to strike a balance between the two ingredients. Using the load rebalancing scenario as a case study, we have designed one cost-aware migration algorithm that utilizes the cost model to guide the choice of possible migration actions. Finally, we demonstrate the effectiveness of the approach using Yahoo! Cloud Serving Benchmark (YCSB).

Purpose: To identify costs deriving from data migration of obsolete digital archives by measuring the workload, and to analyse migration-associated problems. Material and Methods: Two digital archives were used (DTL and MOD) and the capacity of these archives could no longer support the needs of the Medical Imaging Centre. The entire content of the DLT archive and selected data from the MOD archive were transferred to the current higher capacity (17 TB) tape archive. The running time of work processes was measured by self-reporting, and the cost of work was calculated. Results: The transfer of 43,096 studies required 314 working hours over the course of 15 months in total. The work was partly manual, partly automatic. The percentage of non-retrievable MOD images was 35. Less than 0.2% of the DLT image transfers failed due to incorrect patient or image data. The MOD – DLT transfer cost was six times higher per study than the DLT – DLT transmission cost. Conclusion: At present, data migration may be inevitable as the amount of data increases and technology advances. The data transfer proved to be labour intensive, with high fault sensitivity regarding the MOD archive. The cost of work of data migration was 0.4% of estimated digital archiving total yearly cost. Automated data migration is preferable.

International audience

Power-saving has become a central issue for well-configured SOC platforms. In particular, as a high percentage of the total energy is used by the storage systems, the cost effectiveness of data management is equally as important as reliability and availability. To address this issue, we propose the dynamic grid quorum as a method for reducing the power consumption of large-scale distributed storage systems. The basic principle of our approach is to skew the workload toward a small number of quorums. This can be realized using the following three techniques. First, our system allows reconfiguration by exchanging nodes without any data migration, so that high-capacity nodes can be reallocated to busier quorums. Second, for more effective skewing of the workload, we introduce the notion of dual allocation, which makes it possible to consider two distinct allocations in the same grid for write and read quorums. Finally, we present an optimization algorithm to find a pair of a strategy and an allocation of nodes, which minimizes power for a given system setting and its workload. We also demonstrate that the dynamic grid quorum saves, on average, 14---25% energy compared with static configurations, when the intensity of the total workload changes.

Traditionally, active storage devices execute custom application code on large amounts of data by utilizing the unused processing power of the storage nodes. For computation-intensive applications, the performance might be quite low due to insufficient processing power on the storage devices. In this paper we present a reconfigurable computing solution that can provide flexible, high-performance processing capabilities for the storage nodes. We have made a change over the existing active storage framework for object-based storage device by off loading computation-intensive request-driven services to a reconfigurable hardware accelerator. In the experiment, we use two applications (AES encryption and FFT) as the case studies, and the results show that the reconfigurable computing-based system can dramatically increase the performance comparing with the traditional software-based active storage systems.

Cluster-based storage systems are popular for data-intensive applications and it is desirable yet challenging to provide incremental expansion and high availability while achieving scalability and strong consistency. This paper presents the design and implementation of a self-organizing storage cluster called Sorrento, which targets data-intensive workload with highly parallel requests and low write-sharing patterns. Sorrento automatically adapts to storage node joins and departures, and the system can be configured and maintained incrementally without interrupting its normal operation. Data location information is distributed across storage nodes using consistent hashing and the location protocol differentiates small and large data objects for access efficiency. It adopts versioning to achieve single-file serializability and replication consistency. In this paper, we present experimental results to demonstrate features and performance of Sorrento using microbenchmarks, application benchmarks, and application trace replay.

Web 2.0 has been adopted by many as the best way for forming a collaborative framework e.g., sharing resources, experiences, information, knowledge and feedback. A collaborative framework for application to e-health is necessary to provide patients with the awareness that assists in improving their health. Moreover, collaborative framework can be used by physician to exchange experiences and discuss challenge cases. However, the use of Web 2.0 with healthcare applications is not simple as the use of Web 20 with other enterprise applications according to the privacy of healthcare applications, which requires high quality and security of data, availability of resources, maintainability of services, system security, and Quality of Services (QoS). To offer the required requirements, grid computing is proposed here. Grid computing supporting enterprise applications through offering massive resources through resources collaborative framework that is offering power computing, storage devices, and services. The use of grid computing by Web 2.0 requires robust model that is able to deploy, discover, invoke, and integrate resources in open standard format. Therefore, Service Oriented Architecture (SOA) is adopted as a model for managing the mixing between Web 2.0 and grid computing technologies. SOA for Web 2.0 and Grid Computing (SOAW2G) are used throughout this work to offer a fabric for e-health applications.

In relational and complex spreadsheets, relational and non-relational database models, high-speed data migration with scalable structure using visual Basic Excel programming language implementations with proposed migration model. One of the primary purposes behind making a point of interaction through the Visual Basic Application (VBA) is that most PC clients with logical preparation will currently know about Succeed and its significant capabilities (like arranging furthermore, plotting datasets). Even though Excel itself is in many cases utilized as an information storehouse by clients. The time factor, reliability, and credibility of migrating this data from one table to another through programming interfaces were measured using the link codes between the tables. In this paper, the migration and migration of homogeneous and heterogeneous data are investigated by using two types of different migration models of data to measure and match these data and the extent of their integration after migration. Specifies the target data for migration from the input tables in the Excel program to the target tables in larger databases. Furthermore, the two models (A, B) middleware provide an architecture that can be extended to support Relational database management systems (RDBMS) and other graphing databases. Experiments were performed using excel tables, both of which are related as source information bases, and as the data set for the source and target datasets, the migration time between these tables for the two models was calculated While retaining the same characteristics.