Designing a fully interactive video-on-demand server with a novel data placement and retrieval scheme

A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop

This paper examines the requirements for a large multimedia storage system, such as one for a movie-on-demand service. It presents the architecture for a storage system based on these requirements. The paper then presents a scheme for the organization, placement, and retrieval of multimedia data from the server. The data organization and retrieval schemes are analyzed and it is shown that their performance is superior to other schemes found in the literature.

This paper provides a data placement optimization approach for Coarse-Grained Reconfigurable Architecture (CGRA) based computing platform in order to simultaneously optimize the performance of CGRA execution and data transformation between main memory and multi-bank memory. To achieve this goal, we have developed a performance model to evaluate the efficiency of data transformation and CGRA execution. This model is used for comparing the performances difference when using different data placement strategies. We search for the optimal data placement method by firstly choosing the method which generates the best CGRA execution efficiency from the candidates who can generate the optimal data transformation efficiency. Then we choose the best data placement strategy by comparing the performance of the selected strategy with the one generated through existing multi-bank optimization algorithm. Evaluation shows our approach is capable of optimizing the performance to 2.76x of state-of-the-art method when considering both data-transformation and CGRA execution efficiency.

Cloud computing emerges with high performance computing, massive data storage and easy access of the Internet. By deploying cloud computing, scientific workflows can be more cost-effective. During workflow execution, large volume data transmission may occur among multiple datacenters, hence incur large cost. Traditional approaches aim at finding data placement strategies for individual workflows only to reduce data transmission cost. However, workflows may share some datasets. Therefore, optimal data placement considering individual workflows in isolation is not necessarily optimal for the situation of multiple workflows as a whole. In this paper, by facilitation of Particle Swarm Optimization (PSO), we build a novel data transmission cost model for developing a new multi-datacenter cost-effective data placement strategy. The experimental results show that our strategy is much better than its traditional counterparts.

Scientific workflows in collaborative cloud environments are becoming more and more popular.There is an urgent need to address the problem of large amount of data transfer across geo-distributed data centers during workflow execution.By utilizing data dependencies,we propose a two-stage data placement strategy and a task scheduling strategy for efficient workflow execution.With our strategy,the most related datasets can be placed into the same data center based on the data dependence between them at workflow build-time;then the tasks are scheduled to their most closely related data centers for execution and the newly-generated data sets are put into the data center that has the most dependency with them at workflow runtime.The experimental results show that the proposed strategy can significantly reduce the volume of data transfer among different data centers,and hence improve the performance of running scientific workflows and cut down the cost of doing science on the clouds as well.

As one of the most popular frameworks for large-scale analytics processing, Hadoop is facing two challenges: both applications and storage devices become heterogeneous. However, existing data placement and job scheduling schemes pay little attention to such heterogeneity of either application I/O requirements or I/O device capability, thus can greatly degrade system efficiencies. In this paper, we propose ASPS, an Application and Storage-aware data Placement and job Scheduling approach for Hadoop clusters. The idea is to place application data and schedule application tasks considering both application I/O requirements and storage device characteristics. Specifically, ASPS first introduces novel metrics to quantify I/O requirements of applications. Then, based on the quantification, ASPS places data of different applications to the preferred storage devices. Finally, ASPS tries to launch jobs with high I/O requirements on the nodes with the same type of faster devices to improve system efficiency. We have implemented ASPS in Hadoop framework. Experimental results show that ASPS can reduce the completion time of a single application by up to 36% and the average completion time of six concurrent applications by 27%, compared to existing data placement policies and job scheduling approaches.

MapReduce has become an important distributed processing model for large-scale data-intensive applications like data mining and web indexing. Hadoop-an open-source implementation of MapReduce is widely used for short jobs requiring low response time. The current Hadoop implementation assumes that computing nodes in a cluster are homogeneous in nature. Data locality has not been taken into account for launching speculative map tasks, because it is assumed that most maps are data-local. Unfortunately, both the homogeneity and data locality assumptions are not satisfied in virtualized data centers. We show that ignoring the data-locality issue in heterogeneous environments can noticeably reduce the MapReduce performance. In this paper, we address the problem of how to place data across nodes in a way that each node has a balanced data processing load. Given a dataintensive application running on a Hadoop MapReduce cluster, our data placement scheme adaptively balances the amount of data stored in each node to achieve improved data-processing performance. Experimental results on two real data-intensive applications show that our data placement strategy can always improve the MapReduce performance by rebalancing data across nodes before performing a data-intensive application in a heterogeneous Hadoop cluster.

Abstract. As distributed cloud storage technology has become an essential outsourced storage solution, the strategy for mining deeper into data placement has become increasingly significant. Previous analyses of the data predominantly employed existing methods, failing to account for the depth and breadth of data mining concerns. As a result, the outcomes have been imprecise and less than optimal in some instances. In this paper, we will be pursuing two objectives: optimizing the disk space utilization of edge servers and minimizing data transmission latency rates. A novel heuristic data placement strategy based on improved multi-objective whale optimization algorithm for data placement (IWBP) is proposed. To increase the available options for data replica placement, we propose the incorporation of an adaptive weight factor as well as a Lévy flight factor. Additionally, we will use a Gini coefficient to quantify the level of system load balancing. We compared our data placement strategy with current mainstream strategies and demonstrated experimentally that our approach outperforms those strategies in both load balancing and transmission efficiency.

In this data era, massive volumes of data are being generated every second in variety of domains such as Geoscience, Social Web, Finance, e-Commerce, Health Care, Climate modelling, Physics, Astronomy, Government sectors etc. Hadoop has been well-recognized as de factobig data processing platform that have been extensively adopted, and is currently widely used, in many application domains processing Big Data. Even though it is considered as an efficient solution for such complex query processing, it has its own limitation when the data to be processed exhibit interest locality. The data required for any query execution follows grouping behavior wherein only a part of the Big-Data is accessed frequently. During such scenarion, the time taken to execute a queryand return results, increases exponentially as the amount of data increases leading to much waiting time for the user. Since Hadoop default data placement strategy (HDDPS) does not consider such grouping behavior, it does not perform efficiently resulting in lacunas such as decreased local map task execution, increased query execution time etc. Hence proposed an Optimal Data Placement Strategy (ODPS) based on grouping semantics. In this paper we experiment the significance oftwo most promising clustering techniques viz. Hierarchical Agglomerative Clustering (HAC) and Markov Clustering (MCL) in grouping aware data placement for data intensive applications having interest locality. Initially user access pattern is identified by dynamically analyzing history log.Then both clustering techniques (HAC & MCL) are separately applied over the access pattern to obtain independent clusters. These clusters are interpreted and validated to extract the Optimal Data Groupings (ODG). Finally proposed strategy reorganizes the default data layouts in HDFSbased on ODG to achieve maximum parallel execution per group subjective to Load Balancer and Rack Awareness. Our proposed strategy is tested in 10 node cluster placed in a multi rack with Hadoop installed in every node deployed in cloud platform. Proposed strategy reduces the query execution time, significantly improves the data locality and has proved to be more efficient for massive datasets processing in heterogeneous distributed environment. Also MCL shows a marginal improved performance over HAC for queries exhibiting interest localities.

Cloud-based Online Social Networks (OSNs) make it easier for geographically dispersed users to communicate with each other. These users not only demand to quickly access their own data but also hope to access their friends' data with low latency. In order to solve the problem, it is necessary to design a replica placement strategy to manage data on large-scale social networks and reduce the data storage costs while meeting the access latency requirement. In this paper, we propose a novel genetic algorithm-based data placement strategy to find an optimal number of replicas for each user's data and their optimal location. The method can reduce the inter-server traffic load across servers and ensure that users can access data in a tolerable time. Experiments with real Facebook dataset demonstrate that our data placement strategy can significantly reduce the cost of data storage and inter-server traffic.

The traditional method of centralized backup suffers from single point of failure and scalability problem. It is therefore beneficial to carry out the backup service over geographically distributed networks. Hence, redundant copies of data stored in such networks are essential to ensure the availability and reliability of the backup data. The main objective of this work is to balance and evenly distribute large amounts of backup data for efficient use of disks as well as network bandwidth which leads to faster data access and recovery. We develop an efficient method for data placement and replication. Our method has several attractive features: (1) Balanced and efficient data placement in a distributed storage resource environment. (2) Redistributing as few replicas as possible in the event of storage node addition or failure/removal. (3) There is no central mapping table so that the search performance becomes more efficient. We demonstrate the effectiveness of our method through theoretical analysis and simulation results.

Data placement and parallel execution strategies are crucial design issues for high performance transaction processing in a parallel database machine. Declustering (horizontal partitioning) is a most promising solution for data placement. We analyze the trade-offs of three alternative execution strategies in the context of declustered databases. Then we focus on the strategy of executing programs at their data repositories which is highly parallel and offers many opportunities for query optimization. Finally, we elaborate on the implications of this strategy on program loading, associative routing, and decentralized concurrency control.

We consider the problem of optimizing the execution of data-intensive scientific workflows in the Cloud. We address the problem under the following scenario. The tasks of the workflows communicate through files; the output of a task is used by another task as an input file and if these tasks are assigned on different execution sites, a file transfer is necessary. The output files are to be stored at a site. Each execution site is to be assigned a certain percentage of the files and tasks. These percentages, called target weights, are pre-determined and reflect either user preferences or the storage capacity and computing power of the sites. The aim is to place the data files into and assign the tasks to the execution sites so as to reduce the cost associated with the file transfers, while complying with the target weights. To do this, we model the workflow as a hypergraph and with a hypergraph-partitioning-based formulation, we propose a heuristic which generates data placement and task assignment schemes simultaneously. We report simulation results on a number of real-life and synthetically generated scientific workflows. Our results show that the proposed heuristic is fast, and can find mappings and assignments which reduce file transfers, while respecting the target weights.

Many data storage schemes have been proposed in the past for keeping dependable data storage, but they are not designed for Unattended Wireless Sensor Networks (UWSNs). When applying these schemes to UWSNs, most of them have limitations such as high storage requirement, transmission cost, and not resilient to a large number of node compromises. To address the problem of data storage, transmission cost, and resilience of node compromise in UWSNs, we propose a novel Homomorphic Encryption and Homomorphic Secret Sharing based scheme (HEHSS) to accomplish the goals of confidentiality, resilience to node compromises, reliability, and efficiency of storage and transmission. Our scheme exploits the homomorphic secret sharing scheme to enable sensor nodes encode sensed data into several shares, and distribute them to their neighbors. This property allows the mobile sink to reconstruct data even if some of the shares stored in neighbors of a sensor are compromised. Detailed analysis has been conducted to evaluate the scheme in terms of efficiency and reliability.

Distributed Data Storage and Retrieval Schemes in RPL/IPv6-Based Networks