Optimizing the data placement and transformation for multi-bank CGRA computing system

Cloud computing emerges with high performance computing, massive data storage and easy access of the Internet. By deploying cloud computing, scientific workflows can be more cost-effective. During workflow execution, large volume data transmission may occur among multiple datacenters, hence incur large cost. Traditional approaches aim at finding data placement strategies for individual workflows only to reduce data transmission cost. However, workflows may share some datasets. Therefore, optimal data placement considering individual workflows in isolation is not necessarily optimal for the situation of multiple workflows as a whole. In this paper, by facilitation of Particle Swarm Optimization (PSO), we build a novel data transmission cost model for developing a new multi-datacenter cost-effective data placement strategy. The experimental results show that our strategy is much better than its traditional counterparts.

A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop

In this data era, massive volumes of data are being generated every second in variety of domains such as Geoscience, Social Web, Finance, e-Commerce, Health Care, Climate modelling, Physics, Astronomy, Government sectors etc. Hadoop has been well-recognized as de factobig data processing platform that have been extensively adopted, and is currently widely used, in many application domains processing Big Data. Even though it is considered as an efficient solution for such complex query processing, it has its own limitation when the data to be processed exhibit interest locality. The data required for any query execution follows grouping behavior wherein only a part of the Big-Data is accessed frequently. During such scenarion, the time taken to execute a queryand return results, increases exponentially as the amount of data increases leading to much waiting time for the user. Since Hadoop default data placement strategy (HDDPS) does not consider such grouping behavior, it does not perform efficiently resulting in lacunas such as decreased local map task execution, increased query execution time etc. Hence proposed an Optimal Data Placement Strategy (ODPS) based on grouping semantics. In this paper we experiment the significance oftwo most promising clustering techniques viz. Hierarchical Agglomerative Clustering (HAC) and Markov Clustering (MCL) in grouping aware data placement for data intensive applications having interest locality. Initially user access pattern is identified by dynamically analyzing history log.Then both clustering techniques (HAC & MCL) are separately applied over the access pattern to obtain independent clusters. These clusters are interpreted and validated to extract the Optimal Data Groupings (ODG). Finally proposed strategy reorganizes the default data layouts in HDFSbased on ODG to achieve maximum parallel execution per group subjective to Load Balancer and Rack Awareness. Our proposed strategy is tested in 10 node cluster placed in a multi rack with Hadoop installed in every node deployed in cloud platform. Proposed strategy reduces the query execution time, significantly improves the data locality and has proved to be more efficient for massive datasets processing in heterogeneous distributed environment. Also MCL shows a marginal improved performance over HAC for queries exhibiting interest localities.

The paper discusses the design of a video-on-demand server that supports fully interactive operations such as play, fast forward search, and fast backward search, with a novel data placement and retrieval scheme. The main distinctions of this scheme are that (1) it requires neither extra disk bandwidth nor extra memory to support fast forward and fast backward searches; (2) it is based on a general two-level disk array architecture that exploits both parallelism and concurrency offered by fine-grain and coarse-grain disk array structures. >

Abstract. As distributed cloud storage technology has become an essential outsourced storage solution, the strategy for mining deeper into data placement has become increasingly significant. Previous analyses of the data predominantly employed existing methods, failing to account for the depth and breadth of data mining concerns. As a result, the outcomes have been imprecise and less than optimal in some instances. In this paper, we will be pursuing two objectives: optimizing the disk space utilization of edge servers and minimizing data transmission latency rates. A novel heuristic data placement strategy based on improved multi-objective whale optimization algorithm for data placement (IWBP) is proposed. To increase the available options for data replica placement, we propose the incorporation of an adaptive weight factor as well as a Lévy flight factor. Additionally, we will use a Gini coefficient to quantify the level of system load balancing. We compared our data placement strategy with current mainstream strategies and demonstrated experimentally that our approach outperforms those strategies in both load balancing and transmission efficiency.

MapReduce has become an important distributed processing model for large-scale data-intensive applications like data mining and web indexing. Hadoop-an open-source implementation of MapReduce is widely used for short jobs requiring low response time. The current Hadoop implementation assumes that computing nodes in a cluster are homogeneous in nature. Data locality has not been taken into account for launching speculative map tasks, because it is assumed that most maps are data-local. Unfortunately, both the homogeneity and data locality assumptions are not satisfied in virtualized data centers. We show that ignoring the data-locality issue in heterogeneous environments can noticeably reduce the MapReduce performance. In this paper, we address the problem of how to place data across nodes in a way that each node has a balanced data processing load. Given a dataintensive application running on a Hadoop MapReduce cluster, our data placement scheme adaptively balances the amount of data stored in each node to achieve improved data-processing performance. Experimental results on two real data-intensive applications show that our data placement strategy can always improve the MapReduce performance by rebalancing data across nodes before performing a data-intensive application in a heterogeneous Hadoop cluster.

We propose a new optimal data placement technique to improve the performance of MapReduce in cloud data centers by considering not only the data locality but also the global data access costs. We first conducted an analytical and experimental study to identify the performance issues of MapReduce in data centers and to show that MapReduce tasks that are involved in unexpected remote data access have much greater communication costs and execution time, and can significantly deteriorate the overall performance. Next, we formulated the problem of optimal data placement and proposed a generative model to minimize global data access cost in data centers and showed that the optimal data placement problem is NP-hard. To solve the optimal data placement problem, we propose a topology-aware heuristic algorithm by first constructing a replica-balanced distribution tree for the abstract tree structure, and then building a replica-similarity distribution tree for detail tree construction, to construct an optimal replica distribution tree. The experimental results demonstrated that our optimal data placement approach can improve the performance of MapReduce with lower communication and computation costs by effectively minimizing global data access costs, more specifically reducing unexpected remote data access.

One of the grand challenges in the design of portable/wearable electronics is to achieve optimal efficiency and flexibility in a tiny low power package. Coarse grained reconfigurable architectures (CGRAs) hold great promise for low power, high performance, and flexible designs for a domain of applications. CGRAs are very promising due to the ability to highly customize such architectures to an application domain. However, greater customization makes the mapping of applications onto these architectures very challenging. Good tools and fast, effective mapping algorithms are needed to support design space exploration for CGRAs. In particular, the mapping problem has been difficult to solve in a satisfying and general way. In this paper, we present an architectural design flow using crowdsourcing to provide mappings of benchmarks onto new architectures. We show that the crowd can provide high quality, reliable mappings, significantly outperforming our custom Simulated Annealing algorithm in almost all cases. We further show that the crowd can provide other types of feedback that are difficult to obtain from an automatic mapping algorithm. Our proof of concept cross-architectural study supports an 8Way or 4Way1Hop architecture as a top choice, concludes that a custom modification that constrains inputs and outputs consumes less energy but requires more area than its less constrained counterpart, and suggests that Stripe architectures are interesting to consider because they perform nearly as well as our mesh variants and may present a more straightforward mapping problem for the crowd or an automatic mapping algorithm.

As XML is increasingly being used in Web applications, new technologies need to be investigated for processing XML documents with high performance. Parallelism is a promising solution for structured document processing and data placement is a major factor for system performance improvement in parallel processing. This paper describes an effective XML document data placement strategy. The new strategy is based on a multilevel graph partitioning algorithm with the consideration of the unique features of XML documents and query distributions. A new algorithm, which is based on XML query schemas to derive the weighted graph from the labelled directed graph presentation of XML documents, is also proposed. Performance analysis on the algorithm presented in the paper shows that the new data placement strategy exhibits low workload skew and a high degree of parallelism.

Applications processing massive amount of data demand superior time-performance and data-storage capabilities. Many organizations are exploring Cloud Computing to manage such applications because of scalability and convenience of access from different geographic locations, whereas data security and privacy are few of the major concerns preventing them from fully embracing it. Data security while maintaining time-performance becomes an important consideration for designing data placement strategies. We first present the reader with a quick survey on the recent approaches and solutions in data placement oriented problems that address security concern and then characterize the performance for graph based algorithms that are practically realizable. We evaluated the performance of conventional strategies such as, Random, T-coloring and revisited these algorithms in the view of providing maximum security. We refer to our strategy as Data Security Preferential (DSP) data placement strategy and evaluated via rigorous performance evaluation tests to identify which strategy will best suit the requirements of the user on three state-of-the-art hierarchical cloud platforms viz., FatTree, ThreeTier and DCell and non-hierarchical cloud platforms.

Scientific workflows in collaborative cloud environments are becoming more and more popular.There is an urgent need to address the problem of large amount of data transfer across geo-distributed data centers during workflow execution.By utilizing data dependencies,we propose a two-stage data placement strategy and a task scheduling strategy for efficient workflow execution.With our strategy,the most related datasets can be placed into the same data center based on the data dependence between them at workflow build-time;then the tasks are scheduled to their most closely related data centers for execution and the newly-generated data sets are put into the data center that has the most dependency with them at workflow runtime.The experimental results show that the proposed strategy can significantly reduce the volume of data transfer among different data centers,and hence improve the performance of running scientific workflows and cut down the cost of doing science on the clouds as well.

Although benefits from caching in US HEP are well-known, current caching strategies are not adaptive i.e they do not adapt to changing cache access patterns. Newer developments such as the High-Luminosity - Large Hadron Collider (HL-LHC), Deep Underground Neutrino Experiment (DUNE), a steady move toward streaming readout based Data Acquisition systems (DAQs) will increase the data production exponentially and hence burden the storage, compute & network infrastructures. Moreover, existing caching frameworks are optimized to reduce latency, but not optimized for storage. This, in combination with limited cache capacities relative to total data, makes it difficult to achieve data locality. In this work, we present Machine Learning-aided (ML) caching strategies. Specifically, we first present a Long Short-Term Memory-based (LSTM)hourly and multi-step cache usage prediction. Second, we present an hourly file-level access prediction model based on CatboostRegressor. To date, most ML-based cache prediction strategies in HEP have focused on daily cache usage and limited works tackled hourly cache usage and even fewer strategies addressed hourly file-level access prediction. File-level access prediction allows for the design of intelligent prefetching and data placement strategies with fine-grained control. We validated our cache prediction strategies using data collected from SoCal MINI caches in August 2024. We are currently extending the WRENCH simulator to reflect the US HEP ecosystem at the storage, network and compute levels. We plan to deploy our cache prediction strategies into WRENCH and later perform extensive analysis with complex data access patterns and candidate infrastructure configurations.

As one of the most popular frameworks for large-scale analytics processing, Hadoop is facing two challenges: both applications and storage devices become heterogeneous. However, existing data placement and job scheduling schemes pay little attention to such heterogeneity of either application I/O requirements or I/O device capability, thus can greatly degrade system efficiencies. In this paper, we propose ASPS, an Application and Storage-aware data Placement and job Scheduling approach for Hadoop clusters. The idea is to place application data and schedule application tasks considering both application I/O requirements and storage device characteristics. Specifically, ASPS first introduces novel metrics to quantify I/O requirements of applications. Then, based on the quantification, ASPS places data of different applications to the preferred storage devices. Finally, ASPS tries to launch jobs with high I/O requirements on the nodes with the same type of faster devices to improve system efficiency. We have implemented ASPS in Hadoop framework. Experimental results show that ASPS can reduce the completion time of a single application by up to 36% and the average completion time of six concurrent applications by 27%, compared to existing data placement policies and job scheduling approaches.

Coarse-grained reconfigurable architecture (CGRA) is a promising parallel computing platform that provides high performance, high power efficiency and flexibility. However, for imperfect nested loops, the existing loop mapping methods often result in low execution performance and poor hardware utilization. To tackle this problem, this paper makes three contributions: 1) a highly effective and general approach to map imperfect loops on CGRA; 2) a global optimization strategy to search the optimal initiation intervals (IIs); 3) a powerful kernel compression method to reduce the oversized kernel. Experiment results show that our approach can reduce the total computing latency by 20.5, 58.5 and 73.2 percent compared to the state-of-the-art approaches on $2 \times 2$ , $4 \times 4$ and $8 \times 8$ CGRA respectively. Moreover, the compilation time and configuration context size is acceptable in practice.

As a result of the advancements in Industry 4.0, the amount of data collected within industries are continuously expanding to achieve an innovative environment within the industry by maximizing asset usage. Meanwhile, the redundancy rate is increasing in cloud storage, which has an impact on data storage and analysis. To lower the rate of redundancy, the proposed system comprises a Time series-based deduplication technique. In the Time series-based deduplication technique, the Adaptive Multi-Pattern Boyer Moore Horspool (AM-BMH) algorithm, and Merkle tree were used to produce time-series data. Another significant challenge is that the geographically distributed cloud system has encountered that the data placement methodology with high-priced transportation costs for data transmission. To overcome this issue, an optimal data placement strategy using Modified Distribution is proposed. Thus the proposed Time Series-based Deduplication and Optimal Data Placement Strategy (TDOPS) is found to be effective when compared with the existing system. The various parameters like space reduction, efficient retrieval, data transportation costs, and data transmission time are taken into the account in the cloud environment for an evaluation. The proposed scheme saves 98 percent of storage space, 55 percent computation overhead, and improves 60% of cloud storage efficacy.