IWBP: An Efficient Data Storage Method Based on Deep Data Mining and Improved Whale Algorithm

MapReduce has become an important distributed processing model for large-scale data-intensive applications like data mining and web indexing. Hadoop-an open-source implementation of MapReduce is widely used for short jobs requiring low response time. The current Hadoop implementation assumes that computing nodes in a cluster are homogeneous in nature. Data locality has not been taken into account for launching speculative map tasks, because it is assumed that most maps are data-local. Unfortunately, both the homogeneity and data locality assumptions are not satisfied in virtualized data centers. We show that ignoring the data-locality issue in heterogeneous environments can noticeably reduce the MapReduce performance. In this paper, we address the problem of how to place data across nodes in a way that each node has a balanced data processing load. Given a dataintensive application running on a Hadoop MapReduce cluster, our data placement scheme adaptively balances the amount of data stored in each node to achieve improved data-processing performance. Experimental results on two real data-intensive applications show that our data placement strategy can always improve the MapReduce performance by rebalancing data across nodes before performing a data-intensive application in a heterogeneous Hadoop cluster.

A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop

This paper provides a data placement optimization approach for Coarse-Grained Reconfigurable Architecture (CGRA) based computing platform in order to simultaneously optimize the performance of CGRA execution and data transformation between main memory and multi-bank memory. To achieve this goal, we have developed a performance model to evaluate the efficiency of data transformation and CGRA execution. This model is used for comparing the performances difference when using different data placement strategies. We search for the optimal data placement method by firstly choosing the method which generates the best CGRA execution efficiency from the candidates who can generate the optimal data transformation efficiency. Then we choose the best data placement strategy by comparing the performance of the selected strategy with the one generated through existing multi-bank optimization algorithm. Evaluation shows our approach is capable of optimizing the performance to 2.76x of state-of-the-art method when considering both data-transformation and CGRA execution efficiency.

The paper discusses the design of a video-on-demand server that supports fully interactive operations such as play, fast forward search, and fast backward search, with a novel data placement and retrieval scheme. The main distinctions of this scheme are that (1) it requires neither extra disk bandwidth nor extra memory to support fast forward and fast backward searches; (2) it is based on a general two-level disk array architecture that exploits both parallelism and concurrency offered by fine-grain and coarse-grain disk array structures. >

Cloud computing emerges with high performance computing, massive data storage and easy access of the Internet. By deploying cloud computing, scientific workflows can be more cost-effective. During workflow execution, large volume data transmission may occur among multiple datacenters, hence incur large cost. Traditional approaches aim at finding data placement strategies for individual workflows only to reduce data transmission cost. However, workflows may share some datasets. Therefore, optimal data placement considering individual workflows in isolation is not necessarily optimal for the situation of multiple workflows as a whole. In this paper, by facilitation of Particle Swarm Optimization (PSO), we build a novel data transmission cost model for developing a new multi-datacenter cost-effective data placement strategy. The experimental results show that our strategy is much better than its traditional counterparts.

In this data era, massive volumes of data are being generated every second in variety of domains such as Geoscience, Social Web, Finance, e-Commerce, Health Care, Climate modelling, Physics, Astronomy, Government sectors etc. Hadoop has been well-recognized as de factobig data processing platform that have been extensively adopted, and is currently widely used, in many application domains processing Big Data. Even though it is considered as an efficient solution for such complex query processing, it has its own limitation when the data to be processed exhibit interest locality. The data required for any query execution follows grouping behavior wherein only a part of the Big-Data is accessed frequently. During such scenarion, the time taken to execute a queryand return results, increases exponentially as the amount of data increases leading to much waiting time for the user. Since Hadoop default data placement strategy (HDDPS) does not consider such grouping behavior, it does not perform efficiently resulting in lacunas such as decreased local map task execution, increased query execution time etc. Hence proposed an Optimal Data Placement Strategy (ODPS) based on grouping semantics. In this paper we experiment the significance oftwo most promising clustering techniques viz. Hierarchical Agglomerative Clustering (HAC) and Markov Clustering (MCL) in grouping aware data placement for data intensive applications having interest locality. Initially user access pattern is identified by dynamically analyzing history log.Then both clustering techniques (HAC & MCL) are separately applied over the access pattern to obtain independent clusters. These clusters are interpreted and validated to extract the Optimal Data Groupings (ODG). Finally proposed strategy reorganizes the default data layouts in HDFSbased on ODG to achieve maximum parallel execution per group subjective to Load Balancer and Rack Awareness. Our proposed strategy is tested in 10 node cluster placed in a multi rack with Hadoop installed in every node deployed in cloud platform. Proposed strategy reduces the query execution time, significantly improves the data locality and has proved to be more efficient for massive datasets processing in heterogeneous distributed environment. Also MCL shows a marginal improved performance over HAC for queries exhibiting interest localities.

With the development of shallow surface mineral resources in metal mines, it is gradually turning to the stage of deep mining. According to the current mining depth and the average annual depth, during the period of “14th Five‐Year Plan,” one‐third of the underground metal mines will reach or exceed the mining depth of 1,000 m, with the deepest being 2,000 m. In the stage of deep mining, mines will face the conditions of high stress, high temperature, high well depth, and strong mining disturbance, which will greatly increase the difficulty of large‐scale deep mining. Among them, the high ground stress environment is the principal problem of many technical problems in deep mining. The selection of mining method has become a prerequisite for solving the problem of efficient and safe mining of deep deposits. In this paper, the vague set theory was introduced into the selection of mining methods and a vague set model for deep unloading mining schemes was established. Taking the Jinchuan No. 2 mining area as the engineering background, four unloading schemes for deep mining were proposed, and the Vague set model was used for optimization. It is concluded that the mining approach with large‐section unloading is the optimal unloading mining plan. The application shows that it has the advantages of high unloading efficiency, large production capacity, and low loss index. It has been fully promoted in the deep mining of the mining area. It is feasible and effective to use the vague set theory in the selection of deep unloading mining schemes, which provides a proper approach in the selection of deep unloading mining schemes.

The traditional method of centralized backup suffers from single point of failure and scalability problem. It is therefore beneficial to carry out the backup service over geographically distributed networks. Hence, redundant copies of data stored in such networks are essential to ensure the availability and reliability of the backup data. The main objective of this work is to balance and evenly distribute large amounts of backup data for efficient use of disks as well as network bandwidth which leads to faster data access and recovery. We develop an efficient method for data placement and replication. Our method has several attractive features: (1) Balanced and efficient data placement in a distributed storage resource environment. (2) Redistributing as few replicas as possible in the event of storage node addition or failure/removal. (3) There is no central mapping table so that the search performance becomes more efficient. We demonstrate the effectiveness of our method through theoretical analysis and simulation results.

As XML is increasingly being used in Web applications, new technologies need to be investigated for processing XML documents with high performance. Parallelism is a promising solution for structured document processing and data placement is a major factor for system performance improvement in parallel processing. This paper describes an effective XML document data placement strategy. The new strategy is based on a multilevel graph partitioning algorithm with the consideration of the unique features of XML documents and query distributions. A new algorithm, which is based on XML query schemas to derive the weighted graph from the labelled directed graph presentation of XML documents, is also proposed. Performance analysis on the algorithm presented in the paper shows that the new data placement strategy exhibits low workload skew and a high degree of parallelism.

Applications processing massive amount of data demand superior time-performance and data-storage capabilities. Many organizations are exploring Cloud Computing to manage such applications because of scalability and convenience of access from different geographic locations, whereas data security and privacy are few of the major concerns preventing them from fully embracing it. Data security while maintaining time-performance becomes an important consideration for designing data placement strategies. We first present the reader with a quick survey on the recent approaches and solutions in data placement oriented problems that address security concern and then characterize the performance for graph based algorithms that are practically realizable. We evaluated the performance of conventional strategies such as, Random, T-coloring and revisited these algorithms in the view of providing maximum security. We refer to our strategy as Data Security Preferential (DSP) data placement strategy and evaluated via rigorous performance evaluation tests to identify which strategy will best suit the requirements of the user on three state-of-the-art hierarchical cloud platforms viz., FatTree, ThreeTier and DCell and non-hierarchical cloud platforms.

Scientific workflows in collaborative cloud environments are becoming more and more popular.There is an urgent need to address the problem of large amount of data transfer across geo-distributed data centers during workflow execution.By utilizing data dependencies,we propose a two-stage data placement strategy and a task scheduling strategy for efficient workflow execution.With our strategy,the most related datasets can be placed into the same data center based on the data dependence between them at workflow build-time;then the tasks are scheduled to their most closely related data centers for execution and the newly-generated data sets are put into the data center that has the most dependency with them at workflow runtime.The experimental results show that the proposed strategy can significantly reduce the volume of data transfer among different data centers,and hence improve the performance of running scientific workflows and cut down the cost of doing science on the clouds as well.

Although benefits from caching in US HEP are well-known, current caching strategies are not adaptive i.e they do not adapt to changing cache access patterns. Newer developments such as the High-Luminosity - Large Hadron Collider (HL-LHC), Deep Underground Neutrino Experiment (DUNE), a steady move toward streaming readout based Data Acquisition systems (DAQs) will increase the data production exponentially and hence burden the storage, compute & network infrastructures. Moreover, existing caching frameworks are optimized to reduce latency, but not optimized for storage. This, in combination with limited cache capacities relative to total data, makes it difficult to achieve data locality. In this work, we present Machine Learning-aided (ML) caching strategies. Specifically, we first present a Long Short-Term Memory-based (LSTM)hourly and multi-step cache usage prediction. Second, we present an hourly file-level access prediction model based on CatboostRegressor. To date, most ML-based cache prediction strategies in HEP have focused on daily cache usage and limited works tackled hourly cache usage and even fewer strategies addressed hourly file-level access prediction. File-level access prediction allows for the design of intelligent prefetching and data placement strategies with fine-grained control. We validated our cache prediction strategies using data collected from SoCal MINI caches in August 2024. We are currently extending the WRENCH simulator to reflect the US HEP ecosystem at the storage, network and compute levels. We plan to deploy our cache prediction strategies into WRENCH and later perform extensive analysis with complex data access patterns and candidate infrastructure configurations.

As one of the most popular frameworks for large-scale analytics processing, Hadoop is facing two challenges: both applications and storage devices become heterogeneous. However, existing data placement and job scheduling schemes pay little attention to such heterogeneity of either application I/O requirements or I/O device capability, thus can greatly degrade system efficiencies. In this paper, we propose ASPS, an Application and Storage-aware data Placement and job Scheduling approach for Hadoop clusters. The idea is to place application data and schedule application tasks considering both application I/O requirements and storage device characteristics. Specifically, ASPS first introduces novel metrics to quantify I/O requirements of applications. Then, based on the quantification, ASPS places data of different applications to the preferred storage devices. Finally, ASPS tries to launch jobs with high I/O requirements on the nodes with the same type of faster devices to improve system efficiency. We have implemented ASPS in Hadoop framework. Experimental results show that ASPS can reduce the completion time of a single application by up to 36% and the average completion time of six concurrent applications by 27%, compared to existing data placement policies and job scheduling approaches.

The need for increased memory capacity, which also needs to be affordable and sustainable, leads to the adoption of heterogeneous memory hierarchies, combining DRAM and NVM technologies. This work proposes a memory management methodology that relies on multi-objective optimization in terms of performance, energy consumption and impact on NVM’s lifetime, for applications deployed on heterogeneous (i.e., DRAM/NVM) memory systems. We propose a scalable and lightweight data structure exploration flow for supporting data type refinement based on access pattern analysis, enhanced with a weighted-based data placement decision support for multi-objective exploration and optimization. The evaluation of the methodology was performed both on emulated and real DRAM/NVM hardware for different applications and data placement algorithms. The experimental results show up to 58.7% lower execution time and 48.3% less energy consumption compared with the results obtained by the initial versions of the applications. Moreover, we observed 72.6% less NVM write operations, which can significantly extend the lifetime of the NVM memory. Finally, thorough evaluation shows that the methodology is flexible and scalable, as it can integrate different data placement algorithms and NVM technologies and requires reasonable exploration time.

Data placement and parallel execution strategies are crucial design issues for high performance transaction processing in a parallel database machine. Declustering (horizontal partitioning) is a most promising solution for data placement. We analyze the trade-offs of three alternative execution strategies in the context of declustered databases. Then we focus on the strategy of executing programs at their data repositories which is highly parallel and offers many opportunities for query optimization. Finally, we elaborate on the implications of this strategy on program loading, associative routing, and decentralized concurrency control.