Optimizing data migration for cloud-based key-value stores

Key-value (KV) stores based on the Log-Structure Merge tree (LSM-tree) have been widely used in modern storage systems (e.g. LevelDB and RocksDB) to achieve high write throughput. However, conventional LSM-tree design has high latency for random read operations, especially in the workloads with concurrent read and write operations. This paper proposes a dual-grained cache scheme to improve the performance of LSM-tree based KV stores, in which a coarse-grained cache improves read performance by leveraging spatial locality and a fine-grained cache tracks the frequently accessed KV pairs. We implement the dual-grained caching on a LSM-tree simulator and evaluate its performance using various Yahoo! Cloud Service Benchmark (YCSB) workloads. Our experimental results show that the proposed way of dual-grained caches can significantly improve the read performance of KV stores without sacrificing the advantages of the LSM-tree for write-intensive workloads.

In this paper we describe the design and implementation of ACaZoo, a key-value store that combines strong consistency with high performance and high availability. ACaZoo supports the popular column-oriented data model of Apache Cassandra and HBase. It implements strongly-consistent data replication using primary-backup atomic broadcast of a write-ahead log, which records data mutations to a Log-structured Merge Tree (LSM-Tree). ACaZoo scales by horizontally partitioning the key space via consistent primary-key hashing on available replica groups (RGs). LSM-Tree compactions can hamper performance, especially when they take place at RG primaries. ACaZoo addresses this problem by changing RG leadership prior to heavy compactions, a method that can improve throughput by up to 40% in write-intensive workloads. We evaluate ACaZoo using the Yahoo Cloud Serving Benchmark (YCSB) and compare it to Oracle's NoSQL Database and to Cassandra providing serial consistency via an extension of the Paxos algorithm.

SummaryMultitiered storage systems, which are made up of heterogeneous devices, are widely used in distributed environments to accelerate the I/O performance of upper big data applications. It raises new challenges in efficient data migration through smart caching mechanisms among heterogeneous storage levels, such as MEM‐SSD‐HDD. To optimize the cache policy scheduling mechanism on the distributed tiered storage architecture, we proposed a general framework with five layers, including a tiered storage system layer, a cache migration policy layer, a cache policy adaptive scheduling layer, a data access pattern layer, and a big data application layer. The framework prototype has been designed and implemented on the widely used distributed hybrid storage system named Alluxio. To meet the demands of the big data application layer, on the one hand, we designed a couple of cache eviction policies and promotion policies covering various access patterns on the cache migration policy layer (several proposed eviction policies have been adopted by the Alluxio open‐source community). On the other hand, two adaptive cache policy scheduling algorithms for selecting appropriate policies in various scenarios are designed and implemented on the cache policy adaptive scheduling layer. The scheduling algorithms are designed based on the hit ratio statistics and data access pattern model prediction, respectively. Experimental results show that the proposed cache policies are very effective for various big data applications, such as Spark SQL. The proposed cache policy scheduling algorithms with various eviction policies can improve around 20% hit ratio than that with a single eviction policy.

Large-scale storage systems are crucial components in data-intensive applications such as search engine clusters, video-on-demand servers, sensor networks and grid computing. A storage server typically consists of a set of storage devices. In such systems, data layouts may need to be reconfigured over time for load balancing or in the event of system failure/upgrades. It is critical to migrate data to their target locations as quickly as possible to obtain the best performance. Most of the previous results on data migration assume that each storage node can perform only one data transfer at a time. A storage node, however, can typically handle multiple transfers simultaneously and this can reduce the total migration time significantly. Moreover, storage devices tend to have heterogeneous capabilities as devices may be added over time due to storage demand increase. In this paper, we consider the heterogeneous data migration problem, where we assume that each storage node v has different transfer constraint cv, which represents how many simultaneous transfers v can handle. We develop algorithms to minimize the data migration time. We show that it is possible to find an optimal migration schedule when all cvs are even. Furthermore, though the problem is NP-hard in general, we give an efficient algorithm that offers a rigorous (1 + o(1))-approximation guarantee.

Block-based storage devices exhibit different characteristics from main memory, and applications and systems have been optimized for a long time considering the characteristics in mind. However, emerging non-volatile memory technologies are about to change the situation. Persistent Memory (PM) provides a huge, persistent, and byte-addressable address space to the system, thereby enabling new opportunities for systems software. However, existing applications are usually apt to indirectly utilize PM as a storage device on top of file systems. This makes applications and file systems perform unnecessary operations and amplify I/O traffic, thereby under-utilizing the high performance of PM. In this paper, we make the case for an in-Kernel key-value storage service optimized for PM, called InK. While providing the persistence of data at a high performance, InK considers the characteristics of PM to guarantee the crash consistency. To this end, InK indexes key-value pairs with B+ tree, which is more efficient on PM. We implemented InK based on the Linux kernel and evaluated its performance with Yahoo Cloud Service Benchmark (YCSB) and RocksDB. Evaluation results confirms that InK has advantages over LSM-tree-based key-value store systems in terms of throughput and tail latency.

The rise of distributed applications and cloud computing has created a demand for scalable, high-performance key-value storage systems. This paper presents a performance evaluation of three prominent NoSQL key-value stores: Redis, Aerospike, and Dragonfly, using the Yahoo! Cloud Serving Benchmark (YCSB) framework. We conducted extensive experiments across three distinct workload patterns (read-heavy, write-heavy), and balanced while systematically varying client concurrency from 1 to 32 clients. Our evaluation methodology captures both latency, throughput, and memory characteristics under realistic operational conditions, providing insights into the performance trade-offs and scalability behaviour of each system.© The Author(s) 2025. Published by RITHA Publishing. This article is distributed under the terms of the license CC-BY 4.0., which permits any further distribution in any medium, provided the original work is properly cited maintaining attribution to the author(s) and the title of the work, journal citation and URL DOI.

With improvements in computation and storage resources, data access through the network becomes the bottleneck for several cloud applications. Even with high-speed networks, the high latency of the cloud access makes it unfeasible or un-favourable for latency-sensitive applications such as autonomous driving, smart factories, and video streaming. Edge computing provides a solution by utilizing the network edge resources that are closer to the end users. Nevertheless, it is a non-trivial task to design a large-scale edge-capable system that is stable, fault-tolerant, and efficient [1]. In this paper, present the design of EdgeKV: a novel general-purpose distributed key-value store for the network edge. We demonstrate the features of EdgeKV for achieving high efficiency and scalability while providing flexibility, ease of use, and data privacy. We evaluated our prototype on the Grid'5000 framework with multiple realistic Yahoo! Cloud Serving Benchmark (YCSB) workloads. Our initial results show that EdgeKV achieves 72% higher throughput and 47% lower latency on average than centralized cloud storage, for read-dominated workloads.

New business models, constant technological progress, as well as ever-changing legal regulations require that companies replace their business applications from time to time. As a side effect, this demands for migrating the data from the existing source application to a target application. Since the success of the application replacement as a form of IT maintenance is contingent on the underlying data migration project, it is crucial to accomplish the migration in time and on budget. This however, calls for a stringent data migration process model combined with well-defined quality assurance measures. The paper presents, first, a field-tested process model for data migration projects. Secondly, it points out the typical risks we have frequently observed in the course of this type of project. Thirdly, the paper provides practice-based testing and quality assurance techniques to reduce or even eliminate these data migration risks.

During data modelling, NoSQL databases are usually considered schema-less databases. They do not support rigid schema as traditional databases. Instead, offer flexibility and support dynamic schema, which means schema design can be changed at run time to support analytical tasks. NoSQL databases allow storing similar information with multiple schemas, which facilitates data evolution by eliminating restrictions on the schema. It raises the question of whether the schema matters in NoSQL databases, given that explicit schema declaration is not required before data storage. Furthermore, if the schema does matter, what is its potential impact on application performance? This paper examines these questions by taking three types of NoSQL databases: Document store, Column store, and Key-Value store. All possible schema structure alternatives are designed using a sample Entity Relationship (ER) model. Then, a set of common queries are performed on various schema alternatives to analyse the impact of query performance in terms of query response time, speedup factor, read and write Latency, and database size. The results suggest that the scheme alternatives affect the database performance in different NoSQL models.

Multi-tiered storage systems today are integrating Solid State Disks (SSD) on top of traditional rotational hard disks for performance enhancement due to the significant IO improvements in SSD technology. It is widely recognized that automated data migration between SSD and HDD plays a critical role in effective integration of SSD into multi-tiered storage systems. Furthermore, effective data migration has to take into account of application specific workload characteristics, deadlines, and IO profiles. An important and interesting challenge for automated data migration in multi-tiered storage systems is how to fully release the power of data migration while guaranteeing the migration deadline is critical to maximizing the performance of SSD-enabled multi-tiered storage system. In this paper, we present an adaptive look ahead data migration model that can incorporate application specific characteristics and I/O profiles as well as workload deadlines. Our adaptive data migration model has three unique features. First, it incorporates a set of key factors that may impact on the performance of look ahead migration efficiency in our formal model develop. Second, our data migration model can adaptively determine the optimal look ahead window size, based on several parameters, to optimize the effectiveness of look ahead migration. Third, we formally and experimentally show that the adaptive data migration model can improve overall system performance and resource utilization while meeting workload deadlines. Through our trace driven experimental study, we compare the adaptive look ahead migration approach with the basic migration model and show that the adaptive migration model is effective and efficient for continuously improving and tuning of the performance and scalability of multi-tier storage systems.

Nowadays clusters are one of the most used platforms in High Performance Computing and most programmers use the Message Passing Interface (MPI) library to program their applications in these distributed platforms getting their maximum performance, although it is a complex task. On the other side, OpenMP has been established as the de facto standard to program applications on shared memory platforms because it is easy to use and obtains good performance without too much effort. So, could it be possible to join both worlds? Could programmers use the easiness of OpenMP in distributed platforms? A lot of researchers think so. And one of the developed ideas is the distributed shared memory (DSM), a software layer on top of a distributed platform giving an abstract shared memory view to the applications. Even though it seems a good solution it also has some inconveniences. The memory coherence between the nodes in the platform is difficult to maintain (complex management, scalability issues, high overhead and others) and the latency of the remote-memory accesses which can be orders of magnitude greater than on a shared bus due to the interconnection network. Therefore this research improves the performance of OpenMP applications being executed on distributed memory platforms using a DSM with sequential consistency evaluating thoroughly the results from the NAS parallel benchmarks. The vast majority of designed DSMs use a relaxed consistency model because it avoids some major problems in the area. In contrast, we use a sequential consistency model because we think that showing these potential problems that otherwise are hidden may allow the finding of some solutions and, therefore, apply them to both models. The main idea behind this work is that both runtimes, the OpenMP and the DSM layer, should cooperate to achieve good performance, otherwise they interfere one each other trashing the final performance of applications. We develop three different contributions to improve the performance of these applications: (a) a technique to avoid false sharing at runtime, (b) a technique to mimic the MPI behaviour, where produced data is forwarded to their consumers and, finally, (c) a mechanism to avoid the network congestion due to the DSM coherence messages. The NAS Parallel Benchmarks are used to test the contributions. The results of this work shows that the false-sharing problem is a relative problem depending on each application. Another result is the importance to move the data flow outside of the critical path and to use techniques that forwards data as early as possible, similar to MPI, benefits the final application performance. Additionally, this data movement is usually concentrated at single points and affects the application performance due to the limited bandwidth of the network. Therefore it is necessary to provide mechanisms that allows the distribution of this data through the computation time using an otherwise idle network. Finally, results shows that the proposed contributions improve the performance of OpenMP applications on this kind of environments.

Migration is recognised as the essential step in converting seismic, data into a representation of the earth's subsurface structure. Ironically, conventional migration often fails where migration is needed most—when the data are recorded over complex structures. Processing field data shot in Central America, and synthetic data derived for that section, demonstrates that time migration actually degrades the image of the deep structure that lies below a complicated overburden.In the Central American example, velocities increase nearly two-fold across an arched and thrust-faulted interface. Wavefront distortion introduced by this feature gives rise to distorted reflections from depth. Even with interval velocity known perfectly, no velocity is proper for time migrating the data here; time migration is the wrong process because it does not honour Snell's Law. Depth migration of the stacked data, on the other hand, produces a reasonable image of the deeper section. The depth migration, however, leaves artifacts that could be attributed to problems that are common in structurally complicated areas: (1) departures of the stacked section from the ideal, a zero-offset section; (2) incorrect specification of velocities; and (3) loss of energy transmitted through the complex zoneFor such an inhomogeneous velocity structure, shortcomings in CDP stacking are directly related to highly non- hyperbolic moveout. As with migration velocity, no proper stacking velocity can be developed for these data, even from the known interval-velocity model. Proper treatment of nonzero-offset reflection data could be accomplished by depth migration before stacking. Simple ray-theoretical correction of the complex moveouts, however, can produce a stack that is similar to the desired zero-offset section.Overall, the choice of velocity model most strongly influences the results of depth migration. Processing the data with a range of plausible velocity models, however, leads to an important conclusion: although the velocities can never be known exactly, depth migration is essential for clarifying structure beneath complex overburden.

Reverse time migration (RTM) is achieved by a forward and reverse time propagation of source and receiver wavefields respectively, followed by an imaging condition. The quality of the image from any RTM implementation depends directly on the method’s ability to separate the true reflection data from the backscattered correlation noise between the source and receiver wavefields. In this paper we demonstrate the application of an inverse scattering imaging condition that significantly reduces this backscattered correlation noise and results in a much higher quality subsurface image than is achieved by typical RTM imaging conditions.

In relational and complex spreadsheets, relational and non-relational database models, high-speed data migration with scalable structure using visual Basic Excel programming language implementations with proposed migration model. One of the primary purposes behind making a point of interaction through the Visual Basic Application (VBA) is that most PC clients with logical preparation will currently know about Succeed and its significant capabilities (like arranging furthermore, plotting datasets). Even though Excel itself is in many cases utilized as an information storehouse by clients. The time factor, reliability, and credibility of migrating this data from one table to another through programming interfaces were measured using the link codes between the tables. In this paper, the migration and migration of homogeneous and heterogeneous data are investigated by using two types of different migration models of data to measure and match these data and the extent of their integration after migration. Specifies the target data for migration from the input tables in the Excel program to the target tables in larger databases. Furthermore, the two models (A, B) middleware provide an architecture that can be extended to support Relational database management systems (RDBMS) and other graphing databases. Experiments were performed using excel tables, both of which are related as source information bases, and as the data set for the source and target datasets, the migration time between these tables for the two models was calculated While retaining the same characteristics.

This paper presents ALARM, an autonomic load-aware resource management algorithm that can be used to manage physical machines or virtual machines in cloud, which participate in a P2P key-value store. A lot of existing key-value stores claim that they are elastic enough to scale up or down with no downtime or interruption to applications. However, the question that when the scaling up or down should take place has still not been resolved. The situation may get worse if the data store consists of hundreds of machines, for it's unrealistic for a system administrator to monitor the system and add/remove a machine manually. Fortunately, cloud computing and virtualization technology have enabled the real-time provision of virtual machines and a way of managing virtual machines without human interference. By supervising the utilization of multiple resources (CPU, memory, network IO, etc.) in virtual machines hosting the data store, our ALARM algorithm will take effect when some of the machines become overloaded or under loaded. The experiment result shows that ALARM helps the Open Chord data store, an open-source implementation of the Chord protocol, scale up and down according to the resource usage in the virtual machines.