A Contract-aware and Cost-effective LSM Store for Cloud Storage with Low Latency Spikes

Many organizations provide computing resources or services to the client like hardware, software, storage etc. Storage is one of important service that is popular and commonly used today and it provides storage space online to store the data. Service providers give a storage space to consumers on the basis of renting criteria with a trust of high and powerful security. Consumer and organizations uses these storages to secure own personal data. There are many reasons come across that causes to move data from one place to another. Some of them are: reconstruction of storage, backup, new technologies, security etc. So, data migration, data transfer between different cloud storages, often requires. Achieving mechanism of Data migration between clouds is very difficult because of its different architecture to store data. For example Window Azure cloud storage creates a primary key name ID automatically whenever a new table is created. Architecture of cloud storage is depending upon its implementation done by corresponding cloud vendors (like Microsoft). The main vision of this research work is to provide an intermediate service to move the data between both homogeneous and heterogeneous cloud storages. This intermediate service is a two way strategy that built in a SOA technology called WCF that helps the clients, organizations to move data between same and different cloud storages (Window Azure and Go Daddy cloud storages mainly used in this research work).

Now-a-days cloud storage services are growing rapidly and becoming an emerging trend in data storage field. But in cloud not store the petabytes of data. In this reason develop new technology i.e. Big Data and Big File Cloud (BFC) Architecture. By using the Big Data technology store the petabytes of the data in cloud storage engines. But there is a many issues in designing efficient storage of engine for the cloud based system with the requirements. In Big Data using Key Value technique, Key value has dominant role in solving the problems. The main objective is to develop a simple and efficient BFC architecture. By using this easy process the big files up to petabytes, distribute the files, parallel process the I/O files, remove the redundant data, and reduce the file size by using data compression techniques.

Building LSM-Tree-based key-value stores (LSM stores) on Cloud Block Storage (CBS) is becoming increasingly popular due to its convenience compared to local block storage devices. However, the constrained bandwidth of CBS and the competition between bandwidth and paid IOPS results in performance degradation and high tail latency in LSM stores, which cannot provide stable service-level agreements (SLAs). We propose RaKV, an efficient LSM store with stable SLAs for CBS, to improve write performance and reduce latency. This store successfully tackles challenges by utilizing instance memory and managing data flow between components. First, RaKV designs a dynamically partitioned L 0 placed in memory and a parallel L 0 - L 1 compaction mechanism to minimize write amplification and write stalls, mitigating the performance impact of the CBS’s constrained bandwidth. Second, RaKV proposes a bandwidth-aware admission control mechanism that dynamically adjusts the number of incoming user requests based on the capacity of the L 0 layer and the available cloud storage bandwidth, thus preventing write stalls and ensuring SLA consistency. Finally, RaKV implements an efficient read/write I/O isolation and scheduling mechanism to coordinate bandwidth and IOPS demands, thus stabilizing read SLAs and reducing read latency. Our evaluation results show that RaKV improves random write performance by 25.8% compared to the state-of-the-art key-value store ADOC, with more consistent throughput and lower read latency than CruiseDB.

Log-structured merge (LSM) tree key-value (KV) stores have been widely deployed in many NoSQL and SQL systems, serving online big data applications such as social networking, bioinfomatics, graph processing, machine learning, etc. The batch processing of sorted data merging (i.e., compaction) in LSM-tree KV stores greatly improves the efficiency of writing, leading to good write performance and high space efficiency. Recently, some lazy compaction methods were proposed to further promote the system throughput through delaying the compaction to accumulate more data within a compaction batch. However, the batched writing manner also leads to significant tail latency, which is unacceptable for online processing, and the newly proposed lazy approaches worsen the tail latency problem. Furthermore, the unbalanced read/write performance of the widely deployed SSDs make the performance optimization harder. Aiming to optimize both the tail latency and the system throughput, in this paper, we propose a novel Lower-level Driven Compaction (LDC) method for LSM-tree KV stores. LDC breaks the limitations of the traditional upper-level driven compaction manner and triggers practical compaction actions by lower-level data. It has the benefits of both decreasing the compaction granularity effectively for smaller tail latency and reducing the write amplification of LSM-tree compaction for higher throughput. We have implemented LDC in LevelDB; the experimental results indicate that LDC can reduce the 99.9th percentile latency for 2.62 times compared with the traditional upper-level driven compaction mechanism, and achieve 56.7% ~ 72.3% higher system throughput at the same time.

Cloud computing is a group of servers providing the on demand computing services to the users on rental basis. One of the services providing by the cloud is Data Storage as a Service. In Data Storage as a Service user will outsource their sensitive data to cloud storage environment, where the users data will be stored and whenever user needs data, they will access from the cloud storage from anywhere and anytime using any device connected to the internet. One of the critical issues in this is data security. When user outsources their sensitive data to cloud vendor, the cloud vendor may leak the user’s sensitive data to third persons because vendor is not a trusted party. To overcome these problems there are many authors suggested different security algorithms. In this research we studied and evaluating the performances of those security algorithms and proposing a better solution, which provides best security to data stored in cloud databases.

Long-tail latency values significantly affect the user experience and hence they are a major concern in modern systems. However, tail latency characteristics of applications developed using microservices architecture are still unknown. In this paper, we focus on analyzing and characterizing the behaviour of tail latency values of microservices workloads under service's peak sustainable throughput with a closed system model. Our particular focus is to investigate the impact of heap size, garbage collection, concurrency and service demand on the tail latency. Our study has two parts. First, we develop and experiment on four microservices micro benchmarks representing CPU bound, network I/O bound, memory bound and database I/O bound to find the tail characteristics under different concurrency, heap, service demand and garbage collector. Then, in the second part, we explore the tail latency characteristics identified on micro benchmarks by testing those conditions using standard microservices benchmark application, Socks Shop. We observe first that microservices latency distributions exhibit the power law behavior. Second, we systematically prove that microservices workload distributions do not exhibit the heavy tailed behavior. We show that increasing the level of concurrency and service demand increases the tail latency. We use queuing simulation to prove our claims. Finally, we show that tail latency values of memory bound and database I/O bound microservices workloads are correlated with heap size and garbage collection.

The conventional wisdom is that aggressive networking requirements, such as high packet rates for small messages and μs-scale tail latency, are best addressed outside the kernel, in a user-level networking stack. We presentix, a dataplane operating system that provides high I/O performance and high resource efficiency while maintaining the protection and isolation benefits of existing kernels.ixuses hardware virtualization to separate management and scheduling functions of the kernel (control plane) from network processing (dataplane). The dataplane architecture builds upon a native, zero-copy API and optimizes for both bandwidth and latency by dedicating hardware threads and networking queues to dataplane instances, processing bounded batches of packets to completion, and eliminating coherence traffic and multicore synchronization. The control plane dynamically adjusts core allocations and voltage/frequency settings to meet service-level objectives.We demonstrate thatixoutperforms Linux and a user-space network stack significantly in both throughput and end-to-end latency. Moreover,iximproves the throughput of a widely deployed, key-value store by up to 6.4× and reduces tail latency by more than 2× . With three varying load patterns, the control plane saves 46%--54% of processor energy, and it allows background jobs to run at 35%--47% of their standalone throughput.

Key-value stores based on log-structured merge-tree (LSM-tree), e.g. LevelDB by Google and RocksDB by Facebook, have been widely used in a lot of cyber-physical systems scenarios to provide flexible data access and high performance. Compaction is a necessary operation to delete some invalid data, reduce the data size and improve the efficiency of reading and writing. However, compaction would stall write requests, which significantly degrades system performance. This paper studies the effect of compaction on tail latency that shows frequent latency spikes. These spikes would cause poor user experience and have already concerned by industrial researchers. Furthermore, the number of involved SSTables proportionally decides the height of latency spikes. We propose the limited compaction to only allow a part of SSTables involved in the compaction. We implement the limited compaction with a basic method to choose the SSTables randomly and a selective method to choose the SSTables according to their overlapped ranges with the next level. Experimental results on LevelDB with comprehensive benchmarks have shown that the proposed methods can effectively reduce the tail latency, while inducing an acceptable write amplification.

In this paper, we describe a COM between the interior enterprise application and public cloud storage platform which is closer to the client we called-Cloudkey, which is designed to take responsible for enterprise data backup business. Cloudkey stores data persistently in a cloud storage provider such as Amazon S3 [1] or Windows Azure [2], allowing users to take advantages of the reliability and large storage capacity of cloud providers, also avoiding the need for dedicated server hardware. Clients access to the storage through Cloudkey running on-site, which provide lower-latency responses and additional opportunities for optimization through caches data. We describe some of the optimizations which are necessary for achieving good performance and low costs, including separation of metadata and data, database optimization and connection pool technology, and also encryption before data transmission. Cloudkey supports multiple protocols both NFS and CIFS and is portable to different providers.

Consistent hash based storage systems are used in many real world applications for which energy is one of the main cost factors. However, these systems are typically designed and deployed without any mechanisms to save energy at times of low demand. We present an energy conserving implementation of a consistent hashing based key-value store, called PowerCass, based on Apache's Cassandra. In PowerCass, nodes are divided into three groups: active, dormant, and sleepy. Nodes in the active group store cover all the data and running continuously. Dormant nodes are only powered during peak activity time and for replica synchronization. Sleepy nodes are offline almost all the time except for replica synchronization and exceptional peak loads. With this simple and elegant approach we are able to reduce the energy consumption by up to 66% compared to the unmodified key-value store Cassandra.

Besides well-known benefits, commodity cloud storage also raises concerns that include security, reliability, and consistency. We present Hybris key-value store, the first robust hybrid cloud storage system, aiming at addressing these concerns leveraging both private and public cloud resources. Hybris robustly replicates metadata on trusted private premises (private cloud), separately from data, which are dispersed (using replication or erasure coding) across multiple untrusted public clouds. Hybris maintains metadata stored on private premises at the order of few dozens of bytes per key, avoiding the scalability bottleneck at the private cloud. In turn, the hybrid design allows Hybris to efficiently and robustly tolerate cloud outages but also potential malice in clouds without overhead. Namely, to tolerate up to f malicious clouds, in the common case of the Hybris variant with data replication, writes replicate data across f +1 clouds, whereas reads involve a single cloud. In the worst case, only up to f additional clouds are used. This is considerably better than earlier multi-cloud storage systems that required costly 3 f +1 clouds to mask f potentially malicious clouds. Finally, Hybris leverages strong metadata consistency to guarantee to Hybris applications strong data consistency without any modifications to the eventually consistent public clouds. We implemented Hybris in Java and evaluated it using a series of micro and macro-benchmarks. Our results show that Hybris significantly outperforms comparable multi-cloud storage systems and approaches the performance of bare-bone commodity public cloud storage.

Computer systems utilizing byte-addressable Non-Volatile Memory ( NVM ) as memory/storage can provide low-latency data persistence. The widely used key-value stores using Log-Structured Merge Tree ( LSM-Tree ) are still beneficial for NVM systems in aspects of the space and write efficiency. However, the significant write amplification introduced by the leveled compaction of LSM-Tree degrades the write performance of the key-value store and shortens the lifetime of the NVM devices. The existing studies propose new compaction methods to reduce write amplification. Unfortunately, they result in a relatively large read amplification. In this article, we propose NVLSM, a key-value store for NVM systems using LSM-Tree with new accumulative compaction. By fully utilizing the byte-addressability of NVM, accumulative compaction uses pointers to accumulate data into multiple floors in a logically sorted run to reduce the number of compactions required. We have also proposed a cascading searching scheme for reads among the multiple floors to reduce read amplification. Therefore, NVLSM reduces write amplification with small increases in read amplification. We compare NVLSM with key-value stores using LSM-Tree with two other compaction methods: leveled compaction and fragmented compaction. Our evaluations show that NVLSM reduces write amplification by up to 67% compared with LSM-Tree using leveled compaction without significantly increasing the read amplification. In write-intensive workloads, NVLSM reduces the average latency by 15.73%–41.2% compared to other key-value stores.

Cloud object storage has become a cornerstone for managing and analyzing large datasets, offering an efficient and flexible solution for organizations to store and process unstructured and structured data. With the exponential growth of data across various industries, traditional data storage solutions often need help to keep up with the sheer volume and variety of information. With its scalability, durability, and cost-effectiveness, cloud object storage addresses these challenges by providing a centralized platform where businesses can store vast amounts of data and access it seamlessly. This article explores how cloud object storage when paired with modern data analytics tools, enables organizations to unlock valuable insights from massive datasets. The architecture behind cloud object storage is designed to handle high-volume data efficiently while offering flexible access and robust security features. In big data analytics, cloud object storage plays a pivotal role by facilitating data processing at scale & supporting advanced analytics techniques like machine learning and artificial intelligence. Using data lakes and distributed computing frameworks, cloud object storage ensures that large datasets are accessible for real-time analysis, driving faster & more informed decision-making. The article also delves into performance optimization strategies for cloud object storage, such as data tiering and caching, which improve access speed and reduce costs. It also highlights several use cases from industries like finance, healthcare, and e-commerce, demonstrating how organizations leverage cloud object storage to gain competitive advantages. By capitalizing on cloud storage's flexibility, businesses can overcome traditional storage limitations & explore new avenues for innovation and growth. In summary, cloud object storage simplifies the management of massive datasets & enables organizations to harness the power of data-driven decision-making in an ever-evolving digital landscape.

Collaborative modelling of the Internet of Things (IoT) with Artificial Intelligence (AI) has merged into the Intelligence of Things concept. This recent trend enables sensors to track required parameters and store accumulated data in cloud storage, which can be further utilized by AI based predictive models for automatic decision making. In a smart and sustainable environment, effective waste management is a concern. Poor regulation of waste in surrounding areas leads to rapid spread of contagious disease risks. Traditional waste object management requires more working staff, increases effort, consumes time and is relatively ineffective. In this research, an Intelligence of Things Enabled Smart Waste Management (IoT-SWM) model with predictive capabilities is developed. Here, local sinks (LS) are deployed in specified locations. At every instant, the current status of smart bins in each LS is notified to users to determine the priority level of LS to be emptied. Based on aggregated sensor values for the three smart bins, LS weight and poison gas value, the priority order of emptying LS is computed, and decision is made whether to notify the users with an alert message or not. It also helps in predicting the LS, which is likely to be filled up at a faster rate based on assigned timestamp. This model is implemented in real time with many LS and it was observed that bins, which were close to more crowded sites filled up faster compared to sparse populated areas. Random forest algorithm was used to predict whether an alert notification is to be sent or not. An average mean of 95.8% accuracy was noted while using 60 decision trees in random forest algorithm. The average mean execution latency recorded for training and testing sets is 13.06 sec and 14.39 sec respectively. Observed accuracy rate, precision, recall and f1-score parameters were 95.8%, 96.5%, 98.5% and 97.2% respectively. Model buildup and the validation time computed were 3.26 sec and 4.25 sec respectively. It is also noted that at a threshold value of 0.93 in LS level, the maximum accuracy rate reached was 95.8%. Thus, based on the prediction of random forest approach, a decision to notify the users is taken. Obtained outcome indicates that the waste level can be efficiently determined, and the overflow of dustbins can be easily checked in time

Interactive web services increasingly drive critical business workloads such as search, advertising, games, shopping, and finance. Whereas optimizing parallel programs and distributed server systems have historically focused on average latency and throughput, the primary metric for interactive applications is instead consistent responsiveness, i.e., minimizing the number of requests that miss a target latency. This paper is the first to show how to generalize work-stealing, which is traditionally used to minimize the makespan of a single parallel job, to optimize for a target latency in interactive services with multiple parallel requests.We design a new adaptive work stealing policy, called tail-control, that reduces the number of requests that miss a target latency. It uses instantaneous request progress, system load, and a target latency to choose when to parallelize requests with stealing, when to admit new requests, and when to limit parallelism of large requests. We implement this approach in the Intel Thread Building Block (TBB) library and evaluate it on real-world workloads and synthetic workloads. The tail-control policy substantially reduces the number of requests exceeding the desired target latency and delivers up to 58% relative improvement over various baseline policies. This generalization of work stealing for multiple requests effectively optimizes the number of requests that complete within a target latency, a key metric for interactive services.