Enabling near-data processing in distributed object storage systems

The performance and scalability of modern data-intensive systems processing large datasets are limited by unnecessary data movement. Even though near-data processing (NDP) can provably reduce data transfers and increase performance, at present, NDP is utilized primarily in read-only settings. Near-data execution of data-intensive modification operations is currently infeasible due to the lack of transactional consistency and the absence of practicable low-latency synchronization mechanisms between the host database engine and the NDP-engine on smart storage. In this paper, we introduce Update NDP as an approach to offloading modifications to computational storage with transactional guarantees in an NDP database system called neoDBMS . To ensure consistency, we introduce a low-latency shared lock table between the host and computational storage, based on novel cache-coherent interconnects . We also introduce a novel locking protocol that seamlessly integrates the shared lock table within the lock manager of the host NDP-engine. To handle failure recovery, while preserving high and robust performance, we introduce novel extended locking and logging mechanisms that allow the host and computational storage to perform useful work during log-movement. Our evaluation indicates that in-storage modifications in neoDBMS in mixed workload settings are ≥ 6.52 × faster than host-only executions and exhibit robust performance due to lower data movement and better resource utilization.

Approximate nearest neighbor search (ANNS) in high-dimensional vector spaces has become increasingly crucial in database and machine learning applications. Most previous ANNS algorithms require TB-scale memory to store indices of billion-scale datasets, making their deployment extremely expensive for high-performance ANNS services. The emerging SmartSSD technology offers an opportunity to achieve scalable ANNS via near data processing (NDP). However, there remain several challenges to directly adopt existing ANNS algorithms on multiple SmartSSDs. In this paper, we present SmartANNS, a SmartSSD-empowered billion-scale ANNS solution based on a hierarchical indexing methodology. We propose several novel designs to achieve near-linear scaling with multiple SmartSSDs. First, we propose a “host CPUs + SmartSSDs” cooperative architecture incorporated with hierarchical indices to significantly reduce data accesses and computations on SmartSSDs. Second, we propose dynamic task scheduling based on optimized data layout to achieve both load balancing and data reusing for multiple SmartSSDs. Third, we further propose a learning-based shard pruning algorithm to eliminate unnecessary computations on SmartSSDs. We implement SmartANNS using Samsung’s commercial SmartSSDs. Experimental results show that SmartANNS can improve query per second (QPS) by up to 10.7 × compared with the state-of-the-art SmartSSD-based ANNS solution–CSDANNS. Moreover, SmartANNS can achieve near-linear performance scalability for large-scale datasets using multiple SmartSSDs.

With the advent of the era of big data, cloud computing, Internet of things, and other information industries continue to develop. There is an increasing amount of unstructured data such as pictures, audio, and video on the Internet. And the distributed object storage system has become the mainstream cloud storage solution. With the increasing number of distributed applications, data security in the distributed object storage system has become the focus. For the distributed object storage system, traditional defenses are means that fix discovered system vulnerabilities and backdoors by patching, or means to modify the corresponding structure and upgrade. However, these two kinds of means are hysteretic and hardly deal with unknown security threats. Based on mimic defense theory, this paper constructs the principle framework of the distributed object storage system and introduces the dynamic redundancy and heterogeneous function in the distributed object storage system architecture, which increases the attack cost, and greatly improves the security and availability of data.

Data access costs dominate the execution times of most parallel applications and they are expected to be even more important in the future. To address this, recent research has focused on Near Data Processing (NDP) as a new paradigm that tries to bring computation to data, instead of bringing data to computation (which is the norm in conventional computing). This paper explores the potential of compiler support in exploiting NDP in the context of emerging many core systems. To that end, we propose a novel compiler algorithm that partitions the computations in a given loop nest into sub computations and schedules the resulting sub computations on different cores with the goal of reducing the distance-to-data on the on-chip network. An important characteristic of our approach is that it exploits NDP while taking advantage of data locality. Our experiments with 12 multi-threaded applications running on a state of-the-art commercial many core system indicate that the proposed compiler-based approach significantly reduces data movements on the on-chip network by taking advantage of NDP, and these benefits lead to an average execution time improvement of 18.4%. CCS CONCEPTS •Computer systems organization → Multi-core architectures;

Optimizing power efficiency for 3D stacked GPU-in-memory architecture

Data-intensive queries are common in business intelligence, data warehousing and analytics applications. Typically, processing a query involves full inspection of large in-storage data sets by CPUs. An intuitive way to speed up such queries is to reduce the volume of data transferred over the storage network to a host system. This can be achieved by filtering out extraneous data within the storage, motivating a form of near-data processing. This work presents Biscuit, a novel near-data processing framework designed for modern solid-state drives. It allows programmers to write a data-intensive application to run on the host system and the storage system in a distributed, yet seamless manner. In order to offer a high-level programming model, Biscuit builds on the concept of data flow. Data processing tasks communicate through typed and data-ordered ports. Biscuit does not distinguish tasks that run on the host system and the storage system. As the result, Biscuit has desirable traits like generality and expressiveness, while promoting code reuse and naturally exposing concurrency. We implement Biscuit on a host system that runs the Linux OS and a high-performance solid-state drive. We demonstrate the effectiveness of our approach and implementation with experimental results. When data filtering is done by hardware in the solid-state drive, the average speed-up obtained for the top five queries of TPC-H is over 15×.

The upgrade of data processing and storage system for EAST NBI

The ever increasing amount of data being handled in data centers causes an intrinsic inefficiency: moving data around is expensive in terms of bandwidth, latency, and power consumption, especially given the low computational complexity of many database operations. In this paper we explore near-data processing in database engines, i.e., the option of offloading part of the computation directly to the storage nodes. We implement our ideas in Caribou, an intelligent distributed storage layer incorporating many of the lessons learned while building systems with specialized hardware. Caribou provides access to DRAM/NVRAM storage over the network through a simple key-value store interface, with each storage node providing high-bandwidth near-data processing at line rate and fault tolerance through replication. The result is a highly efficient, distributed, intelligent data storage that can be used to both boost performance and reduce power consumption and real estate usage in the data center thanks to the micro-server architecture adopted.

Enterprises usually store data in internal storage and install firewalls to protect against intruders to access the data. They also standardize data access procedures to prevent insiders to disclose the information without permission. In cloud computing, the data will be stored in storage provided by service providers. Service providers must have a viable way to protect their clients' data, especially to prevent the data from disclosure by unauthorized insiders. Storing the data in encrypted form is a common method of information privacy protection. If a cloud system is responsible for both tasks on storage and encryption/decryption of data, the system administrators may simultaneously obtain encrypted data and decryption keys. This allows them to access information without authorization and thus poses a risk to information privacy. This study proposes a business model for cloud computing based on the concept of separating the encryption and decryption service from the storage service. Furthermore, the party responsible for the data storage system must not store data in plaintext, and the party responsible for data encryption and decryption must delete all data upon the computation on encryption or decryption is complete. A CRM (Customer Relationship Management) service is described in this paper as an example to illustrate the proposed business model. The exemplary service utilizes three cloud systems, including an encryption and decryption system, a storage system, and a CRM application system. One service provider operates the encryption and decryption system while other providers operate the storage and application systems, according to the core concept of the proposed business model. This paper further includes suggestions for a multi-party Service- Level Agreement (SLA) suitable for use in the proposed business model.

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.

In clinical genomics, the continuous evolution of bioinformatic algorithms and sequencing platforms makes it beneficial to store patients’ complete aligned genomic data in addition to variant calls relative to a reference sequence. Due to the large size of human genome sequence data files (varying from 30 GB to 200 GB depending on coverage), two major challenges facing genomics laboratories are the costs of storage and the efficiency of the initial data processing. In addition, privacy of genomic data is becoming an increasingly serious concern, yet no standard data storage solutions exist that enable compression, encryption, and selective retrieval. Here we present a privacy-preserving solution named SECRAM (Selective retrieval on Encrypted and Compressed Reference-oriented Alignment Map) for the secure storage of compressed aligned genomic data. Our solution enables selective retrieval of encrypted data and improves the efficiency of downstream analysis (e.g., variant calling). Compared with BAM, the de facto standard for storing aligned genomic data, SECRAM uses 18% less storage. Compared with CRAM, one of the most compressed nonencrypted formats (using 34% less storage than BAM), SECRAM maintains efficient compression and downstream data processing, while allowing for unprecedented levels of security in genomic data storage. Compared with previous work, the distinguishing features of SECRAM are that (1) it is position-based instead of read-based, and (2) it allows random querying of a subregion from a BAM-like file in an encrypted form. Our method thus offers a space-saving, privacy-preserving, and effective solution for the storage of clinical genomic data.

The subject of the study is the methods of intellectual analysis, namely the construction of a decision tree, associative analysis, the identification of patterns between related events based on data presented by a relational model. The purpose of the study is to analyze the features of information units and data structures, using the example of relational systems that affect the technology of knowledge extraction. Tasks: the article solves the following tasks: to consider the relational data model as the most popular and effective data structure used in intelligent information systems for data processing and storage; to analyze the operations of relational algebra, the operational component of the relational data model regarding the application of aggregate functions; to develop a general formal statement of the problem of knowledge extraction from a relational database; to consider the concept of functional associative rules; the ID3 decision tree generation algorithm focused on data processing in relational systems is analyzed. The following methods are implemented: modern view and trends in the field of data mining; features of building information systems based on relational databases, relational algebra, theory of normalization of relations; analysis of literature on the topic of research; comparative analysis. Results achieved: the relational data model is considered as the most effective data structure used in intelligent information systems for data processing and storage. A group of aggregate functions of relational databases is identified and analyzed with respect to key attributes of the relation, which makes it possible to build logical dependencies between information units of the subject area being analyzed. The task of extracting knowledge from the database is formally formulated. The concept of functional associative rules is introduced. The ID3 decision tree generation algorithm focused on data processing in relational systems is carefully analyzed. The semantic network (SN), built on the basis of the proposed approach, allows to increase the efficiency of decision support systems. Conclusions: the universal approach proposed in the article to build a relational data model of an information system for searching for associative patterns in data allows to solve a whole class of typical tasks in which objects are connected by a "many-to-many" relationship or M →N. The relational database model is proposed as a universal information structure for solving associative analysis tasks and presenting knowledge in the form of a semantic network. The examples given in the article confirm the effectiveness of the developed and considered approaches to solving the problem of data mining in the environment of relational systems. Solving the problem of identifying knowledge in data will allow to improve the quality of management decisions made.

Current microscopy techniques hugely profit from modern microscopes producing a massive amount of increasingly complex data which are analysed by sophisticated algorithms. As a result, previously undistinguishable phenomena can be observed. However, this development coincides with new challenges for the biologist executing these experiments. Data storage, data processing, parallelisation, automation, and reproducibility are important factors in mastering these new techniques as they incur additional effort previously of less impact for the biologists. Existing solutions address the mentioned factors separately. Image storage systems manage the storage of data, specialised tool solve individual processing problems, and workflow systems help with automation and ensure reproducibility. Finally, parallelisation is a topic that is slowly gaining traction in the field of the specialised tools. However, there exist gaps between these solutions that the biologist has to bridge by hand, and which lower the overall efficiency. This work introduces a new software, whose design considers the mentioned aspects. It is a plugin to the microscopy images storage system OMERO and is called OPE. This approach eliminates nearly all overhead the biologist faces by integrating a system covering processing, reproducibility, and parallelisation into the data storage.

This paper is motivated by a global online satellite images distribution system operated at the Earth Resources Observation and Science (EROS) center of the U.S Geological Survey. Fundamental objectives of EROS include, but are not limited to, building high-speed and cost-effective massive data processing and storage systems to support online satellite images distribution. Hybrid storage systems -- containing solid-state drives (SSD), hard disks (HDD), and tapes -- can provide an ideal data storage solution for a wide variety of data processing centers like EROS. Large-scale hybrid storage systems will become increasingly popular in the next few years for the following two reasons. First, highly accessed storage objects in a hybrid storage system can be prefetched and cached to high-speed storage components such as solid-state drives. SSD-based hybrid storage system can provide large storage capacity, high I/O performance and data reliability. Second, hybrid storage systems are cost-effective, because inexpensive tapes help in increasing storage capacities at very low cost. Transferring data back and forth among SSDs, HDDs, and tapes plays a critical role in achieving high I/O performance. Thus, we proposed data mining algorithms that can judiciously prefetch data. Our analytical model and the experimental results reveals that our data mining prefetching algorithm increase the performance of the hybrid storage systems.

The data storage and processing system based on cloud computing can take cloud server as the core, make use of sensor technology synthetically, and provide data storage and computing function for client. Combining GPRS/WiFi technology with Ali cloud server, this paper realizes cloud storage and cloud computing processing of remote important environment information and dynamic display of real-time / historical data in Android client. The experimental results show that the system runs stably and the data are reliable.