Data Structures with Unpredictable Timing

Data sets in large applications are often too massive to fit completely inside the computers internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this article we survey the state of the art in the design and analysis of external memory (or EM) algorithms and data structures, where the goal is to exploit locality in order to reduce the I/O costs. We consider a variety of EM paradigms for solving batched and online problems efficiently in external memory. For the batched problem of sorting and related problems such as permuting and fast Fourier transform, the key paradigms include distribution and merging. The paradigm of disk striping offers an elegant way to use multiple disks in parallel. For sorting, however, disk striping can be nonoptimal with respect to I/O, so to gain further improvements we discuss distribution and merging techniques for using the disks independently. We also consider useful techniques for batched EM problems involving matrices (such as matrix multiplication and transposition), geometric data (such as finding intersections and constructing convex hulls), and graphs (such as list ranking, connected components, topological sorting, and shortest paths). In the online domain, canonical EM applications include dictionary lookup and range searching. The two important classes of indexed data structures are based upon extendible hashing and B-trees. The paradigms of filtering and bootstrapping provide a convenient means in online data structures to make effective use of the data accessed from disk. We also reexamine some of the above EM problems in slightly different settings, such as when the data items are moving, when the data items are variable-length (e.g., text strings), or when the allocated amount of internal memory can change dynamically. Programming tools and environments are available for simplifying the EM programming task. During the course of the survey, we report on some experiments in the domain of spatial databases using the TPIE system (transparent parallel I/O programming environment). The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.

Modern embedded systems are now capable of executing complex and demanding applications that are often based on large data structures. The design of the critical data structures of the application affects the performance and the memory requirements of the whole system. Dynamic Data Structure Refinement methodology provides optimizations, mainly in list and array data structures, which are based on the application's features and access patterns. In this work, we extend various aspects of the methodology: First, we integrate radix tree optimizations. Then, we provide a set of platform-aware data structure implementations, for performing optimizations based on the hardware features. The extended methodology is evaluated using a wide set of synthetic and real-world benchmarks, in which we achieved performance and memory trade-offs up to 29.6%. Additionally, Pareto optimal data structure implementations that were not available by the previous methodology, are identified with the extended one.

We present practical learning-based search algorithms that make it possible to automatically design key-value data structures. Our work allows searching within a space of 10100 possible designs for the optimal data structure. The input is a workload specification and the output is an abstract syntax tree which can be absorbed by modern code-generation techniques. Given the vast design space our solution is based on learning and we show that it can find the close to optimal data structure design in a matter of a few seconds.

A class dictionary defines all data structures that appear in a program as well as a language for describing data specified by the data structures. We demonstrate that class dictionaries are ideal for simplifying object-oriented programming. Our class dictionary-based approach to object-oriented programming is independent of any particular programming language, so it is applicable to a large variety of object-oriented systems. The experience in designing and using over one hundred class dictionaries has resulted in a set of useful design techniques. This novel approach to object-oriented programming makes interesting links between language design, data structure design, and data-base design.

The formalism underlying EASYMAP: A precompiler for refinement-based exploration of hierarchical data organizations

Dictionary Look-Up with One Error

Distributed data structures are key to implementing scalable applications for scientific simulations and data analysis. In this paper we look at two implementation styles for distributed data structures: remote direct memory access (RDMA) and remote procedure call (RPC). We focus on operations that require individual accesses to remote portions of a distributed data structure, e.g., accessing a hash table bucket or distributed queue, rather than global operations in which all processors collectively exchange information. We look at the trade-offs between the two styles through microbenchmarks and a performance model that approximates the cost of each. The RDMA operations have direct hardware support in the network and therefore lower latency and overhead, while the RPC operations are more expressive but higher cost and can suffer from lack of attentiveness from the remote side. We also run experiments to compare the real-world performance of RDMA- and RPC-based data structure operations with the predicted performance to evaluate the accuracy of our model, and show that while the model does not always precisely predict running time, it allows us to choose the best implementation in the examples shown. We believe this analysis will assist developers in designing data structures that will perform well on current network architectures, as well as network architects in providing better support for this class of distributed data structures.

The paper deals with the problematics of the dense volume datasets representation, in which voxels are formed as multi-bit scalar values, and is oriented to the use of hierarchical data structures, not only as the medium for encoding and storing of volume datasets, but also for their lossless compression. The main contribution of the paper is in the design of the hierarchical data structure, that allows carving out of subtrees, that are homogenously filled not only by the symbol 0 or symbol 1, but any binary representation of voxel value from defined set of values. Designed data structure provides the possibility of fixed-length and also variable-length (Huffman) encoding of voxel values in leaf node level of the data structure and also in the non-leaf nodes. Results of tests, performed on medical volume datasets that were obtained by various non-invasive imaging techniques, including Computed Tomography and Magnetic Resonance Imaging, along with conclusions that were made, based on the test results, take place in the second part of the paper.

The paper discusses the drawbacks associated with modern web developers' shortage of expertise in working with both unstructured and structured data. While data structures design is generally unnecessary in smaller, highly dynamic and visual presentation-oriented IT projects, it becomes crucial in complex enterprise intranet systems development. Currently, there is a “generation gap” in Russian IT industry, so that younger practitioners are often unaware of long-established patterns in structured data. At the same time, older specialists lack both authority, due to low involvement in real projects, and accessible language to convey their experience. In an attempt to alleviate this problem, the paper provides classification of modern web applications and data design tools, considers relevant problems in information systems development, and proposes some recommendations.

In this paper we will investigate transformations that serve as tools in the design of new data structures. Specifically, we study general methods for converting static structures (in which all elements are known before any searches are performed) to dynamic structures (in which the insertion of a new element can be mixed with searches). We will see three classes of such transformations (each based on a different counting scheme for representing the integers) and then use a combinatorial model to show the optimality of many of the transformations. Issues such as online data structures and deletion of elements are also examined. To demonstrate the applicability of these tools, we will study six new data structures that have been developed by applying the transformations.

The field of stringology studies algorithms and data structures used for processing strings efficiently. The goal of this thesis is to investigate 2-dimensional (2D) variants of some fundamental string problems, including \textit{Exact Pattern Matching} and \textit{Longest Common Substring}. In the 2D pattern matching problem, we are given a matrix $\M[1\dd n,1\dd n]$ that consists of $N = n \times n$ symbols drawn from an alphabet $\Sigma$ of size $\sigma$. The query consists of a $ m \times m$ square matrix $\PP[1\dd m, 1\dd m]$ drawn from the same alphabet, and the task is to find all the locations of $\PP$ in $\M$. For such square patterns, data structures such as suffix trees and suffix arrays exist for the task of efficient pattern matching. However, a suffix tree occupies $O(N \log N)$ bits, which is significantly more than that of the original text's size of $N\log \sigma$ bits. Therefore, the design of compressed data structures, that supports pattern matching queries efficiently and occupies space close to the original text's size, is imperative. In this thesis, we show an interesting result by designing a compact text index of size $O(N \log\log N + N \log\sigma)$ bits that at least supports efficient inverse suffix array queries. Although, the question of designing a compressed text index that would lead to efficient pattern matching is still evasive, this index gives a hope on the existence of a full 2D compressed text index with all functionalities similar to that of 1D case. On the other hand, the Longest Common 2D substring problem consists of two 2D strings (matrices), and the task is to report the size of the longest common 2D substring (submatrix) of these 2D strings. It is interesting to know if there exists a sublinear-time algorithm for solving this task. We answer this question positively by presenting a sublinear-time \textit{quantum} algorithm. In addition to this, we prove that any quantum algorithm requires at least $\tilde{\Omega}(N^{2/3})$ time to solve this problem.

Emergency Assessment and Decision Support System has become an important tool to effectively respond to a nuclear emergency in nuclear power plant, the system mainly used to on-line assessment the real-time reactor conditions, calculate the source term released to the environment, and suggest the protection actions taken to protect the public and emergency workers. This paper describe the development of the system, analysis the business flow chart of nuclear assessment, introduce details of several functional modules, such as core damage assessment, source term calculation and radiological consequence assessment. The system’s data structure and data interface also be describe in detail.

Low-Earth orbit satellite networks have received attention from academia and industry for their advantages in terms of wide coverage and low latency. Meantime deep learning can provide more accurate traffic anomaly detection and has become an important class of methods for LEO satellite network security. However, deep learning is susceptible to adversarial sample attacks, and the LEO satellite network system has not been investigated to find a framework for adversarial sample attacks and defence systems, which poses a potential risk to network communication security. In this paper, we design a framework to generate and defend against adversarial samples in real time. By capturing traffic from LEO satellites, it can generate traffic adversarial samples to detect whether malicious traffic classification models are vulnerable to attacks, and defense against adversarial sample attacks in real time. In this paper, a simple LEO satellite simulation platform is built to generate traffic adversarial samples using four classical adversarial sample generation methods, and a two-classification deep learning model is trained to determine the effectiveness of the attack and defence. Experiments show that the framework proposed in the paper can crawl traffic and perform self-attack and defence tests.KeywordsLEOTraffic classificationAdversarial attack

We present two results for path traversal in trees, where the traversal is performed in an asymptotically optimal number of I/Os and the tree structure is represented succinctly. Our first result is for bottom-up traversal that starts with a node in the tree T and traverses a path to the root. For blocks of size B, a tree on N nodes, and for a path of length K, we design data structures that permit traversal of the bottom-up path in O(K/B) I/Os using only \(2N + \frac{\epsilon N }{\log_B{N}} + o(N)\) bits, for an arbitrarily selected constant, ε, where 0 < ε< 1. Our second result is for top-down traversal in binary trees. We store T using (3 + q)N + o(N) bits, where q is the number of bits required to store a key, while top-down traversal can still be performed in an asymptotically optimal number of I/Os.KeywordsBinary TreeTree BlockDummy NodeDisk BlockReal NodeThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The functional approach to data base design is introduced. In this approach the goal of design is to derive a data structure which is capable of supporting a set of anticipated queries rather than a structure which models the business in some other way. An operational computer program is described which utilizes the functional approach to design data structures conforming to the Data Base Task Group specifications. The automatic programming technology utilized by this program, although typically used to generate procedure, is here used to generate declaratives.