Transforming static data structures to dynamic structures

Data sets in large applications are often too massive to fit completely inside the computers internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this article we survey the state of the art in the design and analysis of external memory (or EM) algorithms and data structures, where the goal is to exploit locality in order to reduce the I/O costs. We consider a variety of EM paradigms for solving batched and online problems efficiently in external memory. For the batched problem of sorting and related problems such as permuting and fast Fourier transform, the key paradigms include distribution and merging. The paradigm of disk striping offers an elegant way to use multiple disks in parallel. For sorting, however, disk striping can be nonoptimal with respect to I/O, so to gain further improvements we discuss distribution and merging techniques for using the disks independently. We also consider useful techniques for batched EM problems involving matrices (such as matrix multiplication and transposition), geometric data (such as finding intersections and constructing convex hulls), and graphs (such as list ranking, connected components, topological sorting, and shortest paths). In the online domain, canonical EM applications include dictionary lookup and range searching. The two important classes of indexed data structures are based upon extendible hashing and B-trees. The paradigms of filtering and bootstrapping provide a convenient means in online data structures to make effective use of the data accessed from disk. We also reexamine some of the above EM problems in slightly different settings, such as when the data items are moving, when the data items are variable-length (e.g., text strings), or when the allocated amount of internal memory can change dynamically. Programming tools and environments are available for simplifying the EM programming task. During the course of the survey, we report on some experiments in the domain of spatial databases using the TPIE system (transparent parallel I/O programming environment). The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.

Range selection is the problem of preprocessing an input array A of n unique integers, such that given a query (i, j, k), one can report the k'th smallest integer in the subarray A[i], A[i + 1], …, A[j]. In this paper we consider static data structures in the word-RAM for range selection and several natural special cases thereof. The first special case is known as range median, which arises when k is fixed to ⌊(j − i + 1)/2⌋. The second case, denoted prefix selection, arises when i is fixed to 0. Finally, we also consider the bounded rank prefix selection problem and the fixed rank range selection problem. In the former, data structures must support prefix selection queries under the assumption that k ≤ κ for some value κ ≤ n given at construction time, while in the latter, data structures must support range selection queries where k is fixed beforehand for all queries. We prove cell probe lower bounds for range selection, prefix selection and range median, stating that any data structure that uses S words of space needs Ω(log n/log(Sw/n)) time to answer a query. In particular, any data structure that uses n logO(1) n space needs Ω(log n/ log log n) time to answer a query, and any data structure that supports queries in constant time, needs n1+Ω(1) space. For data structures that uses n logO(1) n space this matches the best known upper bound. Additionally, we present a linear space data structure that supports range selection queries in O(log k/log log n + log log n) time. Finally, we prove that any data structure that uses S space, needs Ω(log κ/log(Sw/n)) time to answer a bounded rank prefix selection query and Ω(log k/log(Sw/n)) time to answer a fixed rank range selection query. This shows that our data structure is optimal except for small values of k.

Faster proximity searching with the distal SAT

Modern embedded systems are now capable of executing complex and demanding applications that are often based on large data structures. The design of the critical data structures of the application affects the performance and the memory requirements of the whole system. Dynamic Data Structure Refinement methodology provides optimizations, mainly in list and array data structures, which are based on the application's features and access patterns. In this work, we extend various aspects of the methodology: First, we integrate radix tree optimizations. Then, we provide a set of platform-aware data structure implementations, for performing optimizations based on the hardware features. The extended methodology is evaluated using a wide set of synthetic and real-world benchmarks, in which we achieved performance and memory trade-offs up to 29.6%. Additionally, Pareto optimal data structure implementations that were not available by the previous methodology, are identified with the extended one.

This paper presents the analysis of element deletion in different data structures, like linked lists, binary search trees, stacks, and queues. For random data sequence, we build the linked list and binary search tree and we compare the element deletion procedure. We find the much more stable performance of the binary search tree than that of the linked list. Additionally, we analyze the stack as a complex data structure and its basic operations. We present the element deletion with the basic operations for array stack implementation and we note the limitations. Also, we present the element deletion problem for a queue as a complex data structure, implemented with an array. We present the element deletion with the basic queue operations and we point out the limitations.

The field of stringology studies algorithms and data structures used for processing strings efficiently. The goal of this thesis is to investigate 2-dimensional (2D) variants of some fundamental string problems, including \textit{Exact Pattern Matching} and \textit{Longest Common Substring}. In the 2D pattern matching problem, we are given a matrix $\M[1\dd n,1\dd n]$ that consists of $N = n \times n$ symbols drawn from an alphabet $\Sigma$ of size $\sigma$. The query consists of a $ m \times m$ square matrix $\PP[1\dd m, 1\dd m]$ drawn from the same alphabet, and the task is to find all the locations of $\PP$ in $\M$. For such square patterns, data structures such as suffix trees and suffix arrays exist for the task of efficient pattern matching. However, a suffix tree occupies $O(N \log N)$ bits, which is significantly more than that of the original text's size of $N\log \sigma$ bits. Therefore, the design of compressed data structures, that supports pattern matching queries efficiently and occupies space close to the original text's size, is imperative. In this thesis, we show an interesting result by designing a compact text index of size $O(N \log\log N + N \log\sigma)$ bits that at least supports efficient inverse suffix array queries. Although, the question of designing a compressed text index that would lead to efficient pattern matching is still evasive, this index gives a hope on the existence of a full 2D compressed text index with all functionalities similar to that of 1D case. On the other hand, the Longest Common 2D substring problem consists of two 2D strings (matrices), and the task is to report the size of the longest common 2D substring (submatrix) of these 2D strings. It is interesting to know if there exists a sublinear-time algorithm for solving this task. We answer this question positively by presenting a sublinear-time \textit{quantum} algorithm. In addition to this, we prove that any quantum algorithm requires at least $\tilde{\Omega}(N^{2/3})$ time to solve this problem.

A range of attacks on network components, such as algorithmic denial-of-service attacks and cryptanalysis via timing attacks, are enabled by data structures for which an adversary can predict the durations of operations that he will induce on the data structure. In this paper we introduce the problem of designing data structures that confound an adversary attempting to predict the timing of future operations he induces, even if he has adaptive and exclusive access to the data structure and the timings of past operations. We also design a data structure for implementing a set (supporting membership query, insertion, and deletion) that exhibits timing unpredictability and that retains its efficiency despite adversarial attacks. To demonstrate these advantages, we develop a framework by which an adversary tracks a probability distribution on the data structure’s state based on the timings it emitted, and infers invocations to meet his attack goals.KeywordsBinary Search TreeAverage EntropyAbstract Data TypeLinked ListUnpredictable TimingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We study the nondeterministic cell-probe complexity of static data structures. We introduce cell-probe proofs (CPP), a proof system for the cell-probe model, which describes verification instead of computation in the cell-probe model. We present a combinatorial characterization of CPP. With this novel tool, we prove the following lower bounds for the nondeterministic cell-probe complexity of static data structures. –There exists a data structure problem with high nondeterministic cell-probe complexity. –For the exact nearest neighbor search (NNS) problem or the partial match problem in high dimensional Hamming space, for any data structure with Poly( n ) cells, each of which contains O ( n C ) bits where C < 1, the nondeterministic cell-probe complexity is at least Ω(log( d /log n )), where d is the dimension and n is the number of points in the data set. –For the polynomial evaluation problem of d -degree polynomial over finite field of size 2 k where d ≤ 2 k , for any data structure with s cells, each of which contains b bits, the nondeterministic cell-probe complexity is at least min (( k / b ( d − 1)), ( k −log( d −1)/log s )).

The functional approach to data base design is introduced. In this approach the goal of design is to derive a data structure which is capable of supporting a set of anticipated queries rather than a structure which models the business in some other way. An operational computer program is described which utilizes the functional approach to design data structures conforming to the Data Base Task Group specifications. The automatic programming technology utilized by this program, although typically used to generate procedure, is here used to generate declaratives.

This study aims to demonstrate the application of big data models in the evaluation of manufacturing competitiveness across different countries and economic entities. It involves analyzing algorithms and data structures, designing algorithms for evaluating manufacturing competitiveness suitable for different countries and economic entities, and providing algorithmic and theoretical support for building computer assessment models for manufacturing competitiveness in economic entities. The research adopts qualitative and quantitative methods, comparing and analyzing empirical data on manufacturing resource endowment and competitiveness in various countries, combining theoretical analysis and empirical data. By conducting a literature review and empirical research, relevant information and data are collected to establish the relationship between manufacturing resource endowment index and competitiveness. Through case studies and comparative research, this study explores the main factors affecting manufacturing competitiveness in different countries and their interrelationships. It confirms that the competitiveness of manufacturing industries in different countries is influenced by factors such as resource endowment. Further analysis quantifies the relationship between national manufacturing resource endowment and competitiveness, leading to the design of computer algorithms, data structures, and processing logic. The results demonstrate the significant role of resource endowment in manufacturing competitiveness, serving as a crucial basis for formulating comprehensive national strategies. The innovations of this study include: 1. Introducing the manufacturing resource endowment index as a key variable for evaluating competitiveness, enriching the content of competitiveness big data models. 2. Designing comprehensive data structures, algorithms, and processing logic for assessing manufacturing competitiveness from multiple perspectives. 3. In-depth analysis of the relationship and analytical methods between manufacturing strategies, resource endowment, and competitiveness, providing theoretical and model references for the design of computer big data operation platforms.

The paper discusses the drawbacks associated with modern web developers' shortage of expertise in working with both unstructured and structured data. While data structures design is generally unnecessary in smaller, highly dynamic and visual presentation-oriented IT projects, it becomes crucial in complex enterprise intranet systems development. Currently, there is a “generation gap” in Russian IT industry, so that younger practitioners are often unaware of long-established patterns in structured data. At the same time, older specialists lack both authority, due to low involvement in real projects, and accessible language to convey their experience. In an attempt to alleviate this problem, the paper provides classification of modern web applications and data design tools, considers relevant problems in information systems development, and proposes some recommendations.

Sequences occur naturally in many domains such as biology, engineering, finance and scientific research. Since humans have the inherent ability to comprehend and utilize sequences in day to day cognitive tasks such as speech, vision and motor control; biologically inspired sequence learning techniques are used for explanatory data analysis in these domains. Identifying the common substrings which exist in sequences helps in determining the underlying structure and calculating the similarity between sequences. The suffix trie, suffix tree and suffix array are data structures which are used in many solutions to sequence based problems. However, these are static data structures and not flexible tools which can be used for sequence learning. In this paper we present the Adaptive Suffix Trie algorithm, a sequence learning algorithm which can be used for identifying substrings of different lengths and frequencies from a given set of sequences. In contrast to suffix data structures which store all suffixes, the adaptive suffix trie only captures the frequent substrings that occur in the given dataset, resulting in a less complex structure with only the relevant or useful information. We show how the algorithms' learning parameters can be adapted for extracting substrings with the required characteristics and then demonstrate it's application in the classification of biological sequences.

A class dictionary defines all data structures that appear in a program as well as a language for describing data specified by the data structures. We demonstrate that class dictionaries are ideal for simplifying object-oriented programming. Our class dictionary-based approach to object-oriented programming is independent of any particular programming language, so it is applicable to a large variety of object-oriented systems. The experience in designing and using over one hundred class dictionaries has resulted in a set of useful design techniques. This novel approach to object-oriented programming makes interesting links between language design, data structure design, and data-base design.

We present practical learning-based search algorithms that make it possible to automatically design key-value data structures. Our work allows searching within a space of 10100 possible designs for the optimal data structure. The input is a workload specification and the output is an abstract syntax tree which can be absorbed by modern code-generation techniques. Given the vast design space our solution is based on learning and we show that it can find the close to optimal data structure design in a matter of a few seconds.

The formalism underlying EASYMAP: A precompiler for refinement-based exploration of hierarchical data organizations