Dynamic external hashing

The linear-probing hash table is one of the oldest and most widely used data structures in computer science. However, linear probing famously comes with a major draw-back: as soon as the hash table reaches a high memory utilization, elements within the hash table begin to cluster together, causing insertions to become slow. This phenomenon, now known as primary clustering, was first captured by Donald Knuth in 1963; at a load factor of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 -1/x$</tex> , the expected time per insertion is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Theta(x^{2})$</tex> , rather than the more desirable <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Theta(x)$</tex> . We show that there is more to the story than the classic analysis would seem to suggest. It turns out that small design decisions in how deletions are implemented have dramatic effects on the asymptotic performance of insertions. If these design decisions are made correctly, then even a hash table that is continuously at a load factor <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1-\Theta(1/x)$</tex> can achieve average insertion time <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\tilde{O}(x)$</tex> . A key insight is that the tombstones left behind by deletions cause a surprisingly strong “anti-clustering” effect, and that when insertions and deletions are one-for-one, the anti-clustering effects of deletions actually overpower the clustering effects of insertions. We also present a new variant of linear probing, which we call graveyard hashing, that completely eliminates primary clustering on any sequence of operations. If, when an operation is performed, the current load factor is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 -1/x$</tex> for some <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x$</tex> , then the expected cost of the operation is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$O(x)$</tex> . One corollary is that, in the external-memory model with a data block size of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$B$</tex> , graveyard hashing offers the following remarkable guarantee: at any load factor <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 -1/x$</tex> satisfying <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x=o(B)$</tex> , graveyard hashing achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 +o(1)$</tex> expected block transfers per operation. Past external-memory hash tables have only been able to offer a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 +o(1)$</tex> guarantee when the block size <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$B$</tex> is at least <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Omega(x^{2})$</tex> . Our results come with actionable lessons for both theoreticians and practitioners, in particular, that well-designed use of tombstones can completely change the asymptotic landscape of how the linear probing behaves (and if there are no deletions).

Data sets in large applications are often too massive to fit completely inside the computers internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this article we survey the state of the art in the design and analysis of external memory (or EM) algorithms and data structures, where the goal is to exploit locality in order to reduce the I/O costs. We consider a variety of EM paradigms for solving batched and online problems efficiently in external memory. For the batched problem of sorting and related problems such as permuting and fast Fourier transform, the key paradigms include distribution and merging. The paradigm of disk striping offers an elegant way to use multiple disks in parallel. For sorting, however, disk striping can be nonoptimal with respect to I/O, so to gain further improvements we discuss distribution and merging techniques for using the disks independently. We also consider useful techniques for batched EM problems involving matrices (such as matrix multiplication and transposition), geometric data (such as finding intersections and constructing convex hulls), and graphs (such as list ranking, connected components, topological sorting, and shortest paths). In the online domain, canonical EM applications include dictionary lookup and range searching. The two important classes of indexed data structures are based upon extendible hashing and B-trees. The paradigms of filtering and bootstrapping provide a convenient means in online data structures to make effective use of the data accessed from disk. We also reexamine some of the above EM problems in slightly different settings, such as when the data items are moving, when the data items are variable-length (e.g., text strings), or when the allocated amount of internal memory can change dynamically. Programming tools and environments are available for simplifying the EM programming task. During the course of the survey, we report on some experiments in the domain of spatial databases using the TPIE system (transparent parallel I/O programming environment). The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.

The chapter provides a comprehensive and in-depth exploration of two fundamental linear data structures in computer science: Stack and Queue. It begins with an introduction to these data structures, highlighting their key features and fundamental differences. It then explains various operations that can be performed on both data structures, such as push, pop, enqueue, and dequeue. These implementations are thoroughly compared and contrasted, highlighting their advantages and disadvantages. These applications are discussed in detail, showcasing the versatility of Stack and Queue data structures in different scenarios. The chapter includes Python code snippets demonstrating Stack and Queue data structures further to aid the reader's understanding of the material. This chapter provides a comprehensive and practical guide to Stack and Queue data structures, making it a valuable resource for computer science students, researchers, and practitioners.

In this paper, a learning model for prediction is introduced by coupling a static neural network with an external stack memory, creating a new type of recurrent system. We analyze the differences between this external memory recurrent network and recurrent neural network, which possesses internal memory. Internal memory remembers the last state while external memory remembers past useful contents. For a specific automaton, the internal memory is needed if the last state is a variable of the state transition function and the external memory is needed if the past content is a variable of state transition function. Our arguments are verified by comparing the prediction accuracy of different models with internal memory, external memory and the combination of them on counting and reversing tasks. The results shows that: network with an external stack works best for counting tasks since the variables of state transition function is composed of the current input and one past input, while network with the combination of internal and external memory works best for the reversing task since the variables of state transition function is composed of the current input, current state and the past inputs.

Previous chapter Next chapter Full AccessProceedings Proceedings of the 2012 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA)Using Hashing to Solve the Dictionary Problem (In External Memory)John Iacono and Mihai PătraşcuJohn IaconoPolytechnic Institute of NYUAT&T Labs*Research supported by NSF grants CCF-1018370, CCF-0430849 and by an Alfred P. Sloan fellowship. Research partially completed as a visiting professor at MADALGO (Center for Massive Data Algorithmics, a Center of the Danish National Research Foundation), Department of Computer Science, Aarhus University, IT Parken, Åbogade 34, DK-8200 Århus N, Denmark.Search for more papers by this author and Mihai PătraşcuPolytechnic Institute of NYUAT&T LabsSearch for more papers by this authorpp.570 - 582Chapter DOI:https://doi.org/10.1137/1.9781611973099.48PDFBibTexSections ToolsAdd to favoritesDownload CitationsTrack CitationsEmail SectionsAboutAbstract We consider the dictionary problem in external memory and improve the update time of the well-known buffer tree by roughly a logarithmic factor. For any λ > max(lg lg n, logM/B (n/B)}, we can support updates in time O(λ/B) and queries in sublogarithmic time, O(logλ n). We also present a lower bound in the cell-probe model showing that our data structure is optimal. In the RAM, hash tables have been use to solve the dictionary problem faster than binary search for more than half a century. By contrast, our data structure is the first to beat the comparison barrier in external memory. Ours is also the first data structure to depart convincingly from the indivisibility paradigm. Previous chapter Next chapter RelatedDetails Published:2012ISBN:978-1-61197-210-8eISBN:978-1-61197-309-9 https://doi.org/10.1137/1.9781611973099Book Series Name:ProceedingsBook Code:PR141Book Pages:xiii + 1757

Tries and PATRICIA tries are fundamental data structures in computer science with numerous applications. In a recent paper, a general framework for obtaining the mean and variance of additive shape parameters of tries and PATRICIA tries under the Bernoulli model was proposed. In this note, we show that a slight modification of this framework yields a central limit theorem for shape parameters, too. This central limit theorem contains many of the previous central limit theorems from the literature and it can be used to prove recent conjectures and derive new results. As an example, we will consider a refinement of the size of tries and PATRICIA tries, namely, the number of nodes of fixed outdegree and obtain (univariate and bivariate) central limit theorems. Moreover, trivariate central limit theorems for size, internal path length and internal Wiener index of tries and PATRICIA tries are derived as well.

Chapter 8 - Memory and Sound

In this chapter we begin reporting the results of our comprehensive survey study on the interplay of internal and external memory in everyday life, and how that is changing in the early twenty-first century (N = 476 Mechanical Turk participants). After first describing data cleaning and scoring, we report the descriptive statistics for questions about attitudes and beliefs regarding internal memory, external memory, and their interplay. The overall pattern, based on participants’ self-reports, is that most think external memory enhances their everyday functioning, but they also think they would do fine without it, and they do not feel uneasy about their reliance on it. External memory is seen as augmenting human capability, not supplanting it. Furthermore, participants generally put considerable value in their external memories.

Article Free Access Share on Efficient searching with linear constraints Authors: Pankaj K. Agarwal Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Lars Arge Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Jeff Erickson Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Paolo G. Franciosa Dipartimento di Informatica e Sistemistica, Università di Roma 'La Sapienza', Via Salaria 113, 00198, Rome Dipartimento di Informatica e Sistemistica, Università di Roma 'La Sapienza', Via Salaria 113, 00198, RomeView Profile , Jeffry Scott Vitter Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile Authors Info & Claims PODS '98: Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systemsMay 1998 Pages 169–178https://doi.org/10.1145/275487.275506Published:01 May 1998Publication History 30citation392DownloadsMetricsTotal Citations30Total Downloads392Last 12 Months16Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

The paper implements a shift based diamond search architecture, for HDTV videos. High resolution frames like 720p need large memory to store the pixel data. This large pixel data is stored on an external memory and an internal local memory is updated with the pixel data on the reference frame and current frame required for particular diamond search iterations. This is done as internal memory is expensive. However, accessing pixel data from external memory incurs high timing latency. This paper implements a shift based DS architecture to shift data in internal local memory and update from external memory only the pixels which are new to search area. This reduces the redundant operation of accessing same pixel data for successive searches. Even when considering equal latency for internal and external memory access, when simulation was performed on the same sample 720p frame, architectures with no shift block took 173.170ms vs with shift block took 90.564ms. When considering the latency of external memory access this difference in timing becomes largely significant.

Distributed algorithms for graph searching require a high-performance CPU-efficient hash table that supports find-or-put. This operation either inserts data or indicates that it has already been added before. This paper focuses on the design and evaluation of such a hash table, targeting supercomputers. The latency of find-or-put is minimized by using one-sided RDMA operations. These operations are overlapped as much as possible to reduce waiting times for roundtrips. In contrast to existing work, we use linear probing and argue that this requires less roundtrips. The hash table is implemented in UPC. A peak-throughput of 114.9 million op/s is reached on an Infiniband cluster. With a load-factor of 0.9, find-or-put can be performed in 4.5μs on average. The hash table performance remains very high, even under high loads.

Matrisler bilgisayar bilimlerinde sıklıkla kullanılan veri yapılarıdır. Üçgensel matris gibi özel matris türleri için programlama dillerinde hazır yapılar bulunmamaktadır. Bir veri yapısı olarak üçgensel matris kullanmak gerekiyorsa programlama dilinden kaynaklanan bu eksiklik nedeniyle bellek ve zaman karmaşıklığı yönünden etkin olmayan yazılımlar kodlanmaktadır. Bu çalışmada üçgensel matris tipindeki yapıların bir boyutlu dizi olarak temsil edilmesinde kullanılan dönüşüm ve ters dönüşüm formülleri elde edilerek programların yer ve zaman karmaşıklığı açısından etkinliğinin arttırılması amaçlanmaktadır.

Hashing is a very common technique for storing data in such a way that the data can be inserted and retrieved very quickly. Hashing uses a data structure called a hash table . Although hash tables provide fast insertion, deletion, and retrieval, they perform poorly for operations that involve searching, such as finding the minimum or maximum value. For these types of operations, other data structures are preferred (see, for example, Chapter 14 on binary search trees). The.NET Framework library provides a very useful class for working with hash tables, the Hashtable class. We will examine this class in this chapter, but we will also discuss how to implement a custom hash table. Building hash tables is not very difficult and the programming techniques used are well worth knowing. AN OVERVIEW OF HASHING A hash table data structure is designed around an array. The array consists of elements 0 through some predetermined size, though we can increase the size later if necessary. Each data item is stored in the array based on some piece of the data, called the key . To store an element in the hash table, the key is mapped into a number in the range of 0 to the hash table size using a function called a hash function . Ideally, the hash function stores each key in its own cell in the array.

In this chapter we introduce the topic of internal memory versus external memory. Internal memory is information stored in one’s brain, and external memory is information stored outside of one’s brain (either socially or technologically). Humans have a long history of using external memory, and we interact extensively with it in our everyday lives. However, research on this topic has been limited and fragmentary. Here we describe the method of our comprehensive survey study on the interplay of internal and external memory in everyday life, and how that is changing in the early twenty-first century (N = 476 Mechanical Turk participants). The survey included quantitative and qualitative questions about attitudes, beliefs, behaviors, and experiences.

We introduce new dynamic set intersection data structures, which we call 2-3 cuckoo filters and hash tables. These structures differ from the standard cuckoo hash tables and cuckoo filters in that they choose two out of three locations to store each item, instead of one out of two, ensuring that any item in an intersection of two structures will have at least one common location in both structures. We demonstrate the utility of these structures by using them in improved algorithms for listing triangles and answering set intersection queries in internal or external memory. For a graph G of n vertices and m edges, our internal-memory triangle listing algorithm runs in O(m⌈(α(G)log w)/w⌉ + k) expected time, where α(G) is the arboricity of G, w is the number of bits in a machine word, and k is the number of output triangles. Our external-memory algorithm uses O(sort(n,α(G))+ sort(m⌈(α(G)log w)/w⌉) + sort(k)) expected number of I/Os.