External memory algorithms and data structures

Hash tables are one of the most fundamental data structures in computer science, in both theory and practice. They are especially useful in external memory, where their query performance approaches the ideal cost of just one disk access. Knuth [16] gave an elegant analysis showing that with some simple collision resolution strategies such as linear probing or chaining, the expected average number of disk I/Os of a lookup is merely 1+1/2Ω(b), where each I/O can read and/or write a disk block containing b items. Inserting a new item into the hash table also costs 1+1/2Ω(b) I/Os, which is again almost the best one can do if the hash table is entirely stored on disk. However, this requirement is unrealistic since any algorithm operating on an external hash table must have some internal memory (at least Ω(1) blocks) to work with. The availability of a small internal memory buffer can dramatically reduce the amortized insertion cost to o(1) I/Os for many external memory data structures. In this paper we study the inherent query-insertion tradeoff of external hash tables in the presence of a memory buffer. In particular, we show that for any constant c>1, if the expected average successful query cost is targeted at 1+O(1/bc) I/Os, then it is not possible to support insertions in less than 1-O(1/bc-1/6) I/Os amortized, which means that the memory buffer is essentially useless. While if the query cost is relaxed to 1+O(1/bc) I/Os for any constant c

The workshop Algorithm Engineering , organized by Giuseppe Italiano (Roma), Petra Mutzel (Dortmund), Peter Sanders (Karlsruhe), and Martin Skutella (Dortmund) was held May 6th – May 12th, 2007. The goal of the workshop was to bring together researchers with different background, e.g., from combinatorial optimization, algorithmic theory, and algorithm engineering, in order to strengthen and foster collaborations in the area of algorithm engineering and to identify key research directions for the future. There were 47 international participants out of which 23 mainly work in the area of algorithm engineering, 11 in algorithm theory, and 13 in combinatorial optimization. A considerable number of participants visited the Oberwolfach Institute (MFO) for the first time. In five survey talks given by renowned specialists in the field, the state of the art in selected areas of high interest was demonstrated. The introductory talk was presented by Giuseppe Italiano (Roma) who gave “a personal and historical perspective” of algorithm engineering. Bill Cook (Atlanta) presented the newest results as well as engineering efforts for solving huge TSP instances. Friedhelm Meyer auf der Heide (Paderborn) suggested new algorithmic challenges in the area of Computer Graphics. Ian Munro (Waterloo) introduced the audience into the area of succinct data structures which is getting increasingly important with huge data sets. In this context, also external memory and cache-oblivious algorithms are getting growing interest. The survey talk by Gerth Brodal (Aarhus) presented theory and practice for these type of algorithms. In addition, we organized a series of sessions containing shorter contributions of roughly 30 minutes. Here, we particularly encouraged young researchers to present their recent results. The focus of the program lay on recent developments in various areas that are relevant for the broad topic of algorithm engineering. Examples are mathematical programming, external memory algorithms, succinct data structures, resilient data structures, dynamic data structures, online algorithms, graph algorithms, geometric computation, analysis of local search algorithms, and algorithmic game-theory. In total, we had 32 talks of which 15 can be assigned to the area of algorithm engineering, 8 to the area of algorithmic theory, and 7 to the area of combinatorial optimization, which we think was an ideal mixture for the purpose of this workshop. We found it encouraging that after the talks a lot of discussion took place between the participants. We also like to mention that Peter Sanders (Karlsruhe) introduced the new DFG Priority Program 1307 Algorithm Engineering , which will start in fall 2007. As the coordinator of this program he pointed out the specific goals of the program and the chances for the area of algorithm engineering. All of the participants had the chance to discuss different aspects of algorithm engineering in the open problem session on Thursday evening. In our view, and as expressed by many of the participants, the workshop was a great success. We (including many participants) have learned a lot during this week, also from the three different research communities present at the workshop, and got many new ideas for exciting research projects. The program of the workshop was well-received by the participants as the good quality of presentations and the enthusiasm of the speakers and the audience gave raise to joint research among the participants. In particular exchange between young researchers and experienced scientists was promoted. We wish to thank Oberwolfach for this unique opportunity to bring together the three different research groups at this great place in order to strengthen the area of algorithm engineering. The following collection of abstracts in chronological order gives a more detailed overview of the program.

We study the problem of list ranking in the parallel external memory (PEM) model. We observe an interesting dual nature for the hardness of the problem due to limited information exchange among the processors about the structure of the list, on the one hand, and its close relationship to the problem of permuting data, which is known to be hard for the external memory models, on the other hand. By carefully defining the power of the computational model, we prove a permuting lower bound in the PEM model. Furthermore, we present a stronger \Omega(log^2 N) lower bound for a special variant of the problem and for a specific range of the model parameters, which takes us a step closer toward proving a non-trivial lower bound for the list ranking problem in the bulk-synchronous parallel (BSP) and MapReduce models. Finally, we also present an algorithm that is tight for a larger range of parameters of the model than in prior work.

In this paper, a learning model for prediction is introduced by coupling a static neural network with an external stack memory, creating a new type of recurrent system. We analyze the differences between this external memory recurrent network and recurrent neural network, which possesses internal memory. Internal memory remembers the last state while external memory remembers past useful contents. For a specific automaton, the internal memory is needed if the last state is a variable of the state transition function and the external memory is needed if the past content is a variable of state transition function. Our arguments are verified by comparing the prediction accuracy of different models with internal memory, external memory and the combination of them on counting and reversing tasks. The results shows that: network with an external stack works best for counting tasks since the variables of state transition function is composed of the current input and one past input, while network with the combination of internal and external memory works best for the reversing task since the variables of state transition function is composed of the current input, current state and the past inputs.

Chapter 8 - Memory and Sound

We present the parallel buffer tree, a parallel external memory (PEM) data structure for batched search problems. This data structure is a non-trivial extension of Arge's sequential buffer tree to a private-cache multiprocessor environment and reduces the number of I/O operations by the number of available processor cores compared to its sequential counterpart, thereby taking full advantage of multicore parallelism.The parallel buffer tree is a search tree data structure that supports the batched parallel processing of a sequence of N insertions, deletions, membership queries, and range queries in the optimal OhOf(psortN + K/PB) parallel I/O complexity, where K is the size of the output reported in the process and psortN is the parallel I/O complexity of sorting N elements using P processors.

Article Free Access Share on Efficient searching with linear constraints Authors: Pankaj K. Agarwal Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Lars Arge Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Jeff Erickson Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile , Paolo G. Franciosa Dipartimento di Informatica e Sistemistica, Università di Roma 'La Sapienza', Via Salaria 113, 00198, Rome Dipartimento di Informatica e Sistemistica, Università di Roma 'La Sapienza', Via Salaria 113, 00198, RomeView Profile , Jeffry Scott Vitter Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NC Center for Geometric Computing, Department of Computer Science, Duke University, Box 90129, Durham, NCView Profile Authors Info & Claims PODS '98: Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systemsMay 1998 Pages 169–178https://doi.org/10.1145/275487.275506Published:01 May 1998Publication History 30citation392DownloadsMetricsTotal Citations30Total Downloads392Last 12 Months16Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In this chapter we begin reporting the results of our comprehensive survey study on the interplay of internal and external memory in everyday life, and how that is changing in the early twenty-first century (N = 476 Mechanical Turk participants). After first describing data cleaning and scoring, we report the descriptive statistics for questions about attitudes and beliefs regarding internal memory, external memory, and their interplay. The overall pattern, based on participants’ self-reports, is that most think external memory enhances their everyday functioning, but they also think they would do fine without it, and they do not feel uneasy about their reliance on it. External memory is seen as augmenting human capability, not supplanting it. Furthermore, participants generally put considerable value in their external memories.

The paper implements a shift based diamond search architecture, for HDTV videos. High resolution frames like 720p need large memory to store the pixel data. This large pixel data is stored on an external memory and an internal local memory is updated with the pixel data on the reference frame and current frame required for particular diamond search iterations. This is done as internal memory is expensive. However, accessing pixel data from external memory incurs high timing latency. This paper implements a shift based DS architecture to shift data in internal local memory and update from external memory only the pixels which are new to search area. This reduces the redundant operation of accessing same pixel data for successive searches. Even when considering equal latency for internal and external memory access, when simulation was performed on the same sample 720p frame, architectures with no shift block took 173.170ms vs with shift block took 90.564ms. When considering the latency of external memory access this difference in timing becomes largely significant.

We introduce new dynamic set intersection data structures, which we call 2-3 cuckoo filters and hash tables. These structures differ from the standard cuckoo hash tables and cuckoo filters in that they choose two out of three locations to store each item, instead of one out of two, ensuring that any item in an intersection of two structures will have at least one common location in both structures. We demonstrate the utility of these structures by using them in improved algorithms for listing triangles and answering set intersection queries in internal or external memory. For a graph G of n vertices and m edges, our internal-memory triangle listing algorithm runs in O(m⌈(α(G)log w)/w⌉ + k) expected time, where α(G) is the arboricity of G, w is the number of bits in a machine word, and k is the number of output triangles. Our external-memory algorithm uses O(sort(n,α(G))+ sort(m⌈(α(G)log w)/w⌉) + sort(k)) expected number of I/Os.

An external memory data structure is presented for maintaining a dynamic set of N two-dimensional points under the insertion and deletion of points, and supporting unsorted 3-sided range reporting queries and top-k queries, where top-k queries report the k points with highest y-value within a given x-range. For any constant 0 < epsilon <= 1/2, a data structure is constructed that supports updates in amortized O(1/(epsilon * B^{1-epsilon}) * log_B(N)) IOs and queries in amortized O(1/epsilon * log_B(N+K/B)) IOs, where B is the external memory block size, and K is the size of the output to the query (for top-k queries K is the minimum of k and the number of points in the query interval). The data structure uses linear space. The update bound is a significant factor B^{1-epsilon} improvement over the previous best update bounds for these two query problems, while staying within the same query and space bounds.

We present a novel disk-based multiresolution triangle mesh data structure that supports paging and view-dependent rendering of very large meshes at interactive frame rates from external memory. Our approach, called XFastMesh, is based on a view-dependent mesh simplification framework that represents half-edge collapse operations in a binary hierarchy known as a merge-tree forest. The proposed technique partitions the merge-tree forest into so-called detail blocks, which consist of binary subtrees, that are stored on disk. We present an efficient external memory data structure and file format that stores all detail information of the multiresolution triangulation method using significantly less storage then previously reported approaches. Furthermore, we present a paging algorithm that provides efficient loading and interactive rendering of large meshes from external memory at varying and view-dependent level-of-detail. The presented approach is highly efficient both in terms of space cost and paging performance.

In this chapter we introduce the topic of internal memory versus external memory. Internal memory is information stored in one’s brain, and external memory is information stored outside of one’s brain (either socially or technologically). Humans have a long history of using external memory, and we interact extensively with it in our everyday lives. However, research on this topic has been limited and fragmentary. Here we describe the method of our comprehensive survey study on the interplay of internal and external memory in everyday life, and how that is changing in the early twenty-first century (N = 476 Mechanical Turk participants). The survey included quantitative and qualitative questions about attitudes, beliefs, behaviors, and experiences.

We present a new suffix array construction algorithm that aims to build, in external memory, the suffix array for an input string of length n measured in the magnitude of tens of Giga characters over a constant or integer alphabet. The core of this algorithm is adapted from the framework of the original internal memory SA-DS algorithm that samples fixed-size d-critical substrings. This new external-memory algorithm, called EM-SA-DS, uses novel cache data structures to construct a suffix array in a sequential scanning manner with good data spatial locality: data is read from or written to disk sequentially. On the assumed external-memory model with RAM capacity Ω (( nB ) 0.5 ), disk capacity O ( n ), and size of each I/O block B , all measured in log n -bit words, the I/O complexity of EM-SA-DS is O ( n / B ). This work provides a general cache-based solution that could be further exploited to develop external-memory solutions for other suffix-array-related problems, for example, computing the longest-common-prefix array, using a modern personal computer with a typical memory configuration of 4GB RAM and a single disk.

In this chapter we present the second half of our thorough interdisciplinary review of the disparate prior literature on external memory and how it interacts with internal memory. Here we gather and review theoretical perspectives from beyond psychology, including disciplines in the sciences, the arts, and the humanities. Disciplines and topics that we cover include philosophy of mind, dynamic systems, personal information management, library and information science, media and cultural studies, photography, collective memory, and anthropology and archaeology. Along the way, we also weave in relevant insights from our own data, reported in Chapters 2, 3 and 4. This review, along with that of Chapter 5, provides solid guideposts for further work on the interplay of internal and external memory.