Graph Processing with Different Data Structures

The graph model enables a broad range of analysis, thus graph processing is an invaluable tool in data analytics. At the heart of every graph-processing system lies a concurrent graph data structure storing the graph. Such a data structure needs to be highly efficient for both graph algorithms and queries. Due to the continuous evolution, the sparsity, and the scale-free nature of real-world graphs, graph-processing systems face the challenge of providing an appropriate graph data structure that enables both fast analytical workloads and low-memory graph mutations. Existing graph structures offer a hard trade-off between read-only performance, update friendliness, and memory consumption upon updates. In this paper, we introduce CSR++, a new graph data structure that removes these trade-offs and enables both fast read-only analytics and quick and memory-friendly mutations. CSR++ combines ideas from CSR, the fastest read-only data structure, and adjacency lists to achieve the best of both worlds. We compare CSR++ to CSR, adjacency lists from the Boost Graph Library, and LLAMA, a state-of-the-art update-friendly graph structure. In our evaluation, which is based on popular graph-processing algorithms executed over real-world graphs, we show that CSR++ remains close to CSR in read-only concurrent performance (within 10% on average), while significantly outperforming CSR (by an order of magnitude) and LLAMA (by almost 2×) with frequent updates.

We present a framework for compiling recurrence equations into native code. In our framework, users specify a system of recurrences, the types of data structures that store inputs and outputs, and scheduling commands for optimization. Our compiler then lowers these specifications into native code that respects the dependencies in the recurrence equations. Our compiler can generate code over both sparse and dense data structures, and determines if the recurrence system is solvable with the provided scheduling primitives. We evaluate the performance and correctness of the generated code on several recurrences, from domains as diverse as dense and sparse matrix solvers, dynamic programming, graph problems, and sparse tensor algebra. We demonstrate that the generated code has competitive performance to hand-optimized implementations in libraries. However, these handwritten libraries target specific recurrences, specific data structures, and specific optimizations. Our system, on the other hand, automatically generates implementations from recurrences, data formats, and schedules, giving our system more generality than library approaches.

Evolving graph processing has enabled the modeling of many complex network systems, e.g., online social networks and gene networks. Existing in-memory graph data structures cannot effectively exploit the current and ongoing adoption of emerging non-volatile main memory (NVMM) for two reasons. (1) Ephemeral graph data structures are not crash-consistent nor durable for NVMM. Corruption is likely when updating its correlated application-defined data and runtime states in the face of hardware or software failures. (2) NVMM writes and reads may incur higher latency than DRAM. Placing the data structures in NVMM may result in a significant loss of performance. In this paper, we propose a novel persistent evolving graph data structure, named NVGRAPH, for both computing and in-memory storage of evolving graphs in NVMM. We devise NVGRAPH as a multi-version data structure, wherein a minimum of one version of its data is stored in NVMM to provide the desired durability at runtime for failure recovery, and another version is stored in both DRAM and NVMM to reduce the NVMM-induced memory latency. We dynamically transform the layout of NVGRAPH including changing the size of its partition in DRAM and the position of its base snapshot exploiting network properties and data access patterns of workloads. For the evaluation of NVGRAPH, we implement four representative real-world graph applications: pagerank, BFS, influence maximization, and rumor source detection. The experimental results show that the performance of NVGRAPH is comparable to other in-memory data structure (e.g., CSR and LLAMA) while using 70% less DRAM. It scales well up to 10 billion edges and 201 snapshots and supports crash consistency. It offers up to the 21X speedup of execution time compared to the scale-up graph computation approaches (e.g., GraphChi and X-stream).

Graph data structures constitute fundamental components within computer science, serving as versatile models for representing interconnected relationships in a myriad of domains, including social networks, transportation systems, and biological networks.However, comprehending the intricacies inherent to graphs often poses a significant challenge for students enrolled in data structures courses.This paper endeavors to explore the transformative potential of visualization techniques in bolstering comprehension and engagement within the realm of graph data structures education.Drawing upon insights gleaned from cognitive psychology and educational research, we embark on an examination of how visualization methodologies can serve as catalysts for fostering deeper understanding and retention of abstract concepts, thereby elevating overall learning outcomes.Through literature review, we meticulously scrutinize a number of studies, each illuminating the efficacy of graph visualization tools in enhancing student comprehension and problem-solving skills.Moreover, we ardently advocate for the adoption of an active learning paradigm, wherein students are not merely passive recipients of knowledge but are actively encouraged to develop their visualization applications.Such an approach not only nurtures creativity and innovation, but also engenders a profound appreciation for the intricate nuances inherent to graph representations.Additionally, we delve into the underlying principles and methodologies of graph visualization, offering a discourse on techniques for crafting visually compelling graph layouts.By synthesizing a rich tapestry of research findings and practical insights, this paper endeavors to furnish educators with some strategies for seamlessly integrating visualization techniques into the fabric of data structures curricula.Ultimately, our overarching objective is to empower students to navigate the labyrinthine complexities of graph data structures with unwavering confidence, consummate proficiency, and reverence for the elegance imbued within graphical representations.

This paper introduces data structures as applied to computer graphics. Design criteria for computer graphics data structures are discussed, followed by a comparison of general-purpose and tailored graphic data structures. A general graphic data structure is introduced as an example of a structure meeting the preceding criteria. The L6language is then examined as a tool for implementing the above data structure, and is compared to a few other language systems.

The shortest-path problem is an essential component of many applications including Advanced Traveler Information Systems (ATIS), computer networks, etc. A hierarchical routing algorithm decomposes the original graph into a set of fragment graphs and also into a boundary graph which summarizes the fragment graphs. A fully memoized hierarchical routing algorithm pre-computes and stores the shortest-path data structure and the shortest-path-cost data structure for the graph fragments, as well as for the boundary graph. The storage cost of the fully memoized approach can be reduced by a virtual or a hybrid memoization approach, where few or none of the relevant data structures have been pre-computed. This paper achieves two firsts. We use a real graph with 123,000 nodes and 313,000 edges, which is two orders of magnitude larger than the graphs used in related work. Second, the paper analyzes the effect of memoizing individual data structures for the storage overhead and computation time of A/sup */-based hierarchical routing algorithms. Analysis and experiments with the Twin Cities metropolitan road-map show that memoizing the shortest-path-cost data structure for the boundary graph provides the best savings in computation time, for a given amount of storage and a small number of fragments. Memoizing the relevant part of the shortest-path-cost data structure for the fragment graphs provides the next best savings, followed by memoizing the shortest-path data structure for the boundary graph.

In this paper we investigate the problem of building a static data structure that represents a string s using space close to its compressed size, and allows fast access to individual characters of s. This type of data structures was investigated by the recent paper of Bille et al. [3]. Let n be the size of a context-free grammar that derives a unique string s of length L. (Note that L might be exponential in n.) Bille et al. showed a data structure that uses space O(n) and allows to query for the i-th character of s using running time O(logL). Their data structure works on a word RAM with a word size of logL bits.

Efficient indexing for semantic search

To leverage modern hardware platforms to their fullest, more and more database systems embrace compilation of query plans to native code. In the research community, there is an ongoing debate about the best way to architect such query compilers. This is perceived to be a difficult task, requiring techniques fundamentally different from traditional interpreted query execution. In this dissertation, we contribute to this discussion by drawing attention to an old but underappreciated idea known as Futamura projections, which fundamentally link interpreters and compilers. Guided by this idea, we demonstrate that efficient query compilation can actually be very simple, using techniques that are no more difficult than writing a query interpreter in a high-level language. We first develop LB2: a high-level query compiler implemented in this style that is competitive with the best compiled query engines both in sequential and parallel execution on the standard TPC-H benchmark. Query engines process a variety of data types and structures including text, spatial, graphs, etc. Several spatial and graph engines are implemented as extensions to relational query engines to leverage optimized memory, storage, and evaluation. Still, the performance of these extensions is often stymied by the interpretive nature of the underlying data management, generic data structures, and the need to execute domain-specific external libraries. On that basis, compiling spatial and graph queries to native code is a desirable avenue to mitigate existing limitations and improve performance. To support compiling spatial queries, we extend the LB2 main-memory query compiler with spatial predicates, indexing structures, and spatial operators. To support compiling graph queries, we extend LB2 with graph data structures and operators. The spatial extension matches the performance of hand-written code and outperforms relational query engines and map-reduce extensions. Similarly, the graph extension matches, and sometimes outperforms, low-level graph engines.

The data structures that are written in a typical data structures course cannot or would not be used in common applications in industry for two reasons. First, they do not work according to their specifications if they are used in a multi-threaded environment. If two threads are deleting adjacent elements of a linked list at the same time, for example, then one of the deletes may not have any effect. Second, they do not fire change events and thus frustrate common design practices of connecting software components. As a result, it is more difficult for the application programmer to separate the data, the structures, the logic, the user interface, and the application code from each other. Here we discuss some issues involved in including these two topics, threads and components, in a course on data structures. Students who learn these techniques will have less to learn when they begin to use and write data structures in industry.

In classification studies in which pattern-recognition methods are used to distinguish active compounds from inactive ones, a type of data structure which we call "asymmetric" can be observed. This type of data structure can be quite common and its occurrence can have a profound effect on the classification analysis outcome. The origin of asymmetric data structure and a strategy or obtaining meaningful classification results when it is observed are discussed and illustrated with an example of active and inactive antimalarial quinones.

Exact-match overlap graphs have been broadly used in the context of DNA assembly and the shortest super string problem where the number of strings n ranges from thousands to billions. The length ℓ of the strings is from 25 to 1000, depending on the DNA sequencing technologies. However, many DNA assemblers using overlap graphs suffer from the need for too much time and space in constructing the graphs. It is nearly impossible for these DNA assemblers to handle the huge amount of data produced by the next-generation sequencing technologies where the number n of strings could be several billions. If the overlap graph is explicitly stored, it would require Ω(n(2)) memory, which could be prohibitive in practice when n is greater than a hundred million. In this article, we propose a novel data structure using which the overlap graph can be compactly stored. This data structure requires only linear time to construct and and linear memory to store. For a given set of input strings (also called reads), we can informally define an exact-match overlap graph as follows. Each read is represented as a node in the graph and there is an edge between two nodes if the corresponding reads overlap sufficiently. A formal description follows. The maximal exact-match overlap of two strings x and y, denoted by ov(max)(x, y), is the longest string which is a suffix of x and a prefix of y. The exact-match overlap graph of n given strings of length ℓ is an edge-weighted graph in which each vertex is associated with a string and there is an edge (x, y) of weight ω=ℓ-|ov(max)(x, y)| if and only if ω ≤ λ, where |ov(max)(x, y)| is the length of ov(max)(x, y) and λ is a given threshold. In this article, we show that the exact-match overlap graphs can be represented by a compact data structure that can be stored using at most (2λ-1)(2⌈logn⌉+⌈logλ⌉)n bits with a guarantee that the basic operation of accessing an edge takes O(log λ) time. We also propose two algorithms for constructing the data structure for the exact-match overlap graph. The first algorithm runs in O(λℓnlogn) worse-case time and requires O(λ) extra memory. The second one runs in O(λℓn) time and requires O(n) extra memory. Our experimental results on a huge amount of simulated data from sequence assembly show that the data structure can be constructed efficiently in time and memory. Our DNA sequence assembler that incorporates the data structure is freely available on the web at http://www.engr.uconn.edu/~htd06001/assembler/leap.zip

Computationally testable models in linguistics focus on declaring data structures and providing exemplar derivations. This paper outlines a comprehensive model of speech production which goes beyond derivations to show how actual instances of utterances can be formally characterised. Utterances contain a wealth of detail beyond the underlying utterance plan: some of this is a function of the mechanism itself (e.g. coarticulation) and some is the result of carefully supervised control. We develop the notion of managed or supervised speech production to enable the inclusion of EXPRESSIVE content in speech. Building on earlier work the Cognitive Phonetics Agent bridges the gap between the physical and cognitive processes in phonetics by controlling the way phonologically determined utterance plans are phonetically rendered in detail. The model is illustrated using different types of data structure which occur in speech, concentrating in particular on an XML characterisation of appropriate structures. We trace a simple utterance through from its phonological plan to a detailed intrinsic allophonic representation to show how stages in the model work.

In-memory data grid (IMDG) is a new technology that enables scalable and low-latency processing of big data by sharding it over the RAMs of multiple servers. In this paper, we explore the design space of IMDGs to identify their advantages and avoid their drawbacks. We present the performance tradeoffs of IMDGs using unit tests on core distributed operations and data structures. For evaluation, we use large-scale graph processing, a challenging task that requires a high degree of communication and coordination between vertices. We find that while IMDGs cannot compete with specialized distributed frameworks (such as Giraph and GraphLab) for batch-mode graph processing, they excel for online graph processing and offer exciting opportunities for social networks and web services applications.

A simple programming language “extension,” Graspe, for processing directed graphs is defined. Graspe consists of a type of directed graph data structure and a set of primitive operations for manipulating these structures. Graspe may be most easily implemented by embedding it in a host language. Emphasis is placed both on Graspe itself and on its method of definition. Commonly, the definition of a language involves definition of the syntactic elements and explanation of the meaning to be assigned them (the semantics). The definition of Graspe here is solely in terms of its semantics; that is, the data structures and operations are defined precisely but without assignment of a particular syntactic representation. Only when the language is implemented is assignment of an explicit syntax necessary. An example of an implementation of Graspe embedded in Lisp is given as an illustration. The advantages and disadvantages of the definition of a language in terms of its semantics are discussed.