Data structures for computer graphics

To leverage modern hardware platforms to their fullest, more and more database systems embrace compilation of query plans to native code. In the research community, there is an ongoing debate about the best way to architect such query compilers. This is perceived to be a difficult task, requiring techniques fundamentally different from traditional interpreted query execution. In this dissertation, we contribute to this discussion by drawing attention to an old but underappreciated idea known as Futamura projections, which fundamentally link interpreters and compilers. Guided by this idea, we demonstrate that efficient query compilation can actually be very simple, using techniques that are no more difficult than writing a query interpreter in a high-level language. We first develop LB2: a high-level query compiler implemented in this style that is competitive with the best compiled query engines both in sequential and parallel execution on the standard TPC-H benchmark. Query engines process a variety of data types and structures including text, spatial, graphs, etc. Several spatial and graph engines are implemented as extensions to relational query engines to leverage optimized memory, storage, and evaluation. Still, the performance of these extensions is often stymied by the interpretive nature of the underlying data management, generic data structures, and the need to execute domain-specific external libraries. On that basis, compiling spatial and graph queries to native code is a desirable avenue to mitigate existing limitations and improve performance. To support compiling spatial queries, we extend the LB2 main-memory query compiler with spatial predicates, indexing structures, and spatial operators. To support compiling graph queries, we extend LB2 with graph data structures and operators. The spatial extension matches the performance of hand-written code and outperforms relational query engines and map-reduce extensions. Similarly, the graph extension matches, and sometimes outperforms, low-level graph engines.

The graph model enables a broad range of analysis, thus graph processing is an invaluable tool in data analytics. At the heart of every graph-processing system lies a concurrent graph data structure storing the graph. Such a data structure needs to be highly efficient for both graph algorithms and queries. Due to the continuous evolution, the sparsity, and the scale-free nature of real-world graphs, graph-processing systems face the challenge of providing an appropriate graph data structure that enables both fast analytical workloads and low-memory graph mutations. Existing graph structures offer a hard trade-off between read-only performance, update friendliness, and memory consumption upon updates. In this paper, we introduce CSR++, a new graph data structure that removes these trade-offs and enables both fast read-only analytics and quick and memory-friendly mutations. CSR++ combines ideas from CSR, the fastest read-only data structure, and adjacency lists to achieve the best of both worlds. We compare CSR++ to CSR, adjacency lists from the Boost Graph Library, and LLAMA, a state-of-the-art update-friendly graph structure. In our evaluation, which is based on popular graph-processing algorithms executed over real-world graphs, we show that CSR++ remains close to CSR in read-only concurrent performance (within 10% on average), while significantly outperforming CSR (by an order of magnitude) and LLAMA (by almost 2×) with frequent updates.

The shortest-path problem is an essential component of many applications including Advanced Traveler Information Systems (ATIS), computer networks, etc. A hierarchical routing algorithm decomposes the original graph into a set of fragment graphs and also into a boundary graph which summarizes the fragment graphs. A fully memoized hierarchical routing algorithm pre-computes and stores the shortest-path data structure and the shortest-path-cost data structure for the graph fragments, as well as for the boundary graph. The storage cost of the fully memoized approach can be reduced by a virtual or a hybrid memoization approach, where few or none of the relevant data structures have been pre-computed. This paper achieves two firsts. We use a real graph with 123,000 nodes and 313,000 edges, which is two orders of magnitude larger than the graphs used in related work. Second, the paper analyzes the effect of memoizing individual data structures for the storage overhead and computation time of A/sup */-based hierarchical routing algorithms. Analysis and experiments with the Twin Cities metropolitan road-map show that memoizing the shortest-path-cost data structure for the boundary graph provides the best savings in computation time, for a given amount of storage and a small number of fragments. Memoizing the relevant part of the shortest-path-cost data structure for the fragment graphs provides the next best savings, followed by memoizing the shortest-path data structure for the boundary graph.

Graphs play an increasingly important role in various big data applications. However, existing graph data structures cannot simultaneously address the performance bottlenecks caused by the dynamic updates, large scale, and high query complexity of current graphs. This paper proposes a novel data structure for large-scale dynamic graphs called CuckooGraph. It does not require any prior knowledge of the upcoming graphs, and can adaptively resize to the most memory-efficient form while requiring few memory accesses for very fast graph data processing. The key techniques of CuckooGraph include TRANSFORMATION and DENYLIST. TRANSFORMATION fully utilizes the limited memory by designing related data structures that allow flexible space transformations to smoothly expand/tighten the required space depending on the number of incoming items. DENYLIST efficiently handles item insertion failures and further improves processing speed. Our experimental results show that compared with the most competitive solution Spruce, Cuckoo-Graph achieves about 33× higher insertion throughput while requiring only about 68% of the memory space.

Graph data structures constitute fundamental components within computer science, serving as versatile models for representing interconnected relationships in a myriad of domains, including social networks, transportation systems, and biological networks.However, comprehending the intricacies inherent to graphs often poses a significant challenge for students enrolled in data structures courses.This paper endeavors to explore the transformative potential of visualization techniques in bolstering comprehension and engagement within the realm of graph data structures education.Drawing upon insights gleaned from cognitive psychology and educational research, we embark on an examination of how visualization methodologies can serve as catalysts for fostering deeper understanding and retention of abstract concepts, thereby elevating overall learning outcomes.Through literature review, we meticulously scrutinize a number of studies, each illuminating the efficacy of graph visualization tools in enhancing student comprehension and problem-solving skills.Moreover, we ardently advocate for the adoption of an active learning paradigm, wherein students are not merely passive recipients of knowledge but are actively encouraged to develop their visualization applications.Such an approach not only nurtures creativity and innovation, but also engenders a profound appreciation for the intricate nuances inherent to graph representations.Additionally, we delve into the underlying principles and methodologies of graph visualization, offering a discourse on techniques for crafting visually compelling graph layouts.By synthesizing a rich tapestry of research findings and practical insights, this paper endeavors to furnish educators with some strategies for seamlessly integrating visualization techniques into the fabric of data structures curricula.Ultimately, our overarching objective is to empower students to navigate the labyrinthine complexities of graph data structures with unwavering confidence, consummate proficiency, and reverence for the elegance imbued within graphical representations.

Evolving graph processing has enabled the modeling of many complex network systems, e.g., online social networks and gene networks. Existing in-memory graph data structures cannot effectively exploit the current and ongoing adoption of emerging non-volatile main memory (NVMM) for two reasons. (1) Ephemeral graph data structures are not crash-consistent nor durable for NVMM. Corruption is likely when updating its correlated application-defined data and runtime states in the face of hardware or software failures. (2) NVMM writes and reads may incur higher latency than DRAM. Placing the data structures in NVMM may result in a significant loss of performance. In this paper, we propose a novel persistent evolving graph data structure, named NVGRAPH, for both computing and in-memory storage of evolving graphs in NVMM. We devise NVGRAPH as a multi-version data structure, wherein a minimum of one version of its data is stored in NVMM to provide the desired durability at runtime for failure recovery, and another version is stored in both DRAM and NVMM to reduce the NVMM-induced memory latency. We dynamically transform the layout of NVGRAPH including changing the size of its partition in DRAM and the position of its base snapshot exploiting network properties and data access patterns of workloads. For the evaluation of NVGRAPH, we implement four representative real-world graph applications: pagerank, BFS, influence maximization, and rumor source detection. The experimental results show that the performance of NVGRAPH is comparable to other in-memory data structure (e.g., CSR and LLAMA) while using 70% less DRAM. It scales well up to 10 billion edges and 201 snapshots and supports crash consistency. It offers up to the 21X speedup of execution time compared to the scale-up graph computation approaches (e.g., GraphChi and X-stream).

Data-processing methods have evolved because of the high demand due to the development of information technology. Specifically, data created by individuals are being generated very rapidly through social-media services, and thus, they assume importance in service personalization. Personal data involve very complex relations and various lists. It is difficult to develop a data system for complex relational lists, using the traditional relational-table model. There are several approaches to addressing this limitation in the relational-table model. The graph structure is one such emerging approach. In graph data, each object in the data constitutes a node, and relations between objects constitute links. The graph data structure has been used in commercial products, such as Google Cayley and Amazon Neptune. In this study, we generalize the graph data structure to a topological data structure and demonstrate a method to transform topological data into a sentence structure. We suggest the use of query functions and provide relevant examples.

We present a new succinct data structure for graphs that are “tree-like,” in the sense that the number of “additional” edges (w.r.t. a spanning tree) is not too high. Our algorithmic idea is to represent a BFS-spanning tree of the graph with a succinct data structure for trees, and enhance it with additional information that accounts for the non-tree edges. In practical tests, our data structure performs well for graphs containing up to 10% of non-tree edges, reducing the space of a pointer-based representation by a factor of ≈20, while increasing the worst-case running times for the operations by roughly the same factor.

The paper describes graph algorithms performance when using different types of data structures. To achieve that, we developed a multi-level graph processing system, which allows to create graph applications independently of any implementation details such as graph data structure or underlying computational architecture. We measure the performance of breadth-first search, max flow and random graph building algorithms when using compressed sparse row and adjacency matrix data structures. Experiments reveal different graph processing rates for different data structures, which indicates the need of using specific data structures for specific algorithms to achieve highest performance.

As a non-linear data structure consisting of nodes and edges, the graph data span many different domains. In the real world, applications based on such data structures are always time-sensitive, that is, the value of graph data tends to decrease with time. Furthermore, the application based on spatio-temporal graph is one of the typical representatives of time-sensitive, since the time dimension is an inherent feature of spatio-temporal data. The Distributed Stream Processing Engine (DSPE) seems an excellent choice for the above requirement, which is commonly partitioned and concurrently processed by a number of threads to maximize the throughput. However, it is not feasible to do such mission directly using the traditional DSPE. In this paper, we propose a computational model suitable for handling the spatio-temporal graph in DSPE, by reconstructing the DSPE’s parallel processing slots. Specifically, our proposal includes a general processing framework to deal with the data structure of the spatio-temporal graph, a state information compensation mechanism to ensure the correctness of processing such stateful operation in DSPE, a lightweight summary information calculation method to ensure the performance of the system. Empirical studies on real-world stream applications validate the usefulness of our proposals and prove the considerable advantage of our approaches over state-of-the-art solutions in the literature.

Exact-match overlap graphs have been broadly used in the context of DNA assembly and the shortest super string problem where the number of strings n ranges from thousands to billions. The length ℓ of the strings is from 25 to 1000, depending on the DNA sequencing technologies. However, many DNA assemblers using overlap graphs suffer from the need for too much time and space in constructing the graphs. It is nearly impossible for these DNA assemblers to handle the huge amount of data produced by the next-generation sequencing technologies where the number n of strings could be several billions. If the overlap graph is explicitly stored, it would require Ω(n(2)) memory, which could be prohibitive in practice when n is greater than a hundred million. In this article, we propose a novel data structure using which the overlap graph can be compactly stored. This data structure requires only linear time to construct and and linear memory to store. For a given set of input strings (also called reads), we can informally define an exact-match overlap graph as follows. Each read is represented as a node in the graph and there is an edge between two nodes if the corresponding reads overlap sufficiently. A formal description follows. The maximal exact-match overlap of two strings x and y, denoted by ov(max)(x, y), is the longest string which is a suffix of x and a prefix of y. The exact-match overlap graph of n given strings of length ℓ is an edge-weighted graph in which each vertex is associated with a string and there is an edge (x, y) of weight ω=ℓ-|ov(max)(x, y)| if and only if ω ≤ λ, where |ov(max)(x, y)| is the length of ov(max)(x, y) and λ is a given threshold. In this article, we show that the exact-match overlap graphs can be represented by a compact data structure that can be stored using at most (2λ-1)(2⌈logn⌉+⌈logλ⌉)n bits with a guarantee that the basic operation of accessing an edge takes O(log λ) time. We also propose two algorithms for constructing the data structure for the exact-match overlap graph. The first algorithm runs in O(λℓnlogn) worse-case time and requires O(λ) extra memory. The second one runs in O(λℓn) time and requires O(n) extra memory. Our experimental results on a huge amount of simulated data from sequence assembly show that the data structure can be constructed efficiently in time and memory. Our DNA sequence assembler that incorporates the data structure is freely available on the web at http://www.engr.uconn.edu/~htd06001/assembler/leap.zip

The GRClass system, which combines graphics, relations, and classes to provide a visual interface for programming graph data structures within an object-oriented framework, is discussed. This is done by extending the object-oriented model with interobject relations. These relations are then used to directly implement the conceptual model of the graph data structures. Within the GRClass framework, data structures are objects that maintain relation tables. These relations and the objects participating in the relations constitute the form of the data structure. A graphical notation is used to specify the possible relations and to manipulate the relation graph. GRClass is implemented within the Andrew Toolkit programming environment. >

IV.6 - MAINTAINING WINGED-EDGE MODELS

Probabilistic modeling of lexicalized grammars is difficult because these grammars exploit complicated data structures, such as typed feature structures. This prevents us from applying common methods of probabilistic modeling in which a complete structure is divided into sub-structures under the assumption of statistical independence among sub-structures. For example, part-of-speech tagging of a sentence is decomposed into tagging of each word, and CFG parsing is split into applications of CFG rules. These methods have relied on the structure of the target problem, namely lattices or trees, and cannot be applied to graph structures including typed feature structures. This article proposes the feature forest model as a solution to the problem of probabilistic modeling of complex data structures including typed feature structures. The feature forest model provides a method for probabilistic modeling without the independence assumption when probabilistic events are represented with feature forests. Feature forests are generic data structures that represent ambiguous trees in a packed forest structure. Feature forest models are maximum entropy models defined over feature forests. A dynamic programming algorithm is proposed for maximum entropy estimation without unpacking feature forests. Thus probabilistic modeling of any data structures is possible when they are represented by feature forests. This article also describes methods for representing HPSG syntactic structures and predicate-argument structures with feature forests. Hence, we describe a complete strategy for developing probabilistic models for HPSG parsing. The effectiveness of the proposed methods is empirically evaluated through parsing experiments on the Penn Treebank, and the promise of applicability to parsing of real-world sentences is discussed.

We have presented two versions of the software XploRe which implement our ideas about data structures in statistical software. The data structures for graphics and linking are implemented in XploRe 3.2 while the data structures for statistical data are implemented in XploRe 4.0. But we have not implemented all ideas.KeywordsMain MenuMultidimensional ArrayProjection Pursuit RegressionWavelet RegressionGraphical Data StructureThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.