Distributed and parallel processing for real-time and dynamic spatio-temporal graph

Recommender systems are ubiquitous in the modern internet, where they help users find items they might like. A widely deployed recommendation approach is item-based collaborative filtering. This approach relies on analyzing large item cooccurrence matrices that denote how many users interacted with a pair of items. The potentially quadratic number of items to compare poses a scalability bottleneck in analyzing such item cooccurrences. Additionally, this problem intensifies in real world use cases with incrementally growing datasets, especially when the recommendation model is regularly recomputed from scratch. We highlight the connection between the growing cost of item-based recommendation and densification processes in common interaction datasets. Based on our findings, we propose an efficient incremental algorithm for item-based collaborative filtering based on cooccurrence analysis. This approach restricts the number of interactions to consider from 'power users' and 'ubiquitous items' to guarantee a provably constant amount of work per user-item interaction to process. We discuss efficient implementations of our algorithm on a single machine as well as on a distributed stream processing engine, and present an extensive experimental evaluation. Our results confirm the asymptotic benefits of the incremental approach. Furthermore, we find that our implementation is an order of magnitude faster than existing open source recommender libraries on many datasets, and at the same time scales to high dimensional datasets which these existing recommenders fail to process.

Graphs play an increasingly important role in various big data applications. However, existing graph data structures cannot simultaneously address the performance bottlenecks caused by the dynamic updates, large scale, and high query complexity of current graphs. This paper proposes a novel data structure for large-scale dynamic graphs called CuckooGraph. It does not require any prior knowledge of the upcoming graphs, and can adaptively resize to the most memory-efficient form while requiring few memory accesses for very fast graph data processing. The key techniques of CuckooGraph include TRANSFORMATION and DENYLIST. TRANSFORMATION fully utilizes the limited memory by designing related data structures that allow flexible space transformations to smoothly expand/tighten the required space depending on the number of incoming items. DENYLIST efficiently handles item insertion failures and further improves processing speed. Our experimental results show that compared with the most competitive solution Spruce, Cuckoo-Graph achieves about 33× higher insertion throughput while requiring only about 68% of the memory space.

The proliferation of real-time artificial intelligence (AI) and machine learning (ML) systems has amplified the demand for robust, low-latency state management techniques capable of operating at scale. From streaming feature extraction to online model inference and complex event processing, stateful operations lie at the core of intelligent data-driven pipelines. However, managing this state in distributed environments presents numerous challenges, including fault tolerance, efficient recovery, incremental updates, and tight latency budgets. This paper explores RocksDB, a high-performance, embeddable key-value store based on a log-structured merge-tree (LSM) architecture, as a state backend solution for real-time applications. We delve into the internal mechanisms that make RocksDB particularly well-suited for low-latency, high-throughput workloads, such as background compaction, memory/disk tiering, and custom serialization strategies. The article details practical integration techniques with distributed stream processing engines such as Apache Flink and Kafka Streams, emphasizing checkpointing, state TTL, and asynchronous snapshotting. We also introduce a set of design patterns for real-time AI/ML applications — including online feature stores, real-time recommender systems, and stateful anomaly detection — and show how RocksDB enables efficient, fault-tolerant management of evolving application state. Through empirical evaluations, we benchmark performance trade-offs between RocksDB and alternative backends (e.g., in-memory, Redis, Cassandra), and present optimizations that significantly improve state access latency, recovery time, and disk footprint. By providing a comprehensive review of RocksDB’s role in real-time state management, this work serves as both a scholarly reference and a practical guide for engineers, researchers, and system architects building the next generation of AI/ML-driven streaming systems.

The graph model enables a broad range of analysis, thus graph processing is an invaluable tool in data analytics. At the heart of every graph-processing system lies a concurrent graph data structure storing the graph. Such a data structure needs to be highly efficient for both graph algorithms and queries. Due to the continuous evolution, the sparsity, and the scale-free nature of real-world graphs, graph-processing systems face the challenge of providing an appropriate graph data structure that enables both fast analytical workloads and low-memory graph mutations. Existing graph structures offer a hard trade-off between read-only performance, update friendliness, and memory consumption upon updates. In this paper, we introduce CSR++, a new graph data structure that removes these trade-offs and enables both fast read-only analytics and quick and memory-friendly mutations. CSR++ combines ideas from CSR, the fastest read-only data structure, and adjacency lists to achieve the best of both worlds. We compare CSR++ to CSR, adjacency lists from the Boost Graph Library, and LLAMA, a state-of-the-art update-friendly graph structure. In our evaluation, which is based on popular graph-processing algorithms executed over real-world graphs, we show that CSR++ remains close to CSR in read-only concurrent performance (within 10% on average), while significantly outperforming CSR (by an order of magnitude) and LLAMA (by almost 2×) with frequent updates.

Data-processing methods have evolved because of the high demand due to the development of information technology. Specifically, data created by individuals are being generated very rapidly through social-media services, and thus, they assume importance in service personalization. Personal data involve very complex relations and various lists. It is difficult to develop a data system for complex relational lists, using the traditional relational-table model. There are several approaches to addressing this limitation in the relational-table model. The graph structure is one such emerging approach. In graph data, each object in the data constitutes a node, and relations between objects constitute links. The graph data structure has been used in commercial products, such as Google Cayley and Amazon Neptune. In this study, we generalize the graph data structure to a topological data structure and demonstrate a method to transform topological data into a sentence structure. We suggest the use of query functions and provide relevant examples.

The shortest-path problem is an essential component of many applications including Advanced Traveler Information Systems (ATIS), computer networks, etc. A hierarchical routing algorithm decomposes the original graph into a set of fragment graphs and also into a boundary graph which summarizes the fragment graphs. A fully memoized hierarchical routing algorithm pre-computes and stores the shortest-path data structure and the shortest-path-cost data structure for the graph fragments, as well as for the boundary graph. The storage cost of the fully memoized approach can be reduced by a virtual or a hybrid memoization approach, where few or none of the relevant data structures have been pre-computed. This paper achieves two firsts. We use a real graph with 123,000 nodes and 313,000 edges, which is two orders of magnitude larger than the graphs used in related work. Second, the paper analyzes the effect of memoizing individual data structures for the storage overhead and computation time of A/sup */-based hierarchical routing algorithms. Analysis and experiments with the Twin Cities metropolitan road-map show that memoizing the shortest-path-cost data structure for the boundary graph provides the best savings in computation time, for a given amount of storage and a small number of fragments. Memoizing the relevant part of the shortest-path-cost data structure for the fragment graphs provides the next best savings, followed by memoizing the shortest-path data structure for the boundary graph.

This paper introduces data structures as applied to computer graphics. Design criteria for computer graphics data structures are discussed, followed by a comparison of general-purpose and tailored graphic data structures. A general graphic data structure is introduced as an example of a structure meeting the preceding criteria. The L6language is then examined as a tool for implementing the above data structure, and is compared to a few other language systems.

Evolving graph processing has enabled the modeling of many complex network systems, e.g., online social networks and gene networks. Existing in-memory graph data structures cannot effectively exploit the current and ongoing adoption of emerging non-volatile main memory (NVMM) for two reasons. (1) Ephemeral graph data structures are not crash-consistent nor durable for NVMM. Corruption is likely when updating its correlated application-defined data and runtime states in the face of hardware or software failures. (2) NVMM writes and reads may incur higher latency than DRAM. Placing the data structures in NVMM may result in a significant loss of performance. In this paper, we propose a novel persistent evolving graph data structure, named NVGRAPH, for both computing and in-memory storage of evolving graphs in NVMM. We devise NVGRAPH as a multi-version data structure, wherein a minimum of one version of its data is stored in NVMM to provide the desired durability at runtime for failure recovery, and another version is stored in both DRAM and NVMM to reduce the NVMM-induced memory latency. We dynamically transform the layout of NVGRAPH including changing the size of its partition in DRAM and the position of its base snapshot exploiting network properties and data access patterns of workloads. For the evaluation of NVGRAPH, we implement four representative real-world graph applications: pagerank, BFS, influence maximization, and rumor source detection. The experimental results show that the performance of NVGRAPH is comparable to other in-memory data structure (e.g., CSR and LLAMA) while using 70% less DRAM. It scales well up to 10 billion edges and 201 snapshots and supports crash consistency. It offers up to the 21X speedup of execution time compared to the scale-up graph computation approaches (e.g., GraphChi and X-stream).

The analysis of dynamic spatial systems requires an explicit spatio-temporal data model and spatio-temporal analysis tools. Event-based models have been developed to analyze discrete change in continuous and feature-based spatial data. In this paper, a spatio-temporal graph model is described that supports the analysis of continuous change in feature-based polygon spatial data. The spatio-temporal graph edges, called temporal links, track changes in polygon topology through space and time. The model also introduces the concept of a spatial-interaction region that extends a model's focus beyond short-term local events to encompass long-term regional events. The structure of the spatio-temporal graph is used to classify these events into five types of local polygon events and two types of spatial-interaction region events. To illustrate its utility, the model is applied to the ecological question of how patch size influences longevity in underwater plant communities in Chesapeake Bay, USA. Both a short-term local analysis and a longer-term regional analysis showed that patches of plants, or groups of patches, larger than one to two hectares in size were more likely to persist than smaller patches or groups of patches. Overall, the spatio-temporal graph model approach appears applicable to a variety of spatio-temporal questions.

Graph data structures constitute fundamental components within computer science, serving as versatile models for representing interconnected relationships in a myriad of domains, including social networks, transportation systems, and biological networks.However, comprehending the intricacies inherent to graphs often poses a significant challenge for students enrolled in data structures courses.This paper endeavors to explore the transformative potential of visualization techniques in bolstering comprehension and engagement within the realm of graph data structures education.Drawing upon insights gleaned from cognitive psychology and educational research, we embark on an examination of how visualization methodologies can serve as catalysts for fostering deeper understanding and retention of abstract concepts, thereby elevating overall learning outcomes.Through literature review, we meticulously scrutinize a number of studies, each illuminating the efficacy of graph visualization tools in enhancing student comprehension and problem-solving skills.Moreover, we ardently advocate for the adoption of an active learning paradigm, wherein students are not merely passive recipients of knowledge but are actively encouraged to develop their visualization applications.Such an approach not only nurtures creativity and innovation, but also engenders a profound appreciation for the intricate nuances inherent to graph representations.Additionally, we delve into the underlying principles and methodologies of graph visualization, offering a discourse on techniques for crafting visually compelling graph layouts.By synthesizing a rich tapestry of research findings and practical insights, this paper endeavors to furnish educators with some strategies for seamlessly integrating visualization techniques into the fabric of data structures curricula.Ultimately, our overarching objective is to empower students to navigate the labyrinthine complexities of graph data structures with unwavering confidence, consummate proficiency, and reverence for the elegance imbued within graphical representations.

We present a new succinct data structure for graphs that are “tree-like,” in the sense that the number of “additional” edges (w.r.t. a spanning tree) is not too high. Our algorithmic idea is to represent a BFS-spanning tree of the graph with a succinct data structure for trees, and enhance it with additional information that accounts for the non-tree edges. In practical tests, our data structure performs well for graphs containing up to 10% of non-tree edges, reducing the space of a pointer-based representation by a factor of ≈20, while increasing the worst-case running times for the operations by roughly the same factor.

This paper presents and investigates methods for fast and accurate illumination of scenes containing many light sources that have limited spatial influence, e.g. point light sources. For speeding up the computation, current graphics applications use an assumption that the light sources range can be limited using bounding spheres due to their limited spatial influence and illumination is computed only if a surface lies within the sphere. Therefore, we explore the differences in illumination between scenes illuminated with spatially limited light sources and physically more correct computation where the light radius is infinite. We show that the difference can be small if we add appropriate ambient lighting. The contribution of the paper is the method for fast estimation of ambient lighting in scenes illuminated by numerous light sources. We also propose a method for elimination of color discontinuities at the edges of the bounding spheres. Our solution is tested on two different scenes: a procedurally generated city and the Sibenik cathedral. Our approach allows for correct lighting computation in scenes with numerous light sources without any modification of the scene graph or other data structures or rendering procedures. It thus can be applied in various systems without any structural modifications.

To leverage modern hardware platforms to their fullest, more and more database systems embrace compilation of query plans to native code. In the research community, there is an ongoing debate about the best way to architect such query compilers. This is perceived to be a difficult task, requiring techniques fundamentally different from traditional interpreted query execution. In this dissertation, we contribute to this discussion by drawing attention to an old but underappreciated idea known as Futamura projections, which fundamentally link interpreters and compilers. Guided by this idea, we demonstrate that efficient query compilation can actually be very simple, using techniques that are no more difficult than writing a query interpreter in a high-level language. We first develop LB2: a high-level query compiler implemented in this style that is competitive with the best compiled query engines both in sequential and parallel execution on the standard TPC-H benchmark. Query engines process a variety of data types and structures including text, spatial, graphs, etc. Several spatial and graph engines are implemented as extensions to relational query engines to leverage optimized memory, storage, and evaluation. Still, the performance of these extensions is often stymied by the interpretive nature of the underlying data management, generic data structures, and the need to execute domain-specific external libraries. On that basis, compiling spatial and graph queries to native code is a desirable avenue to mitigate existing limitations and improve performance. To support compiling spatial queries, we extend the LB2 main-memory query compiler with spatial predicates, indexing structures, and spatial operators. To support compiling graph queries, we extend LB2 with graph data structures and operators. The spatial extension matches the performance of hand-written code and outperforms relational query engines and map-reduce extensions. Similarly, the graph extension matches, and sometimes outperforms, low-level graph engines.

The paper describes graph algorithms performance when using different types of data structures. To achieve that, we developed a multi-level graph processing system, which allows to create graph applications independently of any implementation details such as graph data structure or underlying computational architecture. We measure the performance of breadth-first search, max flow and random graph building algorithms when using compressed sparse row and adjacency matrix data structures. Experiments reveal different graph processing rates for different data structures, which indicates the need of using specific data structures for specific algorithms to achieve highest performance.

Deep learning approaches, given their low cost and high reliability, have gained much popularity in different subjects, such as computer vision and natural language processing, and more recently in graph data types. Spatiotemporal graph-based neural networks have been more and more developed to solve problems related to spatiotemporal data, mainly for analyzing changes over time and for provisioning purposes. In this systematic literature review, we have aimed to answer the most important questions regarding spatiotemporal graph deep learning architectures in different applications domains, such as traffic related topics, medical imaging, and geographical data analysis. We have selected more than 50 papers that cover a wide range of applications and very different architectures and innovations. We have also noticed that most of them consider the spatiotemporal graphs to be quite classic graphs without any further sophisticated modeling of temporal and spatial data evolution. Moreover, core problems (such as node classification, frequent pattern recognition, etc.) and application domains are not sufficiently addressed by the state-of-the-art. This study thus opens many perspectives to new developments in spatio-temporal graph deep learning with different strategies in order to solve various end-to-end tasks and other problems related to this special kind of graph.