Cuckoo Matrix: A High Efficient and Accurate Graph Stream Summarization on Limited Memory

A graph stream is a continuous sequence of data items, in which each item indicates an edge, including its two endpoints and edge weight. It forms a dynamic graph that changes with every item. Graph streams play important roles in cyber security, social networks, cloud troubleshooting systems and more. Due to the vast volume and high update speed of graph streams, traditional data structures for graph storage such as the adjacency matrix and the adjacency list are no longer sufficient. However, prior art of graph stream summarization, like CM sketches, gSketches, TCM and gMatrix, either supports limited kinds of queries or suffers from poor accuracy of query results. In this paper, we propose a novel Graph Stream Sketch (GSS for short) to summarize the graph streams, which has linear space cost O(|E|) (E is the edge set of the graph) and constant update time cost (O(1)) and supports most kinds of queries over graph streams with the controllable errors. Both theoretical analysis and experiment results confirm the superiority of our solution with regard to the time/space complexity and query results' precision compared with the state-of-the-art.

Graph stream is the model used to represent evolving graph data over time, which can be represented as a sequence of edge streams containing temporal information. To effectively manage an ultra large scale graph stream, existing designs usually use summarization structures based on compressed matrices to support approximate storage and querying of graph streams. However, the state-of-the-art structures are based on limited-sized compressed matrix. When dealing with dynamical graph stream data, they either use an extra adjacency list outside the compressed matrix to store left-over edges whose expected buckets in the matrix have been occupied by other previously inserted edges, or allocate new building blocks of compressed matrices to provide more space capacity. Such designs suffer from linear lookup time caused by the adjacency list or long system blocking time caused by data movement during structure scaling. Moreover, in graph stream applications with dynamically growing data sizes, recent data commonly carries greater significance and value. Existing designs fail to store the time information of items of graph streams in a space efficient way and leave recent data management over graph streams an unsolved problem. To address the dynamically expanding graph stream with the ability to accentuate the importance of recent data, in this work, we propose Sliding-ITeM, a novel adaptive-size graph stream summarization structure with a sliding window model. Two factors contribute to the efficiency of Sliding-ITeM. First, Sliding-ITeM proposes a novel fingerprint suffix index tree (FSIT) structure to efficiently manage the items assigned to a same bucket of a compressed matrix in a fine-grained and scalable way. It thus achieves time and space efficiency for graph stream management as well as avoiding costly blocking time for structure extending. Second, Sliding-ITeM divides continuous time into discrete time slices and stores items belong to different time slices in separate ITeM compressed matrices. Sliding-ITeM organizes the compressed matrices into a chain style chronologically and achieves efficient obtaining of value from recent data as well as removal of expired data following a sliding-window model. We conduct comprehensive experiments over large-scale graph stream data collected from real world systems to evaluate the performance of Sliding-ITeM. Experimental results show that it significantly reduces the operation latency by more than 67% in sliding window queries compared to state-of-the-art designs, while greatly reducing the system blocking duration by three orders of magnitude.

Currently, data in many applications have naturally been modeled as streams over the massive graph infrastructure, e.g., social networks and electronic business. Graph streams are rapidly changing, enormous and endless networks that are too large to maintain in memory or on disks. An important problem in networks is link prediction, which aims to estimate the likelihood of the existence of a specific link. However, in graph streams, predicting the existence of links connected to one vertex is more common. For example, in social networks, we generally want to recommend several friends to a user rather than determining whether a specific user is your friend. Rapidly and accurately predicting groups of links becomes a formidable challenge because of the tremendous size and rapidly updated information of graph streams. In this paper, we propose the problem of link prediction for vertex in graph streams, which aims to predict the top-k vertices, i.e., the top-k links, that are most likely to connect to the target vertex in graph streams. A two-phase selection framework is proposed to predict top-k links with high efficiency and without loss of accuracy. We also propose a novel method for estimating common neighbor in graph streams, which is a very important measure in link prediction. Extensive experiments show that our algorithms are more efficient and more accurate than state-of-the-art methods.

Many dynamic applications are built upon large network infrastructures, such as social networks, communication networks, biological networks and the Web. Such applications create data that can be naturally modeled asgraph streams, in which edges of the underlying graph are received and updated sequentially in a form of a stream. It is often necessary and important to summarize the behavior of graph streams in order to enable effective query processing. However, the sheer size and dynamic nature of graph streams present an enormous challenge to existing graph management techniques. In this paper, we propose a new graph sketch method, gSketch, which combines well studied synopses for traditional data streams with a sketch partitioning technique, to estimate and optimize the responses to basic queries on graph streams. We consider two different scenarios for query estimation: (1) A graph stream sample is available; (2) Both a graph stream sample and a query workload sample are available. Algorithms for different scenarios are designed respectively by partitioning a global sketch to a group of localized sketches in order to optimize the query estimation accuracy. We perform extensive experimental studies on both real and synthetic data sets and demonstrate the power and robustness of gSketch in comparison with the state-of-the-art global sketch method.

Data from emerging applications, such as cybersecurity and social networking, can be abstracted as graphs whose edges are updated sequentially in the form of a stream. The challenging problem of interactive graph stream analytics is the quick response of the queries on terabyte and beyond graph stream data from end users. In this paper, a succinct and efficient double index data structure is designed to build the sketch of a graph stream to meet general queries. A single pass stream model, which includes general sketch building, distributed sketch based analysis algorithms and regression based approximation solution generation, is developed, and a typical graph algorithm—triangle counting—is implemented to evaluate the proposed method. Experimental results on power law and normal distribution graph streams show that our method can generate accurate results (mean relative error less than 4%) with a high performance. All our methods and code have been implemented in an open source framework, Arkouda, and are available from our GitHub repository, Bader-Research. This work provides the large and rapidly growing Python community with a powerful way to handle terabyte and beyond graph stream data using their laptops.

Graph summarization aims to extract critical information from large graphs by creating summaries that represent the original data.Especially, real-world daily applications generate massive dynamic streaming graphs, representing as edge or node streams.How to efficiently generate compact and lossless graph summaries for large streaming graphs is still a challenging problem.This paper proposes an efficient and scalable lossless summarization method for streaming graphs, called Partition and Similarity-based Two-Stage Summarization (PSTSS).PSTSS uses a streaming graph partitioning algorithm as the first stage to generate coarse-grained summaries by on-the-fly subgraph partitioning.The second stage generates fine-grained supernodes within these coarse-grained supernodes through efficient node similarity calculation.Our experiments on six large datasets demonstrate that our method achieves better operational efficiency, compression rate, scalability, and graph query speed compared to state-of-the-art lossless summarization algorithms.

With the rapid development of 5G and mobile internet, various application fields continue to generate diverse dynamic streaming graphs with large scales and complex structures. Analyzing, querying and mining such a streaming graph face great challenges. To address the challenge of low query efficiency and avoid data loss, this paper proposes a two-stage coarsening method that preserves key features of a streaming graph while improves query efficiency. In the initial stage, the size of the streaming graph is not that large yet. To preserve its structure and attributes as much as possible, a linear mapping method is used for graph coarsening. As the size of the streaming graph increases, a graph neural network model based on cluster pooling is constructed in the second stage. Using a large number of graph coarsening results obtained in the initial stage to train and optimize this model. Then, we use this optimized model to obtain the subsequent summaries, which achieves online efficient graph coarsening. Finally, extensive experiments indicate that our algorithm significantly improves the quality of streaming graph coarsening compared to traditional algorithms. Even compared to the state-of-the-art algorithm Auxo, our algorithm shows a better performance on preserving key features.

Many forms of social, information, and communication network activity create large volumes of graph stream data. In such cases, it is often desirable to track interesting properties of the underlying nodes, as they change over time. These dynamic properties can often be represented in the form of time-dependent labels associated with the nodes. Dynamic changes in such node labels may be indicative of important events or patterns of activity. This paper will study the problem of differential classification in graph streams, in which we predict significant classification events; i.e. the changes in classification labels of the nodes. Different from the static collective classification problem, this approach focusses on dynamic and real-time detection of changes in node classification, as opposed to the actual classification of nodes. The differential stream classification problem can also be considered a general form of the nodecentric event detection problem, in which node labels are used in order to supervise the detection process. We present experimental results illustrating the effectiveness of our method.

BackgroundMining logic rules from structured knowledge bases is the basis of knowledge engineering. Due to the NP‐hardness of the rule mining problem, logic rules cannot be efficiently induced from knowledge bases, especially large‐scale ones.IdeaIn this article, we propose a compact and efficient index structure for the maintenance of the intermediate data during top‐down rule mining, such that the memory consumption can be reduced and mining efficiency can be improved.Developing PointsThe index is based on a mapping from constant symbols to integers and the sorting of the mapped integers. Index update has been dissembled into four basic operations. Moreover, the index itself acts as the cache during top‐down mining.ValueMost contributions in existing works employ algorithmic and architectural optimizations to improve efficiency. Data‐oriented optimizations have also been explored to some extent, but the data efficiency is relatively low, and the memory consumption is thus becoming a new challenge for state‐of‐the‐art systems. We tackle this challenge in this article, and our technique has been proven more efficient than state‐of‐the‐art systems. We evaluate our method on six datasets which contain up to 160 K records and are frequently used as benchmarks in tasks related to knowledge engineering. The experimental results show that the proposed technique speeds up the rule mining procedure by on average and reduces memory consumption by up to 70%. The space overhead of the data structure is about twice that of the indexed records, which is more than 80% lower than that of the state‐of‐the‐art technique.

This paper proposes an efficient patch-based image matting approach to reduce memory consumption in processing high-resolution images. Most existing image matting techniques employ a global optimization over the whole set of image pixels, incurring a prohibitively high memory consumption. Inspired by 'divide and conquer', the image is divided into small patches self-adaptively according to the distribution of pixels, then the matting algorithm is applied in patch level. The memory consumption is related to the size of patch. Relationships between patches are also considered by locally linear embedding to maintain consistency through the whole image. Experimental results show that our method significantly reduces memory consumption while maintaining high-fidelity matting results on the benchmark dataset.

Modern graph representation learning works mostly under the assumption of dealing with regularly sampled temporal graph snapshots, which is far from realistic, e.g., social networks and physical systems are characterized by continuous dynamics and sporadic observations. To address this limitation, we introduce the Temporal Graph Ordinary Differential Equation (TG-ODE) framework, which learns both the temporal and spatial dynamics from graph streams where the intervals between observations are not regularly spaced. We empirically validate the proposed approach on several graph benchmarks, showing that TG-ODE can achieve state-of-the-art performance in irregular graph stream tasks.

We will study the problem of analyzing massive data streams in the context of the dynamic network-centered activities. Consider a network in which a time series activity data stream is associated with each edge in a massive network. Examples of such activities could include citation networks with continuously changing edge values, such as road network traffic, social network traffic, or email traffic. We introduce the problem of link regression, which models the prediction of the future stream values associated with a link from the history of all the stream values associated with the different links. This problem is a powerful generalization of both the large scale time series prediction problem and the link prediction problems in network analysis, each of which are known to be quite difficult in their own right. The generality of this framework provides it significant applicability for traffic analysis in road networks, congestion analysis in communication networks, interest trend analysis and bursty feature analysis in social networks. The problem is very challenging because of its massiveness both in network size and stream speed. We present an algorithm which uses streaming graph partitioning in conjunction with predictive regression. We present experimental results illustrating the effectiveness and efficiency of the approach.

Social networks are constantly changing as new members join, existing members leave, and ‘followers’ or ‘friends’ are formed and disappear. The model that captures this constantly changing graph is the streaming graph model. Given a massive graph data stream wherein the number of nodes is in the order of millions and the number of edges is the tens of millions, we propose a simple algorithm to compress this graph without having read in the entire graph into the main memory. Our algorithm uses the quadtree data structure that is implicitly constructed to produce the compressed graph output. As a result of this implicit construction, our algorithm allows for node and edge additions/deletions that directly modifies the output compressed graph. We further develop algorithms to solve edge queries (is there any between two nodes?) and node queries (for a given node, list all its neighbors) that directly operates on the compressed graph. We have performed extensive empirical evaluations of our algorithms using publicly available, large social networks such as LiveJournal, Pokec, Twitter, and others. Our empirical evaluation is based on several parameters including time to compress, memory required by the compression algorithm, size of compressed graph, and time and memory size required to execute queries. We have also presented extensions to the compression algorithm that we have developed.

This article focuses on social media and interactive challenges for emergency organizations during a bioterrorism or agroterrorism incident, and it outlines the dual-use dilemma of social media. Attackers or terrorists can use social media as their modus operandi, and defenders, including emergency organizations in law enforcement and public and animal health, can use it for peaceful purposes. To get a better understanding of the uses of social media in these situations, a workshop was arranged in Stockholm, Sweden, to raise awareness about social media and animal bioterrorism threats. Fifty-six experts and crisis communicators from international and national organizations participated. As a result of the workshop, it was concluded that emergency organizations can collect valuable information and monitor social media before, during, and after an outbreak. In order to make use of interactive communication to obtain collective intelligence from the public, emergency organizations must adapt to social networking technologies, requiring multidisciplinary knowledge in the fields of information, communication, IT, and biopreparedness. Social network messaging during a disease outbreak can be visualized in stream graphs and networks showing clusters of Twitter and Facebook users. The visualization of social media can be an important preparedness tool in the response to bioterrorism and agroterrorism.

Deeply embedded systems powered by microcontrollers are becoming popular with the emergence of Internet-of-Things (IoT) technology. However, these devices primarily run C/C \({+}{+}\) code and are susceptible to memory bugs, which can potentially lead to both control data attacks and non-control data attacks. Existing defense mechanisms (such as control-flow integrity (CFI), dataflow integrity (DFI) and write integrity testing (WIT), etc.) consume a massive amount of resources, making them less practical in real products. To make it lightweight, we design a bitmap-based allowlist mechanism to unify the storage of the runtime data for protecting both control data and non-control data. The memory requirements are constant and small, regardless of the number of deployed defense mechanisms. We store the allowlist in the TrustZone to ensure its integrity and confidentiality. Meanwhile, we perform an offline analysis to detect potential collisions and make corresponding adjustments when it happens. We have implemented our idea on an ARM Cortex-M-based development board. Our evaluation results show a substantial reduction in memory consumption when deploying the proposed CFI and DFI mechanisms, without compromising runtime performance. Specifically, our prototype enforces CFI and DFI at a cost of just 2.09% performance overhead and 32.56% memory overhead on average.