Fruited-Forest: A Reachability Querying Method Based on Spanning Tree Modelling of Reduced DAG

Among uncertain graph queries, reachability, i.e., the probability that one vertex is reachable from another, is likely the most fundamental one. Although this problem has been studied within the field of network reliability, solutions are implemented on a single computer and can only handle small graphs. However, as the size of graph applications continually increases, the corresponding graph data can no longer fit within a single computer's memory and must therefore be distributed across several machines. Furthermore, the computation of probabilistic reachability queries is #P-complete making it very expensive even on small graphs. In this paper, we develop an efficient distributed strategy, called DistR , to solve the problem of reachability query over large uncertain graphs. Specifically, we perform the task in two steps: distributed graph reduction and distributed consolidation . In the distributed graph reduction step, we find all of the maximal subgraphs of the original graph, whose reachability probabilities can be calculated in polynomial time, compute them and reduce the graph accordingly. After this step, only a small graph remains. In the distributed consolidation step, we transform the problem into a relational join process and provide an approximate answer to the #P-complete reachability query. Extensive experimental studies show that our distributed approach is efficient in terms of both computational and communication costs, and has high accuracy.

Shortest distance queries are essential not only in graph analysis and graph mining tasks but also in database applications, when a large graph needs to be dealt with. Such shortest distance queries are frequently issued by end-users or requested as a subroutine in real applications. For intensive queries on large graphs, it is impractical to compute shortest distances on-line from scratch, and impractical to materialize all-pairs shortest distances. In the literature, 2-hop distance labeling is proposed to index the all-pairs shortest distances. It assigns distance labels to vertices in a large graph in a pre-computing step off-line and then answers shortest distance queries on-line by making use of such distance labels, which avoids exhaustively traversing the large graph when answering queries. However, the existing algorithms to generate 2-hop distance labels are not scalable to large graphs. Finding an optimal 2-hop distance labeling is NP-hard, and heuristic algorithms may generate large size distance labels while still needing to pre-compute all-pairs shortest paths. In this paper, we propose a multi-hop distance labeling approach, which generates a subset of the 2-hop distance labels as index off-line. We can compute the multi-hop distance labels efficiently by avoiding pre-computing all-pairs shortest paths. In addition, our multi-hop distance labeling is small in size to be stored. To answer a shortest distance query between two vertices, we first generate the query-specific small set of 2-hop distance labels for the two vertices based on our multi-hop distance labels stored and compute the shortest distance between the two vertices based on the 2-hop distance labels generated on-line. We conducted extensive performance studies on large real graphs and confirmed the efficiency of our multi-hop distance labeling scheme.

Reachability queries are a fundamental type of queries on graphs that find important applications in numerous domains. Although a plethora of techniques have been proposed for reachability queries, most of them require that the input graph is static, i.e., they are inapplicable to the {\em dynamic} graphs (e.g., social networks and the Semantic Web) commonly encountered in practice. There exist a few techniques that can handle dynamic graphs, but none of them can scale to sizable graphs without significant loss of efficiency. To address this deficiency, this paper presents a novel study on reachability indices for large dynamic graphs. We first introduce a general indexing framework that summarizes a family of reachability indices with the best performance among the existing techniques for static graphs. Then, we propose general and efficient algorithms for handling vertex insertions and deletions under the proposed framework. In addition, we show that our update algorithms can be used to improve the existing reachability techniques on static graphs, and we also propose a new approach for constructing a reachability index from scratch under our framework. We experimentally evaluate our solution on a large set of benchmark datasets, and we demonstrate that our solution not only supports efficient updates on dynamic graphs, but also provides even better query performance than the state-of-the-art techniques for static graphs.

Continuously Evolving Graphs (CEGs) are graphs whose connectivity constantly changes over time is uniquely important for many domains such as social networks, evolutionary genomics, communication networks etc. In many of these it is often important to keep track of connectivity among the nodes of interest as the underlying structure changes over time. While interval-based indexing has been a popular strategy for testing reach ability in static graphs, it cannot be directly applied in the context of evolving graphs. In this paper, we propose CoUPE (Continuous qUery Processing Engine), which, to our best knowledge, is the first time-efficient framework for answering continuous reach ability queries in evolving graphs. The main idea here is to maintain the indices of the evolving graph and recalculate only a subset of reach ability queries using a novel heuristic to determine the change in the state of queries because for the most recent change in the graph. We make three novel contributions while designing CoUPE. First, we introduce a generic indexing and querying framework for answering continuous queries in large time-evolving graphs. Second, we design a highly efficient, scalable and provably correct algorithm for updating the indices of graph by analyzing the changes happening on the graph. Third, we present a novel heuristic-based technique for identifying which subset of existing queries might get effected because of the most recent edit. This paper also presents a detailed experimental study demonstrating the scalability and efficiency of the processing engine.

This thesis delves into critical challenges associated with subgraph analysis in large-scale graphs. The proliferation of data in domains such as social networks, Internet applications, and biological networks necessitates a deep understanding of relationships modeled as graphs. The complexity and scale of these graphs pose challenges for efficient extraction of meaningful information. The thesis focuses on three interconnected research problems: subgraph counting in fully-dynamic graph streams, k-clique listing in large general graphs, and maximal clique enumeration in large general graphs. The first research problem addresses subgraph counting in fully-dynamic graph streams, where graphs evolve dynamically with edge insertions and deletions. Traditional ap- proaches sample edges uniformly, overlooking the varying importance of edges. The thesis introduces a novel weighted sampling framework, WSD, for fully-dynamic graph streams, which samples the edges based on their weights that indicate their importance and reflect their properties. It determines the weights of edges in a data-driven fashion, using a novel method based on reinforcement learning. We conduct extensive experi- ments to verify that our technique can produce estimates with smaller errors while often running faster compared with existing algorithms. The second research problem focuses on k-clique listing in large general graphs, a critical task for applications such as community detection and network analysis. The state-of-the-art algorithms all adopt a branch-and-bound (BB) framework with a vertex-oriented branching strategy (called VBBkC), which forms a sub-branch by expanding a partial k-clique with a vertex. These algorithms have the time complexity of O(km(\delta/2)^(k-2)), where m is the number of edges in the graph and \delta is the degeneracy of the graph. In this thesis, we propose a BB framework with a new edge-oriented branching (called EBBkC), which forms a sub-branch by expanding a partial k-clique with two vertices that connect each other (which correspond to an edge). We explore various edge orderings for EBBkC such that it achieves a time complexity of O(m\delta + km(\tau/2)^(k-2), where \tau is an integer related to the maximum truss number of the graph and we have \tau<\delta. The time complexity of EBBkC is better than that of VBBkC algorithms for k>3 since both O(m\delta) and O(km(\tau/2)^(k-2) are bounded by O(km(\delta/2)^(k-2)). Furthermore, we develop specialized algorithms for sub-branches on dense graphs so that we can early-terminate them and apply the specialized algorithms. We conduct extensive experiments on 19 real graphs, and the results show that our newly developed EBBkC based algorithms with the early termination technique consistently and largely outperform the state-of-the- art (VBBkC based) algorithms. The third research problem centers on maximal clique enumeration (MCE) in large general graphs, crucial for tasks like community detection and biological network analysis. Existing algorithms typically adopt the branch-and-bound framework with the vertex-oriented Bron-Kerbosch (BK) branching strategy, which forms the sub-branches by expanding the partial clique with a vertex. The state-of-the-arts have a time complexity no better than O(n\delta 3^(\delta/3)), where n is the number of vertices in the graph and \delta is the degeneracy of the graph. We introduce a novel approach, HBBMC, a hybrid framework combining vertex-oriented and edge-oriented BK branching, where the latter adopts a new branch-and-bound framework which forms the sub-branches by expanding the partial clique with a edge. This hybrid strategy enables more efficient pruning of sub-branches, resulting in a worst-case time complexity of O(m\delta + m\tau 3^(\tau/3)), where \tau is related to the maximum truss number. This theoretical result is better than that of the state-of-the-arts when \delta > \tau + (3/ln3) \rho, a condition satisfied by many real-world graphs, where \rho is the edge density of the graph defined as \rho = m/n. To further enhance efficiency, we introduce an early termination technique applicable to all branch-and-bound frameworks, which leverages the topological information of the graphs within the branches. This technique accelerates the enumeration process without affecting worst-case time complexity. Extensive experiments on real and synthetic graphs demonstrate the superior performance of our approach. The proposed HBBMC algorithm with the early termination technique consistently outperforms existing vertex-oriented algorithms. In summary, this thesis makes significant contributions by introducing novel algo- rithms, frameworks, and methodologies to address subgraph-related problems. The pro- posed approaches show superior performance across various real-world and synthetic datasets, providing efficient solutions for graph analysis in large-scale, dynamic, and complex scenarios.

Vertex graph coloring (VGC) is a well known problem in graph theory and has a large number of applications in various domains such as telecommunications, bioinformatics, and Internet. It is one of the 21 NP-complete problems of Karp. Several large graph treatment frameworks have emerged and are effective options to deal with the VGC problem. Examples of those frameworks include Pregel, Graphx and Giraph. The latter is one of the most popular large graph processing frameworks both in industry and academia. In this paper, we present a novel graph coloring algorithm designed for utilizing the simple parallelization technique provided by the Giraph framework or any other vertex-centric paradigm. We have compared our algorithm to existing Giraph graph coloring algorithms with regard to solution quality (number of used colors) and CPU runtime, using several large graph datasets. The obtained results have shown that the proposed algorithm is much more efficient than existing Giraph algorithms.

An important component of current research in big data is graph analytics on very large graphs. Of the many problems of interest in this domain, graph pattern matching is both challenging and practically important. The problem is, given a relatively small query graph, finding matching patterns in a large data graph. Algorithms to address this problem are used in large social networks and graph databases. Though fast querying is highly desirable, the scalability of pattern matching algorithms is hindered by the NP-completeness of the subgraph isomorphism problem. This paper presents a conceptually simple, memory-efficient, pruning-based algorithm for the subgraph isomorphism problem that outperforms commonly used algorithms on large graphs. The high performance is due in large part to the effectiveness of the pruning algorithm, which in many cases removes a large percentage of the vertices not found in isomorphic matches. In this paper, the runtime of the algorithm is tested alongside other algorithms on graphs of up to 10 million vertices and 250 million edges.

Measures of graph similarity have a broad range of applications but involve compute-intensive process. Similarity flooding algorithm is an efficient algorithm for comparing the similarity of graphs of small size and small datasets. However, nowadays more and more large-scale graph applications emerge and existing stand-alone similarity flooding algorithm cannot efficiently conduct the similarity comparison process for large scale graph datasets in acceptable time. This paper presents a parallelized similarity flooding algorithm with MapReduce for large-scale graph datasets. The experimental results demonstrate that the parallelized algorithm achieves significant performance improvement compared to the stand-alone similarity flooding algorithm. Experimental results also reveal that the parallelized algorithm can obtain excellent speedup when the size of cluster increases.

The need to efficiently store and query large scale graph datasets is evident in the growing number of data-intensive applications, particularly to maximize the mining of intelligence from these data (e.g., to inform decision making). However, directly releasing graph dataset for analysis may leak sensitive information of an individual even if the graph is anonymized, as demonstrated by the re-identification attacks on the DBpedia datasets. A key challenge in the design of graph sanitization methods is scalability, as existing execution models generally have significant memory requirements. In this paper, we propose a novel k-decomposition algorithm and define a new information loss matrix designed for utility measurement in massively large graph datasets. We also propose a novel privacy preserving framework that can be seamlessly integrated with graph storage, anonymization, query processing, and analysis. Our experimental studies show that the proposed solution achieves privacy-preserving, utility, and efficiency.

It is common to find that many real world data graphs are edge-labeled, i.e., each edge attaches a label to indicate the relationship of the two vertices connected by the edge. A basic research issue on these graphs is how to make a query to test if the vertex u can reach the vertex v through the path constrained by a series of labels. In this paper, we propose a novel index framework to retain all the path label information of a data graph and answer the regular expression based reachability queries efficiently. Our index is a generalized-list-structured index and constructed based on the method of bread first search and depth first search. To improve the query efficiency, we also put forward a labeling mechanism for our index. A detailed experimental evaluation on both real and synthetic datasets shows that the performance of our method is improved in term of both index size and query time.

Correlation clustering (CC) is a graph based clustering method. Edges of the graph are labeled either positive or negative depending on the similarity/dissimilarity between the pair of vertices. The objective of CC is to group vertices of the induced complete graph so as to maximize the positively labeled edges that lie within a group and to maximize negatively labeled edges that lie across groups. This objective function is formulated as a semidefinite programming (SDP) problem which is well studied theoretically producing encouraging approximation values. In this work we propose a scalable solution for the SDP formulation of correlation clustering (SDP-CC) by reducing the number of constraints. The proposed formulation is solved efficiently using SDP-NAL tool. The proposed scalable formulation is compared with other scalable variants namely variable reduction based CC. Experimental results on synthetic, real world data sets whose graph sizes range from 100 vertices to 13000 vertices are tested with both the scalable formulations. Large scale bench mark graph data sets are also tested whose sizes range from 2395 vertices to 13992 vertices. The proposed formulation is shown to have an edge over the original SDP-CC formulation, variable reduction variant of SDP-CC and a constraint clustering method, namely constrained spectral clustering.

A link server is a system designed to support efficient implementations of graph computations on the web graph. In this work, we present a compression scheme for the web graph specifically designed to accommodate community queries and other random access algorithms on link servers. We use a frequent pattern mining approach to extract meaningful connectivity formations. Our Virtual Node Miner achieves graph compression without sacrificing random access by generating virtual nodes from frequent itemsets in vertex adjacency lists. The mining phase guarantees scalability by bounding the pattern mining complexity to O(E log E). We facilitate global mining, relaxing the requirement for the graph to be sorted by URL, enabling discovery for both inter-domain as well as intra-domain patterns. As a consequence, the approach allows incremental graph updates. Further, it not only facilitates but can also expedite graph computations such as PageRank and local random walks by implementing them directly on the compressed graph. We demonstrate the effectiveness of the proposed approach on several publicly available large web graph data sets. Experimental results indicate that the proposed algorithm achieves a 10- to 15-fold compression on most real word web graph data sets

GraphChi is the first reported disk-based graph engine that can handle billion-scale graphs on a single PC efficiently. GraphChi is able to execute several advanced data mining, graph mining and machine learning algorithms on very large graphs. With the novel technique of parallel sliding windows (PSW) to load subgraph from disk to memory for vertices and edges updating, it can achieve data processing performance close to and even better than those of mainstream distributed graph engines. GraphChi mentioned that its memory is not effectively utilized with large dataset, which leads to suboptimal computation performances. In this paper we are motivated by the concepts of "pin " from TurboGraph and "ghost" from GraphLab to propose a new memory utilization mode for GraphChi, which is called Part-in-memory mode, to improve the GraphChi algorithm performance. The main idea is to pin a fixed part of data inside the memory during the whole computing process. Part-in-memory mode is successfully implemented with only about 40 additional lines of code to the original GraphChi engine. Extensive experiments are performed with large real datasets (including Twitter graph with 1.4 billion edges). The preliminary results show that Part-in-memory mode memory management approach effectively reduces the GraphChi running time by up to 60% in PageRank algorithm. Interestingly it is found that a larger portion of data pinned in memory does not always lead to better performance in the case that the whole dataset cannot be fitted in memory. There exists an optimal portion of data which should be kept in the memory to achieve the best computational performance.

There exist many graph-based applications including bioinformatics, social science, link analysis, citation analysis, and collaborative work. All need to deal with a large data graph. Given a large data graph, in this paper, we study finding top-k answers for a graph query, and in particular, we focus on top-k cyclic graph queries where a graph query is cyclic and can be complex. The capability of supporting top-k cyclic graph queries over a data graph provides much more flexibility for a user to search graphs. And the problem itself is challenging. After investigating a direct yet infeasible solution, we propose a new twig query approach. In our approach, we first identify a spanning tree of the cyclic graph query, which is used to generate a list of ranked twig answers on-demand. Then we identify the top-k answers for the graph query based on the twig answer list. In order to find the best twig query in solving a given cyclic graph query, cost-based optimization for twig query selection is studied. We conducted extensive performance studies using a real dataset, and we report our findings in this paper.

Reachability queries processing has been extensively studied during the past decades. Many approaches have followed the line of designing 2-hop labels to ensure acceleration. Considering its index size cannot be bounded, researchers have proposed to use a part of nodes to construct partial 2-hop labels (p2HLs) to cover as much reachability information as possible. We achieved better query performance using p2HLs with a limited index size and index construction time. However, the adoption of p2HLs was based on intuition, and the number of nodes used to generate p2HLs was fixed in advance blindly, without knowing its applicability. In this paper, we focused on the problem of efficiently computing a reachability ratio (RR) in order to obtain RR-aware p2HLs. Here, RR denoted the ratio of the number of reachable queries that could be answered by p2HLs over the total number of reachable queries involved in a given graph. Based on the RR, users could determine whether p2HLs should be used to answer the reachability queries for a given graph and how many nodes should be chosen to generate p2HLs. We discussed the difficulties of RR computation and propose an incremental-partition algorithm for RR computation. Our rich experimental results showed that our algorithm could efficiently obtain the RR and the overall effects on query performance by different p2HLs. Based on the experimental results, we provide our findings on the use p2HLs for a given graph for processing reachability queries.