On compressing massive streaming graphs with Quadtrees

Emerging real-world graph problems include detecting community structure in large social networks, improving the resilience of the electric power grid, and detecting and preventing disease in human populations. Unlike traditional applications in computational science and engineering, solving these problems at scale often raises new challenges because of sparsity and the lack of locality in the data, the need for additional research on scalable algorithms and development of frameworks for solving these problems on high performance computers, and the need for improved models that also capture the noise and bias inherent in the torrential data streams. The explosion of real-world graph data poses a substantial challenge: How can we analyze constantly changing graphs with billions of vertices? Our approach leverages the Cray XMT’s fine-grained parallelism and flat memory model to scale to massive graphs. On the Cray XMT, our static graph characterization package GraphCT summarizes such massive graphs, and our ongoing STINGER streaming work updates clustering coefficients on massive graphs at a rate of tens of thousands updates per second.

Currently, data in many applications have naturally been modeled as streams over the massive graph infrastructure, e.g., social networks and electronic business. Graph streams are rapidly changing, enormous and endless networks that are too large to maintain in memory or on disks. An important problem in networks is link prediction, which aims to estimate the likelihood of the existence of a specific link. However, in graph streams, predicting the existence of links connected to one vertex is more common. For example, in social networks, we generally want to recommend several friends to a user rather than determining whether a specific user is your friend. Rapidly and accurately predicting groups of links becomes a formidable challenge because of the tremendous size and rapidly updated information of graph streams. In this paper, we propose the problem of link prediction for vertex in graph streams, which aims to predict the top-k vertices, i.e., the top-k links, that are most likely to connect to the target vertex in graph streams. A two-phase selection framework is proposed to predict top-k links with high efficiency and without loss of accuracy. We also propose a novel method for estimating common neighbor in graph streams, which is a very important measure in link prediction. Extensive experiments show that our algorithms are more efficient and more accurate than state-of-the-art methods.

Are users of an online social network interested equally in all connections in the network? If not, how can we obtain a summary of the network personalized to specific users? Can we use the summary for approximate query answering? As massive graphs (e.g., online social networks, hyperlink networks, and road networks) have become pervasive, graph compression has gained importance for the efficient processing of such graphs with limited resources. Graph summarization is an extensively-studied lossy compression method. It provides a summary graph where nodes with similar connectivity are merged into supernodes, and a variety of graph queries can be answered approximately from the summary graph. In this work, we introduce a new problem, namely personalized graph summarization, where the objective is to obtain a summary graph where more emphasis is put on connections closer to a given set of target nodes. Then, we propose Pegasus, a linear-time algorithm for the problem. Through experiments on six real-world graphs, we demonstrate that Pegasus is (a) Effective: node-similarity queries for target nodes can be answered significantly more accurately from personalized summary graphs than from non-personalized ones of similar size, (b) Scalable: it summarizes graphs with up to one billion edges, and (c) Applicable to distributed multi-query answering: it successfully replaces graph partitioning for communication-free multi-query processing.

A graph stream is a continuous sequence of data items, in which each item indicates an edge, including its two endpoints and edge weight. It forms a dynamic graph that changes with every item. Graph streams play important roles in cyber security, social networks, cloud troubleshooting systems and more. Due to the vast volume and high update speed of graph streams, traditional data structures for graph storage such as the adjacency matrix and the adjacency list are no longer sufficient. However, prior art of graph stream summarization, like CM sketches, gSketches, TCM and gMatrix, either supports limited kinds of queries or suffers from poor accuracy of query results. In this paper, we propose a novel Graph Stream Sketch (GSS for short) to summarize the graph streams, which has linear space cost O(|E|) (E is the edge set of the graph) and constant update time cost (O(1)) and supports most kinds of queries over graph streams with the controllable errors. Both theoretical analysis and experiment results confirm the superiority of our solution with regard to the time/space complexity and query results' precision compared with the state-of-the-art.

Community detection task aims to find communities for a given network while community search focuses on finding a densely connected community with some given query nodes. Community search methods have the advantage of speed as they find a community for a set of query nodes instead of finding all the communities. They also have better performance on handling dynamic networks compared to community detection algorithms. However, current studies of community search are not well designed to handle spatial sensitive networks. We propose a new community search method, Spatial-based Search using PageRank Vector (SSPPV), which aims to find a community for a given query node over a large spatial social network. SSPPV finds a community with not only good cluster quality but also a smaller spatial size of the resulting community, compared to existing methods.

Stochastic model for emotion contagion in social networks security based on machine learning

A linear threshold-hurdle model for product adoption prediction incorporating social network effects

Prior research has established a positive association between social support and psychological resilience. In this study, we seek to examine whether and to what extent aspects of individuals’ social network – specifically size (how many relatives and friends one has) and strength (how often did one communicate with close network members and at the time of important decisions) – are related to greater psychological resilience. We use data pertaining to 1,609 respondents from the Panel on Ageing and Transitions in Health Survey (PATHS), 2016–2017, a national study of 1,654 older midlife adults, aged 50–59, in Singapore. We estimate the relationship between social networks and psychological resilience, using inverse probability weighted regression adjustment to account for the possibility of a selection bias whereby individuals with larger or stronger social networks may be more resilient at the outset. We find that strong social networks are associated with greater psychological resilience among older midlife adults, regardless of the size of the network. Having a large social network is associated with greater resilience only if it is also a strong network. Maintaining stronger, even if small, social networks may enable individuals at the cusp of older ages to be better prepared to deal with stressful life events and challenges associated with older midlife.

The k-core of a graph is the largest subgraph in which every vertex is connected to at least k other vertices within the subgraph. Core decomposition finds the k-core of the graph for every possible k. Past studies have shown important applications of core decomposition such as in the study of the properties of large networks (e.g., sustainability, connectivity, centrality, etc.), for solving NP-hard problems efficiently in real networks (e.g., maximum clique finding, densest subgraph approximation, etc.), and for large-scale network fingerprinting and visualization. The k-core is a well accepted concept partly because there exists a simple and efficient algorithm for core decomposition, by recursively removing the lowest degree vertices and their incident edges. However, this algorithm requires random access to the graph and hence assumes the entire graph can be kept in main memory. Nevertheless, real-world networks such as online social networks have become exceedingly large in recent years and still keep growing at a steady rate. In this paper, we propose the first external-memory algorithm for core decomposition in massive graphs. When the memory is large enough to hold the graph, our algorithm achieves comparable performance as the in-memory algorithm. When the graph is too large to be kept in the memory, our algorithm requires only O(k max ) scans of the graph, where k max is the largest core number of the graph. We demonstrate the efficiency of our algorithm on real networks with up to 52.9 million vertices and 1.65 billion edges.

With the rapid development of 5G and mobile internet, various application fields continue to generate diverse dynamic streaming graphs with large scales and complex structures. Analyzing, querying and mining such a streaming graph face great challenges. To address the challenge of low query efficiency and avoid data loss, this paper proposes a two-stage coarsening method that preserves key features of a streaming graph while improves query efficiency. In the initial stage, the size of the streaming graph is not that large yet. To preserve its structure and attributes as much as possible, a linear mapping method is used for graph coarsening. As the size of the streaming graph increases, a graph neural network model based on cluster pooling is constructed in the second stage. Using a large number of graph coarsening results obtained in the initial stage to train and optimize this model. Then, we use this optimized model to obtain the subsequent summaries, which achieves online efficient graph coarsening. Finally, extensive experiments indicate that our algorithm significantly improves the quality of streaming graph coarsening compared to traditional algorithms. Even compared to the state-of-the-art algorithm Auxo, our algorithm shows a better performance on preserving key features.

Social Network Marketing techniques employ pre-existing social networks to increase brands or products awareness through word-of-mouth promotion. Full understanding of social network marketing and the potential candidates that can thus be marketed to certainly offer lucrative opportunities for prospective sellers. Due to the complexity of social networks, few models exist to interpret social network marketing realistically. We propose to model social network marketing using Heat Diffusion Processes. This paper presents three diffusion models, along with three algorithms for selecting the best individuals to receive marketing samples. These approaches have the following advantages to best illustrate the properties of real-world social networks: (1) We can plan a marketing strategy sequentially in time since we include a time factor in the simulation of product adoptions; (2) The algorithm of selecting marketing candidates best represents and utilizes the clustering property of real-world social networks; and (3) The model we construct can diffuse both positive and negative comments on products or brands in order to simulate the complicated communications within social networks. Our work represents a novel approach to the analysis of social network marketing, and is the first work to propose how to defend against negative comments within social networks. Complexity analysis shows our model is also scalable to very large social networks.

Many dynamic applications are built upon large network infrastructures, such as social networks, communication networks, biological networks and the Web. Such applications create data that can be naturally modeled asgraph streams, in which edges of the underlying graph are received and updated sequentially in a form of a stream. It is often necessary and important to summarize the behavior of graph streams in order to enable effective query processing. However, the sheer size and dynamic nature of graph streams present an enormous challenge to existing graph management techniques. In this paper, we propose a new graph sketch method, gSketch, which combines well studied synopses for traditional data streams with a sketch partitioning technique, to estimate and optimize the responses to basic queries on graph streams. We consider two different scenarios for query estimation: (1) A graph stream sample is available; (2) Both a graph stream sample and a query workload sample are available. Algorithms for different scenarios are designed respectively by partitioning a global sketch to a group of localized sketches in order to optimize the query estimation accuracy. We perform extensive experimental studies on both real and synthetic data sets and demonstrate the power and robustness of gSketch in comparison with the state-of-the-art global sketch method.

The graph stream is defined as rapid edge streams on a huge domain of nodes. Nowadays, graph streams play important roles in network traffic, social networks, and cloud troubleshooting. Therefore, various summary structures for graph streams are proposed to obtain approximate evaluation results. However, these structures either sacrifice accuracy for guaranteed throughput or compromise memory consumption for high precision. In view of the limitations, we propose Cuckoo Matrix. It only uses one adjacency matrix to complete high accuracy queries while assuring large throughput. Meanwhile, Cuckoo Matrix is capable of preserving the connectivity of edges for the purpose of supporting both structural queries and weight-based estimations. The experimental results show that Cuckoo Matrix improves insertion throughput by 25% and reduces memory consumption by 25% compared to the state-of-the-art, which meets the current requirements of graph stream summarization.

Data from emerging applications, such as cybersecurity and social networking, can be abstracted as graphs whose edges are updated sequentially in the form of a stream. The challenging problem of interactive graph stream analytics is the quick response of the queries on terabyte and beyond graph stream data from end users. In this paper, a succinct and efficient double index data structure is designed to build the sketch of a graph stream to meet general queries. A single pass stream model, which includes general sketch building, distributed sketch based analysis algorithms and regression based approximation solution generation, is developed, and a typical graph algorithm—triangle counting—is implemented to evaluate the proposed method. Experimental results on power law and normal distribution graph streams show that our method can generate accurate results (mean relative error less than 4%) with a high performance. All our methods and code have been implemented in an open source framework, Arkouda, and are available from our GitHub repository, Bader-Research. This work provides the large and rapidly growing Python community with a powerful way to handle terabyte and beyond graph stream data using their laptops.

Graphs can be used to represent a wide variety of data belonging to different domains. Graphs can capture the relationship among data in an efficient way, and have been widely used. In recent times, with the advent of Big Data, there has been a need to store and compute on large data sets efficiently. However, considering the size of the data sets in question, finding optimal methods to store and process the data has been a challenge. Therefore, we study different graph compression techniques and propose novel algorithms to do the same in this paper. Specifically, given a graph G = (V, E), where V is the set of vertices and E is the set of edges, and |V| = n, we propose techniques to compress the adjacency matrix representation of the graph. Our algorithms are based on finding patterns within the adjacency matrix data, and replacing the common patterns with specific markers. All the techniques proposed here are lossless compression of graphs. Based on the experimental results, it is observed that our proposed techniques achieve almost 70% compression as compared to the adjacency matrix representation. The results show that large graphs can be efficiently stored in smaller memory and exploit the parallel processing power of compute nodes as well as efficiently transfer data between resources.