DynamicDFEP

Dynamic Graph Neural Network (DGNN) has shown a strong capability of learning dynamic graphs by exploiting both spatial and temporal features. Although DGNN has recently received considerable attention by AI community and various DGNN models have been proposed, building a distributed system for efficient DGNN training is still challenging. It has been well recognized that how to partition the dynamic graph and assign workloads to multiple GPUs plays a critical role in training acceleration. Existing works partition a dynamic graph into snapshots or temporal sequences, which only work well when the graph has uniform spatio-temporal structures. However, dynamic graphs in practice are not uniformly structured, with some snapshots being very dense while others are sparse. To address this issue, we propose DGC, a distributed DGNN training system that achieves a 1.25× - 7.52× speedup over the state-of-the-art in our testbed. DGC's success stems from a new graph partitioning method that partitions dynamic graphs into chunks, which are essentially subgraphs with modest training workloads and few inter connections. This partitioning algorithm is based on graph coarsening, which can run very fast on large graphs. In addition, DGC has a highly efficient run-time, powered by the proposed chunk fusion and adaptive stale aggregation techniques. Extensive experimental results on 3 typical DGNN models and 4 popular dynamic graph datasets are presented to show the effectiveness of DGC.

Continuous pattern detection plays an important role in monitoring-related applications. The large size and dynamic update of graphs, along with the massive search space, pose huge challenges in developing an efficient continuous pattern detection system. In this paper, we leverage a distributed graph processing framework to approximately detect a given pattern over a large dynamic graph. We aim to improve the scalability and precision, and reduce the response time and message cost in the detection. We convert a given query pattern into a Single-Sink DAG (Directed Acyclic Graph), and propose an evaluation plan with message transitions on the DAG, which is shorten by SSD plan, to detect the pattern in a large dynamic graph. SSD plan can guide the data graph exploration via messages, and the messages will converge at data sink vertices, which then detect existences of the query pattern. We also conduct join operations over partial vertices during the graph exploration to improve the precision of pattern detection. In addition, we show that SSD plan can support the continuous query over dynamic graphs with slight extensions. We further design various sink vertex selection strategies and neighborhood based transition rule attachment to lower the evaluation cost. The experiments on billion-edge real-life graphs using Giraph, an open source implementation of Pregel, illustrate the efficiency and effectiveness of our method.

Libya is the African country with the highest proportion of migrants and it is the major migration gateway to Europe. Since the fall of Gaddafi’s regime in 2011, Libya has been experiencing a prolonged period of instability and conflict which has also affected migration flows. This paper provides the first analytical study of the networks of irregular migrants toward, within, and from Libya since the beginning of the Second Civil War in 2014. We use longitudinal-geolocalised data on migrants’ movements and combine tools taken from spatial statistics and network analysis to reconstruct the network of migrants’ international movements towards and from Libya, and the network of migrants’ internal movements within Libya. Employing the information obtained from these networks, we map the different migrant routes within the country, we provide evidence that migrants sort into routes according to their nationality, and we show how these routes have evolved over time. Moreover, using a Separable Temporal Exponential Random Graph (STERG) model, we document that migrants are more likely to move towards areas with higher economic opportunities, lower level of conflict intensity, more same country-of-origin migrants, and connecting to multiple routes. Our analysis shows how to combine migration data with insights from spatial statistics and network analysis in order to reconstruct migrants’ movements.

Predicting how changes to the urban environment layout will affect the spatial distribution of pedestrian flows is important for environmental, social and economic sustainability. We present longitudinal evaluation of a model of the effect of urban environmental layout change in a city centre (Cardiff 2007–2010), on pedestrian flows. Our model can be classed as regression based direct demand using Multiple Hybrid Spatial Design Network Analysis (MH-sDNA) assignment, which bridges the gap between direct demand models, facility-based activity estimation and spatial network analysis (which can also be conceived as a pedestrian route assignment based direct demand model). Multiple theoretical flows are computed based on retail floor area: everywhere to shops, shop to shop, railway stations to shops and parking to shops. Route assignment, in contrast to the usual approach of shortest path only, is based on a hybrid of shortest path and least directional change (most direct) with a degree of randomization. The calibration process determines a suitable balance of theoretical flows to best match observed pedestrian flows, using generalized cross-validation to prevent overfit. Validation shows that the model successfully predicts the effect of layout change on flows of up to approx. 8000 pedestrians per hour based on counts spanning a 1 km2 city centre, calibrated on 2007 data and validated to 2010 and 2011. This is the first time, to our knowledge, that a pedestrian flow model with assignment has been evaluated for its ability to forecast the effect of urban layout changes over time.

Incremental maximal clique enumeration (IMCE), which maintains maximal cliques in dynamic graphs, is a fundamental problem in graph analysis. A maximal clique has a solid descriptive power of dense structures in graphs. Real-world graph data is often large and dynamic. Studies on IMCE face significant challenges in the efficiency of incremental batch computation and hybrid edge changes. Moreover, with growing graph sizes, new requirements occur on indexing global maximal cliques and obtaining maximal cliques under specific vertex scope constraints. This work presents a new data structure SOMEi to maintain intermediate maximal cliques during construction. SOMEi serves as a space-efficient index to retrieve scope-constrained maximal cliques on the fly. Based on SOMEi, we design a procedure-oriented IMCE algorithm to deal with hybrid edge changes within a unified algorithm framework. In particular, the algorithm is able to process a large batch of edge changes and significantly improve the average processing time of a single edge change through an efficient pruning strategy. Experimental results on real and synthetic graph data demonstrate that the proposed algorithm outperforms all the baselines and achieves good efficiency through pruning.

The exploration and analysis of Web graphs has flourished in the recent past, producing a large number of relevant and interesting research results. However, the unique characteristics of the Tor network demand for specific algorithms to explore and analyze it. Tor is an anonymity network that allows offering and accessing various Internet resources while guaranteeing a high degree of provider and user anonymity. So far the attention of the research community has focused on assessing the security of the Tor infrastructure. Most research work on the Tor network aimed at discovering protocol vulnerabilities to de-anonymize users and services, while little or no information is available about the topology of the Tor Web graph or the relationship between pages' content and topological structure. With our work we aim at addressing such lack of information. We describe the topology of the Tor Web graph measuring both global and local properties by means of well-known metrics that require due to the size of the network, high performance algorithms. We consider three different snapshots obtained by extensively crawling Tor three times over a 5 months time frame. Finally we present a correlation analysis of pages' semantics and topology, discussing novel insights about the Tor Web organization and its content. Our findings show that the Tor graph presents some of the characteristics of social and surface web graphs, along with a few unique peculiarities.

Reachability queries are a fundamental type of queries on graphs that find important applications in numerous domains. Although a plethora of techniques have been proposed for reachability queries, most of them require that the input graph is static, i.e., they are inapplicable to the {\em dynamic} graphs (e.g., social networks and the Semantic Web) commonly encountered in practice. There exist a few techniques that can handle dynamic graphs, but none of them can scale to sizable graphs without significant loss of efficiency. To address this deficiency, this paper presents a novel study on reachability indices for large dynamic graphs. We first introduce a general indexing framework that summarizes a family of reachability indices with the best performance among the existing techniques for static graphs. Then, we propose general and efficient algorithms for handling vertex insertions and deletions under the proposed framework. In addition, we show that our update algorithms can be used to improve the existing reachability techniques on static graphs, and we also propose a new approach for constructing a reachability index from scratch under our framework. We experimentally evaluate our solution on a large set of benchmark datasets, and we demonstrate that our solution not only supports efficient updates on dynamic graphs, but also provides even better query performance than the state-of-the-art techniques for static graphs.

This article considers problems that can be characterized by large dynamic graphs. Communication networks provide the prototypical example of such problems where nodes in the graph are network IDs and the edges represent communication between pairs of network IDs. In such graphs, nodes and edges appear and disappear through time so that methods that apply to static graphs are not sufficient. Our definition of a dynamic graph is procedural. We introduce a data structure and an updating scheme that captures, in an approximate sense, the graph and its evolution through time. The data structure arises from a bottom-up representation of the large graph as the union of small subgraphs centered on every node. These subgraphs are interesting in their own right and can be enhanced to form what we call communities of interest (COI). We discuss an application in the area of telecommunications fraud detection to help motivate the ideas.

Understanding the correlation between two different scores for the same set of items is a common problem in graph analysis and information retrieval. The most commonly used statistics that quantifies this correlation is Kendall's tau; however, the standard definition fails to capture that discordances between items with high rank are more important than those between items with low rank. Recently, a new measure of correlation based on average precision has been proposed to solve this problem, but like many alternative proposals in the literature it assumes that there are no ties in the scores. This is a major deficiency in a number of contexts, and in particular when comparing centrality scores on large graphs, as the obvious baseline, indegree, has a very large number of ties in social networks and web graphs. We propose to extend Kendall's definition in a natural way to take into account weights in the presence of ties. We prove a number of interesting mathematical properties of our generalization and describe an O(n\log n) algorithm for its computation. We also validate the usefulness of our weighted measure of correlation using experimental data on social networks and web graphs.

Using prefecture-level panel data and social media data, this study investigates how industrial agglomeration, environmental regulations, and technology affect the pollutant intensity and spillover channels of pollutant emissions by integrating social and economic networks into a Spatial Durbin Model. The results show that industrial agglomeration, environmental regulations, and technological inputs facilitate the emissions intensity abatement. The outcomes also confirm that these factors affect the intensity of pollutant emissions in neighboring regions through social, economic, and spatial networks. Agglomeration has a negative spillover effect on the intensity of pollutant emissions in surrounding cities via social and spatial networks, while environmental regulations affect pollutant emissions intensity in related cities through social networks. Technology can effectively lower pollutant emissions through economic networks. These findings highlight the network linkages and spillover channels affecting the intensity of pollutant emissions.

Modeling, analysis, control, and management of complex social networks represent an important area of interdisciplinary research in an advanced digitalized world. In the last decade social networks have produced significant online applications which are running on top of a modern Internet infrastructure and have been identified as major driver of the fast growing Internet traffic. The "Second International Workshop on Modeling, Analysis and Management of Social Networks and Their Applications" (SOCNET 2018) held at Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany, on February 28, 2018, has covered related research issues of social networks in modern information society. The Proceedings of SOCNET 2018 highlight the topics of a tutorial on "Network Analysis in Python" complementing the workshop program, present an invited talk "From the Age of Emperors to the Age of Empathy", and summarize the contributions of eight reviewed papers. The covered topics ranged from theoretical oriented studies focusing on the structural inference of topic networks, the modeling of group dynamics, and the analysis of emergency response networks to the application areas of social networks such as social media used in organizations or social network applications and their impact on modern information society. The Proceedings of SOCNET 2018 may stimulate the readers' future research on monitoring, modeling, and analysis of social networks and encourage their development efforts regarding social network applications of the next generation.

In this paper, we present our study on building a Web graph for the Malaysian Web based on the crawling of Malaysian Web sites. Given the constructed Web graph, interesting characteristics have been studied, such as the in-degree, out-degree, the distribution of power law, bow-tie structure and the strongly-connected components (SCCs). Besides, more important insight could be obtained from analyzing the hyperlinks among the hosts. Similar to Web graph, host graph serves as an informative source for enhancing crawling and searching methodologies, predicting Web growth and for Web sociology study. To date, Web or host graphs have been built for the global web and other nations. Hence, being a subset of the global Web, it would also be interesting to compare the characteristics of Malaysian host graph with the global web and other nations'. This research outputs a Malaysian Web graph, a Malaysian host graph and studies on the characteristics of these graphs. The graphs portray characteristics that are consistent with the graphs created for other nations. There are irregularities to the graphs where it is discussed at length to justify to anomalies.

Social network analysis has become a major research area that has impact in diverse applications ranging from search engines to product recommendation systems. A major problem in implementing social network analysis algorithms is the sheer size of many social networks, for example, the Facebook graph has more than 900 million vertices and even small networks may have tens of millions of vertices. One solution to dealing with these large graphs is dimensionality reduction using spectral or SVD analysis of the adjacency matrix of the network, but these global techniques do not necessarily take into account local structures or clusters of the network that are critical in network analysis. A more promising approach is clustered low-rank approximation: instead of computing a global low-rank approximation, the adjacency matrix is first clustered, and then a low-rank approximation of each cluster (i.e., diagonal block) is computed. The resulting algorithm is challenging to parallelize not only because of the large size of the data sets in social network analysis, but also because it requires computing with very diverse data structures ranging from extremely sparse matrices to dense matrices. In this paper, we describe the first parallel implementation of a clustered low-rank approximation algorithm for large social network graphs, and use it to perform link prediction in parallel. Experimental results show that this implementation scales well on large distributed-memory machines; for example, on a Twitter graph with roughly 11 million vertices and 63 million edges, our implementation scales by a factor of 86 on 128 processes and takes less than 2300 seconds, while on a much larger Twitter graph with 41 million vertices and 1.2 billion edges, our implementation scales by a factor of 203 on 256 processes with a running time about 4800 seconds.KeywordsSocial network analysislink predictionparallel computinggraph computationsclustered low-rank approximation

Distributed frameworks, such as MapReduce and Spark, have been developed by industry and research groups to analyze the vast amount of data that is being generated on a daily basis. Many graphs of interest, such as the Web graph and Social Networks, increase their size daily at an unprecedented scale and rate. To cope with this vast amount of data, researchers have been using distributed processing frameworks to analyze these graphs extensively. Most of these graph algorithms are iterative in nature. In our previous work, we introduced an efficient design pattern to handle a family of iterative graph algorithms in a distributed framework. Unfortunately, in most of these iterative algorithms, such as for Page-Rank, if the graph is modified with the addition or deletion of edges or vertices, the Page-Rank has to be recomputed from scratch. In this paper, we are introducing an improved design pattern for such algorithms to handle graph updates in an incremental fashion. Our method is to separate the graph topology from the graph analysis results. At each iteration step, each node participating in this graph analysis task, in addition to reading a single graph partition, it reads all the current analysis results from the distributed file system (DFS). These results are correlated with the local graph partition using a special merge-join and the new improved analysis results are calculated and stored to the DFS, one partition from each worker node. To handle continuous updates, an update function collects the changes to the graph and applies them to the graph partitions in a streaming fashion. Once the changes are made, the iterative algorithm is resumed to process the new updated data. Since a large part of the graph analysis task has already been completed on the existing data, the new updates require fewer iterations to compute the new graph analysis results as the iterative algorithm will converge faster.

Numerous applications focus on the analysis of entities and the connections between them, and such data are naturally represented as graphs. In particular, the detection of a small subset of vertices with anomalous coordinated connectivity is of broad interest, for problems such as detecting strange traffic in a computer network or unknown communities in a social network. These problems become more difficult as the background graph grows larger and noisier and the coordination patterns become more subtle. In this paper, we discuss the computational challenges of a statistical framework designed to address this cross-mission challenge. The statistical framework is based on spectral analysis of the graph data, and three partitioning methods are evaluated for computing the principal eigenvector of the graph's residuals matrix. While a standard one-dimensional partitioning technique enables this computation for up to four billion vertices, the communication overhead prevents this method from being used for even larger graphs. Recent two-dimensional partitioning methods are shown to have much more favorable scaling properties. A data-dependent partitioning method, which has the best scaling performance, is also shown to improve computation time even as a graph changes over time, allowing amortization of the upfront cost.