RBSEP: a reassignment and buffer based streaming edge partitioning approach

We will study the problem of analyzing massive data streams in the context of the dynamic network-centered activities. Consider a network in which a time series activity data stream is associated with each edge in a massive network. Examples of such activities could include citation networks with continuously changing edge values, such as road network traffic, social network traffic, or email traffic. We introduce the problem of link regression, which models the prediction of the future stream values associated with a link from the history of all the stream values associated with the different links. This problem is a powerful generalization of both the large scale time series prediction problem and the link prediction problems in network analysis, each of which are known to be quite difficult in their own right. The generality of this framework provides it significant applicability for traffic analysis in road networks, congestion analysis in communication networks, interest trend analysis and bursty feature analysis in social networks. The problem is very challenging because of its massiveness both in network size and stream speed. We present an algorithm which uses streaming graph partitioning in conjunction with predictive regression. We present experimental results illustrating the effectiveness and efficiency of the approach.

Partitioning approach oriented to the decentralised predictive control of large-scale systems

Graph partitioning is a mandatory step in large-scale distributed graph processing. When partitioning real-world power-law graphs, the edge partitioning algorithm performs better than the traditional vertex partitioning algorithm, because it can cut a single vertex into multiple replicas to apportion the computation. Many advanced edge partitioning methods are designed for partitioning a static graph from scratch. However, the real-world graph structure changes continuously, which leads to a decrease in partition quality and affects the performance of the graph applications. Some studies are devoted to offline repartitioning or batch incremental partitioning, but how to deal with dynamics in real-time is still worthy of in-depth study. In this article, we discuss the impact of dynamic change on partition and discover that both insertion and deletion will lead to local suboptimal partitioning, which is the reason for the degradation of partition quality. As a solution, a dynamic edge partitioning algorithm is proposed to partition dynamics in real-time. Specifically, we deal with dynamics by a distributed stream and improve partition quality by reassigning some closely connected edges. Experiments show that it is robust to initial partition quality, dynamic scale and type, and distributed scale. Compared with the state-of-the-art dynamic partitioner, it can reduce vertex-cuts by 29.5 percent. Compared with the repartitioning algorithms, it can save the partitioning time by 91.0 percent. Applied on the graph task, it can reduce the increase of communication cost and the increase of the total time of task by 41.5 and 71.4 percent.

A drift aware adaptive method based on minimum uncertainty for anomaly detection in social networking

k-way graph partitioning is an NP-complete problem, which is applied to various tasks such as route planning, image segmentation, community detection, and high-performance computing. The approximate methods constitute a useful solution for these types of problems. Thus, many research studies have focused on developing meta-heuristic algorithms to tackle the graph partitioning problem. Local search is one of the earliest methods that has been applied efficiently to this type of problem. Recent studies have explored various types of local search methods and have improved them such that they can be used with the partitioning process. Moreover, local search methods are widely integrated with population-based approaches, to provide the best diversification and intensification for the problem space. This study emphasizes the local search approaches, as well as their combination with other graph partitioning approaches. At present, none of the surveys in the literature has focused on this class of state of the art approaches in much detail. In this study, the vital parts of these approaches including neighborhood structure, acceptance criterion, and the ways of combining them with other approaches, are highlighted. Additionally, we provide an experimental comparison that shows the variance in the performance of the reviewed methods. Hence, this study clarifies these methods to show their advantages and limitations for the targeted problem, and thus can aid in the direction of research flow towards the area of graph partitioning.

Streaming graph partitioning methods have recently gained attention due to their ability to scale to very large graphs with limited resources. However, many such methods do not consider workload and graph characteristics. This may degrade the performance of queries by increasing inter-node communication and computational load imbalance. Moreover, existing workload-aware methods cannot consistently provide good performance as they do not consider dynamic workloads that keep emerging in graph applications. We address these issues by proposing a novel workload-adaptive streaming partitioner named WASP, that aims to achieve low-latency and high-throughput online graph queries. As each workload typically contains frequent query patterns, WASP exploits the existing workload to capture active vertices and edges which are frequently visited and traversed, respectively. This information is used to heuristically improve the quality of partitions either by avoiding the concentration of active vertices in a few partitions proportional to their visit frequencies or by reducing the probability of the cut of active edges proportional to their traversal frequencies. In order to assess the impact of WASP on a graph store and to show how easily the approach can be plugged on top of the system, we exploit it in a distributed graph-based RDF store. Our experiments over three synthetic and real-world graph datasets and the corresponding static and dynamic query workloads show that WASP achieves a better query performance against state-of-the-art graph partitioners, especially in dynamic query workloads.

Graph partitioning is a fundamental problem in several scientific and engineering applications. In this paper, we describe heuristics that improve the state-of-the-art practical algorithms used in graph-partitioning software in terms of both partitioning speed and quality. An important use of graph partitioning is in ordering sparse matrices for obtaining direct solutions to sparse systems of linear equations arising in engineering and optimization applications. The experiments reported in this paper show that the use of these heuristics results in a considerable improvement in the quality of sparse-matrix orderings over conventional ordering methods, especially for sparse matrices arising in linear programming problems. In addition, our graph-partitioning-based ordering algorithm is more parallelizable than minimum-degree-based ordering algorithms, and it renders the ordered matrix more amenable to parallel factorization.

Over the last few years, graph partitioning has been recognized as a suitable technique for optimizing cellular network structure. For example, in a recent paper, the authors proposed a classical graph partitioning algorithm to optimize the assignment of cells to Packet Control Units (PCUs) in GSM-EDGE Radio Access Network. Based on this approach, the quality of packet data services in a live environment was increased by reducing the number of cell re-selections between different PCUs. To learn more about the potential of graph partitioning in cellular networks, in this paper, a more sophisticated, yet computationally efficient, partitioning algorithm is proposed for the same problem. The new method combines multi-level refinement and adaptive multi-start techniques with algorithms to ensure the connectivity between cells under the same PCU. Performance assessment is based on an extensive set of graphs constructed with data taken from a live network. During the tests, the new method is compared with classical graph partitioning approaches. Results show that the proposed method outperforms classical approaches in terms of solution quality at the expense of a slight increase in computing time, while providing solutions that are easier to check by the network operator.

SummaryThis paper presents how parallel advanced graph partitioning algorithms can be designed and improved with the use of global application states monitoring of distributed programs. The proposed algorithms have been implemented inside a novel distributed program design framework Program Execution Governed by Asynchronous Supervision of States in Distributed Applications (PEGASUS DA). This framework provides system support for the design of execution control in distributed applications based on automated global state monitoring. Two strategies for the control design of advanced parallel/distributed graph partitioning algorithms are presented and discussed. In the first one, the parallel algorithm control runs on top of the popular graph partitioning METIS tool. The second control strategy is based on genetic programming so that partitioning primitives and the overall algorithmic control can be freely designed by the user. Advanced partitioning methods have been conveniently embedded inside the global state monitoring driven PEGASUS DA framework which controls partitioning distributed actions at the level of processes and threads. The use of the framework allowed easy design and testing of different graph optimization strategies. The presented graph partitioning methods are illustrated by experiments with benchmark graphs. The experiments have comparatively assessed the obtained graph partitioning quality (visible improvement has been observed) and have identified benefits of the proposed approach for programmers.

The 2017 Grand Challenge focused on the problem of automatic detection of anomalies for manufacturing equipment. This paper reports the technical details of a solution focused on particular optimizations of the processing stages. These included customized input parsing, fine tuning of a k-means clustering algorithm and probability analysis using a lazy flavor of a Markov chain. We have observed in our custom implementation that carefully tweaking these processing stages at single node level by leveraging various data stream characteristics can yield good performance results. We start the paper with several observations concerning the input data stream, following with our solution description with details on particular optimizations, and we conclude with evaluation and a discussion of obtained results.

We consider the read/write streams model, an extension of the standard data stream model in which an algorithm can create and manipulate multiple read/write streams in addition to its input data stream. We show that any randomized read/write stream algorithm with a fixed number of streams and a sublogarithmic number of passes that produces a constant factor approximation of the k-th frequency moment Fk of an input sequence of length of at most N from {1, ..., N} requires space Omega(N1-4/k-delta) for any delta > 0. For comparison, it is known that with a single read-only data stream there is a randomized constant- factor approximation for Fk using O(N1-2/k) space and that there is a deterministic algorithm computing Fk exactly using 3 read/write streams, O(log N) passes, and O(log N) space. Therefore, although the ability to manipulate multiple read/write streams can add substantial power to the data stream model, with a sub-logarithmic number of passes this does not significantly improve the ability to approximate higher frequency moments efficiently. Our lower bounds also apply to (1 + epsi)-approximations of Fk for epsi ges 1/N.

Nowadays, many applications that involve big data can be modelled as graphs. In many cases these graphs may be too large to be loaded and processed on a single commodity computer. This necessitated the development of frameworks that allow processing large graphs by distributing the graph among nodes in a cluster. To process a graph using these frameworks, first, the graph is partitioned into smaller components called subgraphs or partitions, and then assign these smaller subgraphs to different nodes for parallel processing. Depending on the type of processing (example, computing pagerank, counting number of triangles etc.), there will be some communication between nodes during the execution, this communication affects execution time. Therefore, graph partitioning is an important step in distributed graph processing. Being able to determine the quality of a partition prior to processing is important as this will allow us to predict the execution time before the actual processing. A number of metrics for evaluating the quality of a graph partitions exist, but studies show that these metrics may not serve as accurate predictors in many cases. In this work, we reviewed published papers about graph partitioning and we were able to identify and defined more metrics in order to have a catalogue of these metrics.

Alignment and comparison of protein 3D structures is an important and fundamental task in structural biology to study evolutionary, functional and structural relatedness among proteins. Since two decades, the research on protein structure alignment has been taken up on priority and numbers of research articles are being published. There are incremental advances over previous efforts, and still these methods continue to improve over the time and still this is an open problem in structural biology. A novel methodology has been developed for comparing protein 3D structure by employing conversion of pair of protein 3D structures into 2D graphs (undirected weighted graph), partitioning of 2D graphs into sub-graphs, matching sub-graphs with main graphs and finally these sub-graphs matches calculates similarity between the pair of proteins. The proposed method has been implemented in MATLAB and R Package. The performance of the developed methodology is tested with four existing best methods such as CE, jFATCAT, TM_Align and Dali on 100 proteins benchmark dataset with SCOP database. The proposed method is efficient in terms of time complexity, accuracy, grouping of proteins in relevant structural groups and provides additional information towards non-bonded interactions and sub-graphs indicates the dominance of secondary structure.

In the last years, large-scale graph processing has gained increasing attention, with most recent systems placing particular emphasis on latency. One possible technique to improve runtime performance in a distributed graph processing system is to reduce network communication. The most notable way to achieve this goal is to partition the graph by minimizing the number of edges that connect vertices assigned to different machines, while keeping the load balanced. However, real-world graphs are highly dynamic, with vertices and edges being constantly added and removed. Carefully updating the partitioning of the graph to reflect these changes is necessary to avoid the introduction of an extensive number of cut edges, which would gradually worsen computation performance. In this paper we show that performance degradation in dynamic graph processing systems can be avoided by adapting continuously the graph partitions as the graph changes. We present a novel highly scalable adaptive partitioning strategy, and show a number of refinements that make it work under the constraints of a large-scale distributed system. The partitioning strategy is based on iterative vertex migrations, relying only on local information. We have implemented the technique in a graph processing system, and we show through three real-world scenarios how adapting graph partitioning reduces execution time by over 50% when compared to commonly used hash-partitioning.

The m‐way graph partitioning problem is of central importance in combinatorial optimization. It has many important applications in fields such as VLSI circuit design, task allocation in distributed computing systems, and network partitioning. In this paper, we propose an efficient genetic algorithm to solve this problem. The proposed method searches a large solution space and finds the best possible solution by adjusting the intensification and diversification automatically during the optimization process. The proposed method is tested on a large number of instances and compared with some existing algorithms. The experimental results show that the proposed algorithm is superior to its competitors in terms of computation time and solution quality. © 2011 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.