Probabilistic estimates of attribute statistics and match likelihood for people entity resolution

Zuckerli is a scalable compression system meant for large real-world graphs. Graphs are notoriously challenging structures to store efficiently due to their linked nature, which makes it hard to separate them into smaller, compact components. Therefore, effective compression is crucial when dealing with large graphs, which can have billions of nodes and edges. Furthermore, a good compression system should give the user fast and reasonably flexible access to parts of the compressed data without requiring full decompression, which may be unfeasible on their system. Compared to WebGraph, the de-facto standard in compressing real-world graphs, Zuckerli improves multiple aspects by using advanced compression techniques and novel heuristic graph algorithms. Zuckerli can produce both a compressed representation for storage and one which allows fast direct access to the adjacency lists of the compressed graph without decompressing the entire graph. We validate its effectiveness on real-world graphs with up to a billion nodes and 90 billion edges, conducting an extensive experimental evaluation of both compression density and decompression performance. We show that Zuckerli-compressed graphs are 10% to 29% smaller, and more than 20% in most cases, with a resource usage for decompression comparable to that of WebGraph.

Graphs appear in numerous applications including cyber security, the Internet, social networks, protein networks, recommendation systems, citation networks, and many more. Graphs with millions or even billions of nodes and edges are common-place. How to store such large graphs efficiently? What are the core operations/queries on those graph? How to answer the graph queries quickly? We propose Gbase, an efficient analysis platform for large graphs. The key novelties lie in (1) our storage and compression scheme for a parallel, distributed settings and (2) the carefully chosen graph operations and their efficient implementations. We designed and implemented an instance of Gbase using Mapreduce/Hadoop. Gbase provides a parallel indexing mechanism for graph operations that both saves storage space, as well as accelerates query responses. We run numerous experiments on real and synthetic graphs, spanning billions of nodes and edges, and we show that our proposed Gbase is indeed fast, scalable, and nimble, with significant savings in space and time.

This book discusses the development and implementation of data structures and algorithms using C#. The data structures we use in this book are found in the .NET Framework class library System.Collections. In this chapter, we develop the concept of a collection by first discussing the implementation of our own Collection class (using the array as the basis of our implementation) and then by covering the Collection classes in the .NET Framework. An important addition to C# 2.0 is generics. Generics allow the C# programmer to write one version of a function, either independently or within a class, without having to overload the function many times to allow for different data types. C# 2.0 provides a special library, System.Collections.Generic, that implements generics for several of the System.Collections data structures. This chapter will introduce the reader to generic programming. Finally, this chapter introduces a custom-built class, the Timing class, which we will use in several chapters to measure the performance of a data structure and/or algorithm. This class will take the place of Big O analysis, not because Big O analysis isn't important, but because this book takes a more practical approach to the study of data structures and algorithms. COLLECTIONS DEFINED A collection is a structured data type that stores data and provides operations for adding data to the collection, removing data from the collection, updating data in the collection, as well as operations for setting and returning the values of different attributes of the collection.

Rough set theory has been proposed by Pawlak in the early 80s to deal with inconsistency problems following from information granulation. It operates on an information table composed of a set U of objects described by a set Q of condition and decision attributes. Decision attributes make a partition of U into decision classes. Basic concepts of rough set theory are: indiscernibility relation on U, lower and upper (rough) approximations of decision classes, dependence and reduction of attributes from Q, and decision rules induced from rough approximations of decision classes. The original rough set idea was failing, however, to handle preferential ordering of domains of attributes (scales of criteria), as well as preferential ordering of decision classes. In order to deal with multiple criteria decision problems a number of methodological changes to the original rough set theory were necessary. The main change is the substitution of the indiscernibility relation by a dominance relation, which permits approximation of ordered sets. In multiple criteria decision context, the information table is composed of decision examples given by a decision maker. The Dominance-based Rough Set Approach (DRSA) applied to this information table results with a set of decision rules, being a preference model of the decision maker. It is more general than the classical multiple attribute utility model or outranking model, and it is more understandable because of its natural syntax. In this chapter, after recalling the classical rough set approach and DRSA, we review their fuzzy set extensions. Moreover, we characterize the dominance-based rough approximation of a fuzzy set, and we show that the classical rough approximation of a crisp set is its particular case. In this sense, DRSA is also relevant in the case where preferences are not considered, but just a kind of monotonicity relating values of different attributes is meaningful for the analysis of data at hand. In general terms, monotonicity concerns relationship between different aspects of a phenomenon described by data: for example, “the larger the house, the higher its price” or “the closer the house to the city centre, the higher its price”. In this perspective, DRSA gives a very general framework for reasoning about data using only monotonicity relationships.

Global citizenship education (GCE), like many educational concepts that are now garnering attention as useful extensions to mainstay instructional targets such as literacy and numeracy, faces ongoing problems of how to define and measure it. After providing an overview of those problems, I offer discrete-choice measures (DCMs) and situational-judgment tests (SJTs) as potential solutions. DCMs—also known as forced-choice measures—are more commonly found in economic and market research than in education. They prompt respondents to select from a set of choices so researchers can quantify the relative value of different attributes that those choices represent. SJTs offer respondents a scenario with a list of possible responses so their chosen response enables researchers to analyze attributes underlying their choices. Overall, I intend for readers to think of DCMs and SJTs as encouraging possibilities, not silver bullets.

Salinity is an ancient environmental phenomenon and reflected as the most important process of land degradation. It is widespread at variable degrees across the world. A sand culture study was conducted in order to investigate the performance of exogenously applied triacontanol on two tolerant (Green long and Marketmore) and two sensitive (Summer green and 20252) genotypes of cucumber (Cucumis sativus L.) under salinity stress (NaCl 50 mM). The foliar application of triacontanol was carried out @ 0.20, 0.40, 0.60, 0.80, 1.00 and 1.20 mg L-1. Salinity caused significant reduction in growth rate, gas exchange and other physiological attributes. Results revealed that triacontanol seemed to relieve the harmful impact of salt stress by improving morpho-physiological attributes and decreasing membrane leakage. Genotypes Green long and Marketmore performed better under salt stress regarding all studied parameters than Summer green and 20252. However, foliar feeding of triacontanol significantly enriched the efficiency of sensitive genotypes under saline conditions. The highest values of different attributes of cucumber plants were observed with foliar application of 0.80 mg L-1 triacontanol. Hence, triacontanol can be effectively used as a mitigating agent to alleviate phytotoxic effects in plants under saline stress.

Distributed-memory clusters are used for in-memory processing of very large graphs with billions of nodes and edges. This requires partitioning the graph among the machines in the cluster. When a graph is partitioned, a node in the graph may be replicated on several machines, and communication is required to keep these replicas synchronized. Good partitioning policies attempt to reduce this synchronization overhead while keeping the computational load balanced across machines. A number of recent studies have looked at ways to control replication of nodes, but these studies are not conclusive because they were performed on small clusters with eight to sixteen machines, did not consider work-efficient data-driven algorithms, or did not optimize communication for the partitioning strategies they studied. This paper presents an experimental study of partitioning strategies for work-efficient graph analytics applications on large KNL and Skylake clusters with up to 256 machines using the Gluon communication runtime which implements partitioning-specific communication optimizations. Evaluation results show that although simple partitioning strategies like Edge-Cuts perform well on a small number of machines, an alternative partitioning strategy called Cartesian Vertex-Cut (CVC) performs better at scale even though paradoxically it has a higher replication factor and performs more communication than Edge-Cut partitioning does. Results from communication micro-benchmarks resolve this paradox by showing that communication overhead depends not only on communication volume but also on the communication pattern among the partitions. These experiments suggest that high-performance graph analytics systems should support multiple partitioning strategies, like Gluon does, as no single graph partitioning strategy is best for all cluster sizes. For such systems, a decision tree for selecting a good partitioning strategy based on characteristics of the computation and the cluster is presented.

We have entered the age of big data. Massive datasets are now common in science, government and enterprises. Yet, making sense of these data remains a fundamental challenge. Where do we start our analysis? Where to go next? How to visualize our findings? We answers these questions by bridging Data Mining and Human- Computer Interaction (HCI) to create tools for making sense of graphs with billions of nodes and edges, focusing on: (1) Attention Routing: we introduce this idea, based on anomaly detection, that automatically draws people’s attention to interesting areas of the graph to start their analyses. We present three examples: Polonium unearths malware from 37 billion machine-file relationships; NetProbe fingers bad guys who commit auction fraud. (2) Mixed-Initiative Sensemaking: we present two examples that combine machine inference and visualization to help users locate next areas of interest: Apolo guides users to explore large graphs by learning from few examples of user interest; Graphite finds interesting subgraphs, based on only fuzzy descriptions drawn graphically. (3) Scaling Up: we show how to enable interactive analytics of large graphs by leveraging Hadoop, staging of operations, and approximate computation. This thesis contributes to data mining, HCI, and importantly their intersection, including: interactive systems and algorithms that scale; theories that unify graph mining approaches; and paradigms that overcome fundamental challenges in visual analytics. Our work is making impact to academia and society: Polonium protects 120 million people worldwide from malware; NetProbe made headlines on CNN, WSJ and USA Today; Pegasus won an opensource software award; Apolo helps DARPA detect insider threats and prevent exfiltration. We hope our Big Data Mantra “Machine for Attention Routing, Human for Interaction” will inspire more innovations at the crossroad of data mining and HCI.

In this paper, we revisit the problem of influence maximization with fairness, which aims to select k influential nodes to maximise the spread of information in a network, while ensuring that selected sensitive user attributes (e.g., gender, location, origin, race, etc.) are fairly affected, i.e., are proportionally similar between the original network and the affected users. Recent studies on this problem focused only on extremely small networks, hence the challenge remains on how to achieve a scalable solution, applicable to networks with millions or billions of nodes. We propose an approach that is based on learning node representations (embeddings) for fair spread from diffusion cascades, instead of the social connectivity, and in this way we can deal with very large graphs. We propose two data-driven approaches: (a) fairness-based participant sampling (FPS), and (b) fairness as context (FAC). Spread related user features, such as the probability of diffusing information to others, are derived from the historical information cascades, using a deep neural network. The extracted features are then used in selecting influencers that maximize the influence spread, while being also fair with respect to the chosen sensitive attributes. In FPS, fairness and cascade length information are considered independently in the decision-making process, while FAC considers these information facets jointly and takes into account correlations between them. The proposed algorithms are generic and represent the first policy-driven solutions that can be applied to arbitrary sets of sensitive attributes at scale. We evaluate the performance of our solutions on a real-world public dataset (Sina Weibo) and on a hybrid real-synthetic dataset (Digg), which exhibit all the facets that we exploit, namely diffusion network, diffusion traces, and user profiles. These experiments show that our methods outperform the state-the-art solutions in terms of spread, fairness, and scalability.

Given a large graph with millions or billions of nodes and edges, like a who-follows-whom Twitter graph, how do we scalably compute its statistics, summarize its patterns, spot anomalies, visualize and make sense of it? We present OPAvion, a graph mining system that provides a scalable, interactive workflow to accomplish these analysis tasks. OPAvion consists of three modules: (1) The Summarization module (Pegasus) operates off-line on massive, disk-resident graphs and computes graph statistics, like PageRank scores, connected components, degree distribution, triangles, etc.; (2) The Anomaly Detection module (OddBall) uses graph statistics to mine patterns and spot anomalies, such as nodes with many contacts but few interactions with them (possibly telemarketers); (3) The Interactive Visualization module (Apolo) lets users incrementally explore the graph, starting with their chosen nodes or the flagged anomalous nodes; then users can expand to the nodes' vicinities, label them into categories, and thus interactively navigate the interesting parts of the graph.In our demonstration, we invite our audience to interact with OPAvion and try out its core capabilities on the Stack Overflow Q&A graph that describes over 6 million questions and answers among 650K users.

A reachability query is a fundamental graph operation in real graph applications, which answers whether a node can reach another node through a path in a graph. However, the increasingly large amounts of real graph data make it more challenging for query efficiency and scalability. In this paper, we propose a Fruited-Forest (FF) approach to accelerate reachability queries in large graphs by constructing four kinds of fruited-forests from a reduced DAG in different traversal orders. We build different binary-label schemes for the four kinds of fruited-forests to cover reachability between nodes as much as possible, and create a corresponding index for the deleted edges which are deleted during the construction of fruited-forests. Our experimental results on 18 large real graph datasets show that our FF approach requires less index construct time and a smaller index size, which is more scalable to answer reachability queries compared with other existing works.

Subgraph matching, which belongs to NP-hard, faces significant challenges on a large scale graph with billions of nodes, and existing methods are usually confronted with greater challenges from both stability and efficiency. In this article, a subgraph matching method in a distributed system, tree model-based subgraph matching method (TBSGM) is proposed. The authors provide a transformed efficient query tree as a replacement for a query graph. In order to get the tree, they present a cost evaluation model which may help to generate the efficient query tree according to network communication-cost and calculation-cost evaluation. Also, a key set based indexing strategy for intermediate results is given to simplify the matching results during network communication. Extensive experiments with real-world datasets show that TBSGM significantly outperforms other methods in the aspects of scalability and efficiency.

How do we find patterns and anomalies, on graphs with billions of nodes and edges, which do not fit in memory? How to use parallelism for such terabyte-scale graphs? In this work, we focus on inference, which often corresponds, intuitively, to "guilt by association" scenarios. For example, if a person is a drug-abuser, probably its friends are so, too; if a node in a social network is of male gender, his dates are probably females. We show how to do inference on such huge graphs through our proposed HADOOP Line graph Fixed Point (HA-LFP), an efficient parallel algorithm for sparse billion-scale graphs, using the HADOOP platform. Our contributions include (a) the design of HA-LFP, observing that it corresponds to a fixed point on a line graph induced from the original graph; (b) scalability analysis, showing that our algorithm scales up well with the number of edges, as well as with the number of machines; and (c) experimental results on two private, as well as two of the largest publicly available graphs - the Web Graphs from Yahoo! (6.6 billion edges and 0.24 Tera bytes), and the Twitter graph (3.7 billion edges and 0.13 Tera bytes). We evaluated our algorithm using M45, one of the top 50 fastest supercomputers in the world, and we report patterns and anomalies discovered by our algorithm, which would be invisible otherwise.

Graph summarization aims to extract critical information from large graphs by creating summaries that represent the original data.Especially, real-world daily applications generate massive dynamic streaming graphs, representing as edge or node streams.How to efficiently generate compact and lossless graph summaries for large streaming graphs is still a challenging problem.This paper proposes an efficient and scalable lossless summarization method for streaming graphs, called Partition and Similarity-based Two-Stage Summarization (PSTSS).PSTSS uses a streaming graph partitioning algorithm as the first stage to generate coarse-grained summaries by on-the-fly subgraph partitioning.The second stage generates fine-grained supernodes within these coarse-grained supernodes through efficient node similarity calculation.Our experiments on six large datasets demonstrate that our method achieves better operational efficiency, compression rate, scalability, and graph query speed compared to state-of-the-art lossless summarization algorithms.

This paper describes a new congestive heart failure (CHF) treatment performance measure information extraction system - CHIEF - developed as part of the Automated Data Acquisition for Heart Failure project, a Veterans Health Administration project aiming at improving the detection of patients not receiving recommended care for CHF. CHIEF is based on the Apache Unstructured Information Management Architecture framework, and uses a combination of rules, dictionaries, and machine learning methods to extract left ventricular function mentions and values, CHF medications, and documented reasons for a patient not receiving these medications. The training and evaluation of CHIEF were based on subsets of a reference standard of various clinical notes from 1083 Veterans Health Administration patients. Domain experts manually annotated these notes to create our reference standard. Metrics used included recall, precision, and the F 1 -measure. In general, CHIEF extracted CHF medications with high recall (>0.990) and good precision (0.960-0.978). Mentions of Left Ventricular Ejection Fraction were also extracted with high recall (0.978-0.986) and precision (0.986-0.994), and quantitative values of Left Ventricular Ejection Fraction were found with 0.910-0.945 recall and with high precision (0.939-0.976). Reasons for not prescribing CHF medications were more difficult to extract, only reaching fair accuracy with about 0.310-0.400 recall and 0.250-0.320 precision. This study demonstrated that applying natural language processing to unlock the rich and detailed clinical information found in clinical narrative text notes makes fast and scalable quality improvement approaches possible, eventually improving management and outpatient treatment of patients suffering from CHF.