Sorted Nearest Neighborhood Clustering for Efficient Private Blocking

Record linkage represents the process of identifying records that refer to the same real-world entity across multiple sources. In the recent past, the large explosion of data from organizations and individuals has created critical challenges in record linkage process, leading the scientific community to develop a rich series of techniques. Among these recent developments, the design of record linkage solutions for medical data is a cornerstone for health-care systems, as personal identifying information, such as name and date of birth, is often used for linkage. In this setting, record linkage solutions are expected to preserve individual patients’ privacy in addition to be effective and efficient. In this chapter, we provide a broad review of the recent works in privacy preserving record linkage (PPRL). From a privacy-centric perspective, we summarize a comprehensive framework of the PPRL process and describe the privacy assurance techniques adopted for each step in the framework. With the applications in biomedical domain in mind, we identify several challenges for PPRL and discuss future research directions.

ABSTRACT
 ObjectivesWhile record linkage has become a strategic research priority within Australia and internationally, legal and administrative issues prevent data linkage in some situations due to privacy concerns. Even current best practices in record linkage carry some privacy risk as they require the release of personally identifying information to trusted third parties. Application of record linkage systems that do not require the release of personal information can overcome legal and privacy issues surrounding data integration. Current conceptual and experimental privacy-preserving record linkage (PPRL) models show promise in addressing data integration challenges but do not yet address all of the requirements for real-world operations. This paper aims to identify and address some of the challenges of operationalising PPRL frameworks.
 ApproachTraditional linkage processes involve comparing personally identifying information (name, address, date of birth) on pairs of records to determine whether the records belong to the same person. Designing appropriate linkage strategies is an important part of the process. These are typically based on the analysis of data attributes (metadata) such as data completeness, consistency, constancy and field discriminating power. Under a PPRL model, however, these factors cannot be discerned from the encrypted data, so an alternative approach is required. This paper explores methods for data profiling, blocking, weight/threshold estimation and error detection within a PPRL framework.
 ResultsProbabilistic record linkage typically involves the estimation of weights and thresholds to optimise the linkage and ensure highly accurate results. The paper outlines the metadata requirements and automated methods necessary to collect data without compromising privacy. We present work undertaken to develop parameter estimation methods which can help optimise a linkage strategy without the release of personally identifiable information. These are required in all parts of the privacy preserving record linkage process (pre-processing, standardising activities, linkage, grouping and extracting).
 ConclusionsPPRL techniques that operate on encrypted data have the potential for large-scale record linkage, performing both accurately and efficiently under experimental conditions. Our research has advanced the current state of PPRL with a framework for secure record linkage that can be implemented to improve and expand linkage service delivery while protecting an individual’s privacy. However, more research is required to supplement this technique with additional elements to ensure the end-to-end method is practical and can be incorporated into real-world models.

ABSTRACTObjectiveIn most European settings, record linkage across different institutions has to be based on personal identifiers such as names, birthday or place of birth. To protect the privacy of research subjects, the identifiers have to be encrypted. In practice, these identifiers show error rates up to 20% per identifier, therefore linking on encrypted identifiers usually implies the loss of large subsets of the databases. In many applications, this loss of cases is related to variables of interest for the subject matter of the study. Therefore, this kind of record-linkage will generate biased estimates.
 These problems gave rise to techniques of Privacy Preserving Record Linkage (PPRL). Many different PPRL techniques have been suggested within the last 10 years, very few of them are suitable for practical applications with large database containing millions of records as they are typical for administrative or medical databases. One proven technique for PPRL for large scale applications is PPRL based on Bloom filters.MethodUsing appropriate parameter settings, Bloom filter approaches show linkage results comparable to linkage based on unencrypted identifiers. Furthermore, this approach has been used in real-world settings with data sets containing up to 100 Million records. By the application of suitable blocking strategies, linking can be done in reasonable time.ResultHowever, Bloom filters have been subject of cryptographic attacks. Previous research has shown that the straight application of Bloom filters has a nonzero re-identification risk. We will present new results on recently developed techniques to defy all known attacks on PPRL Bloom filters. These computationally simple algorithms modify the identifiers by different cryptographic diffusion techniques. The presentation will demonstrate these new algorithms and show their performance concerning precision, recall and re-identification risk on large databases.

BackgroundData on newborns is regularly linked for epidemiological research. However, hospital data often suffers from incomplete data. We report on a linkage of two population-covering administrative health databases containing neonatal and perinatal data without unique personal identifiers and with incomplete information in standard patient identifiers.
 GoalTo study the effects of a policy-induced change from linking a national database without standard patient identifiers to a privacy-preserving Record Linkage method, we compare the linkage system in use to clear-text and privacy-preserving Record Linkage techniques. We expected large proportions of missing identifiers since they are not needed for clinical practice. Therefore, we expected missing links caused by missing identifiers. To study the impact of these missing identifiers on these successful links, we compared several linkage methods. Furthermore, we study the variations of linkage success between hospitals.
 MethodsPerinatal and neonatal data from population-covering real-world administrative databases was linked using several variants of state of the art methods, including Privacy-preserving Record Linkage (PPRL) techniques such as multiple match keys and Bloom filter methods.
 Results We report on the variation of linkage results between the hospitals and give possible explanations for the differences.
 The resulting linkage success is reported for each method. The impact of incomplete data on linkage success for each method is documented.
 Finally, we report on the relative performance of the modified techniques compared to standard linkage procedures used in practice.
 ConclusionImplementing a record linkage system based on identifiers not required for clinical practice caused a large number of missing identifiers. Since this information is essential for successful clear-text and private linkage methods, emphasizing the need for documenting patient identifiers, especially in cases where auxiliary information (such as stable addresses, date of birth or health insurance numbers) are missing, is of central importance for implementing a privacy-preserving Record Linkage system.

In privacy-preserving record linkage, a number of data custodians encode their records and submit them to a trusted third-party who is responsible for identifying those records that refer to the same real-world entity. In this paper, we propose FEDERAL, a novel record linkage framework that implements methods for anonymizing both string and numerical data values, which are typically present in data records. These methods rely on a strong theoretical foundation for rigorously specifying the dimensionality of the anonymization space, into which the original values are embedded, to provide accuracy and privacy guarantees under various models of privacy attacks. A key component of the applied embedding process is the threshold that is required by the distance computations, which we prove can be formally specified to guarantee accurate results. We evaluate our framework using three real-world data sets with varying characteristics. Our experimental findings show that FEDERAL offers a complete and effective solution for accurately identifying matching anonymized record pairs (with recall rates constantly above 93 percent) in large-scale privacy-preserving record linkage tasks.

Big Data projects often require data from several sources to be integrated before they can be used for analysis. Once data have been integrated, they allow more detailed analysis that would otherwise not be possible. Accordingly, recent years have seen an increasing interest in techniques that facilitate the integration of data from diverse sources. Whenever data about individuals, or otherwise sensitive data, are to be integrated across organizations, privacy and confidentiality have to be considered. Domains where privacy preservation during data integration is of importance include business collaborations, health research, national censuses, the social sciences, crime and fraud detection, and homeland security. Increasingly, applications in these domains require data from diverse sources (both internal and external to an organization) to be integrated.Consequently, in the past decade, various techniques have been developed that aim to facilitate data integration without revealing any private or confidential information about the databases and records that are integrated. These techniques either provably prevent leakage of any private information, or they provide some empirical numerical measure of the risk of disclosure of private information.In the first part of this presentation we provide a background on data integration, and illustrate the importance of preserving privacy during data integration with several application scenarios. We then given an overview of the main concepts and techniques that have been developed to facilitate data integration in such ways that no private or confidential information is being revealed. We focus on privacy-preserving record linkage (PPRL), where so far most research has been conducted. We describe the basic protocols used in PPRL, and several key technologies employed in these protocols. Finally, we discuss the challenges privacy poses to data integration in the era of Big Data, and we discuss directions and opportunities in this research area.

The process of identifying record pairs that represent the same real-world entity in multiple databases, commonly known as record linkage, is one of the important initial steps in many data mining applications. Record linkage of millions of records is a computationally expensive task. Various blocking methods have been used in record linkage systems to reduce the number of record pairs for comparison. A good blocking key is critical to the success of a blocking method and will ideally result in lot of small blocks. However, in practice, there are almost always large blocks no matter how good the blocking key is. For example, when blocking on surname for an Anglo-Celtic population, ‘Smith’ and ‘Taylor’ are populous and result in very large block sizes. The efficiency of a blocking method is hindered by these large blocks since the resulting number of record pairs is dominated by the sizes of these large blocks. In this paper, we present an adaptive filtering algorithm to post-process large blocks to enhance the blocking efficiency. Experimental results show that our filtering algorithm can reduce the number of record pairs produced by the standard blocking method by 88% on a small real-world data set. The algorithm also reduces the number of record pairs generated by a 3-pass standard blocking method by 50% on several synthetic test data sets, with minimal loss of accuracy.

IntroductionBusinesses worldwide are increasingly adopting the storage, compute and analytical services provided by cloud computing. Yet, few operational linkage units are keeping pace with this world of technological change - most use legacy systems approaching their limits with the rapidly increasing size and range of datasets now required for linkage.
 Objectives and ApproachTo meet the demands of linkage for the near future, it is important that new solutions for linkage consider the services provided by public cloud infrastructure for compute, storage and analytics. We examined Platform as a Service (PaaS) offerings for use in the development of a cost-effective cloud model for scalable, privacy-preserving record linkage (PPRL). PPRL techniques were adapted to maximise the quality of linkage and to automate as much of the process as possible. Finally, a prototype was created to demonstrate the capabilities and potential of the model.
 ResultsWe present our cloud model for PPRL, a platform for record linkage that provides rapid scaling of resources to meet demand, and the results of how our prototype performed on massive datasets.
 Conclusion/ImplicationsThe application of record linkage using relatively inexpensive cloud infrastructure represents a significant step towards providing an efficient and scalable record linkage service to researchers and government. Larger datasets can be linked efficiently, including national or cross-jurisdictional datasets, with little investment in private infrastructure, and improved turnaround times for researchers.

Privacy-preserving record linkage (PPRL) facilitates the matching of records that correspond to the same real-world entities across different databases while preserving the privacy of the individuals in these databases. A Bloom filter (BF) is a space efficient probabilistic data structure that is becoming popular in PPRL as an efficient privacy technique to encode sensitive information in records while still enabling approximate similarity computations between attribute values. However, BF encoding is susceptible to privacy attacks which can re-identify the values that are being encoded. In this paper we propose two novel techniques that can be applied on BF encoding to improve privacy against attacks. Our techniques use neighbouring bits in a BF to generate new bit values. An empirical study on large real databases shows that our techniques provide high security against privacy attacks, and achieve better similarity computation accuracy and linkage quality compared to other privacy improvements that can be applied on BF encoding.

ObjectiveTo introduce Linkja, an open-source, certified, Privacy-Preserving Record Linkage (PPRL) tool with the primary objective of facilitating secure and efficient record linkage across diverse datasets, while preserving individual privacy, providing a comprehensive and privacy-centric approach to program assessments. ApproachLinkja has been used to evaluate Chicago’s Flexible Housing Pool program by securely linking various datasets, including Homeless Management Information Systems (HMIS), Electronic Health Records, Jail records, Medical Examiner reports, and Public Health data. Data contributors used separate honest brokers for record linkage, and generation of salt and cryptography keys. ResultsLinkja enabled this multi-sector linkage by facilitating legal and data sharing agreements, avoided licensing fees and procurement processes for all parties, and provided a de-identified for analysis. The insights gleaned from the analysis motivated the FHP program to plan a follow-up evaluation that will add emergency serviced records from a separate City of Chicago department. The successful linkage provided a nuanced understanding of the association between stable housing and reduced healthcare utilization. ConclusionsLinkja's cryptographic certification ensures the highest standards of security during data linkage, safeguarding sensitive information. The matching rules, validated on hard-to-match populations, prove essential in maintaining accuracy and reliability in the linkage process. ImplicationsThe open-source nature of Linkja promotes transparency and collaboration in the development and enhancement of privacy-preserving record linkage techniques. This tool has broad implications for sectors dealing with sensitive data, fostering responsible data-sharing practices, and enabling secure collaborative research initiatives without compromising individual privacy.

The process of merging data of different data sources is referred to as record linkage. A medical environment with increased preconditions on privacy protection demands the transformation of clear-text attributes like first name or date of birth into one-way encrypted pseudonyms. When performing an automated or privacy preserving record linkage there might be the need of a binary classification deciding whether two records should be classified as the same entity. The classification is the final of the four main phases of the record linkage process: Preprocessing, indexing, matching and classification. The choice of binary classification techniques in dependence of project specifications in particular data quality has not extensively been studied yet. The aim of this work is the introduction and evaluation of an automatable semi-supervised binary classification system applied within the field of record linkage capable of competing or even surpassing advanced automated techniques of the domain of unsupervised classification. This work describes the rationale leading to the model and the final implementation of an automatable semi-supervised binary classification system and the comparison of its classification performance to an advanced active learning approach out of the domain of unsupervised learning. The performance of both systems has been measured on a broad variety of artificial test sets (n = 400), based on real patient data, with distinct and unique characteristics. While the classification performance for both methods measured as F-measure was relatively close on test sets with maximum defined data quality, 0.996 for semi-supervised classification, 0.993 for unsupervised classification, it incrementally diverged for test sets of worse data quality dropping to 0.964 for semi-supervised classification and 0.803 for unsupervised classification. Aside from supplying a viable model for semi-supervised classification for automated probabilistic record linkage, the tests conducted on a large amount of test sets suggest that semi-supervised techniques might generally be capable of outperforming unsupervised techniques especially on data with lower levels of data quality.

Record linkage from multiple, independent and heterogeneous databases is a major area of database research. However, little attention has been paid to the privacy concerns of colluding databases. Activities such as sharing data for scientific research, monitoring disease symptoms over a region or developing large e-commerce market place, all require large scale data integration among different organizations. Legitimate concerns over data security and privacy have caused organizations to be reluctant to collaborate in any data sharing protocol; thus a data integration protocol applicable for private databases is needed. Privacy-preserving Record Linkage (PRL) deals with the development of such a protocol. This paper presents the PRL key ideas and research challenges.

Privacy-preserving record linkage on large real world datasets

Nearly 90% of today’s data have been produced only in the last two years! These data come from a multitude of human activities, including social networking sites, mobile phone applications, electronic medical records systems, e-commerce sites, etc. Integrating and analyzing this wealth and volume of data offers remarkable opportunities in sectors that are of high interest to businesses, governments, and academia. Given that the majority of the data are proprietary and may contain personal or business sensitive information, Privacy-Preserving Record Linkage (PPRL) techniques are essential to perform data integration. In this paper, we review existing work in PPRL, focusing on the computational aspect of the proposed algorithms, which is crucial when dealing with Big data. We propose an analysis tool for the computational aspects of PPRL, and characterize existing PPRL techniques along five dimensions. Based on our analysis, we identify research gaps in current literature and promising directions for future work.

Privacy Preserving Record Linkage (PPRL) is the scientific field that explores methods of linking datasets in order to identify common entities efficiently and accurately by simultaneously preserving the privacy of the underlying data. In this paper we present a distributed Locality Sensitive Hashing-based framework for linking huge collections of records, by grouping similar records efficiently and by distributing computations among underutilized commodity hardware resources uniformly, without imposing an extra overhead on the existing infrastructure, thus promoting scalability. We also propose two methods of assessing computational cost, aiming to distribute workload evenly among compute nodes.