Comparing Record Linkage methods for real-world perinatal and neonatal data without unique identifiers

ABSTRACTObjectiveIn most European settings, record linkage across different institutions has to be based on personal identifiers such as names, birthday or place of birth. To protect the privacy of research subjects, the identifiers have to be encrypted. In practice, these identifiers show error rates up to 20% per identifier, therefore linking on encrypted identifiers usually implies the loss of large subsets of the databases. In many applications, this loss of cases is related to variables of interest for the subject matter of the study. Therefore, this kind of record-linkage will generate biased estimates.
 These problems gave rise to techniques of Privacy Preserving Record Linkage (PPRL). Many different PPRL techniques have been suggested within the last 10 years, very few of them are suitable for practical applications with large database containing millions of records as they are typical for administrative or medical databases. One proven technique for PPRL for large scale applications is PPRL based on Bloom filters.MethodUsing appropriate parameter settings, Bloom filter approaches show linkage results comparable to linkage based on unencrypted identifiers. Furthermore, this approach has been used in real-world settings with data sets containing up to 100 Million records. By the application of suitable blocking strategies, linking can be done in reasonable time.ResultHowever, Bloom filters have been subject of cryptographic attacks. Previous research has shown that the straight application of Bloom filters has a nonzero re-identification risk. We will present new results on recently developed techniques to defy all known attacks on PPRL Bloom filters. These computationally simple algorithms modify the identifiers by different cryptographic diffusion techniques. The presentation will demonstrate these new algorithms and show their performance concerning precision, recall and re-identification risk on large databases.

Nearly 90% of today’s data have been produced only in the last two years! These data come from a multitude of human activities, including social networking sites, mobile phone applications, electronic medical records systems, e-commerce sites, etc. Integrating and analyzing this wealth and volume of data offers remarkable opportunities in sectors that are of high interest to businesses, governments, and academia. Given that the majority of the data are proprietary and may contain personal or business sensitive information, Privacy-Preserving Record Linkage (PPRL) techniques are essential to perform data integration. In this paper, we review existing work in PPRL, focusing on the computational aspect of the proposed algorithms, which is crucial when dealing with Big data. We propose an analysis tool for the computational aspects of PPRL, and characterize existing PPRL techniques along five dimensions. Based on our analysis, we identify research gaps in current literature and promising directions for future work.

IntroductionBusinesses worldwide are increasingly adopting the storage, compute and analytical services provided by cloud computing. Yet, few operational linkage units are keeping pace with this world of technological change - most use legacy systems approaching their limits with the rapidly increasing size and range of datasets now required for linkage.
 Objectives and ApproachTo meet the demands of linkage for the near future, it is important that new solutions for linkage consider the services provided by public cloud infrastructure for compute, storage and analytics. We examined Platform as a Service (PaaS) offerings for use in the development of a cost-effective cloud model for scalable, privacy-preserving record linkage (PPRL). PPRL techniques were adapted to maximise the quality of linkage and to automate as much of the process as possible. Finally, a prototype was created to demonstrate the capabilities and potential of the model.
 ResultsWe present our cloud model for PPRL, a platform for record linkage that provides rapid scaling of resources to meet demand, and the results of how our prototype performed on massive datasets.
 Conclusion/ImplicationsThe application of record linkage using relatively inexpensive cloud infrastructure represents a significant step towards providing an efficient and scalable record linkage service to researchers and government. Larger datasets can be linked efficiently, including national or cross-jurisdictional datasets, with little investment in private infrastructure, and improved turnaround times for researchers.

Linking information across databases fosters new research in the medical sciences. Recent European privacy regulations recommend encrypting personal identifiers used for linking. Bloom filter based methods are an increasingly popular Record Linkage method. However, basic Bloom filter encodings are prone to cryptographic attacks. Therefore, hardening methods against these attacks are required. In this paper, a new method for such a hardening method for Privacy-preserving Record Linkage (PPRL) technique is presented. By using a Markov chain-based language model of bigrams of identifiers during the encryption, protection against attacks is increased. Based on real-world mortality data, we compare unencrypted and state of the art PPRL methods with the results of the proposed hardening method.

ABSTRACT
 ObjectivesWhile record linkage has become a strategic research priority within Australia and internationally, legal and administrative issues prevent data linkage in some situations due to privacy concerns. Even current best practices in record linkage carry some privacy risk as they require the release of personally identifying information to trusted third parties. Application of record linkage systems that do not require the release of personal information can overcome legal and privacy issues surrounding data integration. Current conceptual and experimental privacy-preserving record linkage (PPRL) models show promise in addressing data integration challenges but do not yet address all of the requirements for real-world operations. This paper aims to identify and address some of the challenges of operationalising PPRL frameworks.
 ApproachTraditional linkage processes involve comparing personally identifying information (name, address, date of birth) on pairs of records to determine whether the records belong to the same person. Designing appropriate linkage strategies is an important part of the process. These are typically based on the analysis of data attributes (metadata) such as data completeness, consistency, constancy and field discriminating power. Under a PPRL model, however, these factors cannot be discerned from the encrypted data, so an alternative approach is required. This paper explores methods for data profiling, blocking, weight/threshold estimation and error detection within a PPRL framework.
 ResultsProbabilistic record linkage typically involves the estimation of weights and thresholds to optimise the linkage and ensure highly accurate results. The paper outlines the metadata requirements and automated methods necessary to collect data without compromising privacy. We present work undertaken to develop parameter estimation methods which can help optimise a linkage strategy without the release of personally identifiable information. These are required in all parts of the privacy preserving record linkage process (pre-processing, standardising activities, linkage, grouping and extracting).
 ConclusionsPPRL techniques that operate on encrypted data have the potential for large-scale record linkage, performing both accurately and efficiently under experimental conditions. Our research has advanced the current state of PPRL with a framework for secure record linkage that can be implemented to improve and expand linkage service delivery while protecting an individual’s privacy. However, more research is required to supplement this technique with additional elements to ensure the end-to-end method is practical and can be incorporated into real-world models.

Novel record linkage (RL) methods have the potential to enhance clinical informatics by integrating patient data from multiple sources-including electronic health records, insurance claims, and digital health devices-to inform patient-centered care. Engaging patients and other stakeholders in the use of RL methods in patient-centered outcomes research (PCOR) is a key step in ensuring RL methods are viewed as acceptable, appropriate, and useful. The University of Colorado Record Linkage (CURL) platform empowers the use of RL in PCOR. This study aimed to describe the process of engaging patients and other stakeholders in the design of an RL dissemination package to support the use of RL methods in PCOR. Customer discovery, value proposition design, and user experience methods were used to iteratively develop an RL dissemination package that includes animated explainer videos for patients and an RL research planning workbook for researchers. Patients and other stakeholders (researchers, data managers, and regulatory officials) were engaged in the RL dissemination package design. Patient partners emphasized the importance of conveying how RL methods may benefit patients and the rules researchers must follow to protect the privacy and security of patient data. Other stakeholders described accuracy, flexibility, efficiency, and data security compared with other available RL solutions. Dissemination package communication products reflect the value propositions identified by key stakeholders. As prioritized by patients, the animated explainer videos emphasize the data privacy and security processes and procedures employed when performing research using RL. The RL workbook addresses researchers' and data managers' needs to iteratively design RL projects and provides accompanying resources to alleviate leadership and regulatory officials' concerns about data regulation compliance. Dissemination products to promote adoption and use of CURL include materials to facilitate patient engagement in RL research and investigator step-by-step decision-making materials about the integration of RL methods in PCOR.

ObjectiveTo introduce Linkja, an open-source, certified, Privacy-Preserving Record Linkage (PPRL) tool with the primary objective of facilitating secure and efficient record linkage across diverse datasets, while preserving individual privacy, providing a comprehensive and privacy-centric approach to program assessments. ApproachLinkja has been used to evaluate Chicago’s Flexible Housing Pool program by securely linking various datasets, including Homeless Management Information Systems (HMIS), Electronic Health Records, Jail records, Medical Examiner reports, and Public Health data. Data contributors used separate honest brokers for record linkage, and generation of salt and cryptography keys. ResultsLinkja enabled this multi-sector linkage by facilitating legal and data sharing agreements, avoided licensing fees and procurement processes for all parties, and provided a de-identified for analysis. The insights gleaned from the analysis motivated the FHP program to plan a follow-up evaluation that will add emergency serviced records from a separate City of Chicago department. The successful linkage provided a nuanced understanding of the association between stable housing and reduced healthcare utilization. ConclusionsLinkja's cryptographic certification ensures the highest standards of security during data linkage, safeguarding sensitive information. The matching rules, validated on hard-to-match populations, prove essential in maintaining accuracy and reliability in the linkage process. ImplicationsThe open-source nature of Linkja promotes transparency and collaboration in the development and enhancement of privacy-preserving record linkage techniques. This tool has broad implications for sectors dealing with sensitive data, fostering responsible data-sharing practices, and enabling secure collaborative research initiatives without compromising individual privacy.

Privacy-preserving record linkage on large real world datasets

Record linkage aims to identify records referring to the same individual across different datasets, but often involves sensitive personal identifiers. Privacy-Preserving Record Linkage (PPRL) techniques, particularly those based on Bloom filter encoding, enable secure linkage while protecting privacy. However, existing methods suffer from information loss and lack rigorous privacy guarantees. To address these limitations, we propose a novel (Hash)-A method that retains position and frequency information of q-grams, enhancing matching accuracy. We also introduce the Utility-optimized Bloom Filter (UBF), a privacy-preserving mechanism grounded in User-Level Local Differential Privacy (ULDP), which offers formal and quantifiable privacy guarantees. These components are integrated into a complete three-party PPRL protocol that supports missing value handling. Experimental results on real-world datasets show that our method achieves improved linkage accuracy and robust privacy protection against inference attacks, demonstrating a practical balance between utility and privacy.

Matching records that refer to the same entity across data-bases is becoming an increasingly important part of many data mining projects, as often data from multiple sources needs to be matched in order to enrich data or improve its quality. Significant advances in record linkage techniques have been made in recent years. However, many new techniques are either implemented in research proof-of-concept systems only, or they are hidden within expensive 'black box' commercial software. This makes it difficult for both researchers and practitioners to experiment with new record linkage techniques, and to compare existing techniques with new ones. The Febrl (Freely Extensible Biomedical Record Linkage) system aims to fill this gap. It contains many recently developed techniques for data cleaning, deduplication and record linkage, and encapsulates them into a graphical user interface (GUI). Febrl thus allows even inexperienced users to learn and experiment with both traditional and new record linkage techniques. Because Febrl is written in Python and its source code is available, it is fairly easy to integrate new record linkage techniques into it. Therefore, Febrl can be seen as a tool that allows researchers to compare various existing record linkage techniques with their own ones, enabling the record linkage research community to conduct their work more efficiently. Additionally, Febrl is suitable as a training tool for new record linkage users, and it can also be used for practical linkage projects with data sets that contain up to several hundred thousand records.

Record linkage represents the process of identifying records that refer to the same real-world entity across multiple sources. In the recent past, the large explosion of data from organizations and individuals has created critical challenges in record linkage process, leading the scientific community to develop a rich series of techniques. Among these recent developments, the design of record linkage solutions for medical data is a cornerstone for health-care systems, as personal identifying information, such as name and date of birth, is often used for linkage. In this setting, record linkage solutions are expected to preserve individual patients’ privacy in addition to be effective and efficient. In this chapter, we provide a broad review of the recent works in privacy preserving record linkage (PPRL). From a privacy-centric perspective, we summarize a comprehensive framework of the PPRL process and describe the privacy assurance techniques adopted for each step in the framework. With the applications in biomedical domain in mind, we identify several challenges for PPRL and discuss future research directions.

The authors present a method for estimating the number of persons (as opposed to the number of admissions) admitted to hospital for a defined time period for specific diagnoses when unique personal identifiers are not available. A simple computer algorithm (algorithm 1), employing a family-based scrambled health insurance number in conjunction with diagnosis, sex, and age was used to estimate the rates of readmission to hospital specific diagnoses. A proportion of the hospital morbidity records used in the study did not contain a health insurance number; therefore, estimated readmission ratios for specific diagnoses, calculated from admissions with health insurance numbers, were applied to all admissions for those diagnoses to obtain the estimates of total persons admitted to hospital in the given time period. The validity of the methodology was tested for selected diagnoses for a specific year by 1) comparing estimates of persons obtained by applying algorithm 1 with manual counts of persons obtained through examination of printed lists of all Ontario admissions to hospital for the specific diagnoses, and 2) comparing actual counts of persons admitted to a specific hospital with computer estimates of persons admitted to that hospital. The conclusions drawn were that the method, using algorithm 1, is a valid one for obtaining estimates of the number of persons admitted to hospital in a given year with a specific diagnosis and that the information derived is potentially useful for conducting epidemiologic studies. Readmission ratios were also estimated through the use of a second algorithm (algorithm 2) which was not dependent on a scrambled health insurance number. Sensitivity and specificity of these ratios were found to be low when compared with ratios estimated by algorithm 1.

Research driven by real-world clinical data is increasingly vital to enabling learning health systems, but integrating such data from across disparate health systems is challenging. As part of the NCATS National COVID Cohort Collaborative (N3C), the N3C Data Enclave was established as a centralized repository of deidentified and harmonized COVID-19 patient data from institutions across the US. However, making this data most useful for research requires linking it with information such as mortality data, images, and viral variants. The objective of this project was to establish privacy-preserving record linkage (PPRL) methods to ensure that patient-level EHR data remains secure and private when governance-approved linkages with other datasets occur. Separate agreements and approval processes govern N3C data contribution and data access. The Linkage Honest Broker (LHB), an independent neutral party (the Regenstrief Institute), ensures data linkages are robust and secure by adding an extra layer of separation between protected health information and clinical data. The LHB's PPRL methods (including algorithms, processes, and governance) match patient records using "deidentified tokens," which are hashed combinations of identifier fields that define a match across data repositories without using patients' clear-text identifiers. These methods enable three linkage functions: Deduplication, Linking Multiple Datasets, and Cohort Discovery. To date, two external repositories have been cross-linked. As of March 1, 2023, 43 sites have signed the LHB Agreement; 35 sites have sent tokens generated for 9 528 998 patients. In this initial cohort, the LHB identified 135 037 matches and 68 596 duplicates. This large-scale linkage study using deidentified datasets of varying characteristics established secure methods for protecting the privacy of N3C patient data when linked for research purposes. This technology has potential for use with registries for other diseases and conditions.

Record linkage is an emerging research area which is required by various real-world applications to identify which records in different data sources refer to the same real-world entities. Often privacy concerns and restrictions prevent the use of traditional record linkage applications across different organizations. Linking records in situations where no private or confidential information can be revealed is known as privacy-preserving record linkage (PPRL). As with traditional record linkage applications, scalability is a main challenge in PPRL. This challenge is generally addressed by employing a blocking technique that aims to reduce the number of candidate record pairs by removing record pairs that likely refer to non-matches without comparing them in detail. This paper presents an efficient private blocking technique based on a sorted neighborhood approach that combines k-anonymous clustering and the use of public reference values. An empirical study conducted on real-world databases shows that this approach is scalable to large databases, and that it can provide effective blocking while preserving k-anonymous characteristics. The proposed approach can be up-to two orders of magnitude faster than two state-of-the-art private blocking techniques, k-nearest neighbor clustering and Hamming based locality sensitive hashing.

The process of merging data of different data sources is referred to as record linkage. A medical environment with increased preconditions on privacy protection demands the transformation of clear-text attributes like first name or date of birth into one-way encrypted pseudonyms. When performing an automated or privacy preserving record linkage there might be the need of a binary classification deciding whether two records should be classified as the same entity. The classification is the final of the four main phases of the record linkage process: Preprocessing, indexing, matching and classification. The choice of binary classification techniques in dependence of project specifications in particular data quality has not extensively been studied yet. The aim of this work is the introduction and evaluation of an automatable semi-supervised binary classification system applied within the field of record linkage capable of competing or even surpassing advanced automated techniques of the domain of unsupervised classification. This work describes the rationale leading to the model and the final implementation of an automatable semi-supervised binary classification system and the comparison of its classification performance to an advanced active learning approach out of the domain of unsupervised learning. The performance of both systems has been measured on a broad variety of artificial test sets (n = 400), based on real patient data, with distinct and unique characteristics. While the classification performance for both methods measured as F-measure was relatively close on test sets with maximum defined data quality, 0.996 for semi-supervised classification, 0.993 for unsupervised classification, it incrementally diverged for test sets of worse data quality dropping to 0.964 for semi-supervised classification and 0.803 for unsupervised classification. Aside from supplying a viable model for semi-supervised classification for automated probabilistic record linkage, the tests conducted on a large amount of test sets suggest that semi-supervised techniques might generally be capable of outperforming unsupervised techniques especially on data with lower levels of data quality.