Aggregate queries on probabilistic record linkages

This thesis presents a methodology of data preparation with probabilistic record linkage and information fusion for improving and enriching information visualizations of biomedical citation data. The problem of record linkage of citation databases where only non-unique identifiers such as author names and document titles are available as common identifiers to be linked was investigated. This problem in citation data parallels problems in clinical data and Knowledge Discovery in Databases (KDD) methods from clinical data mining are evaluated. Probabilistic and deterministic (exact-match) record linkage models were developed and compared through the use of a gold standard or truth dataset. Empirical comparison with ROC analysis of record linkage models showed a significant difference (p=.000) in performance of a probabilistic model over deterministic models. The methodology was evaluated with probabilistic linkage of records from the Web of Science, Medline, and CINAHL citation databases in the knowledge domains of medical informatics, HIV/AIDS, and nursing informatics. Data quality metrics for datasets prepared with probabilistic record linkage and information fusion showed improvement in completeness of key variables and reduction in sample bias. The resulting visualizations offered a richer information space for users through an increase in terms entering the visualization. The significant contributions of this work include the development of a novel model of probabilistic record linkage for biomedical citation databases which improves upon existing deterministic models. In addition a methodology for improving and enriching knowledge domain visualizations though a data preparation approach has been validated with analyses of multiple citation databases and knowledge domains. The data preparation methodology of probabilistic record linkage with information fusion offers a remedy for data quality problems, and the opportunity to enrich visualizations with added content for user exploration, which in turn improves the utility of knowledge domain visualizations as a medium for assessing available evidence and forming hypotheses.

Record linkage (RL) is the process of identifying pairs of records that correspond to the same entity, for example the same patient. The basic approach assigns to each pair of records a similarity weight, and then determines a certain threshold, above which the two records are considered to be a match. Three different RL methods were applied under privacy-preserving conditions on hospital admission data: deterministic RL (DRL), probabilistic RL (PRL), and Bloom filters. The patient characteristics like names were one-way encrypted (DRL, PRL) or transformed to a cryptographic longterm key (Bloom filters). Based on one year of hospital admissions, the data set was split randomly in 30 thousand new and 1,5 million known patients. With the combination of the three RL-methods, a positive predictive value of 83 % (95 %-confidence interval 65 %-94 %) was attained. Thus, the application of the presented combination of RL-methods seem to be suited for other applications of population-based research.

Data integration has become one of the main challenges in the era of Big Data analytics. Often to enable decision-making, data from different sources have to be integrated and linked together. For example, multi-source data integration is vital to police, counter terrorism and national security to allow efficient and accurate verification of people. One of the key challenges in the data integration process is matching records that represent the same real-world entity (e.g. person). This process is referred to as record linkage. In many cases, data sets do not share a unique identifier (e.g. National Insurance Number), hence records need to be matched by comparing their corresponding attributes. Most of the existing record linkage methods require assistance from a domain expert for handcrafting domain-specific linking rules. More automatic approaches, based on using machine learning, were also proposed. However, those approaches relay on having a substantial set of manually labelled records, which makes them inapplicable in real-world scenarios. Given the importance of the problem, record linkage has witnessed a strong interest in the past decade. As a result, significant progress has been made in this area. In particular, the problem of reducing the manual effort and the amount of labelled data required for constructing record linkage models has been addressed in many studies. In this chapter, we review the most recently proposed approaches to semi-supervised and unsupervised record linkage.

Record linkage is an essential task in data integration in the fields of healthcare, law enforcement, fraud detection, transportation, biology, and supply chain management. The problem of record linkage is to cluster records from various sources such that each cluster belongs to a single entity. Scalability in record linking is limited by the large number of pairwise comparisons required. Blocking addresses this challenge by partitioning data into smaller parts, substantially reducing the computational cost. With the advancement of Large Language Models (LLMs), there are several possibilities to improve record linkage by leveraging their semantic understanding of textual attributes. LLM-based record linkage algorithms in the literature have very large runtimes. In this paper, we show that the employment of blocking can result in significant improvements not only in the runtime but also in the accuracy. Specifically, we propose a record linkage algorithm that combines LLMs with blocking. Experimental evaluation demonstrates that our algorithm achieves lower runtimes while simultaneously improving F1 scores compared to the approaches relying solely on LLMs. These findings demonstrate the importance of blocking even in the era of advanced machine learning models.

The process of merging data of different data sources is referred to as record linkage. A medical environment with increased preconditions on privacy protection demands the transformation of clear-text attributes like first name or date of birth into one-way encrypted pseudonyms. When performing an automated or privacy preserving record linkage there might be the need of a binary classification deciding whether two records should be classified as the same entity. The classification is the final of the four main phases of the record linkage process: Preprocessing, indexing, matching and classification. The choice of binary classification techniques in dependence of project specifications in particular data quality has not extensively been studied yet. The aim of this work is the introduction and evaluation of an automatable semi-supervised binary classification system applied within the field of record linkage capable of competing or even surpassing advanced automated techniques of the domain of unsupervised classification. This work describes the rationale leading to the model and the final implementation of an automatable semi-supervised binary classification system and the comparison of its classification performance to an advanced active learning approach out of the domain of unsupervised learning. The performance of both systems has been measured on a broad variety of artificial test sets (n = 400), based on real patient data, with distinct and unique characteristics. While the classification performance for both methods measured as F-measure was relatively close on test sets with maximum defined data quality, 0.996 for semi-supervised classification, 0.993 for unsupervised classification, it incrementally diverged for test sets of worse data quality dropping to 0.964 for semi-supervised classification and 0.803 for unsupervised classification. Aside from supplying a viable model for semi-supervised classification for automated probabilistic record linkage, the tests conducted on a large amount of test sets suggest that semi-supervised techniques might generally be capable of outperforming unsupervised techniques especially on data with lower levels of data quality.

A Conceptual Algorithm to Link Police and Hospital Records Based on Occurrence of Values

Entity identification deals with matching records from different datasets or within one dataset that represent the same real-world entity when unique identifiers are not available. Enabling data integration at record level as well as the detection of duplicates, entity identification plays a major role in data preprocessing, especially concerning data quality. This paper presents a framework for statistical entity identification in particular focusing on probabilistic record linkage and string matching and its implementation in R. According to the stages of the entity identification process, the framework is structured into seven core components: data preparation, candidate selection, comparison, scoring, classification, decision, and evaluation. Samples of real-world CRM datasets serve as illustrative examples.

Results from simulated data sets: probabilistic record linkage outperforms deterministic record linkage

To describe an efficient, generalizable approach to validate probabilistic record linkage results, in particular by a model-guided detection of linking errors, and to apply this approach to validate linkage of admissions of newborns. Our double-blind validation procedure consisted of three steps: sample selection, data collection and data analysis. The linked Dutch national newborn admission registry contained 30,082 records for 2001 including readmissions (7.4%) and twins (9.7%). A highly informative sample was selected from the linked file by oversampling uncertain links based on model-derived linking weight. Four hundred and eight fax forms with minimal registry information (admissions of 191 children) were sent out to different pediatric units. The pediatricians were asked to create a short detailed patient history from independent sources. The linkage status and additional record data was validated against this external information. Response rate was 97% (395/408 faxes). Accuracy of the linkage of singleton admissions was high: except for some expected errors in the uncertain area (0.02% of record pairs), linkage was error-free. Validation of multiple birth readmissions showed 37% linkage errors due to low data quality of the multiple birth variables. The quality of the linked registry file was still high; only 1.7% of the children were from a multiple birth with multiple admissions, resulting in less than 1% linking error. Our external validation procedure of record linkage was feasible, efficient, and informative about identifying the source of the errors.

IntroductionThe importance of Indigenous data sovereignty and Indigenous-led research processes is increasingly being recognized in Canada and internationally. For First Nations in Ontario, Canada, access to routinely-collected demographic and health systems data is critical to planning and measuring health status and outcomes in their populations. Linkage of this data with the Indian Register (IR), under First Nations data governance, has unlocked data for use by First Nations organizations and communities. ObjectivesTo describe the linkage of the IR database to the Ontario Registered Persons Database (RPDB) within the context of Indigenous data sovereignty principles.MethodsDeterministic and probabilistic record linkage methods were used to link the IR to the RPDB. There is no established population of First Nations people living in Ontario with which we could establish a linkage rate. Accordingly, several approaches were taken to determine a denominator that would represent the total population of First Nations we would hope to link to the RPDB. ResultsOverall, 201,678 individuals in the national IR database matched to Ontario health records by way of the RPDB, of which 98,562 were female and 103,116 were male. Of those First Nations individuals linked to the RPDB, 90.2% (n=181,915) lived in Ontario when they first registered with IR, or were affiliated with an Ontario First Nation Community. The proportion of registered First Nations people linking to the RPDB improved across time, from 62.8% in the 1960s to 94.5% in 2012.ConclusionThis linkage of the IR and RPDB has resulted in the creation of the largest First Nations health research study cohort in Canada. The linked data are being used by First Nations communities to answer questions that ultimately promote wellbeing, effective policy, and healing

To evaluate the impact on tuberculosis (TB) incidence rates of removal of improper duplicate records from the notification system. Data from the Sistema de Informação de Agravos de Notificação (Brazilian Information System for Tuberculosis Notification) from 2000 to 2004 were analyzed. Repeat records were identified through probabilistic record linkage and classified into six mutually exclusive categories and then kept, combined or removed from database. Of all TB records, 73.7% had no duplicate, 18.9% were duplicate, 4.7% were triplicate, and 2.7% were quadruplicate or more. Of all repeat records, 47.3% were classified as transfer in/out; 23.6% return after default, 16.4% true duplicates, 10% relapse, 2.5% inconclusive and 0.2% had missing data. These proportions were different in Brazilian states. Removal of improper duplicate records reduced TB incidence rate per 100.000 inhabitants by 6.1% in the year 2000 (from 44 to 41.3), 8.3% in 2001 (from 44.5 to 40.8), 9.4% in 2002 (from 45.8 to 41.5), 9.2% in 2003 (from 46.9 to 42.6) and 8.4% in 2004 (from 45.4 to 41.6). The study results indicate that the observed tuberculosis incidence rates represent estimates that would be closer to the actual rates than those obtained from the raw database at state and country level. The use of record linkage approach should be promoted for better quality of notification system data.

Extending the Fellegi-Sunter record linkage model for mixed-type data with application to the French national health data system

IntroductionResearch data combined with administrative data provides a robust resource capable of answering unique research questions. However, in cases where personal health data are encrypted, due to ethics requirements or institutional restrictions, traditional methods of deterministic and probabilistic record linkages are not feasible. Instead, privacy-preserving record linkages must be used to protect patients’ personal data during data linkage.ObjectivesTo determine the feasibility and validity of a deterministic privacy preserving data linkage protocol using homomorphically encrypted data.MethodsFeasibility was measured by the number of records that successfully matched via direct identifiers. Validity was measured by the number of records that matched with multiple indirect identifiers. The threshold for feasibility and validity were both set at 95%. The datasets shared a single, direct identifier (health card number) and multiple indirect identifiers (sex and date of birth). Direct identifiers were encrypted in both datasets and then transferred to a third-party server capable of linking the encrypted identifiers without decrypting individual records. Once linked, the study team used indirect identifiers to verify the accuracy of the linkage in the final dataset.ResultsWith a combination of manual and automated data transfer in a sample of 8,128 individuals, the privacy-preserving data linkage took 36 days to match to a population sample of over 3.2 million records. 99.9% of the records were successfully matched with direct identifiers, and 99.8% successfully matched with multiple indirect identifiers. We deemed the linkage both feasible and valid.ConclusionsAs combining administrative and research data becomes increasingly common, it is imperative to understand options for linking data when direct linkage is not feasible. The current linkage process ensured the privacy and security of patient data and improved data quality. While the initial implementations required significant computational and human resources, increased automation keeps the requirements within feasible bounds.

IntroductionHepatitis C virus (HCV) infection is a major health problem associated with considerable risk of mortality in different regions of the world. The purpose of this study was to investigate the contribution of HCV infection on all-cause and liver-related mortality, in a large cohort of blood donors in Brazil.MethodsThis is a retrospective cohort study of blood donors from 1994 to 2013, at Fundação Pró-Sangue—Hemocentro de São Paulo (FPS). This cohort included 2,892 and 5,784 HCV antibody seropositive and seronegative donors, respectively. Records from the FPS database and the Mortality Information System (SIM: a national database in Brazil) were linked through a probabilistic record linkage (RL). Mortality outcomes were defined based on ICD-10 (10th International Statistical Classification of Diseases and Related Health Problems) codes listed as the cause of death on the death certificate. Hazard ratios (HRs) were estimated for outcomes using Cox multiple regression models.ResultsWhen all causes of death were considered, RL identified 209 deaths (7.2%) among seropositive blood donors and 190 (3.3%) among seronegative blood donors. Donors seropositive for HCV infection had a 2.5 times higher risk of death due to all causes (95% CI: 1.76–2.62; p<0.001). When only liver-related causes of death were considered, RL identified 73 deaths among seropositive blood donors and only 6 among seronegative blood donors. Donors seropositive for HCV infection had a 23.4 times higher risk of death due to liver related causes (95% CI: 10.2–53.9; p<0.001). Donors seropositive for HCV had a 29.5 (95%CI: 3.9–221.7), 2.8 (95% CI: 1.4–5.5) and a 1.9 (95% CI: 1.2–3.0) times higher risk of death due to hepatocellular carcinoma, infection or trauma, respectively, compared to seronegative donors.ConclusionsAll-cause and liver-related mortality rate was increased among blood donors seropositive for HCV compared with the mortality rate among seronegative blood donors. Our data confirms HCV as a relevant cause of death in Brazil and also suggest that interventions directed at following patients even after access to specific drug treatment are urgent and necessary.

Record (or entity) matching or linkage is the process of identifying records in one or more data sources, that refer to the same real world entity or object. In record linkage, the ultimate goal of a decision model is to provide the decision maker with a tool for making decisions upon the actual matching status of a pair of records (i.e., documents, events, persons, cases, etc.). Existing models of record linkage rely on decision rules that minimize the probability of subjecting a case to clerical review, conditional on the probabilities of erroneous matches and erroneous non-matches. In practice though, (a) the value of an erroneous match is, in many applications, quite different from the value of an erroneous non-match, and (b) the cost and the probability of a misclassification, which is associated with the clerical review, is ignored in this way. In this paper, we present a decision model which is optimal, based on the cost of the record linkage operation, and general enough to accommodate multi-class or multi-decision case studies. We also present an example along with the results from applying the proposed model to large comparison spaces.