Efficient apache spark-based approach for a probabilistic entity resolution

Entity resolution is the process of identifying the records that refer to the same entity. Blocking is the most important step in the entity resolution process, which allows optimizing the matching step by grouping the records into blocks in a way that only the records in the same block will be compared to each other. However, the blocking process can’t be done with certainty and confidence, such that records may be assigned to the wrong blocks causing the ignorance of many necessary comparisons. In this paper, we propose a Fuzzy/Probabilistic approach to overcome the uncertainty problem. Our approach defines two steps: Fuzzy Block Generator and Probabilistic filtering which aim to minimize the effect of uncertainty by assigning records to multiple possible blocks, and to reduce the number of unnecessary comparisons based on a probabilistic approach, respectively. Conducted experiments showcased the effectiveness and efficiency of our solution.

Entity Resolution (ER) is increasingly being used to identify and link names across archival collections. We describe a framework for unlocking and linking biographical data from WWII Japanese American Incarceration Camps using Entity Resolution and other computational approaches. We demonstrate the construction of social graphs that link people, places, and events and which support further scholarship and reveal hidden stories in historical events, especially given contested archival sources. Finally, we show the power of computational analysis to recreate event networks and represent movement of people using maps. This type of modeling is captured through interactive Jupyter Notebooks that integrate these various elements and document our interpretation of Japanese American experiences and events at the Tule Lake concentration camp.

This paper proposes a Deep Neural Networks (DNN) based approach for entity resolution in databases. This approach is mainly based on a record linkage process which aims to detect records that refer to the same entity. First, record pairs are represented by their word embedding using an N-gram embedding based method. Then, they are classified into matching or unmatching pairs using a DNN model. Three DNN architectures: Multi-Layer Perceptron, Long Short Term Memory networks and Convolutional Neural Networks are investigated and compared for this purpose. The approach is experimented on two databases. The results exceed \(97\%\) for recall and \(96\%\) for precision. The comparison with similarity measure and classical classifier based approaches shows a significant improvement in the results on the two databases.

The real-time precise orbit is an essential prerequisite for a real-time precise point positioning service. Focusing on the impact of ambiguity resolution on real-time orbital precision, we propose an efficient approach for real-time ambiguity resolution, which consists of two modules. The first module resolves the double-differenced ambiguities according to their proximity to the nearest integer. The second module sequentially adds the resolved integer ambiguity constraints to the square-root information filter process, after confirming that the same constraints have not been imposed before the current epoch. To validate our method, GPS data collected from 100 globally distributed stations are used to simulate a real-time precise orbit determination. The convergence performance is analyzed, and the accuracy of the orbit is evaluated. The results show that: (1) Almost 90% of the double-differenced ambiguities are fixed correctly in real time for baselines shorter than 1000 km. (2) After application of the proposed approach, the root-mean-square errors of all the satellite orbits are reduced from (5.9, 3.4, 2.3) cm to (4.7, 2.6, 2.2) cm for the along-track, cross-track and radial directions, respectively, with improvements of about 22% for the along-track and cross-track directions, and 6% for the radial direction. (3) Simulated real-time orbits determined with this method can obtain almost the same accuracy as some ultra-rapid products and, particularly, better accuracy can be achieved for eclipsing satellites.

Entity Resolution (ER) is one of the techniques used to disambiguate the various manifestations of same object to improve search results in databases. Recently, Crowd sourcing has been utilized to improve entity resolution to gain positive impact when searching for particular information in a database. In this paper, we consider the domain of Customer Relationship Management (CRM) and utilize the approach of Crowd sourcing to enrich the process of achieving ER. Specifically our focus is to identify the right customer that has been manifested in various ways under a common name in a database using In-house Crowd sourcing-based Entity Resolution approach (ICER). The ICER takes the list of possible duplicates into consideration (which are pre-determined) and identifies the pair of record that has the maximum impact in achieving ER. Then, this pair is crowd sourced to Customer Service Representatives (CSRs) to have their input (labeling). ICER incorporates the principles of Human Intelligence Task (HIT) that aims to keep the questions asked to the CSR to a minimum. Two ICER approaches are proposed in this study based on probabilistic (modified approach of Whang et al) and active learning schemas. The applicability of the proposed ICER approaches and comparison of their results have been highlighted by using an example.

In this paper, we propose an efficient conflict-resolution approach based on the multi-disk architecture for the insertion/deletion operations on continuous media that are split up into blocks and placed in various locations on the disk, without reorganizing the whole data. When a new subobject is inserted after subobject i, it will be assigned with an identification number (i + 1) and be inserted into a disk in which the retrieval of the new subobject does not conflict with the retrieval of any other subobject, where a conflict means a pair of two consecutive subobjects that are stored in the same disk have to be retrieved simultaneously. However, a new conflict on the same disk may occur since all the identification numbers of subobjects after subobject i are increased by one. Only when such a new conflict occurs, one movement operation is required, so does the case of a deletion operation. Moreover, to reduce those additional movement cost, a deferring approach is proposed at the cost of an additional buffer. In this approach, n data insertions are deferred and stored in a buffer. Then, the system starts to insert those data after an optimal insertion sequence is determined. Based on this approach, two strategies are proposed: the conflict-resolved-first-deferring strategy (the CRFD strategy) and the conflict-resolved-last-deferring strategy (the CRLD strategy). From our performance analysis, we will prove that the CRLD strategy with an insertion sequence according to the ascending order of identification numbers is an optimal strategy based on the proposed deferring approach.

Differential expression analysis of RNA sequencing (RNA-seq) data typically relies on reconstructing transcripts or counting reads that overlap known gene structures. We previously introduced an intermediate statistical approach called differentially expressed region (DER) finder that seeks to identify contiguous regions of the genome showing differential expression signal at single base resolution without relying on existing annotation or potentially inaccurate transcript assembly.We present the derfinder software that improves our annotation-agnostic approach to RNA-seq analysis by: (i) implementing a computationally efficient bump-hunting approach to identify DERs that permits genome-scale analyses in a large number of samples, (ii) introducing a flexible statistical modeling framework, including multi-group and time-course analyses and (iii) introducing a new set of data visualizations for expressed region analysis. We apply this approach to public RNA-seq data from the Genotype-Tissue Expression (GTEx) project and BrainSpan project to show that derfinder permits the analysis of hundreds of samples at base resolution in R, identifies expression outside of known gene boundaries and can be used to visualize expressed regions at base-resolution. In simulations, our base resolution approaches enable discovery in the presence of incomplete annotation and is nearly as powerful as feature-level methods when the annotation is complete.derfinder analysis using expressed region-level and single base-level approaches provides a compromise between full transcript reconstruction and feature-level analysis. The package is available from Bioconductor at www.bioconductor.org/packages/derfinder.

Cloud and edge-computing based platforms have enabled rapid development of distributed business process (BP) applications in a plug and play manner. However, these platforms do not provide the needed capabilities for identifying or repairing faults in BPs. Faults in BP may occur due to errors made by BP designers because of their lack of understanding of the underlying component services, misconfiguration of these services, or incorrect/incomplete BP workflow specifications. Such faults may not be discovered at design or development stage and may occur at runtime. In this paper, we present a unified framework for automated fault resolution in BPs. The proposed framework employs a novel and efficient fault resolution approach that extends the generate-and-validate program repair approach. In addition, we propose a hybrid approach that performs fault resolution by analyzing a faulty BP in isolation as well as by comparing with other BPs using similar services. This hybrid approach results in improved accuracy and broader coverage of fault types. We also perform an extensive experimental evaluation to compare the effectiveness of the proposed approach using a dataset of 208 faulty BPs.

Human trafficking is a challenging law enforcement problem, and traces of victims of such activity manifest as ‘escort advertisements’ on various online forums. Given the large, heterogeneous and noisy structure of this data, building models to predict instances of trafficking is a convoluted task. In this paper we propose an entity resolution pipeline using a notion of proxy labels, in order to extract clusters from this data with prior history of human trafficking activity. We apply this pipeline to 5M records from backpage.com and report on the performance of this approach, challenges in terms of scalability, and some significant domain specific characteristics of our resolved entities.

The historical record of the trans-Atlantic slave trade remains profoundly incomplete, particularly regarding the lives of African women whose identities were disproportionately erased, anonymized, or collapsed into generic labor categories. Existing archival collections including ship manifests, plantation inventories, baptismal rolls, court testimonies, sale ledgers, and manumission documents encode gender, kinship, ethnicity, and geographic origin unevenly due to colonial recordkeeping practices. As a result, women’s labor roles, reproductive histories, and social networks often disappear into ambiguous descriptors such as “girl,” “wench,” “daughter,” or unnamed household dependents. This research project proposes a machine learning framework to recover African women’s identities and reconstruct relational networks across fragmented documentary sources. The method integrates optical character recognition (OCR) to digitize handwritten and degraded archival documents; natural language processing (NLP) models trained to detect gender-coded vocabulary, kinship relational markers, and African naming patterns; and probabilistic entity resolution to match individuals across dispersed archival collections. Rather than displacing human interpretation, the approach is designed to augment historical reasoning, acting as a recovery tool that flags overlooked individuals and generates new research leads. Collaborative integration with existing digital humanities infrastructures especially Slave Voyages and Freedom on the Move enables scale, interoperability, and standardized metadata exchange. This project contributes to ongoing efforts in reparative archival work, feminist historiography, and Black Atlantic studies by systematically addressing archival silences. By leveraging computational approaches to illuminate erased presences, it advances a historically grounded, ethically sensitive framework for restoring African women to the narrative of Atlantic world history.