Partitioning the data space before applying hashingusing clustering algorithms

Overlap detection is a key step in de novo genome assembly pipelines based on the Overlap-Layout-Consensus (OLC) paradigm. Existing methods for overlap detection either rely on heuristic seed-and-extension strategies or locality-sensitive hashing (LSH), both of which struggle to handle repetitive genomic regions and the computational burden of large-scale datasets. Here, we present FEDRANN, a novel strategy for overlap graph construction that integrates feature extraction, dimensionality reduction (DR), and approximate nearest neighbor (ANN) search. We find the pipeline combining inverse document frequency (IDF) transformation, sparse random projection (SRP), and NNDescent enables accurate detection of overlaps across diverse datasets. We developed an efficient open-source implementation of this pipeline named Fedrann (https://github.com/jzhang-dev/fedrann). Through systematic benchmarking on real long-read sequencing data, we demonstrate that Fedrann produces overlap graphs comparable to or better than those generated by existing state-of-the-art tools, including MECAT2, minimap2, and wtdbg2, while maintaining competitive runtime. By integrating Fedrann into the Shasta assembler, we successfully reconstructed human whole genomes, achieving high assembly contiguity and quality. Despite being implemented primarily in Python, Fedrann achieves performance parity with tools written in compiled languages by leveraging C-accelerated numerical libraries and optimized batch-based matrix operations. Our results suggest that the combination of dimensionality reduction and ANN techniques offers a robust, scalable framework for accurate overlap detection in long-read assembly and broader sequence similarity search tasks.

Background‘One Health’ recognizes the interconnectivity of humans with their production and companion animals, and the environment. Emergence and transmission of antimicrobial resistance (AMR) within and between these compartments is a recognized global threat that requires further understanding to design interventions protecting both human and animal health. In this study we identified resistance gene targets and clonotypes of Escherichia coli recovered from human, canine and bovine hosts and applied non-linear dimensionality reduction and visualization techniques to identify genetic relationships that may otherwise be unobservable within the data. MethodsNon-duplicative E. coli isolates (N=3398; see Figure captions) were collected from humans, canines, bovines from the Midwest USA. We identified beta-lactamase gene targets for third-generation cephem multidrug resistant isolates and performed clonotype analysis on each. Uniform Manifold Approximation (UMAP) was used to create a two-dimensional “map” of the high dimensional space of the genetic results to identify similarities between both infecting and colonizing isolates, and between susceptible and resistant isolates in humans and animals in the study region (see Figure captions). ResultsThe resulting “map” highlights similarities in: 1) genetic patterns of AMR among animals and humans, and 2) links between isolates that are infecting and colonizing in humans and canines (Figures 1-2). Our results suggest that there is strong genetic overlap linking human and animal patterns of AMR. UMAP also identified genetic segments that are unique to humans, distinct outliers, and suggest limited exchange among the neighboring counties (Figure 3).Figure 1. Distribution of infection and surveillance isolates shows distinct clusters and distribution within host species in the UMAP space.Each panel of the figure shows the same UMAP space with the labeled species in color and the other points in grey as a reference. The UMAP space is a non-linear two-dimensional representation of the genetic information contained in the clonotype analysis. UMAP is a dimensionality reduction technique similar to principal component analysis (PCA), except that it uses a non-linear combination of the underlying dimensions, which highlights the local structure and grouping of the cases. For more details see: Diaz-Papkovich, A., Anderson-Trocmé, L., & Gravel, S. (2021). A review of UMAP in population genetics. Journal of Human Genetics, 66(1), 85–91. Infection isolates: no bovine isolates tested, canine n=190, human n=115. Surveillance isolates: bovine n=175, canine n=747, human n=2171.Figure 2. Distribution of resistant and susceptible isolates shows the resistant cases are distributed in small clusters surrounding a large cluster of predominantly susceptible cases.This figure plots the same cases on the same UMAP space as Figure 1. The only difference is the color that distinguishes between resistant and susceptible cases. Resistant isolates: bovine n=91, canine n=300; human n=238. Susceptible isolates: bovine n=84. canine=637, human n=2048.Figure 3. The proportion of cases from each cluster in four adjoining counties varies considerably.The dark bars show the proportion of cases falling into each cluster for each county. The light bars provide a reference point for interpreting the dark bars by showing the proportion of cases falling into each cluster across all four counties. When the dark bars exceed the light bars it indicates that the proportion of cases in that cluster exceeds that of the neighboring counties, such as Cluster 2 for Taylor county and Cluster 3 for Marathon county. All counties shown include a population of at least 20,000. These stipulations are in compliance with federal (HIPAA) guidelines.ConclusionThe results support that UMAP is a valuable tool for visualizing genetic AMR links across species. Human-animal transmission is likely for disparate and common clonotypes. DisclosuresAll Authors: No reported disclosures

Recently, hashing based Approximate Nearest Neighbor (ANN) techniques have been attracting lots of attention in computer vision. The data-dependent hashing methods, e.g., Spectral Hashing, expects better performance than the data-blind counterparts, e.g., Locality Sensitive Hashing (LSH). However, most data-dependent hashing methods only employ a single hash table. When higher recall is desired, they have to retrieve exponentially growing number of hash buckets around the bucket containing the query, which may drag down the precision rapidly. In this paper, we propose a so-called complementary hashing approach, which is able to balance the precision and recall in a more effective way. The key idea is to employ multiple complementary hash tables, which are learned sequentially in a boosting manner, so that, given a query, its true nearest neighbors missed from the active bucket of one hash table are more likely to be found in the active bucket of the next hash table. Compared with LSH that also can exploit multiple hash tables, our approach is more effective to find true NNs, thanks to the complementarity property of the hash tables from our approach. Experimental results on large scale ANN search show that the proposed method significantly improves the performance and outperforms the state-of-the-art.

Many large multimedia applications require efficient processing of nearest neighbor queries. Often, multimedia data are represented as a collection of important high-dimensional feature vectors. Existing Locality Sensitive Hashing (LSH) techniques require users to find top-k similar feature vectors for each of the feature vectors that represent the query object. This leads to wasted and redundant work due to two main reasons: 1) not all feature vectors may contribute equally in finding the top-k similar multimedia objects, and 2) feature vectors are treated independently during query processing. Additionally, there is no theoretical guarantee on the returned multimedia results. In this work, we propose a practical and efficient indexing approach for finding top-k approximate nearest neighbors for multimedia data using LSH called mmLSH, which can provide theoretical guarantees on the returned multimedia results. Additionally, we present a buffer-conscious strategy to speed up the query processing. Experimental evaluation shows significant gains in performance time and accuracy for different real multimedia datasets when compared against state-of-the-art LSH techniques.

Private approximate nearest neighbor search for on-chain data based on locality-sensitive hashing

Nearest Neighbor search is one of the simplest and most intuitive ideas in data mining. Due to it's simplicity and diverse utility, Nearest Neighbor search is often found to be the workhorse of a variety of data mining, machine learning, and computer vision algorithms. For very high dimensional data, the naive linear search tends to be optimal. This is due to the so called curse of dimensionality in database search. To accelerate the search time, may researchers have turned to the power and efficiency of GPGPU computing; mainly for computing the data intensive distance metrics. Though promising speedups are achieved through this method, it fails to capitalize on many of the recent advances in algorithms for bounded error approximate nearest neighbor searches. Approximate nearest neighbor methods give satisfactory results, especially in very large and possibly redundant datasets, while boasting sub-linear search complexities. In this paper we present a c-approximate r-nearest neighbor search algorithm for CUDA using Locality Sensitive Hash with Nearest Neighbor search (LSH-NN). We implement this system in CUDA and test it on real world image SIFT vector data. Our tests show that we are able to achieve significant speedup over the serial version, with good approximations on scalability, while achieving the near-optimal search complexity of LSH-NN.

Finding nearest neighbors in high-dimensional spaces is a fundamental operation in many multimedia retrieval applications. Exact tree-based indexing approaches are known to suffer from the notorious curse of dimensionality for high-dimensional data. Approximate searching techniques sacrifice some accuracy while returning good enough results for faster performance. Locality Sensitive Hashing (LSH) is a very popular technique for finding approximate nearest neighbors in high-dimensional spaces. Apart from providing theoretical guarantees on the query results, one of the main benefits of LSH techniques is their good scalability to large datasets because they are external memory based. The most dominant costs for existing LSH techniques are the algorithm time and the index I/Os required to find candidate points. Existing works do not compare both of these dominant costs in their evaluation. In this experimental survey paper, we show the impact of both these costs on the overall performance of the LSH technique. We compare three state-of-the-art techniques on four real-world datasets, and show that, in contrast to recent works, C2LSH is still the state-of-the-art algorithm in terms of performance while achieving similar accuracy as its recent competitors.

Hashing-based approximate nearest neighbor (ANN) search in huge databases has become popular due to its computational and memory efficiency. The popular hashing methods, e.g., Locality Sensitive Hashing and Spectral Hashing, construct hash functions based on random or principal projections. The resulting hashes are either not very accurate or are inefficient. Moreover, these methods are designed for a given metric similarity. On the contrary, semantic similarity is usually given in terms of pairwise labels of samples. There exist supervised hashing methods that can handle such semantic similarity, but they are prone to overfitting when labeled data are small or noisy. In this work, we propose a semi-supervised hashing (SSH) framework that minimizes empirical error over the labeled set and an information theoretic regularizer over both labeled and unlabeled sets. Based on this framework, we present three different semi-supervised hashing methods, including orthogonal hashing, nonorthogonal hashing, and sequential hashing. Particularly, the sequential hashing method generates robust codes in which each hash function is designed to correct the errors made by the previous ones. We further show that the sequential learning paradigm can be extended to unsupervised domains where no labeled pairs are available. Extensive experiments on four large datasets (up to 80 million samples) demonstrate the superior performance of the proposed SSH methods over state-of-the-art supervised and unsupervised hashing techniques.

This paper proposes a novel local feature approach for human verification using 2D ear imaging based on Polar Sine Transform (PST). The proposed approach consists mainly of four steps. Firstly, normalizing the training and testing images, then, combining the normalized images together. Secondly, dividing the fused image into blocks, then, PST is used to extract the invariant features from each block. Thirdly, the Approximate Nearest Neighbors (ANN) searching criterion is adopted to collect the most similar blocks by means of Locality Sensitive Hashing (LSH). Finally, some morphological operations are used to reduce the number of false matching blocks, then, the system verifies the human ear. False Reject Rate (FRR) versus False Acceptance Rate (FAR) and ROC curve are used to evaluate the performance of the proposed approach. The experiments are performed on IIT Delhi database to validate the proposed approach. The results demonstrate that the proposed approach has better performance compared with the existing methods.

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

Recent years, the stereo image has become an emerging media in the field of 3D technology, which leads to an urgent demand of stereo image retrieval. In this paper, we attempt to introduce a framework for object-based stereo image retrieval (OBSIR), which retrieves images containing the similar objects to the one captured in the query image by the user. The proposed approach consists of both online and offline procedures. In the offline procedure, we propose a salient object segmentation method making use of both color and depth to extract objects from each image. The extracted objects are then represented by multiple visual feature descriptors. In order to improve the image search efficiently, we construct an approximate nearest neighbor (ANN) index using cluster-based locality sensitive hashing (LSH). In the online stage, the user may supply the query object by selecting a region of interest (ROI) in the query image, or clicking one of the objects recommended by the salient object detector. For the image retrieval evaluation we build a new dataset containing over 10K stereo images. The experiments on this dataset show that the proposed method can effectively recommend the correct object and the final retrieval result is also better than other baseline methods.

The present work strives to investigate the effect of using dimensionality reduction techniques (DRTs) on breast cancer (BC) classification problem. Primarily, we focused on the following five (DRTs): Auto-Encoders (AE), T-Distributed Stochastic Neighbor Embedding (T- SNE), Recursive Feature Elimination (RFE), Isometric Feature Mapping (Isomap), and Principle Component analysis (PCA). These methods are combined with two famous classifiers that are Support Vector Machine (SVM) and Multilayer perceptron (MLP). They are used for BC classification. Breast Cancer Wisconsin Diagnostic (WDBC) data set was used to validate the experiments of this work. The former was provided by the University of California, Irvine (UCI) machine learning repository. The results demonstrated that combining MLP with the chosen (DRTs) methods increased the classification accuracy for almost all built models by at least 0.7%. In addition, they revealed a decrease in the classification accuracy using SVM as a classifier for almost all built models.KeywordsMachine learningDeep learningDimensionality reductionNeural networksBreast cancer predictionMedical dataset

Evaluation of soil liquefaction using AI technology incorporating a coupled ENN / t-SNE model

A crucial component of intuitive data visualization is presenting a hierarchical tree structure with interactive functions. For example, single-cell transcriptomics studies may generate gene expression values with developmental trajectories or cell lineage structures. Two common visualization methods, t-Distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP), require two separate figures to depict the distribution of cell types and gene expression data, with low-dimension projections that may not capture the hierarchical structures among cells. Here, we present a JavaScript framework and an interactive web app named Collapsible Tree, which presents values jointly with interactive, expandable, and collapsible lineage structures. For example, the Collapsible Tree presents cellular states and gene expression from single-cell transcriptomics within a single hierarchical plot, enabling comparisons of gene expression across lineages and subtle patterns between sub-lineages. Our framework can facilitate the exploration of complicated value patterns that are not evident in UMAP or t-SNE plots. The Collapsible Tree web interface is available at https://collapsibletree.data2in.net. The JavaScript library source code is available at https://github.com/data2intelligence/collapsible_tree.

We introduce a framework for using contrastive learning for dimensionality reduction on genetic datasets to create principal component analysis (PCA)-like population visualizations. Contrastive learning is a self-supervised deep learning method that uses similarities between samples to train the neural network to discriminate between samples. Many of the advances in these types of models have been made for computer vision, but some common methodology does not translate well from image to genetic data. We define a loss function that outperforms loss functions commonly used in contrastive learning, and a data augmentation scheme tailored specifically towards SNP genotype datasets. We compare the performance of our method to PCA and contemporary nonlinear methods with respect to how well they preserve local and global structure, and how well they generalize to new data. Our method displays good preservation of global structure and has improved generalization properties over t-distributed stochastic neighbor embedding, Uniform Manifold Approximation and Projection, and popvae, while preserving relative distances between individuals to a high extent. A strength of the deep learning framework is the possibility of projecting new samples and fine-tuning to new datasets using a pretrained model without access to the original training data, and the ability to incorporate more domain-specific information in the model. We show examples of population classification on two datasets of dog and human genotypes.