A new variant of radial visualization for supervised visualization of high dimensional data

High-dimensional data sets are often available in genome-enabled predictions. Such data sets include nonlinear relationships with complex dependence structures. For such situations, vine copula-based (quantile) regression is an important tool. However, the current vine copula-based regression approaches do not scale up to high and ultra-high dimensions. To perform high-dimensional sparse vine copula-based regression, we propose 2 methods. First, we show their superiority regarding computational complexity over the existing methods. Second, we define relevant, irrelevant, and redundant explanatory variables for quantile regression. Then, we show our method's power in selecting relevant variables and prediction accuracy in high-dimensional sparse data sets via simulation studies. Next, we apply the proposed methods to the high-dimensional real data, aiming at the genomic prediction of maize traits. Some data processing and feature extraction steps for the real data are further discussed. Finally, we show the advantage of our methods over linear models and quantile regression forests in simulation studies and real data applications.

DBFS: An effective Density Based Feature Selection scheme for small sample size and high dimensional imbalanced data sets

Hierarchically clustered heatmaps are a popular visualization technique to display high-dimensional biological datasets as figures in research publications. Most tools developed to visualize hierarchically clustered heatmaps generate static images. Clustergrammer is a web-based matrix visualization tool built, using the D3 JavaScript library, to enable experimental and computational-biologists to easily generate highly interactive heatmaps. Clustergrammer implements many interactive features including: zooming, panning, filtering, sorting, reordering, dimensionality reduction, gene set enrichment analysis, gene description text, and shareable persistent links. Clustergrammer can be used to visualize and analyze a diverse set of high-dimensional biological data including: gene expression, protein-protein interactions, and CyTOF single-cell proteomics data. Clustergrammer provides interactive visualizations for several web-tools including: Enrichr, GEN3VA, GEO2Enrichr, Harmonizome, and L1000CDS2. In addition, Clustergrammer can be embedded within IPython Jupyter Notebooks or as a standalone open-source JavaScript or Python library.

Distributed feature selection (DFS) strategy for microarray gene expression data to improve the classification performance

Optimizing and benchmarking data reduction methods for dynamic or spatial visualization and interpretation (DSVI) face challenges due to many factors, including data complexity, lack of ground truth, time-dependent metrics, dimensionality bias and different visual mappings of the same data. Current studies often focus on independent static visualization or interpretability metrics that require ground truth. To overcome this limitation, we propose the MIBCOVIS framework, a comprehensive and interpretable benchmarking and computational approach. MIBCOVIS enhances the visualization and interpretability of high-dimensional data without relying on ground truth by integrating five robust metrics, including a novel time-ordered Markov-based structural metric, into a semi-supervised hierarchical Bayesian model. The framework assesses method accuracy and considers interaction effects among metric features. We apply MIBCOVIS using linear and nonlinear dimensionality reduction methods to evaluate optimal DSVI for four distinct dynamic and spatial biological processes captured by three single-cell data modalities: CyTOF, scRNA-seq and CODEX. These data vary in complexity based on feature dimensionality, unknown cell types and dynamic or spatial differences. Unlike traditional single-summary score approaches, MIBCOVIS compares accuracy distributions across methods. Our findings underscore the joint evaluation of visualization and interpretability, rather than relying on separate metrics. We reveal that prioritizing average performance can obscure method feature performance. Additionally, we explore the impact of data complexity on visualization and interpretability. Specifically, we provide optimal parameters and features and recommend methods, like the optimized variational contractive autoencoder, for targeted DSVI for various data complexities. MIBCOVIS shows promise for evaluating dynamic single-cell atlases and spatiotemporal data reduction models.

Display and visualization of high dimensional data are typically performed with a well-chosen linear projection of the data or by displaying many linear projections to form an animation. This study presents an evolutionary algorithm for producing nonlinear projections of high dimensional data with cues, in the drawing of the projection, as to the types of distortions introduced. Such projections can provide drawings closer to the true high dimensional distances of the displayed data than any single linear drawing. This permits a researcher to view a good analog to a scatter plot for high dimensional data. The system is demonstrated on a synthetic four dimensional fitness landscape and on distance data derived from RNA folds. Because fitness landscapes often have more dimensions than can be easily visualized it is difficult to gain an intuitive understanding of a fitness landscape. The nonlinear projection algorithm is applied to an abstraction of the fitness landscape called a fitness web. Fitness webs can be used to display the relative quality of optima, the frequency with which they were found by different evolutionary runs, or other factors of interest. In addition to displaying the relative position of optima in a fitness landscape, a graph of the fitness function along the edges a fitness web displays important slices of the fitness landscape. Called fitness morphs these plots can provide intuition about the fitness landscapes as well as direction for subsequent evolutionary searches. The second demonstration of the nonlinear projection algorithm is to data generated from an ad hoc metric on RNA folds. The algorithm yields drawings that permit a researcher to correctly distinguish two different types of folds for iron response elements.

BackgroundReducing TB surveillance incurs significant costs, as effective control and societal impact minimization rely on robust, ongoing surveillance systems that integrate and analyze comprehensive data for prompt action. Our study supports integrating artificial intelligence (LSI) into TB and other notifiable disease monitoring frameworks to improve trackingFigure 1Binary Matrix of 125 Terms: Presence vs. Absence of Root Keywords Retrieved from the Patient ElectronicRecord for the LSI Automatic Retrieval System.Binary Matrix of 125 Terms: Presence vs. Absence of Root Keywords Retrieved from the Patient Electronic Record for the LSI Automatic Retrieval System.MethodsIn 2024, we developed a novel application within Datamart A.R.G.U.S.—Assistant for Recovery and Guarding of Urgent and Epidemiological Sentinels—a cloud-based platform hosted on AWS and developed by our team. This application integrates with hospital systems (MV) and laboratory systems (MATRIX and GAL) to enhance the surveillance of healthcare-associated infections (HAI) and notifiable diseases. Data from both hospital and outpatient care are encoded into a binary matrix of 125 terms (root keywords) extracted from patient electronic records. This matrix is then processed using Singular Value Decomposition (SVD) within an LSI-based information retrieval system (Fig. 1). In our model, each patient is represented as a point in a 125-dimensional hyperdimensional space. When projected into three-dimensional space, this framework allows for effective clustering and classification of notifiable Diseases (Fig. 2), specifically TB (Fig. 3).Figure 2:3D Visualization of High-Dimensional Data: Comparing Mandatory Notifiable Diseases versus Non-Notifiable Diseases. Both patient types are well separated in the space, enabling automatic clustering and classification.3D Visualization of High-Dimensional Data: Comparing Mandatory Notifiable Diseases versus Non-Notifiable Diseases. Both patient types are well separated in the space, enabling automatic clustering and classification.ResultsThe LSI-based matrix was constructed using data from January 2023 to February 2024, encompassing a sample of 17,895 cases (lines) across 125 presence versus absence terms, with each case classified as either a notifiable or non-notifiable disease. The 125-dimensional matrix was reduced to 28 factors using SVD. The sensitivity and specificity of the Euclidean distance were better than those of cosine similarity when automatically identifying TB cases in 1,970 independent queries (Fig. 4).Figure 3:3D Visualization of High-Dimensional Data: Comparing Tuberculosis versus Other Mandatory Notifiable Diseases versus Non-Notifiable Diseases. The three types of patients are well separated in the space, enabling automatic clustering and classification.3D Visualization of High-Dimensional Data: Comparing Tuberculosis versus Other Mandatory Notifiable Diseases versus Non-Notifiable Diseases. The three types of patients are well separated in the space, enabling automatic clustering and classification.ConclusionArtificial Intelligence (LSI), significantly enhances the surveillance of TB and other notifiable diseases by enabling real-time, automated case identification up to one day prior (D-1). This advancement allows surveillance personnel to complete their tasks more efficiently, thereby preventing outbreaks and reducing the risk of unforeseen fatalities.Figure 4:Performance Comparison of Cosine vs. Euclidean Distance Metrics in Automatically Identifying TB Cases Using LSI: Euclidean Distance Demonstrates Superior Accuracy and Has Been Integrated into Datamart A.R.G.U.S. for Real-Time Identification of TB and Other Notifiable Diseases as of the Previous Workday (D-1).Performance Comparison of Cosine vs. Euclidean Distance Metrics in Automatically Identifying TB Cases Using LSI: Euclidean Distance Demonstrates Superior Accuracy and Has Been Integrated into Datamart A.R.G.U.S. for Real-Time Identification of TB and Other Notifiable Diseases as of the Previous Workday (D-1).DisclosuresAll Authors: No reported disclosures

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

The development of high-dimensional imaging systems in image-guided radiotherapy provides important pathways to the ultimate goal of real-time full volumetric motion monitoring. Effective motion management during radiation treatment usually requires prediction to account for system latency and extra signal/image processing time. It is challenging to predict high-dimensional respiratory motion due to the complexity of the motion pattern combined with the curse of dimensionality. Linear dimension reduction methods such as PCA have been used to construct a linear subspace from the high-dimensional data, followed by efficient predictions on the lower-dimensional subspace. In this study, we extend such rationale to a more general manifold and propose a framework for high-dimensional motion prediction with manifold learning, which allows one to learn more descriptive features compared to linear methods with comparable dimensions. Specifically, a kernel PCA is used to construct a proper low-dimensional feature manifold, where accurate and efficient prediction can be performed. A fixed-point iterative pre-image estimation method is used to recover the predicted value in the original state space. We evaluated and compared the proposed method with a PCA-based approach on level-set surfaces reconstructed from point clouds captured by a 3D photogrammetry system. The prediction accuracy was evaluated in terms of root-mean-squared-error. Our proposed method achieved consistent higher prediction accuracy (sub-millimeter) for both 200 ms and 600 ms lookahead lengths compared to the PCA-based approach, and the performance gain was statistically significant.

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

The information in high-dimensional datasets is often too complex for human users to perceive directly. Hence, it may be helpful to use dimensionality reduction methods to construct lower dimensional representations that can be visualized. The natural question that arises is how do we construct a most informative low dimensional representation? We study this question from an information-theoretic perspective and introduce a new method for linear dimensionality reduction. The obtained model that quantifies the informativeness also allows us to flexibly account for prior knowledge a user may have about the data. This enables us to provide representations that are subjectively interesting. We title the method Subjectively Interesting Component Analysis (SICA) and expect it is mainly useful for iterative data mining. SICA is based on a model of a user’s belief state about the data. This belief state is used to search for surprising views. The initial state is chosen by the user (it may be empty up to the data format) and is updated automatically as the analysis progresses. We study several types of prior beliefs: if a user only knows the scale of the data, SICA yields the same cost function as Principal Component Analysis (PCA), while if a user expects the data to have outliers, we obtain a variant that we term t-PCA. Finally, scientifically more interesting variants are obtained when a user has more complicated beliefs, such as knowledge about similarities between data points. The experiments suggest that SICA enables users to find subjectively more interesting representations.

Regarding the magnificent ability of the fuzzy logic in explanation and analyzing the real phenomena, using fuzzy data is growing rapidly. But the tools for doing data mining in a fuzzy data set; especially in high dimension, are not sufficient. Considering the remarkable ability of the LoOP (Local Outlier Probabilities) method in outlier detecting in high dimensional crisp data set, like supplying for each data object, determining outlier probability as scar which is easily explainable competitor on various synthetic and real world data sets, the aim of this article is to present an extended version of this method for using in high dimensional fuzzy database. In order to provide a completely fuzzy strategy, first we propose a new approach to assigning distance between fuzzy high dimension data based on Hausdorff and Vertex distance. After proving that this distance measuring is a metric, since LoOP is a distance based method, we redefined the LoOP algorithm step by step based on the new meter. Then we allocate an outlier score in the range of [0,1] to each data based on LoOP idea, to identify the outliers. The presented numerical tests show the efficiency of the new method, for random cases of knowing and unknowing the outlier and a real case. The results shows also the stability and robustness of the presented method more that its accuracy.

In this paper, we discuss dimension reduction methods for 2D visualization of high dimensional clustered data. We propose a two-stage framework for visualizing such data based on dimension reduction methods. In the first stage, we obtain the reduced dimensional data by applying a supervised dimension reduction method such as linear discriminant analysis which preserves the original cluster structure in terms of its criteria. The resulting optimal reduced dimension depends on the optimization criteria and is often larger than 2. In the second stage, the dimension is further reduced to 2 for visualization purposes by another dimension reduction method such as principal component analysis. The role of the second-stage is to minimize the loss of information due to reducing the dimension all the way to 2. Using this framework, we propose several two-stage methods, and present their theoretical characteristics as well as experimental comparisons on both artificial and real-world text data sets.

High dimensionality has become a typical feature of biomolecular data. In this paper, a novel dimension reduction method named p-norm singular value decomposition (PSVD) is proposed to seek the low-rank approximation matrix to the biomolecular data. To enhance the robustness to outliers, the Lp-norm is taken as the error function and the Schatten p-norm is used as the regularization function in the optimization model. To evaluate the performance of PSVD, the Kmeans clustering method is then employed for tumor clustering based on the low-rank approximation matrix. Extensive experiments are carried out on five gene expression data sets including two benchmark data sets and three higher dimensional data sets from the cancer genome atlas. The experimental results demonstrate that the PSVD-based method outperforms many existing methods. Especially, it is experimentally proved that the proposed method is more efficient for processing higher dimensional data with good robustness, stability, and superior time performance.

<ns3:p>Background Visualization and analysis of high-dimensional data are essential in biomedical research. There is a need for secure, scalable, and reproducible tools to facilitate data exploration and interpretation. Results We introduce DataMap, a browser-based application designed for the visualization of high-dimensional data using heatmaps and dimensionality reduction plots. DataMap operates directly in the web browser, ensuring data privacy without the requirement of installations or server setups. The application features an intuitive user interface for data transformation, annotation, and generation of reproducible R code. Conclusions Freely accessible as a GitHub page (https://gexijin.github.io/datamap), DataMap is a secure, user-friendly, and reproducible solution for visualizing high-dimensional data.</ns3:p>