VR-visualization of High-dimensional Urban Data

Due to the intricate relationship between different dimensions of high-dimensional data, subspace analysis is often conducted to decompose dimensions and give prominence to certain subsets of dimensions, i.e. subspaces. Exploring and comparing subspaces are important to reveal the underlying features of subspaces, as well as to portray the characteristics of individual dimensions. To date, most of the existing high-dimensional data exploration and analysis approaches rely on dimensionality reduction algorithms (e.g. principal component analysis and multi-dimensional scaling) to project high-dimensional data, or their subspaces, to two-dimensional space and employ scatterplots for visualization. However, the dimensionality reduction algorithms are sometimes difficult to fine-tune and scatterplots are not effective for comparative visualization, making subspace comparison hard to perform. In this article, we aggregate high-dimensional data or their subspaces by computing pair-wise distances between all data items and showing the distances with matrix visualizations to present the original high-dimensional data or subspaces. Our approach enables effective visual comparisons among subspaces, which allows users to further investigate the characteristics of individual dimensions by studying their behaviors in similar subspaces. Through subspace comparisons, we identify dominant, similar, and conforming dimensions in different subspace contexts of synthetic and real-world high-dimensional data sets. Additionally, we present a prototype that integrates parallel coordinates plot and matrix visualization for high-dimensional data exploration and incremental dimensionality analysis, which also allows users to further validate the dimension characterization results derived from the subspace comparisons.

Today cities are the main humankind settlement relying on intricate systems. The technological growth in the context of urban produces a vast amount of data. Analyzing and visualizing this data have provided insights into this complex environment. However, these mere approaches are inert and, due to technical constraints, hard to be integrated into urban planning.Consequently, we adopt a research hypothesis in which the adaptability and complexity of urban systems can be replicated, or partially replicated, by the machine-learning algorithms.The current study aims to define a process for evaluating the capabilities for analyzing and predicting urban data with machine-learning (ML) algorithms. This process starts with constructing a data structure for inputting into the ML algorithm. Following this, different tests are applied to identify valid combinations of data and models that allow understanding of urban patterns. The bicycle-sharing system is used as a case study. The process ends discussing the options to replicate the experiment in different urban areas as well as to adapt it to different problems.Different datasets from different cities have been explored and considered for this experiment. Across many cities' open dataset platforms, the NYD platform offered the most reliable data. From the sub-systems of the city, the mobility network was selected as a case study for exploration. More specifically, data on shared-bicycle mobility and use were selected as a result of its exciting raise as travel choices reported by New York City's department of transportation. The urban data analysis and prediction process of this research paper identifies the neighborhood as the unit of the model. Additionally, to illustrate and analyze the relationship between the selected mobility sub-system and other urban systems, contextual indicators such as land use indexes were added in the modeling process.Despite the prediction modeling machine for the bike-sharing system coming out of this study, the main achievement is the introduction of a collection of analysis and prediction processes for urban data beyond mobility. For further advancement, implementing this approach in a different urban system and context is crucial. By this means, the replicability of the process could be evaluated and tested.

Local and global business interests assemble images of neighbourhoods from localised knowledge, including disparate forms of public data such as reviews, blog posts, and open data from municipalities and other organisations. (In)visible forms of working with and worrying about neighbourhood data can be understood as an engagement with the neighbourhood's reputation, or rather its symbolic trajectory: a set of tangible and intangible indicators through which an urban space is known and treated accordingly over time. This paper addresses ethical concerns that emerge from contemporary datafied urban ethnography. We consider a combination of large-scale and bespoke, quantitative, and qualitative analyses of available sources with sustained ethnographic engagement with a Dutch neighbourhood coping with a troubled reputation. While the latter activities can mitigate ethical issues stemming from the former, ethnography in turn raises further concerns of exploitation and risk exposure and should not be treated as a kind of 'ethical panacea' for big, open, or public data projects. A multifaceted and interrogative approach to data collection may offer a more rounded account of contemporary urban data practices by drawing upon distinct and possibly conflicting accounts of social life. The challenge is to prioritise under-represented and otherwise marginalised voices in both the design and the dissemination of research on urban data analytics.

With the rapid development of information technology and the popularity of the Internet, the era of big data has come. The urban traffic system continuously generates a large amount of traffic data, which are so large and diverse that the traditional data processing methods have been unable to effectively process and analyze them. Therefore, it is necessary to process and analyze urban traffic data with the help of emerging big data technologies and methods. Starting from introducing the characteristics of big data visualization platforms and common big data visualization tools and technologies, the article analyzes the scale and characteristics of urban traffic data and the limitations of traditional data analysis methods, discusses the advantages and application prospects of visualization platforms, and focuses on the four perspectives of data cleaning and pre-processing, traffic flow and congestion analysis, traffic behavior and trend analysis, traffic simulation, and planning support. The application of big data visualization platforms in urban traffic data analysis is discussed.

Contemporary single-cell technologies produce data with a vast number of variables at a rapid pace, making large volumes of high-dimensional data available. The exploratory analysis of such high dimensional data can be aided by intuitive low dimensional visualizations. In this work, we investigate how both discrete and continuous structures in single cell data can be captured using the recently proposed dimensionality reduction method SONG, and compare the results with commonly used methods UMAP and PHATE. Using simulated and real-world datasets, we observed that SONG preserves a variety of patterns including discrete clusters, continuums, and branching structures. More importantly, SONG produced more/equally insightful visualizations compared to UMAP and PHATE in all considered datasets. We also quantitatively validate the high-dimensional pairwise distance preservation ability of these visualization methods in the low dimensional space for the generated visualizations.

Applications of medical image analysis are often faced with the challenge of modelling high-dimensional data with relatively few samples. In many settings, normal or healthy samples are prevalent while pathological samples are rarer, highly diverse, and/or difficult to model. In such cases, a robust model of the normal population in the high-dimensional space can be useful for characterizing pathologies. In this context, there is utility in hybrid models, such as probabilistic PCA, which learns a low-dimensional model, commensurates with the available data, and combines it with a generic, isotropic noise model for the remaining dimensions. However, the isotropic noise model ignores the inherent correlations that are evident in so many high-dimensional data sets associated with images and shapes in medicine. This paper describes a method for estimating a Gaussian model for collections of images or shapes that exhibit underlying correlations, e.g., in the form of smoothness. The proposed method incorporates a Gaussian-process noise model within a generative formulation. For optimization, we derive a novel expectation maximization (EM) algorithm. We demonstrate the efficacy of the method on synthetic examples and on anatomical shape data.

This chapter covers the state-of-the-art multivariate statistical methods designed for high dimensional multiset omics data analysis. Recent biotechnological developments have enabled large-scale measurement of various biomolecular data, such as genotypic and phenotypic data, dispersed over various omics domains. An emergent research direction is to analyze these data sources using an integrated approach to better model and understand the underlying biology of complex disease conditions. However, comprehensive analysis techniques that can handle both the size and complexity, and at the same time can account for the hierarchical structure of such data, are lacking. An overview of some of the developments in multivariate techniques for high-dimensional omics data analysis, highlighting two well-known multivariate methods, canonical correlation analysis (CCA) and redundancy analysis (RDA), is provided in this chapter. Penalized versions of CCA are widespread in the omics data analysis field, and there is recent work on multiset penalized RDA that is applicable to multiset omics data. How these methods meet the statistical challenges that come with high-dimensional multiset omics data analysis and help to further our understanding of the human condition in terms of health and disease are presented. Additionally, the current challenges to be resolved in the field of omics data analysis are discussed.

Uncovering driver genes is crucial for understanding heterogeneity in cancer. L1-type regularization approaches have been widely used for uncovering cancer driver genes based on genome-scale data. Although the existing methods have been widely applied in the field of bioinformatics, they possess several drawbacks: subset size limitations, erroneous estimation results, multicollinearity, and heavy time consumption. We introduce a novel statistical strategy, called a Recursive Random Lasso (RRLasso), for high dimensional genomic data analysis and investigation of driver genes. For time-effective analysis, we consider a recursive bootstrap procedure in line with the random lasso. Furthermore, we introduce a parametric statistical test for driver gene selection based on bootstrap regression modeling results. The proposed RRLasso is not only rapid but performs well for high dimensional genomic data analysis. Monte Carlo simulations and analysis of the “Sanger Genomics of Drug Sensitivity in Cancer dataset from the Cancer Genome Project” show that the proposed RRLasso is an effective tool for high dimensional genomic data analysis. The proposed methods provide reliable and biologically relevant results for cancer driver gene selection.

Numerous variable selection techniques have been developed for complete high-dimensional data but very few of them for censored data. The techniques for complete data must be modified if censoring is present. In this paper, we consider the variable selection technique for accelerated failure time (AFT) models by extending the ranking-based variable selection (RBVS) algorithm and its iterative procedure as proposed in the work of Baranowski et al. through the Stute’s weighted least square technique. Simulation studies are conducted to demonstrate the performance of the proposed methods. We further illustrate the performance of this method with a mantle cell lymphoma microarray example. When there is no correlation among the covariates, the proposed method outperforms the iterative sure independence screening and stability selection methods in terms of overall performance for high-dimensional data. Real data analysis also suggests that the proposed method can be chosen for high-dimensional censored data analysis in parallel to other methods in the literature.

With the explosive growth of social media data, the need for effective high-dimensional data analysis techniques has become increasingly pressing. Current research in this field faces challenges such as data sparsity, noise, and scalability issues. To address these challenges, this paper proposes a novel approach utilizing Principal Component Analysis (PCA) for high-dimensional data analysis in social media. By applying PCA to reduce the dimensionality of the data while preserving essential information, our method aims to enhance the efficiency and accuracy of social media data analysis. Through comprehensive experimentation and evaluation, this paper demonstrates the effectiveness and potential of the proposed approach in improving the analysis of high-dimensional social media data, contributing valuable insights to the field of social media analytics.

<ns3:p>Background Visualization and analysis of high-dimensional data are essential in biomedical research. There is a need for secure, scalable, and reproducible tools to facilitate data exploration and interpretation. Results We introduce DataMap, a browser-based application designed for the visualization of high-dimensional data using heatmaps and dimensionality reduction plots. DataMap operates directly in the web browser, ensuring data privacy without the requirement of installations or server setups. The application features an intuitive user interface for data transformation, annotation, and generation of reproducible R code. Conclusions Freely accessible as a GitHub page (https://gexijin.github.io/datamap), DataMap is a secure, user-friendly, and reproducible solution for visualizing high-dimensional data.</ns3:p>

Popular clustering algorithms based on usual distance functions (e.g., the Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances and violation of neighborhood structure have adverse effects on their performance. In this article, we use a new data-driven dissimilarity measure, called MADD, which takes care of these problems. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform well for high dimensional data. We establish it using theoretical as well as numerical studies. We also address the problem of estimating the number of clusters. This is a challenging problem in cluster analysis, and several algorithms are available for it. We show that many of these existing algorithms have superior performance in high dimensions when they are constructed using MADD. We also construct a new estimator based on a penalized version of the Dunn index and prove its consistency in the HDLSS asymptotic regime. Several simulated and real data sets are analyzed to demonstrate the usefulness of MADD for cluster analysis of high dimensional data.

Partial least squares (PLS) regression is a very useful tool for the analysis of high dimensional data (HDD). Choosing the ideal number of PLS components is a vital step in developing the best model. The accuracy of the model will be affected if there are too many or too few PLS components being selected. Numerous classical methods, such as the leave-one-out cross-validation (LOOCV) and K-fold cross-validation (K-FoldCV) are developed to determine the optimal number of PLS components. Nonetheless, they are easily affected by high leverage points (HLPs). Thus, robust cross validation techniques, denoted as RMD- MRCD-PCA-LOOCV and RMD-MRCD-PCA-K-FoldCV are proposed to remedy this problem. The results of the simulation study and real data set indicate that the proposed methods successfully select the appropriate number of PLS components.

Dimension reduction of data is an important issue in the data processing and it is needed for the analysis of higher dimensional data in the application domain. Rough set is fundamental and useful to reduce higher dimensional data to lower one for the classification. We develop generation of reducts based on nearest neighbor relation for the classification. In this paper, the nearest neighbor relation is shown to play a fundamental role for the classification from the geometric easoning of reducts by convex cones. Then, it is shown that reducts are generated based on the convex cones construction. Finally, using nearest neighbor relation, algebraic operations are derived on the degenerate convex cones.

ABSTRACT‘Zaha’ is a retrieval‐augmented generation question answering system integrated with The World Avatar knowledge graph, designed to support urban planning and smart city initiatives by enabling intuitive natural language queries for complex urban data. Zaha facilitates the querying of diverse domains within the urban environment, offering an accessible and effective tool for urban data analysis. By simplifying access to complex dataset, Zaha addresses a critical barrier in urban planning and management: the need for technical expertise to query data effectively. Urban data, encompassing geospatial, environmental, and regulatory information, is pivotal in informing decision‐making processes. However, challenges such as data silos and the technical complexity of query tools and languages hinder the accessibility and utilisation of urban data. The integration of Zaha with The World Avatar knowledge graph further mitigates the issue of data silos by unifying urban data from diverse sources and formats into a single framework. Leveraging knowledge graph technology, Zaha facilitates efficient data retrieval based on the relationships defined between entities. By bridging the gap between data and users, Zaha empowers urban planners and other stakeholders to access and query complex urban data intuitively, enabling them to make informed decisions without requiring technical expertise.