수평 분할 방식을 이용한 병렬 셀-기반 필터링 기법의 설계 및 성능 평가

The vector approximation file (VA-file) approach is an efficient high-dimensional indexing method using compression technique to overcome the difficulty of 'curse of dimensionality'. The VA-file method combined with tree-based index structure can improve the querying efficiency, but it still succumbs to the 'curse of dimensionality'. In this paper, a new high-dimensional indexing structure called PCR-tree for non-uniform distributed data sets was presented, which employs R-tree to manage the approximate vectors in the reduced-dimensionality space. The approximate vectors can be built in the KL transform domain, and low dimensional MBRs (minimum bounding rectangles) can be used to manage the approximations on the first few principal components. When performing k-nearest neighbor search, a lower-bound filtering algorithm is used to reject the improper nodes of PCR-tree, which can reduce the computational complexity and I/O cost without any dismissals. The experiment results on large image databases show that the new approach provides a faster search speed than other tree-structured vector approximation approaches.

This chapter introduces a concurrency control algorithm based on link-technique for high-dimensional index structures. In high-dimensional index structures, search operations are generally more frequent than insert or delete operations and need to access many more nodes than those in other index structures, such as B+-tree, B-tree, hashing techniques, and so on, due to the properties of queries. This chapter proposed an algorithm that minimizes the delay of search operations in all cases. The proposed algorithm also supports concurrency control on reinsert operations for the high-dimensional index structures employing reinsert operations to improve their performance. The authors hope that this chapter will give helpful information for studying multidimensional index structures and their concurrency control problems to researchers.

When SIFT algorithm is applied to image feature matching on the mobile platform, it has been found some issues such as slow search rate, high memory consumption and frequency I/O, etc. Aiming at such problems, this paper studies on high dimensional data index structure for SIFT image feature matching, proposed a K-Dvd-tree index structure. This structure divides high dimensional data into multiple subspace using vantage dimension in order to increase the fan-out degree of the tree, controls the number of feature points in leaf node in order to reduce the height of the tree, and ultimately raise the search rate, at the same time, it stores multiple internal nodes at the same level within one page, and makes each leaf node correspond to one page in order to reduce memory consumption, and improve I/O efficiency as well. the experiments show that the performance of this index structure is relatively increased in terms of search rate and I/O efficiency.

The technique of machinery condition monitoring has been greatly enhanced over recent years with the application of many effective classifiers. However, these classification methods suffer from the 'curse of dimensionality' when applied to high-dimensional condition monitoring data. Actually, many classification algorithms are simply intractable when the number of features in the data is sufficiently large. In order to solve the problem, engineers have to resort to complicated feature extraction methods and other statistical theories to reduce the data dimensionality. However, features extracted using these methods lose their original engineering meaning and become obscure for engineers. In this study, a novel feature selection algorithm is presented to help to identify machinery condition quickly, based only on frequency spectrum data and without considering any complicated feature extraction methods. This brings many significant benefits: it can not only help engineers out of difficulties with complicated feature extraction methods, but it also gives a clear insight into the real condition of machinery without any loss of engineering meaning from the features used. The algorithm forms a functional extension of a relevance vector machine to feature selection, with a fast step-variable sequential backward search algorithm to search relevant scale parameters within a kernel function. Two case studies on different levels of condition monitoring are conducted to demonstrate the potential of applying the algorithm to high-dimensional engineering data.

Given a very large multimedia database, how to process k-nearest-neighbor queries efficiently? While the sequential scan is one of the most obvious solutions for small-to-moderate multimedia databases, it becomes practically infeasible when the database size grows. Concomitant with the volume and velocity of data, multimedia databases are frequently endowed with a complex distance-based similarity model that supports content-based data access in an adjustable and adaptive manner. Typical for many state-of-the-art distance-based similarity models is an at least quadratic computation time complexity for a single distance evaluation between two multimedia objects. Thus the search for the most query-like multimedia objects is still one of the major challenges.In this paper, we address the problem of optimal k-nearest-neighbor query processing via multiple lower bound approximations in very large multimedia databases. To this end, we propose the concepts of filter-optimality and refinement-optimality and present the Cascading Multi-Step Algorithm and the Interleaved Multi-Step Algorithm for fast query processing. Besides the algorithms’ properties, we study their query processing performance with respect to the number of CPU and I/O operations on large-scale benchmark multimedia databases. Our performance analysis shows how to process k-nearest-neighbor queries in multimedia databases efficiently and provides a guide for further research.

Fast media search and browsing is today a growing need to face the challenge of managing large collections of personal media. Traditional databases, (e.g. MySQL), or text databases (e.g. Lucene), do not address an important aspect of multimedia data: the high-dimensionality of data. In this paper, we describe the implementation and evaluation of high-dimensional data indexing structures for fast search and browsing. Index structures for high-dimensional data have been researched in the literature and several proposals exist in the literature. We compare five popular index structures on a large-scale image retrieval scenario with different visual features of varying dimensionality. Both indexing and search aspects are evaluated: indexing time, search time and the trade-off between precision and performance.

A topological approach for cancer subtyping from gene expression data.

Most clustering methods for data mining applications do not work efficiently when dealing with large, high-dimensional data. This is caused by so-called ‘curse of dimensionality’ and the limitation of available memory. In this paper, we propose an efficient clustering method for handling of large amounts of high-dimensional data. Our clustering method provides both an efficient cell creation and a cell insertion algorithm. To achieve good retrieval performance on clusters, we also propose a filtering-based index structure using an approximation technique. We compare the performance of our clustering method with the CLIQUE method. The experimental results show that our clustering method achieves better performance on cluster construction time and retrieval time.

Techniques of Content-Based Image Retrieval (CBIR) employ a mathematical representation of an image (also called feature vector), to characterize the image in the retrieval process. The feature vector-based representation of an image in CBIR systems causes the "semantic gap" problem, which is the inconsistency between the low-level image feature representation and the high-level image interpretation. However, the usage of a large number of features to represent an image, which seems to be a solution for the semantic gap, leads to the "dimensionality curse" problem. In this paper, we propose to amend the semantic gap along with the dimensionality curse by a dimensionality reduction method called FTK (Feature Transformation based on K-means). FTK performs feature transformation by clustering the feature vector. It employs the clustering principle of k-means to compact the feature vector space. The results indicate that clustering is an approach well-suited to perform dimensionality reduction in CBIR systems.

The incidence of bladder cancer is on the rise, and its molecular heterogeneity presents significant challenges for personalized cancer therapy. Transcriptome data can characterize the variability among patients. Traditional machine-learning methods often struggle with high-dimensional genomic data, falling into the ’curse of dimensionality’. To address this challenge, we have developed MVMSGAT, an innovative predictive model tailored for forecasting responses to neoadjuvant therapy in bladder cancer patients. MVMSGAT significantly enhances model performance by incorporating multi-perspective biological prior knowledge. It initially utilizes the Boruta algorithm to select key genes from transcriptome data, subsequently constructing a comprehensive graph of gene co-expression and protein–protein interactions. MVMSGAT further employs a graph convolutional neural network to integrate this information within a multiview knowledge graph, amalgamating biological knowledge maps from various scales using an attention mechanism. For validation, MVMSGAT was tested using a five-fold cross-validation approach on two specific GEO datasets, GSE169455 and GSE69795, involving a total of 210 bladder cancer samples. MVMSGAT demonstrated superior performance, with the following metrics (mean ± standard deviation): AUC-ROC of 0.8724±0.0511, accuracy of 0.7789±0.068, F1 score of 0.8529±0.0338, and recall of 0.9231±0.0719. These results underscore the potential of MVMSGAT in advancing personalized treatment and precision medicine in bladder cancer.

Memorizing and retrieval of a face of board is problem on constructing a board game agent because board games have problem of state explosion, “The curse of dimensionality.” We propose to memorize by Self-Organizing Maps and to retrieve from it. The problem of memorizing by SOM is determination of the thresholds which retrieve properly a state of a board from memory of SOM. In this paper, we propose to determine the thresholds by statistical information and by a GA. The results show that the thresholds are determined by statistical information and a GA are the best performance of retrieval from memory of SOM.

Images can be represented as vectors in a high-dimensional Image space with components specifying light intensities at image pixels. To avoid the ‘curse of dimensionality’, the original high-dimensional image data are transformed into their lower-dimensional features preserving certain subject-driven data properties. These properties can include ‘information-preserving’ when using the constructed low-dimensional features instead of original high-dimensional vectors, as well preserving the distances and angles between the original high-dimensional image vectors. Under the commonly used Manifold assumption that the high-dimensional image data lie on or near a certain unknown low-dimensional Image manifold embedded in an ambient high-dimensional ‘observation’ space, a constructing of the lower-dimensional features consists in constructing an Embedding mapping from the Image manifold to Feature space, which, in turn, determines a low-dimensional parameterization of the Image manifold. We propose a new geometrically motivated Embedding method which constructs a low-dimensional parameterization of the Image manifold and provides the information-preserving property as well as the locally isometric and conformal properties.

Along with the rapid advancement of computer science and data engineering, the complexity and dimension of the data considered are bursting. With the increase of the data dimension, the performance of high dimensional index structure drops rapidly. And these high-dimensional data usually contains many redundant, its essence dimension often is much smaller than the original data, so the high-dimensional data processing question boils down to the related dimension reduction method to cut some less relevant data and reduce its dimension. This paper integrates the ameliorated supportive vector machine to propose the novel high-dimensional data dimension reduction algorithm. Our algorithm optimises the traditional methodologies from the three layers: 1) data cleaning and pre-processing steps are added to enhance the feasibility of the captured data; 2) SVM is modified to hold the feature of higher robustness; 3) application scenarios are built to test the performance. The experimental result proves that the proposed method outperforms compared with other state-of-theart algorithms theoretically and numerically.

We prove that the gradient descent training of a two-layer neural network on empirical or population risk may not decrease population risk at an order faster than $t^{-4/(d-2)}$ under mean field scaling. The loss functional is mean squared error with a Lipschitz-continuous target function and data distributed uniformly on the $d$ -dimensional unit cube. Thus gradient descent training for fitting reasonably smooth, but truly high-dimensional data may be subject to the curse of dimensionality. We present numerical evidence that gradient descent training with general Lipschitz target functions becomes slower and slower as the dimension increases, but converges at approximately the same rate in all dimensions when the target function lies in the natural function space for two-layer ReLU networks. Impact Statement –Artificial neural networks perform well in many real life applications, but may suffer from the curse of dimensionality on certain problems. We provide theoretical and numerical evidence that this may be related to whether a target function lies in the hypothesis class described by infinitely wide networks. The training dynamics are considered in the fully non-linear regime and not reduced to neural tangent kernels. We believe that it will be essential to study these hypothesis classes in detail to choose an appropriate machine learning models for a given problem. The goal of the article is to illustrate this in a mathematically sound and numerically convincing fashion.

Chapter 9 - Effective dimensionality reduction model with machine learning classification for microarray gene expression data