Document Classification after Dimension Reduction through a Modified Gram-Schmidt Process

Document categorization is the process of classifying documents from many mixed documents automatically, and the main problem is how to express document content in vector space completely. This paper proposes a new model named Latent Semantic Analysis (LSA) + word2vec to categorize documents. This is the first attempt of combining word2vec with LSA at document categorization and it can map document to vector space under the premise of keeping document contents fully. At first, we create a term by document matrix and the element of which is decided by Term Frequency-Inverse Document Frequency (TF-IDF) weighting and word vector trained by word2vec. This matrix is a 3-dimensional matrix and it can describe the meaning of every word and the content of every document exactly. Secondly, Singular Value Decomposition (SVD) is executed on the matrix and lower computational complexity is gained from this. The model is named LSA + word2vec. Then, document vector gained from the new model are put into Convolutional Neural Network (CNN) to train. CNN is an efficient deep learning algorithm, which improves the accuracy of classification greatly. We evaluate the performance based on the 20newsgroups corpus. The results show that our new model achieves better effects on document categorization tasks, and the accuracy made about 15% improvement than traditional methods, such as LSA and Vector Space Model (VSM).

Latent semantic analysis (LSA) is a generalized vector space method that uses dimension reduction to generate term correlations for use during the information retrieval process. We hypothesized that even though the dimension reduction establishes correlations between terms, the dimension reduction is causing a degradation in the correlation of a term to itself (self-correlation). In this article, we have proven that there is a direct relationship to the size of the LSA dimension reduction and the LSA self-correlation. We have also shown that by altering the LSA term self-correlations we gain a substantial increase in precision, while also reducing the computation required during the information retrieval process.

Dimensionality reduction is the process of deriving an approximate representation of a dataset, that can reflect most of the correlations underlying within the dataset. In the context of text processing, dimensionality reduction is used for transforming any text to a precise representation that efficiently identifies the main insights of the original text. LSA(Latent Semantic Analysis) is a technique that is used to find correlations between words and sentences based on the usage of words within the text. This paper addresses the issue of dimensionality reduction in representing relevant data from Hindi text using LSA. An empirical evaluation is performed to find the influence of language complexity and influence of various weighting schemes on dimensionality reduction. The results are presented using the standard measures such as recall, precision and F-score.

Latent semantic analysis (LSA) has been used in several intelligent tutoring systems(ITS's) for assessing students' learning by evaluating their answers to questions in the tutoring domain. It is based on word-document co-occurrence statistics in the training corpus and a dimensionality reduction technique. However, it doesn't consider the word-order or syntactic information, which can improve the knowledge representation and therefore lead to better performance of an ITS. We present here an approach called Syntactically Enhanced LSA (SELSA) which generalizes LSA by considering a word along with its syntactic neighborhood given by the part-of-speech tag of its preceding word, as a unit of knowledge representation. The experimental results on Auto-Tutor task to evaluate students' answers to basic computer science questions by SELSA and its comparison with LSA are presented in terms of several cognitive measures. SELSA is able to correctly evaluate a few more answers than LSA but is having less correlation with human evaluators than LSA has. It also provides better discrimination of syntactic-semantic knowledge representation than LSA.

Latent Semantic Analysis (LSA) is a method that allows us to automatically index and retrieve information from a set of objects by reducing the term-by-document matrix using the Singular Value Decomposition (SVD) technique. However, LSA has a high computational cost for analyzing large amounts of information. The goals of this work are (i) to improve the execution time of semantic space construction, dimensionality reduction, and information retrieval stages of LSA based on heterogeneous systems and (ii) to evaluate the accuracy and recall of the information retrieval stage. We present a heterogeneous Latent Semantic Analysis (hLSA) system, which has been developed using General-Purpose computing on Graphics Processing Units (GPGPUs) architecture, which can solve large numeric problems faster through the thousands of concurrent threads on multiple CUDA cores of GPUs and multi-CPU architecture, which can solve large text problems faster through a multiprocessing environment. We execute the hLSA system with documents from the PubMed Central (PMC) database. The results of the experiments show that the acceleration reached by the hLSA system for large matrices with one hundred and fifty thousand million values is around eight times faster than the standard LSA version with an accuracy of 88% and a recall of 100%.

A text classification model based on Latent Semantic Analysis and Improved Hyper-sphere Support Vector Machine, is proposed in order to improve the accuracy and efficiency of text classification. Latent Semantic Analysis is used in this model for feature extraction, eliminating the text representation errors caused by synonyms and polysemes, and reducing the dimension of text vector. At the same time, a new decision-making approach based on concentration, is designed for improving the classification of texts in overlapping regions. Experimental results show that the model will give good classification results when the number of classes is small. And the improved algorithm is effective and feasible.

This undergraduate honors thesis focuses on the problem that machine translators face in choosing the correct translation of a polysemous English noun in a foreign language. For instance, general-purpose internet translators such as Google Translate cannot distinguish between the meanings of the English noun bat, generating the French word for the baseball implement rather than that for the flying mammal - even when provided with such contextual hints as wings and cave. Latent Semantic Analysis (LSA; Foltz & Laham 1998) offers a solution for this issue. LSA is a well-developed technique and theory for relating words and meanings by analyzing text corpora. A multidimensional vector represents each word and each sentence or other contextual block, and similarity or disparity of meaning can then be calculated by the relative angles of these vectors. LSA uses singular value decomposition (Golub & vanLohn 1996) on matrices constructed from words and passages in the learning texts to reduce the number of dimensions in these vectors. Such a dimensional reduction has been shown to be capable of producing simulations of human contextual associations significantly better than simple proximity frequencies.

Semantic image retrieval using text such keywords or captions at different semantic levels has attracted considerable research attention in recent years. Automatic image annotation (AIA) has been proved to be an effective and promising solution to automatically deduce the high-level semantics from low-level visual features. Due to the complexity of image-label mapping, it has become a challenge to systematically develop precise models with better performance. In this paper, we try to address the problem using Latent Semantic Indexing (LSI) together with mixed Bagging (Bootstrap Aggregating). Given a training set of annotated images, we first perform vector quantizing of visual space to construct a visual vocabulary consisting of visual terms based on K-means constrained by the semantic relationships derived from the associated annotations using Latent Semantic Indexing; then the statistical hidden correlation between the visual terms and keywords is modeled by a two-level mixed Bagging model. At the base level, each multi-class SVM is trained independently using the replicate training set via Bootstrap technique, and the decision of each multi-class SVM is integrated based on Average aggregating to generate a combined confidence label vector (CLV) associated with each visual term. All the CLVs are then propagated to the subsequent language model fusion level. At his level, WordNet is used for re-weighting the CLV for each visual term by incorporating the word correlations to obtain an appropriate and consistent image annotation. We carried out experiments on a medium-sized image collection with about 1000 images from Corel Stock Photo CDs. The experimental results demonstrated that the annotation performance of this method outperforms some traditional approaches, showing the feasibility and effectiveness of the proposed unified framework into which semantic image classification and language models are seamlessly integrated.

ldquoCurse of dimensionalityrdquo is a common problem in the area of information retrieval. It was verified that points in a vector space are projected to a random subspace of suitably high dimension, and then the distances between the points are approximately preserved. Although such a random projection can be used to reduce the dimension of the document space, it does not bring together semantically related documents. Latent Semantic Indexing (LSI) projects documents to lower dimensional LSI space from higher dimensional term space with singular-value decomposition (SVD) for the purpose of reducing the dimensions of the document space and bringing together semantically related documents. But the computation time of SVD is a bottleneck because of the higher dimensions of documents. In this paper, a novel method of dimension reduction for improving LSI is provided. A term-to-concept projection matrix based on domain ontology was created in this method. This way documents were projected to lower dimensional concept space by the projection matrix. LSI pre-computation was performed not on the original term by document matrix, but on the lower dimensional concept by document matrix at great computational savings. Experiments indicate that this method improves the efficiency of LSI. And the similarity judgment between documents is not disturbed.

Latent semantic indexing (LSI) has been shown to be extremely useful in information retrieval, but it is not an optimal representation for text classification. It always drops the text classification performance when being applied to the whole training set (global LSI) because this completely unsupervised method ignores class discrimination while only concentrating on representation. Some local LSI methods have been proposed to improve the classification by utilizing class discrimination information. However, their performance improvements over original term vectors are still very limited. In this paper, we propose a new local LSI method called "local relevancy weighted LSI" to improve text classification by performing a separate single value decomposition (SVD) on the transformed local region of each class. Experimental results show that our method is much better than global LSI and traditional local LSI methods on classification within a much smaller LSI dimension.

Land Use/Land Cover (LULC) classification data have proven to be valuable assets for various governmental agencies, park managers, and natural resource managers. Traditional pixel-based classification methods have difficulty with high resolution imagery, resulting in a “salt and pepper” appearance. Newer object-based methods may prove to be more accurate. This study compared an object based classification procedure utilizing Feature Analyst© software with a traditional pixel-based methodology (supervised classification) when applied to medium-spatial resolution satellite imagery merged with high-spatial resolution aerial imagery. This study utilized two multi-spectral SPOT-5 satellite images, leaf-on and leaf-off, merged with a color infrared aerial image. Because of correlation between some of the bands of the merged image, Principal Component Analysis (PCA) was used to reduce redundancy in the data. Field data was collected in the study area to serve as a reference for the accuracy assessment. A training set was produced by selecting and identifying specific LULC class-types using 1-foot high-spatial resolution aerial imagery. This training set was used by both of the classification methods (supervised and object-based) to identify the various cover types within the study area. An accuracy assessment was performed on each image utilizing error matrices, the Kappa coefficient, and a two-tailed Z-test. Results indicate that the overall accuracy of the object-based classification was 82.0%, while the pixel-based classification was 66.9%. A Kappa analysis and a two tailed Z test were calculated. These values indicated a significant difference in the overall accuracies of the classifications.

Exploring potentially useful information from huge amount of textual data produced by microblogging services has attracted much attention in recent years. An important preprocessing step of microblog text mining is to convert natural language texts into proper numerical representations. Due to the short-length characteristics of microblog texts, using term frequency vectors to represent microblog texts will cause “sparse data” problem. Finding proper representations of microblog texts is a challenging issue. In this paper, we apply deep networks to map the high-dimensional representations of microblog texts to low-dimensional representations. To improve the result of dimensionality reduction, we take advantage of the semantic similarity derived from two types of microblogspecific information, namely the retweet relationship and hashtags. Two types of approaches, including modifying training data and modifying the training objective of deep networks, are proposed to make use of microblog-specific information. Experiment results show that the deep models perform better than traditional dimensionality reduction methods such as latent semantic analysis and latent Dirichlet allocation topic model, and the use of microblog-specific information can help to learn better representations.

Latent semantic analysis (LSA) is a model of knowledge representation for words. It works by applying dimension reduction to local co-occurrence data from a large collection of documents after performing singular value decomposition on it. When the reduction is applied, the system forms condensed representations for the words that incorporate higher order associations. The higher order associations are primarily responsible for any semantic similarity between words in LSA. In this article, a memory model is described that creates semantic representations for words that are similar in form to those created by LSA. However, instead of applying dimension reduction, the model builds the representations by using a retrieval mechanism from a well-known account of episodic memory.

Search engines are an integral part of web information gathering and retrieval process, but they are significantly dependent upon the words or word phrases input by the end user. The search engine usually contributes no additional semantic data toward the information acquisition process. This paper presents an intelligent search agent "Web host access tool" (WHAT) based on support vector machines (SVM), which introduces the notion of queries conducted within a specific contextual meaning. Given a context and associated keywords that personalize the search history and preferences of the user, WHAT performs more intelligent resource filtering than conventional search engines, providing more relevant results while filtering the irrelevant references. Search results obtained in the form of text from different search engines are processed by the SVM based word classifier that arrange the results obtained according to user preference obtained from previous search processes. The text materials are processed by Latent Semantic Indexing (LSI) for creating a document matrix that gives the probability of a word occurrence in a specified context. For simplicity, this paper considers 5 different contexts: business, education, entertainment, news and information and tourism. The LSI coefficients were used by SVM to yield confidence levels for each search result and according to that the results were sorted and presented to the end user. As an alternative, least square support vector machines (LS-SVM) are also studied in this paper. The system is updated by the user that provides feedback regarding the search relevance, based on which the LSI coefficients are updated to ensure increased relevance in future searches. Three different types of kernel function were considered in this paper -Linear. Radial basis function (RBF) and polynomial kernel. Results claim that Linear SVM perform better than the others, not only in terms of classification accuracy but also in terms of training speed.

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.