Effective News Text Summarization Techniques

The aim of this study is to evaluate how effective the Lexrank algorithm and Latent semantic analysis (LSA) are in automatic text summarization for the Indonesian language. This research focuses on natural language processing and handling of excessive data. We applied both algorithms to generate text summaries using the INDOSUM dataset, which contains about 20,000 news articles in Indonesian with manual summaries. To assess performance, the ROUGE metric was used, which includes aspects of precision, recall, and F1 score. In all tested metrics, LSA outperformed Lexrank. LSA had a precision of 0.57, recall of 0.67, and an F1 score of 0.59, whereas Lexrank had a precision of 0.46, recall of 0.52, and an F1 score of 0.48. These result indicate that LSA is better at gathering important information from the original text than Lexrank.

The emotion classification of text is an important research direction of text mining. Application on emotion text classification, latent semantic analysis algorithm has advantage of small occupied space, applicable to a large scale of text classifications. Compared with the traditional vector space model, latent semantic analysis algorithms reduce the search space for text classification by means of singular value decomposition for term and document matrix. Moreover, latent semantic analysis algorithms solve the problem of words with multiple meanings by analyzing the term at the semantic level. Using an improved latent semantic analysis algorithm to classify the test set by their emotion. The new cluster centroid is the average vector for each emotion category, and access to emotions classification for training dataset by calculating similarity of the average vector and test textual. The experimental results show that the improved latent semantic analysis algorithm have high precision and recall rate as same as the original algorithm, the efficiency of text emotion classification improved 4 percentage points. Keywords-Latent Semantic Analysis; Vector Space Model; Text Emotion Classification;

Context Based errors are those errors which are wrongly used in sentences, though it seems valid. These errors would turn sentences into meaningless, which in turned to invalid documents. These words would not rectify in context or would do an automatic spelling checking. This paper explores the idea of Document Semantic called Latent Semantic analysis which can correct the context from this minor difference. This paper also reflects the importance of using this terminology. Keywords: stop word removal; stemming; term-frequency; singular value decomposition; cosine similarity.

Retrieving information from different languages may lead to many problems like polysemy and synonymy, which can be resolved by Latent Semantic Indexing (LSI) techniques. This paper uses the Singular Value Decomposition (SVD) of LSI technique to achieve effective indexing for English and Hindi languages. Parallel corpus consisting of both Hindi and English documents is created and is used for training and testing the system. Removing stop words from the documents is performed followed by stemming and normalization in order to reduce the feature space and to get language relations. Then, cosine similarity method is applied on query document and target document. Based on our experimental results it is proved that LSI based CLIR gets over the non-LSI based retrieval which have retrieval successes of 67% and 9% respectively.

In this chapter, we introduce latent semantic analysis (LSA), which uses singular value decomposition (SVD) to reduce the dimensionality of the document-term representation. This method reduces the large matrix to an approximation that is made up of fewer latent dimensions that can be interpreted by the analyst. Two important concepts in LSA, cosine similarity and queries, are explained. Finally, we discuss decision-making in LSA.

Nowadays, searching for a topic on the Internet can be a frustrating experience because of all the excessive information. Thus, a strategy for automatically classifying the results can improve user experience and work efficiency. Latent Semantic Indexing (LSI) algorithm is used to classify documents by meaning due to its effectiveness. However, there is a problem with the implementation of this algorithm. LSI is computationally intensive because the cost is directly related to the number of documents. In particular, the Singular Value Decomposition (SVD) that is mainly used in LSI is unscalable in terms of both memory and computation time. One possible solution is to use more powerful computational resources, such as multiple computing nodes. In this paper, a novel distributed architecture for the LSI algorithm is proposed. It is based on the use of microservices in a Google Cloud environment. We evaluated the performances of the proposed Cloud-based LSI, and comparison is made with standalone LSI. The results show the benefits of using distributed systems based on runtime, concurrency, and processing.

We present an application of a novel methodology called Text Influenced Molecular Indexing (TIMI) to mine the information in the scientific literature. TIMI is an extension of two existing methodologies: (1) Latent Semantic Structure Indexing (LaSSI), a method for calculating chemical similarity using two-dimensional topological descriptors, and (2) Latent Semantic Indexing (LSI), a method for generating correlations between textual terms. The singular value decomposition (SVD) of a feature/object matrix is the fundamental mathematical operation underlying LSI, LaSSI, and TIMI and is used in the identification of associations between textual and chemical descriptors. We present the results of our studies with a database containing 11,571 PubMed/MEDLINE abstracts which show the advantages of merging textual and chemical descriptors over using either text or chemistry alone. Our work demonstrates that searching text-only databases limits retrieved documents to those that explicitly mention compounds by name in the text. Similarly, searching chemistry-only databases can only retrieve those documents that have chemical structures in them. TIMI, however, enables search and retrieval of documents with textual, chemical, and/or text- and chemistry-based queries. Thus, the TIMI system offers a powerful new approach to uncovering the contextual scientific knowledge sought by the medical research community.

Singular value decomposition is an important method to factorize a matrix, which is useful for many applications, such as principal component analysis in statistics and latent semantic analysis in natural language text processing. In order to diminish the computational complexity, incremental update algorithm is developed, especially for analyzing relationships between text documents. However, it is at the cost of a reduction in precision. In this paper, we study the effect of incremental update on latent semantic analysis; and, by experiments, to investigate the trend downward of computing precision and then try to find the balance point between re-computation and incremental update. The results indicate that when increased documents are less than original documents, incremental update method has an approximate precision to re-computing method, and then its recall ratio will obviously decline. It could be helpful for deciding when the matrix factorization must be recomputed.

Evaluation Of Recommendation System On A Movie Dataset

The standard application of Latent Semantic Indexing (LSI), a well-known technique for information retrieval, requires the computation of a partial Singular Value Decomposition (SVD) of the term-document matrix. This computation becomes infeasible for large document collections, since it is very demanding both in terms of arithmetic operations and in memory requirements. This paper discusses two divide and conquer strategies, with the goal of alleviating these difficulties. Both strategies divide the document collection in subsets, perform relevance analysis on each subset, and conquer the analysis results to form the query response. Since each sub-problem resulting from the recursive division process has a smaller size, the processing of large scale document collections requires much fewer resources. In addition, the computation is highly parallel and can be easily adapted to a parallel computing environment. To reduce the computational cost, we perform the analysis on the subsets by using the Lanczos vectors instead of singular vectors as in the standard LSI method. This technique is far more efficient than the computation of the truncated SVD, while its accuracy is comparable. Experimental results confirm that the proposed divide and conquer strategies are effective for information retrieval problems.

The exponential growth of data demands scalable infrastructures capable of indexing and searching rich content such as text, music, and images. A promising direction is to combine information re-trieval with peer-to-peer technology for scalability, fault-tolerance, and low administration cost. One pioneering work along this di-rection is pSearch [32, 33]. pSearch places documents onto a peer-to- peer overlay network according to semantic vectors produced using Latent Semantic Indexing (LSI). The search cost for a query is reduced since documents related to the query are likely to be co-located on a small number of nodes. Unfortunately, because of its reliance on LSI, pSearch also inherits the limitations of LSI. (1) When the corpus is large and heterogeneous, LSI's retrieval quality is inferior to methods such as Okapi. (2) The Singular Value Decomposition (SVD) used in LSI is unscalable in terms of both memory consumption and computation time.This paper addresses the above limitations of LSI and makes the following contributions. (1) To reduce the cost of SVD, we reduce the size of its input matrix through document clustering and term selection. Our method retains the retrieval quality of LSI but is several orders of magnitude more efficient. (2) Through extensive experimentation, we found that proper normalization of semantic vectors for terms and documents improves recall by 76%. (3) To further improve retrieval quality, we use low-dimensional subvectors of semantic vectors to cluster documents in the overlay and then use Okapi to guide the search and document selection.

Latent semantic analysis (LSA) usually uses the singular value decomposition (SVD) of the term-document matrix for discovering the latent relationships within the document collection. With the SVD, by disregarding the smaller singular values of the term-document matrix a vector space cleaned from noises that distort the meaning is obtained. The latent semantic structure of the terms and documents is obtained by examining the relationship of representative vectors in the vector space. However, the computational time of re-computing or updating the SVD of the term-document is high when adding new terms and/or documents to pre-existing document collection. Thus, the need a method not only has low computational complexity but also creates the correct semantic structure when updating the latent semantic structure is arisen. This study shows that the truncated ULV decomposition is a good alternative to the SVD in LSA modelling about cost and producing the correct semantic structure.

Search engine technology was used to find information as needed easily, quickly and efficiently, including in searching the information about the hadith which was a second guideline of life for muslim besides the Holy Qur'an. This study was aim to build a specialized search engine to find information about a complete and eleven hadith in Indonesian language. In this research, search engines worked by using latent semantic analysis (LSA) and cosine similarity based on the keywords entered. The LSA and cosine similarity methods were used in forming structured representations of text data as well as calculating the similarity of the keyword text entered with hadith text data, so the hadith information was issued in accordance with what was searched. Based on the results of the test conducted 50 times, it indicated that the LSA and cosine similarity had a success rate in finding high hadith information with an average recall value was 87.83%, although from all information obtained level of precision hadith was found semantically not many, it was indicated by the average precision value was 36.25%.

News articles categorization is a supervised learning approach in which news articles are assigned category labels based on likelihood demonstrated by a training set of labeled articles. A system for automatic categorization of news articles into a standard set of categories has been implemented. The proposed work will use Term Frequency–Inverse Document Frequency (TF-IDF) term weighting scheme for optimization of classification techniques to get more optimized results and use two supervised learning approaches, i.e., Support Vector Machine (SVM) and K-Nearest neighbor (kNN) and compare the performances of both classifiers. Each news document is preprocessed and transformed into a term-document matrix (Tsoumakas et al. in Data mining and knowledge discovery handbook. Springer, Berlin, pp 667–685 (2010) [1]). After preprocessing and transforming each news article into a vector of weights, TF-IDF term weighting scheme was used for weighting the word. TF-IDF weighted the words calculating the number of words that appear in a document. An unknown news item is also transformed into a vector of keyword weights, and then categorized into suitable categories such as Sports, Business, and Science and Technology. The system purposed in research work was trained on the collection of approximately 300 categorized news articles extracted from the various Indian newspaper websites and tested on a different set of 60 randomly extracted news items from the same sources (Trstenjak et al. in Proc Eng 69:1356–1364 (2014) [2], Buana et al. Int J Comput Appl 50:37–42 [3]). It has been observed that the performances of both algorithms improve when TF-IDF approach is used.

Text summarization is a technique for reducing lengthy passages of text into smaller portions. The goal is to develop a logical and fluent summary that only includes the document's major ideas. It's a important task in Natural Language Processing (NLP) that undoubtedly has a significant impact on synthesization of lengthy documents. With the rise of the digital documentation and publication, devoting time to thoughtfully read an article, document, or book in order to determine its relevance is no longer an option, especially given time constraints. In machine learning and NLP, automatic text summarization is a generic problem. A comparison of text summarization applying the Term Frequency ~ Inverse Document Frequency, Latent Semantic Analysis and Text Rank algorithms for the Malayalam language is presented in this research paper.