Comparative Analysis of Hindi Text Summarization for Multiple Documents by Padding of Ancillary Features

Text Summarization as a phenomenon has always been present and rather an evolving one with the advent of new technologies both in terms of data collection as well for the processing of this data. One reason of using text summarization is the huge amount of data floating over the internet in the form of text files, comments which is though potent enough to be used to extract useful information. but since the amount of text present in these sources is too huge, so the need of text summarization becomes justified by every argument. Some of the areas where text summarization is vastly used is applications involved in providing capsule information such as compact news applications, or websites providing academic notes for various examinations This paper presents an auto text summarizer application which takes the URL of a web page as input, performs summarization on the selected elements and then presents this summarized text content on the front end of a web application. At the backend, the process of scraping of web page content (if an http URL is provided as input) using beautiful soup library or reading of text provided takes place. news in short forms, or micro blogging websites. The scraped content after being preprocessed properly is summarized using a suitable library which in our case is one among NLTK, Spacy, Genism and Sumy. The summarized content is presented at the frontend using flask framework of Python. The results produced using different libraries are compared in the end in terms of reading time of the summarized content. The application uses extractive text summarization technique in order to achieve its result which is a compact summary of the textual data prepared from the keywords already present in the document.

Every day, the amount of textual data created increases exponentially, both in terms of complexity and volume. Massive amounts of information are generated by social media, news articles, emails, text messages and other resources, making it difficult to read lengthy language materials. Our main objective in the paper is to obtain a short understandable and fluent abstractive summary of any given text. The Abstractive Text Summarizer automatically gives the summary of the text by generating new phrase, rephrasing or using the new words which are not present in the original text. In this paper, a machine learning architecture i.e. Stacked LSTM based on attention mechanism using Sequence-to-Sequence model is proposed, to generate the summary using abstractive approach for Amazon reviews of fine foods dataset. Our approach allows the model to accept content and provide a concise summary that may clearly describe the gist of the original text. The experiments on Amazon reviews of fine foods dataset show that our model obtained BLEU Score as 0.91 for a test set.

Text summary has become essential in an era of information overload from various sources, including social media, news articles, emails, and text messages. Using the most recent developments in Deep Learning and Natural Language Processing, this project builds a model that can efficiently summarize large amounts of text into clear, concise summaries that improve comprehension and save time. The project is on abstractive text summarization using seq2seq and self-attention mechanism models like BART, PEGASUS, and T5 from Transformers and LSTM in Convolution Neural Networks. Utilizing the auto-regressive and bidirectional properties of the BART encoder-decoder framework, the proposed system design extensively integrates it to enhance summarization performance. The model is ready for sequential data processing after the code implementation, including text pretreatment and tokenization. Sophisticated optimization techniques are applied to improve model performance, and strategies like padding are employed to ensure consistent input sequence lengths. Testing outcomes validate the efficacy of BART's attention mechanisms, demonstrating the significance of the research as a breakthrough in precise and context-aware text summarization

A textual data processing task that involves the automatic extraction of relevant and salient keyphrases from a document that expresses all the important concepts of the document is called keyphrase extraction. Due to technological advancements, the amount of textual information on the Internet is rapidly increasing as a lot of textual information is processed online in various domains such as offices, news portals, or for research purposes. Given the exponential increase of news articles on the Internet, manually searching for similar news articles by reading the entire news content that matches the user’s interests has become a time-consuming and tedious task. Therefore, automatically finding similar news articles can be a significant task in text processing. In this context, keyphrase extraction algorithms can extract information from news articles. However, selecting the most appropriate algorithm is also a problem. Therefore, this study analyzes various supervised and unsupervised keyphrase extraction algorithms, namely KEA, KP-Miner, YAKE, MultipartiteRank, TopicRank, and TeKET, which are used to extract keyphrases from news articles. The extracted keyphrases are used to compute lexical and semantic similarity to find similar news articles. The lexical similarity is calculated using the Cosine and Jaccard similarity techniques. In addition, semantic similarity is calculated using a word embedding technique called Word2Vec in combination with the Cosine similarity measure. The experimental results show that the KP-Miner keyphrase extraction algorithm, together with the Cosine similarity calculation using Word2Vec (Cosine-Word2Vec), outperforms the other combinations of keyphrase extraction algorithms and similarity calculation techniques to find similar news articles. The similar articles identified using KPMiner and the Cosine similarity measure with Word2Vec appear to be relevant to a particular news article and thus show satisfactory performance with a Normalized Discounted Cumulative Gain (NDCG) value of 0.97. This study proposes a method for finding similar news articles that can be used in conjunction with other methods already in use.

Text summarization is the process of minimizing the text of a document to obtain a description that preserves the most important points of the original document, known as a text summary. Text summarization is the process of minimizing the text of a document to obtain a description that preserves the most important points of the original document, known as a text summary. Humans are finding it increasingly difficult to manually summarize vast documents of text as the amount of data grows by the day. This paper investigates record summarization in two ways i.e., Abstract and Extract text summarization.

This research aimed at finding relevances Vector Machine for summarization. The needed of producing an automatics text summarization create the research of text summarization continues to develop. One way to create an automatic summarization is by choosing the sentences which contain the main topics and reassembled them into a summary. The usage of Supports Vector Machine method (SVM) able to select summary sentences. The Relevance Vector Machine (RVM) appears as a further development of the SVM. This method performs a good result in a classification of Magnetic Resonance Imaging (MRI) data. Therefore, in this research, it examined the ability of RVM in the text summarization. Extracting the sentences used eight features, they are the length of the sentence, the sentence position, the containing of numerical data, the thematic words in the sentence, the similarity of the title, the sentence similarity, the sentence lexical cohesion before and after. There are 1509 training sentences and 214 testing sentences from 100 text documents. The result showed that using Radial Basis Function the accuracy of the RVM reached 63.084%. The RVM performance shows a better result than the SVM, 2% higher than the SVM result and uses fewer vector supports.

Text summarization is the process of creating a short, accurate and fluent summary of a longer text document. As plenty of digital data is available online, automatic text summarization methods greatly needed to help and understand the lengthy & complex documents quickly by discovering the relevant information. This paper proposes the text summarization method for short news articles and long scientific papers using unsupervised neural network model. The proposed method works in four main steps: input document pre-processing, feature extraction, feature enhancement and final summary generation. We have extracted combination of various statistical and linguistic features from input document, which helps in improving the quality of sentence selection. Further Restricted Boltzmann Machine (RBM) model is used to capture & enhance the discriminative, abstract features in an unsupervised way to improve the overall performance without losing any significant information. Sentences are scored based on enhanced feature set and top sentences are selected for final extractive summary. Performance of the proposed method is evaluated using Rouge score and compared with TextRank, LexRank, LSA & Luhn baseline methods and the results demonstrates that proposed methodology performs better compared to other methods.

Generating high quality gene clusters and identifying the underlying biological mechanism of the gene cluster are the important goals of clustering gene expression analysis. To get high quality cluster results, most of the current approaches rely on choosing the best cluster algorithm whose design biases and assumptions meet the underlying distribution of the data set. There are two issues for this approach: (1) usually the underlying data distribution of the gene expression data sets is unknown, and (2) there are so many clustering algorithms available and it is very challenging to choose the proper one. To provide a textual summary of the gene clusters, the most explored approach is the extractive approach that essentially builds upon techniques borrowed from the information retrieval, in which the objective is to provide terms to be used for query expansion, and not to act as a stand alone summary for the entire document sets. Another drawback is that the clustering quality and cluster interpretation are treated as two isolated research problems and are studied separately. But cluster quality and cluster interpretation are closely related and must be addressed in a coherent and unified way. It is essential to have relatively high quality clusters first, in order to get a correct, informative biological explanation of the gene cluster, otherwise, the biological explanation will be incorrect or misleading, no matter how good or robust the text summarization technique is. Based on this consideration, we design and develop a unified system GE-Miner (gene expression miner) to address these challenging issues in a principled and general manner by integrating cluster ensemble and text summarization and provide an environment for comprehensive gene expression data analysis. Experimental results demonstrate that our system can obtain high quality clusters and provide concise and informative textual summary for the gene clusters.

Text summarization is an important task in natural language processing (NLP). Neural summary models summarize information by understanding and rewriting documents through the encoder-decoder structure. Recent studies have sought to overcome the bias that cross-entropy-based learning methods can have through reinforcement learning (RL)-based learning methods or the problem of failing to learn optimized for metrics. However, the ROUGE metric with only <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> -gram matching is not a perfect solution. The purpose of this study is to improve the quality of the summary statement by proposing a reward function used in text summarization based on RL. We propose ROUGE-SIM and ROUGE-WMD, modified functions of the ROUGE function. ROUGE-SIM enables meaningfully similar words, in contrast to ROUGE-L. ROUGE-WMD is a function adding semantic similarity to ROUGE-L. The semantic similarity between articles and summary text was computed using Word Mover’s Distance (WMD) methodology. Our model with two proposed reward functions demonstrated superior performance on ROUGE-1, ROUGE-2, and ROUGE_L than on ROUGE-L as a reward function. Our two models, ROUGE-SIM and ROUGE-WMD, scored 0.418 and 0.406 for ROUGE-L, respectively, for the Gigaword dataset. The two reward functions outperformed ROUGE-L even in the abstractiveness and grammatical aspects.

In recent years, due to the popularity of the Internet, digital documents are growing at an exponential rate on the Web. To save time and quickly know about the document(s), a text summary system is required because the manual text summary takes time, effort, and cost, that produces a summary automatically from the source document(s). A summary contains key phrases and other related essential text material with no alteration of the key information and the general context of the source document(s). The text summary process began in the 1958 s, and researchers continue to try and improve the summary of texts. The process of summary is either extractive or abstractive. The extractive text summarization extracts the most appropriate sentences, phrases, or word from the text/document(s), then incorporates them in the summary while the abstractive text summarization system results in a summary of phrases other than the from the input text/document(s). This review paper describes preprocessing, features, methods, evaluations, and future directions in the extractive text summarization research. This study describes the advantages and shortcomings of each method and compares them using precision, recall, and F-score and finds that deep learning-based methods produce excellent results when adequate training summaries are available.

Automatic text summarization has very efficient techniques to create the summarization (important information form document) of the text document. As we know the information on the internet is growing rapidly and we cannot go through all the documents on the internet. If we want to read the news we have to identify which news we have to read then the news article can be summarized in only one paragraph by information extraction from the news article and by reading this short paragraph we can identify which article we should read. The automatic text summarization can be done by three methods. First one is the extraction method in which we just find the informative sentences from the whole document text. The second approach is the abstraction technique in which we find the meaning of the sentences and try to formulate the paragraph in the same meaning manner or we can say the same thing express the fewer words. The third and last approach is compression technique in which compress each sentence of the document. We compress the sentence by having the words related to the meaning of the sentence by title or first sentence of the graph. We can apply the hybrid approach to create the summarization in which we first compress the whole text document and then apply knowledge extraction. Knowledge extraction can be done by giving the score to each sentence and then pick the best k sentences according to compression ratio.

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics

Evaluation of Content Compaction in Assamese Language

The aim of this study is to evaluate how effective the Lexrank algorithm and Latent semantic analysis (LSA) are in automatic text summarization for the Indonesian language. This research focuses on natural language processing and handling of excessive data. We applied both algorithms to generate text summaries using the INDOSUM dataset, which contains about 20,000 news articles in Indonesian with manual summaries. To assess performance, the ROUGE metric was used, which includes aspects of precision, recall, and F1 score. In all tested metrics, LSA outperformed Lexrank. LSA had a precision of 0.57, recall of 0.67, and an F1 score of 0.59, whereas Lexrank had a precision of 0.46, recall of 0.52, and an F1 score of 0.48. These result indicate that LSA is better at gathering important information from the original text than Lexrank.

Existence of large amount of textual information available on the internet emerged serious research in the area of machine generated summarization. Manual summarization of these online text documents is a very difficult task for human beings. So we need an automatic text summarizer. Automatic Text Summarization (ATS) is “condensing the source text into a shorter version, while preserving its information content and overall meaning”. Even though the work of automatic text summarization started in 1950's, still it is lacking to achieve more coherent and meaningful summaries. The proposed approach provides automatic feature based extractive heading wise text summarizer to improve the coherence thereby improving the understandability of the summary text. It summarizes the given input document using local scoring and local ranking that is it provides heading wise summary. Headings of a document give contextual information and permit visual scanning of the document to find the search contents. The proposed approach applies the same features to all document sentences. But it ranks the sentences heading wise and selects top n sentences from each heading where n depends upon compression ratio. The final heading wise summary produced by this approach is a collection of summary of individual headings. Since the heading wise summary contains the equal proportion of sentences from each heading, it reduces the coherent gap of the summary text. Also it improves the overall meaning and understanding of the summary text. The outcomes of the experiment clearly show that heading wise summarizer provides better precision, recall and f-measure over the main summarizer, Ms-word summarizer, free summarizer and Auto summarizer.