Automatic text summarization with statistical and linguistic features using successive thresholds

Automatic Text Summarization techniques in Assamese language is still in an immature stage compared to other Indian languages. In this study, we have reviewed some of the studies in other Indian languages for having a better understanding of the problems. Furthermore, we have also put in our efforts to understand keeping in view the opinion of Assamese linguists as how we could overcome the shortcomings of Assamese Text Summarization and provide an effective solution that could serve as fundamental concepts in this particular field. An extractive method for text summarization is proposed in this paper to evaluate the sentences of the input document based on the merging of semantic and statistical characteristics. It is applied considering the importance of sentence and diversity. Moreover, we have applied Synsets from Assamese WordNet to calculate the word frequency. Besides identifying and weighing the salience sentences, the concept of vector centrality and cosine similarity is used and the result is represented in graph representation. A threshold value to indicate the important sentences is applied. The effectiveness of our proposed method is demonstrated through a set of experiments conducted using ROUGE measure and the evaluation is showed in terms of Recall.

Text summarization is the process of creating a short, accurate and fluent summary of a longer text document. As plenty of digital data is available online, automatic text summarization methods greatly needed to help and understand the lengthy & complex documents quickly by discovering the relevant information. This paper proposes the text summarization method for short news articles and long scientific papers using unsupervised neural network model. The proposed method works in four main steps: input document pre-processing, feature extraction, feature enhancement and final summary generation. We have extracted combination of various statistical and linguistic features from input document, which helps in improving the quality of sentence selection. Further Restricted Boltzmann Machine (RBM) model is used to capture & enhance the discriminative, abstract features in an unsupervised way to improve the overall performance without losing any significant information. Sentences are scored based on enhanced feature set and top sentences are selected for final extractive summary. Performance of the proposed method is evaluated using Rouge score and compared with TextRank, LexRank, LSA & Luhn baseline methods and the results demonstrates that proposed methodology performs better compared to other methods.

As a typical unsupervised learning method, the TextRank algorithm performs well for large-scale text mining, especially for automatic summarization or keyword extraction. However, TextRank only considers the similarities between sentences in the processes of automatic summarization and neglects information about text structure and context. To overcome these shortcomings, the authors propose an improved highly-scalable method, called iTextRank. When building a TextRank graph in their new method, the authors compute sentence similarities and adjust the weights of nodes by considering statistical and linguistic features, such as similarities in titles, paragraph structures, special sentences, sentence positions and lengths. Their analysis shows that the time complexity of iTextRank is comparable with TextRank. More importantly, two experiments show that iTextRank has a higher accuracy and lower recall rate than TextRank, and it is as effective as several popular online automatic summarization systems.

Evaluation of Content Compaction in Assamese Language

Due to the exponential overflow of textual information in various fields of knowledge and on the internet, it is very challenging to extract important information or to generate a summary from some multi-document collection in a specific field. With such a gigantic amount of textual content, human text summarization becomes impractical since it is expensive and consumes a lot of time and effort. So, developing automatic text summarization (ATS) systems is becoming increasingly essential. ATS approaches are either extractive or abstractive. The extractive approach is simpler and faster than the abstractive approach. This work proposes an extractive ATS system that aims to extract a small subset of sentences from a large multi-document text. First, the whole text is preprocessed by applying some natural language processing techniques such as sentences segmentation, words tokenization, removal of stop-words, and stemming to provide a structured representation of the original document collection. Based on this structured representation, the ATS problem is formulated as a multi-objective optimization (MOO) problem that optimizes the extracted summary to maintain the coverage of the main text content while avoiding redundant information. Secondly, an evolutionary sparse multi-objective algorithm is developed to solve the formulated large-scale MOO. The output of this algorithm is a set of non-dominated summaries (Pareto front). A novel criterion is proposed to select the target summary from the Pareto front. The proposed ATS system has been examined using (DUC) datasets, and the output summaries have been evaluated using (ROUGE) metrics and compared with the literature.

Extractive text summarization selects the most important sentences from a document, preserves their original meaning, and produces an objective and fact-based summary. It is faster and less computationally intensive than abstract summarization techniques. Learning cross-sentence relationships is crucial for extractive text summarization. However, most of the language models currently in use process text data sequentially, which makes it difficult to capture such inter-sentence relations, especially in long documents. This paper proposes an extractive summarization model based on the graph neural network (GNN) to address this problem. The model effectively represents cross-sentence relationships using a graph-structured document representation. In addition to sentence nodes, we introduce two nodes with different granularity in the graph structure, words and topics, which bring different levels of semantic information. The node representations are updated by the graph attention network (GAT). The final summary is obtained using the binary classification of the sentence nodes. Our text summarization method was demonstrated to be highly effective, as supported by the results of our experiments on the CNN/DM and NYT datasets. To be specific, our approach outperformed baseline models of the same type in terms of ROUGE scores on both datasets, indicating the potential of our proposed model for enhancing text summarization tasks.

How to Improve Text Summarization and Classification by Mutual Cooperation on an Integrated Framework

The explosion of online and offline data has changed how we gather, evaluate, and understand data. It is frequently difficult and time-consuming to comprehend large text documents and extract crucial information from them. Text summarization techniques address the mentioned problems by compressing long texts while retaining their essential contents. These techniques rely on the fast delivery of filtered, high-quality content to their users. Due to the massive amounts of data generated by technology and various sources, automated text summarization of large-scale data is challenging. There are three types of automatic text summarization techniques: extractive, abstractive, and hybrid. Regardless of these previous techniques, the generated summaries are a long way from the summarization produced by human experts. Although Arabic is a widely spoken language that is frequently used for content sharing on the web, Arabic text summarization of Arabic content is limited and still immature because of several problems, including the Arabic language’s morphological structure, the variety of dialects, and the lack of adequate data sources. This paper reviews text summarization approaches and recent deep learning models for this approach. Additionally, it focuses on existing datasets for these approaches, which are also reviewed, along with their characteristics and limitations. The most often used metrics for summarization quality evaluation are ROUGE1, ROUGE2, ROUGE L, and Bleu. The challenges that are encountered during Arabic text summarizing methods and approaches and the solutions proposed in each approach are analyzed. Many Arabic text summarization methods have problems, such as the lack of golden tokens during testing, being out of vocabulary (OOV) words, repeating summary sentences, lack of standard systematic methodologies and architectures, and the complexity of the Arabic language. Finally, providing the required corpora, improving evaluation using semantic representations, the lack of using rouge metrics in abstractive text summarization, and using recent deep learning models to adopt them in Arabic summarization studies is an essential demand.

Text summarization is a process of distilling the most important content from text documents. While human beings have proven to be extremely capable summarizers, computer based automatic abstracting and summarizing has proven to be extremely challenging tasks. In this paper we report our experience with applying extractive summarization techniques to process news articles, economic reports and nursing narratives. We present analysis of the effect of different summarization methods and parameters on the summarization results. We also compare the performance of the summarizers across the three different document genres. The learned lessons are discussed and the possibilities for applying the theory of Computing with Words in text summarization are elaborated.

Recent advances in technology have made tremendous amounts of multimedia information available to the general population. An efficient way of dealing with this new development is to develop browsing tools that distill multimedia data as information oriented summaries. Such an approach will not only suit resource poor environments such as wireless and mobile, but also enhance browsing on the wired side for applications like digital libraries and repositories. Automatic summarization and indexing techniques will give users an opportunity to browse and select multimedia document of their choice for complete viewing later. In this paper, we present a technique by which we can automatically gather the frames of interest in a video for purposes of summarization. Our proposed technique is based on using Delaunay Triangulation for clustering the frames in videos. We represent the frame contents as multi-dimensional point data and use Delaunay Triangulation for clustering them. We propose a novel video summarization technique by using Delaunay clusters that generates good quality summaries with fewer frames and less redundancy when compared to other schemes. In contrast to many of the other clustering techniques, the Delaunay clustering algorithm is fully automatic with no user specified parameters and is well suited for batch processing. We demonstrate these and other desirable properties of the proposed algorithm by testing it on a collection of videos from Open Video Project. We provide a meaningful comparison between results of the proposed summarization technique with Open Video storyboard and K-means clustering. We evaluate the results in terms of metrics that measure the content representational value of the proposed technique.

Text Summarization is condensing the source text into a shorter version preserving its information content and overall meaning. It is very difficult for human beings to manually summarize large documents of text. The text summarization method consists of selecting important sentences, paragraphs etc. from the original document and converting them into a star map. The importance of sentences is decided based on statistical and linguistic features of sentences. A star map summarization method consists of understanding the original text and re-telling it in fewer words. This star map is used for duplicate elimination, exam paper evaluator, lesson planning, Identify Shingling. A Pioneering Tool For Text Summarization using Star Map has been presented.

Text summarization is a very important function in next generation e-business-intelligence-service. While human beings have proven to be extremely capable summarizers, computer based automated abstracting and summarizing has proven to be extremely challenging tasks. The dominant approach to text summarization is selection-based, by which the most content-bearing sentences or passages are identified and selected to compose a summary. However, the results from such a process often suffer from flaws such as incoherent content and poor readability due to unclear relationships between the selected text excerpts, dangling references, and so on. In this paper, we present a process model of topic guided text summarization - TIDE. By directing the content selection process with identified topic structure of the text, the logical relation between the selected sentences is captured and better presented, which is also valuable resource for a more complete and precise understanding of text meaning in the next stage of processing.

Abstractive text summarization: State of the art, challenges, and improvements

We present a new task of identifying core term (CT) of knowledge unit (KU) from text for knowledge management and service. Two kinds of approaches, including binary classification using naive bayesian, decision tree, logistic regression and SVM, as well as competition learning based on pairwise classification, are investigated for this specific task, combined with presented rich feature set from position, token features to statistic and linguistic features. Experimental results show that simple classification method can effectively address this task with desirable performance at 82.7% KU accuracy. However, since the recognition of core term relies on the KU as an integer and all its inner terms, competition learning based on pairwise classification achieves better result at 89.6%. We also empirically show that all of the presented types of features are useful for our task, and the combination of position and linguistic features is essential for information extraction on short text.

Text Summarization as a phenomenon has always been present and rather an evolving one with the advent of new technologies both in terms of data collection as well for the processing of this data. One reason of using text summarization is the huge amount of data floating over the internet in the form of text files, comments which is though potent enough to be used to extract useful information. but since the amount of text present in these sources is too huge, so the need of text summarization becomes justified by every argument. Some of the areas where text summarization is vastly used is applications involved in providing capsule information such as compact news applications, or websites providing academic notes for various examinations This paper presents an auto text summarizer application which takes the URL of a web page as input, performs summarization on the selected elements and then presents this summarized text content on the front end of a web application. At the backend, the process of scraping of web page content (if an http URL is provided as input) using beautiful soup library or reading of text provided takes place. news in short forms, or micro blogging websites. The scraped content after being preprocessed properly is summarized using a suitable library which in our case is one among NLTK, Spacy, Genism and Sumy. The summarized content is presented at the frontend using flask framework of Python. The results produced using different libraries are compared in the end in terms of reading time of the summarized content. The application uses extractive text summarization technique in order to achieve its result which is a compact summary of the textual data prepared from the keywords already present in the document.