Natural Language Processing-Driven Document Summarization Using Attention-Guided Memory-Augmented Transformer

Abstract

Translate Article

In recent years, Document summarization has become an important area in Natural Language Processing (NLP) with the growth of long-rich texts, such as the Government Report (GovReport). Traditional transformer-based models achieve strong results on short texts, although they face challenges with very long documents owing to the loss of contextual information; thus, the segmentation methods and chain-of-thought prompting improved coverage. Hence, this research proposes an Attention-Guided Memory-Augmented Transformer (AGMGT) for long-document summarization with the help of the GovReport dataset, which contains government research reports and their expert-written summaries. The documents are then segmented and encoded using transformer encoders to manage the context length. Furthermore, a memory bank is constructed by storing salient information identified through attention scoring. Subsequently, cross-memory attention is applied for chunk-level micro-summarization to maintain consistency across sections. Then, the micro-summaries are aggregated into a global outline to generate a final summary using hierarchical attention over the outlines. Finally, the model is trained with cross-entropy and auxiliary coverage losses to generate concise and factually correct summaries of long government reports. The proposed AGMGT achieved better results in terms of ROUGE-1 (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{0. 8 2 2 1}$</tex>), ROUGE-2 (0.6432), and ROUGE-L (0.7932) than the existing multilingual transformer 5 (mT5-Large) model.