Comparative Analysis of Pretrained Encoder-Decoder Transformer Models for Extreme Text Summarization

The exponential growth of textual data, automatic text summarization has become an essential tool for extracting key information. It presents an end-to-end implementation of an abstractive text summarization model based on transformer architectures. The proposed system leverages a pre-trained language model, such as BART or T5 to generate coherent and concise summaries. The deployment is automated using GitHub Actions, ensuring a seamless CI/CD pipeline for real-world applications. Experimental results demonstrate the effectiveness of the model in generating high-quality summaries with improved ROUGE scores compared to baseline methods. This implementation provides a robust and automated solution for text summarization, making it adaptable for various NLP applications in academia and industry. Hence, automatic text summarization has become a desirable tool in today’s information age. It produces concise, fluent and readable summaries from larger bodies of text. Keywords: NLP, Abstractive Summarization, Transformers, Docker, GitHub-Actions, Cloud Deployment

Accurate genomic prediction of complex phenotypes is crucial for accelerating genetic progress in swine breeding. However, conventional methods like Genomic Best Linear Unbiased Prediction (GBLUP) face limitations in capturing complex non-additive effects that contribute significantly to phenotypic variation, restricting the potential accuracy of phenotype prediction. To address this challenge, we introduce a novel framework based on a self-supervised, pre-trained encoder-only Transformer model. Its core novelty lies in tokenizing SNP sequences into non-overlapping 6-mers (sequences of 6 SNPs), enabling the model to directly learn local haplotype patterns instead of treating SNPs as independent markers. The model first undergoes self-supervised pre-training on the unlabeled version of the same SNP dataset used for subsequent fine-tuning, learning intrinsic genomic representations through a masked 6-mer prediction task. Subsequently, the pre-trained model is fine-tuned on labeled data to predict phenotypic values for specific economic traits. Experimental validation demonstrates that our proposed model consistently outperforms baseline methods, including GBLUP and a Transformer of the same architecture trained from scratch (without pre-training), in prediction accuracy across key economic traits. This outperformance suggests the model's capacity to capture non-linear genetic signals missed by linear models. This research contributes not only a new, more accurate methodology for genomic phenotype prediction but also validates the potential of self-supervised learning to decipher complex genomic patterns for direct application in breeding programs. Ultimately, this approach offers a powerful new tool to enhance the rate of genetic gain in swine production by enabling more precise selection based on predicted phenotypes.

Biomedical text summarization is a critical task for comprehension of an ever-growing amount of biomedical literature. Pre-trained language models (PLMs) with transformer-based architectures have been shown to greatly improve performance in biomedical text mining tasks. However, existing methods for text summarization generally fine-tune PLMs on the target corpora directly and do not consider how fine-grained domain knowledge, such as PICO elements used in evidence-based medicine, can help to identify the context needed for generating coherent summaries. To fill the gap, we propose KeBioSum, a novel knowledge infusion training framework, and experiment using a number of PLMs as bases, for the task of extractive summarization on biomedical literature. We investigate generative and discriminative training techniques to fuse domain knowledge (i.e., PICO elements) into knowledge adapters and apply adapter fusion to efficiently inject the knowledge adapters into the basic PLMs for fine-tuning the extractive summarization task. Experimental results from the extractive summarization task on three biomedical literature datasets show that existing PLMs (BERT, RoBERTa, BioBERT, and PubMedBERT) are improved by incorporating the KeBioSum knowledge adapters, and our model outperforms the strong baselines.

Coffee bean production can encounter challenges due to fluctuations in global coffee prices, impacting the economic stability of some countries that heavily depend on coffee production. The primary objective is to evaluate how effectively various pre-trained models can predict coffee types using advanced deep learning techniques. The selection of an optimal pre-trained model is crucial, given the growing popularity of specialty coffee and the necessity for precise classification. We conducted a comprehensive comparison of several pre-trained models, including AlexNet, LeNet, HRNet, Google Net, Mobile V2 Net, ResNet (50), VGG, Efficient, Darknet, and DenseNet, utilizing a coffee-type dataset. By leveraging transfer learning and fine-tuning, we assess the generalization capabilities of the models for the coffee classification task. Our findings emphasize the substantial impact of the pre-trained model choice on the model's performance, with certain models demonstrating higher accuracy and faster convergence than conventional alternatives. This study offers a thorough evaluation of pre-trained architectural models regarding their effectiveness in coffee classification. Through the evaluation of result metrics, including sensitivity (1.0000), specificity (0.9917), precision (0.9924), negative predictive value (1.0000), accuracy (1.0000), and F1 score (0.9962), our analysis provides nuanced insights into the intricate landscape of pre-trained models.

While we have seen significant advances in automatic summarization for text, research on speech summarization is still limited. In this work, we address the challenge of automatically generating teasers for TED talks. In the first step, we create a corpus for automatic summarization of TED and TEDx talks consisting of the talks' recording, their transcripts and their descriptions. The corpus is used to build a speech summarization system for the task. We adapt and combine pre-trained models for automatic speech recognition (ASR) and text summarization using the collected data. This initial work shows that is more important to adapt the summarization model to the ASR transcripts than to adapt the ASR model to the talks.

The pre-training and fine-tuning paradigm has demonstrated its effectiveness and has become the standard approach for tailoring language models to various tasks. Currently, community-based platforms offer easy access to various pre-trained models, as anyone can publish without strict validation processes. However, a released pre-trained model can be a privacy trap for fine-tuning datasets if it is carefully designed. In this work, we propose PreCurious framework to reveal the new attack surface where the attacker releases the pre-trained model and gets a black-box access to the final fine-tuned model. PreCurious aims to escalate the general privacy risk of both membership inference and data extraction on the fine-tuning dataset. The key intuition behind PreCurious is to manipulate the memorization stage of the pre-trained model and guide fine-tuning with a seemingly legitimate configuration. While empirical and theoretical evidence suggests that parameter-efficient and differentially private fine-tuning techniques can defend against privacy attacks on a fine-tuned model, PreCurious demonstrates the possibility of breaking up this invulnerability in a stealthy manner compared to fine-tuning on a benign pre-trained model. While DP provides some mitigation for membership inference attack, by further leveraging a sanitized dataset, PreCurious demonstrates potential vulnerabilities for targeted data extraction even under differentially private tuning with a strict privacy budget e.g. . Thus, PreCurious raises warnings for users on the potential risks of downloading pre-trained models from unknown sources, relying solely on tutorials or common-sense defenses, and releasing sanitized datasets even after perfect scrubbing.

This paper introduces an approach to employ deep features for person re-identification. In contrast to existing works, we focus on using pre-trained deep models and their concept-based output to enhance attribute presentations of person images. There are two main contributions. First, we investigate recent state-of-the-art deep learning models for the task and provide a comprehensive evaluation. Second, we present an approach to improve identification accuracy of a standard attribute-based person re-identification approach. By using pre-trained models, we avoid re-training new deep models which always require a huge amount of training data and high computational cost. The idea is to utilize the correlation between generic concepts learned by the deep models and specific pre-defined attributes commonly used for person re-identification. We employ the deeply learned features from generic concepts to represent person images. These images and their manually annotated attributes are then used to train attribute classifiers. Given the classifiers, attributes of the probe and gallery images can be automatically extracted to compose attribute-based feature vectors. Re-identification is done by matching the vectors. Experiments conducted on several benchmark datasets demonstrated the effectiveness of the proposed approach.

Automatic text summarization is a subarea of natural language processing that generates a summary of the text by keeping its key points. The research work done on summarizing low-resourced language text is very limited. In India, the Hindi language is being spoken by central and north Indian people and only a few research works have been done on abstractive summarization of Hindi language. Having matras in Hindi makes it difficult to tokenize so it is difficult to summarize Hindi text using abstractive text summarization. In the proposed method, abstractive Hindi text summarization is done using transliteration and fine-tuning. In this work, the model is trained to generate both summaries and headlines. ROUGE-score and BERT-score have been utilized to check summary quality. A new semantic similarity score-based performance measure is also proposed to measure semantic similarity between reference summaries and predicted summaries. Using the proposed method, we have achieved the highest 55.16 ROUGE score, 0.80 BERT score, and 0.98 similarity score. Along with these performance measures, human evaluation of predicted summaries is also done and it is found that summaries and headlines were generated at a human-acceptable level.

目的 基于计算机的胸腔X线影像疾病检测和分类目前存在误诊率高，准确率低的问题。本文在视觉Transformer（vision Transformer，ViT）预训练模型的基础上，通过迁移学习方法，实现胸腔X线影像辅助诊断，提高诊断准确率和效率。方法 选用带有卷积神经网络（convolutional neural network，CNN）的ViT模型，其在超大规模自然图像数据集中进行了预训练；通过微调模型结构，使用预训练的ViT模型参数初始化主干网络，并迁移至胸腔X线影像数据集中再次训练，实现疾病多标签分类。结果 在IU X-Ray数据集中对ViT迁移学习前、后模型平均AUC（area under ROC curve）得分进行对比分析实验。结果表明，预训练ViT模型平均AUC得分为0.774，与不使用迁移学习相比提升了0.208。并针对模型结构和数据预处理进行了消融实验，对ViT中的注意力机制进行可视化，进一步验证了模型有效性。最后使用Chest X-Ray14和CheXpert数据集训练微调后的ViT模型，平均AUC得分为0.839和0.806，与对比方法相比分别有0.014～0.031的提升。结论 与其他方法相比，ViT模型胸腔X线影像的多标签分类精确度更高，且迁移学习可以在降低训练成本的同时提升ViT模型的分类性能和泛化性。消融实验与模型可视化表明，包含CNN结构的ViT模型能重点关注有意义的区域，高效获取胸腔X线影像的视觉特征。;Objective The chest X-ray-relevant screening and diagnostic method is essential for radiology nowadays. Most of chest X-ray images interpretation is still restricted by clinical experience and challenged for misdiagnose and missed diagnoses. To detect and identify one or more potential diseases in images automatically，it is beneficial for improving diagnostic efficiency and accuracy using computer-based technique. Compared to natural images，multiple lesions are challenged to be detected and distinguished accurately in a single image because abnormal areas have a small proportion and complex representations in chest X-ray images. Current convolutional neural network（CNN）based deep learning models have been widely used in the context of medical imaging. The structure of the CNN convolution kernel has sensitive to local detail information，and it is possible to extract richer image features. However，the convolution kernel cannot be used to get global information，and the features-extracted are restricted of redundant information like its relevance of background， muscles，and bones. The model’s performance in multi-label classification tasks are affected to a certain extent. At present，the vision Transformer（ViT）model has achieved its priorities in computer vision-related tasks. The ViT can be used to capture information simultaneously and effectively for multiple regions of the entire image. However，it is required to use large-scale dataset training to achieve good performance. Due to some factors like patient privacy and manual annotate costs，the size of the chest X-ray image data set has been limited. To reduce the model's dependence on data scale and improve the performance of multi-label classification，we develop the CNN-based ViT pre-training model in terms of the transfer learning method for diagnosis-assisted of chest X-ray image and multi-label classification. Method The CNN-based ViT model is pre-trained on a huge scale ground truth dataset，and it is used to obtain the initial parameters of the model. The model structure is fine-tuned according to the features of chest X-ray dataset. A 1×1 convolution layer is used to convert the chest X-ray images channels between 1 to 3. The number of output nodes of the linear layer in the classifier is balanced from 1 000 to the number of chest X-ray classification labels，and the Sigmoid is used as an activation function. The parameters of the backbone network are initialized in terms of the pre-trained ViT model parameters，and it is trained in the chest X-ray dataset after that to complete multi-label classification. The experiment is configured of Python3. 7 and PyTorch1. 8 to construct the model and RTX3090 GPU for training. Stochastic gradient descent（SGD）optimizer，binary cross-entropy（BCE）loss function，an initial learning rate of 1E-3，the cosine annealing learning rate decay are used. For training，each image is scaled to a size of 512×512 pixels，and a 224×224 pixels area and it is then cropped in random as the model input，and data augmentation is performed randomly by some of the flipping，perspective transformation， shearing，translation，zooming，and changing brightness. For testing，the chest X-ray image is scaled to 256×256 pixels and center crop a 224×224 area to input the trained model. Result The experiment is performed on the IU X-Ray，which is a small-scale chest X-ray dataset. This model is evaluated in quantitative using the average of area under ROC curve （AUC）scores across all classification labels. The results show that the average AUC score of the pre-trained ViT model is 0. 774. The accuracy and training efficiency of the non-pre-trained ViT model is dropped significantly. The average AUC score is reached to 0. 566 only，which is 0. 208 lower. In addition，the attention mechanism heat map is generated based on the ViT model，which can strengthen the interpretability of the model. A series of ablation experiments are carried out for data augmentation，model structure，and batch size design. The fine-tuned ViT model is trained on the Chest-Ray14 and CheXpert dataset as well. The average AUC score is reached to 0. 839 and 0. 806，which is optimized by 0. 014 and 0. 031. Conclusion A pre-trained ViT model is used for the multi-label classification of chest X-ray images via transfer learning. The experimental results illustrate that the ViT has its stronger multi-label classification performance in chest Xray images，and its attention mechanism is beneficial for lesions precision-focused like the interior of the chest cavity and the heart. Transfer learning is potential to improve the classification performance and model generalization of the ViT in small-scale datasets，and the training cost is reduced greatly. Ablation experiments demonstrate that the incorporated model of CNN and Transformer has its priority beyond single-structure model. Data enhancement and the batch size cutting can improve the performance of the model，but smaller scale of batch is still interlinked to longer training span. To improve the model's ability，we predict that future research direction can be focused on the extraction for complex disease and highlevel semantic information，such as their small lesions，disease location，and severity.

This Mental Wellness Chatbot is a software-based solution created for the purpose of solving the incomplete alleviation of depression symptoms in a person. This mental wellness Companion is a chatbot that act as a normal chatbot while mainly focus on the mental health of the user. To accomplish this, we are going to be use various techniques and algorithms to create this chatbot. We are going to use API token of a pre-trained hugging face model in order to create this chatbot rather than using the Large Language Model (LLM) locally since executing the chatbot locally requires a lot of GPU power in the device in order to create and embed the data that we are providing as a dataset. Mainly we are going to be using a pre-trained model from the open-source hugging face model and we are not going to be using OpenAI for completing it. For the database we are going to be using the vector database Facebook AI Similarity Search (FAISS) instead of Mangodb. The mental health Chatbot app is like a reliable friend in the pocket, always ready to lend an ear and offer support whenever you need it. Through gentle questions and personalized suggestions, it helps to navigate the emotions and find ways to feel better. By tracking persons progress over time, it creates a valuable record of emotional journey, empowering to understand oneself better. With 24/7 availability and a commitment to privacy and confidentiality, it provides a safe space for a person to express openly and honestly. Whether the person looking for coping strategies, educational insights, or connections to additional resources, the Chatbot app is there to guide a person on their path to emotional well-being.

This research introduces a summarization app using Flask and Hugging Face API. It summarizes audio, PDFs, and text efficiently. It features robust audio summarization by converting speech to text and then summarizing it. It also supports PDF and text summarization. The pre-trained NLP models like DistilBART, offers accurate and concise summaries. The user interface is intuitive, allowing seamless interaction. Overall, it's a comprehensive tool for extracting insights from various content sources, enhancing productivity.

Comparative analysis of deep convolution neural network models on small scale datasets

With the overflowing of textual data cross digital platforms, automatic text summarization has become increasingly essential for efficient information retrieval, knowledge distillation, and content comprehension. Abstractive summarization, in particular, seeks to produce concise and coherent summaries by paraphrasing and reinterpreting the original content rather than simply extracting key sentences.In this work, we propose an abstractive summarization approach based on large language models (LLMs), combining the strengths of Mistral 7B and BART. The proposed hybrid method aims to improve summary quality by generating outputs from both models, segmenting the summaries into individual sentences, and comparing them using cosine similarity to identify semantically aligned content. Subsequently, BERTScore is applied to select the most relevant sentence for inclusion in the final summary.To further enhance performance, the Mistral 7B model was fine-tuned on the CNN/DailyMail dataset using optimization techniques such as QLoRA, Gradient Checkpointing, LoRA, and GPTQ. Experimental results demonstrate that our approach produces more accurate and informative summaries, outperforming the individual models in both ROUGE and BERTScore metrics. These findings underline the potential of hybrid LLM fusion for generating higher-quality abstractive summaries.

Dealing with vast amounts of textual data requires the use of efficient systems. Automatic summarization systems are capable of addressing this issue. Therefore, it becomes highly essential to work on the design of existing automatic summarization systems and innovate them to make them capable of meeting the demands of continuously increasing data, based on user needs. This study tends to survey the scientific literature to obtain information and knowledge about the recent research in automatic text summarization specifically abstractive summarization based on neural networks. A review of various neural networks based abstractive summarization models have been presented. The proposed conceptual framework includes five key elements identified as encoder-decoder architecture, mechanisms, training strategies and optimization algorithms, dataset, and evaluation metric. A description of these elements is also included in this article. The purpose of this research is to provide an overall understanding and familiarity with the elements of recent neural networks based abstractive text summarization models with an up-to-date review as well as to render an awareness of the challenges and issues with these systems. Analysis has been performed qualitatively with the help of a concept matrix indicating common trends in the design of recent neural abstractive summarization systems. Models employing a transformer-based encoder-decoder architecture are found to be the new state-of-the-art. Based on the knowledge acquired from the survey, this article suggests the use of pre-trained language models in complement with neural network architecture for abstractive summarization task.

Automatic text summarization remains a very challenging task, especially in producing summaries that are semantically faithful and structurally well formed. Transformer models like T5 excel at capturing global contextual knowledge but tend to perform less effectively in handling subtle sequential dependencies in long documents. On the other hand, recurrent models like bidirectional LSTMs are proficient at modeling sequential order but are limited in capturing broader contextual relationships. To enhance these aspects, there is a proposed novel hybrid architecture HyT5 that combines the contextual power of Transformers with the sequential precision of LSTMs. By merging these complementary strengths, HyT5 generates summaries that are globally coherent while maintaining key sequential continuity. The model is trained and tested on the CNN/Daily Mail dataset, with preprocessing involving tokenization, truncation, and normalization to standardize input. Model optimization was stabilized through iterative check pointing during training. ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore evaluations show that HyT5 outperforms existing architectures consistently. The results highlight the strength of novel architectures in text summarization and position HyT5 as a promising approach for improved accuracy, fluency, and semantic fidelity in automatic summarization.