A survey of low-bit large language models: Basics, systems, and algorithms.

Picture description tasks, such as the Cookie Theft task, are widely used in neuropsychological assessments to detect cognitive impairment. However, manual scoring is time-consuming, requires specialized training, and is subject to interrater variability. Recent advancements in natural language processing, particularly large language models (LLMs), offer a promising solution to automate and standardize the scoring process. This study evaluated the performance and reliability of five LLMs (GPT-4 Turbo, GPT-4o, Claude 3 Opus, Claude 3 Sonnet, and Llama 3 70b) in scoring the Cookie Theft picture description task. A subset of 25 participants were selected from the DementiaBank corpus. The LLMs were tasked with scoring 22 content units in the participants' responses using various prompt strategies, including few-shot learning, prompt chaining, and self-consistency. LLM performance was compared to the consensus score of three human raters. LLMs demonstrated comparable accuracy to human raters in scoring the Cookie Theft task, with no significant differences in mean absolute error (MAE) between the best performing models and human raters. Few-shot learning significantly improved LLM performance, while prompt chaining and self-consistency showed limited benefits. Claude 3 Opus and GPT-4o exhibited the highest accuracy and reliability. Notably, LLMs showed significantly higher interrater reliability compared to human raters. The findings demonstrate the potential of LLMs to accurately and reliably score picture description tasks, offering a promising approach to streamline and standardize neuropsychological assessments. By automating the scoring process, clinicians and researchers can benefit from increased efficiency, reduced subjectivity, and improved scalability in evaluating cognitive functions.

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

The rapid development of Large Language Models (LLMs) has brought significant advancements in natural language processing, but their high demands for computational resources, memory, and energy pose significant challenges. This report addresses the need for more energy-efficient LLMs by exploring the interconnected factors of computation, memory, energy, cost, and network communication in optimizing these models. Through a detailed literature review, the report evaluates existing strategies in model architecture, pre-training, fine-tuning, and inference to improve LLM efficiency. Furthermore, it introduces a novel unified metric designed to assess resource efficiency across various dimensions. The report also examines optimization techniques such as model pruning, quantization, and hardware acceleration to improve LLM performance and sustainability. The findings emphasize that achieving scalable, efficient, and sustainable LLMs requires a balanced approach that includes energy-conscious design, optimized training processes, and effective resource management.

Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

Two main objectives of Computational Textual Criticism are the development of algorithms for tree and text reconstruction under conditions of imperfect copying. Despite recent developments in the field, few comparative studies or benchmarks have been performed, particularly in the case of text reconstruction. On the other hand, recent advancements in Natural Language Processing (NLP) have begun to impact various aspects of the humanities and social sciences. In this paper, we incorporate various NLP techniques (including Large Language Models (LLMs)) to simulate text transmission and benchmark different tree and text reconstruction algorithms. In addition, for text reconstruction, we incorporate LLMs to improve the final result. Our results show that the Unweighted Pair Group Method with Arithmetic Mean (UPGMA) or Neighbor Joining (NJ) methods combined with the Levenshtein metric achieved superior comparative results for tree reconstruction. Moreover, for text reconstruction, we found that the Simple Majority Rule (SMR), UR, and Roos-Heikkilä-Myllymäki (RHM) methods yielded consistent results, and in most cases, the incorporation of LLMs improved the final output.

Large Language Models (LLMs) have ushered in a transformative era in Natural Language Processing (NLP), reshaping research and extending NLP's influence to other fields of study. However, there is little to no work examining the degree to which LLMs influence other research fields. This work empirically and systematically examines the influence and use of LLMs in fields beyond NLP. We curate 106 LLMs and analyze ∼148k papers citing LLMs to quantify their influence and reveal trends in their usage patterns. Our analysis reveals not only the increasing prevalence of LLMs in non-CS fields but also the disparities in their usage, with some fields utilizing them more frequently than others since 2018, notably Linguistics and Engineering together accounting for ∼45% of LLM citations. Our findings further indicate that most of these fields predominantly employ task-agnostic LLMs, proficient in zero or few-shot learning without requiring further fine-tuning, to address their domain-specific problems. This study sheds light on the cross-disciplinary impact of NLP through LLMs, providing a better understanding of the opportunities and challenges.

Mental health diagnoses possess unique challenges that often lead to nuanced difficulties in managing an individual’s well-being and daily functioning. Self-report questionnaires are a common practice in clinical settings to help mitigate the challenges involved in mental health disorder screening. However, these questionnaires rely on an individual’s subjective response which can be influenced by various factors. Despite the advancements of Large Language Models (LLMs), quantifying self-reported experiences with natural language processing has resulted in imperfect accuracy. This project aims to demonstrate the effectiveness of zero-shot learning LLMs for screening and assessing item scales for depression using LLMs. The DAIC-WOZ is a publicly available mental health dataset that contains textual data from clinical interviews and self-report questionnaires with relevant mental health disorder labels. The RISEN prompt engineering framework was utilized to evaluate LLMs’ effectiveness in predicting depression symptoms based on individual PHQ-8 items. Various LLMs, including GPT models, Llama3_8B, Cohere, and Gemini were assessed based on performance. The GPT models, especially GPT-4o, were consistently better than other LLMs (Llama3_8B, Cohere, Gemini) across all eight items of the PHQ-8 scale in accuracy (M = 75.9%), and F1 score (0.74). GPT models were able to predict PHQ-8 items related to emotional and cognitive states. Llama 3_8B demonstrated superior detection of anhedonia-related symptoms and the Cohere LLM’s strength was identifying and predicting psychomotor activity symptoms. This study provides a novel outlook on the potential of LLMs for predicting self-reported questionnaire scores from textual interview data. The promising preliminary performance of the various models indicates there is potential that these models could effectively assist in the screening of depression. Further research is needed to establish a framework for which LLM can be used for specific mental health symptoms and other disorders. As well, analysis of additional datasets while fine-tuning models should be explored.

Large Language Models (LLMs) recently demonstrated extraordinary capability, including natural language processing (NLP), language translation, text generation, question answering, etc. Moreover, LLMs are a new and essential part of computerized language processing, having the ability to understand complex verbal patterns and generate coherent and appropriate replies for the situation. Though this success of LLMs has prompted a substantial increase in research contributions, rapid growth has made it difficult to understand the overall impact of these improvements. Since a lot of new research on LLMs is coming out quickly, it is getting tough to get an overview of all of them in a short note. Consequently, the research community would benefit from a short but thorough review of the recent changes in this area. This article thoroughly overviews LLMs, including their history, architectures, transformers, resources, training methods, applications, impacts, challenges, etc. This paper begins by discussing the fundamental concepts of LLMs with its traditional pipeline of the LLMs training phase. It then provides an overview of the existing works, the history of LLMs, their evolution over time, the architecture of transformers in LLMs, the different resources of LLMs, and the different training methods that have been used to train them. It also demonstrated the datasets utilized in the studies. After that, the paper discusses the wide range of applications of LLMs, including biomedical and healthcare, education, social, business, and agriculture. It also illustrates how LLMs create an impact on society and shape the future of AI and how they can be used to solve real-world problems. Then it also explores open issues and challenges to deploying LLMs in real-world scenario. Our review paper aims to help practitioners, researchers, and experts thoroughly understand the evolution of LLMs, pre-trained architectures, applications, challenges, and future goals.

Recent breakthroughs in artificial intelligence (AI) research include advancements in natural language processing (NLP) achieved by large language models (LLMs), and; in particular, generative pre–trained transformer (GPT) architectures. The latest GPT developed by OpenAI, GPT–4, has shown remarkable intelligence across various domains and tasks. It exhibits capabilities in abstraction, comprehension, vision, computer coding, mathematics, and more, suggesting it to be a significant step towards artificial general intelligence (AGI), a level of AI that possesses capabilities similar to human intelligence. This paper explores this AGI, its knowledge diffusive and societal influences, and its governance. In addition to coverage of the major associated topics studied in the literature, and making up for their loopholes, we scrutinize how GPT-4 can facilitate the diffusion of knowledge across different areas of science by promoting their interpretability and explainability (IE) to inexperts. Where applicable, the topics are also accompanied by their specific potential implications on medical imaging.

Place semantics refers to the collective socio-cultural values, experiences, and functions that people associate with specific geographic locations, turning mere spaces into "places imbued with meaning" (Tuan, 1977) . Studying place semantics has traditionally spanned several disciplines, including geography, sociology, environmental psychology, and geographic information science, each contributing different perspectives on how places are perceived and valued. The advent of user-generated content platforms, which provide vast quantities of geographically referenced textual data, has opened new avenues for the study of place semantics. Geosocial media data, such as tweets or textual metadata from geotagged images shared on Flickr and Instagram, offers insights into human activities, behaviors, and perceptions, thereby offering new opportunities to extract place semantics and analyze spatiotemporal interactions between people and places. (Purves & Derungs, 2015; Adams & McKenzie, 2013) . Nevertheless, processing natural language data at scale presents significant challenges and necessitates the use of efficient text mining algorithms to manage the vast amounts of data involved. Recent advancements in natural language processing (NLP), especially the development of transformer based and large language models, have introduced powerful new methods for efficient information retrieval, greatly enhancing our ability to derive meaningful insights from extensive geosocial media datasets.

Conventional requirements engineering (RE) is often a manual process that is tedious, time-consuming, susceptible to errors, and hard to scale in case of continuous updates. Recent advances in natural language processing (NLP) transformed the RE process via automation. However, automation works better with structured data and formal requirement formats, limiting the ability to elicit and extract requirements from unstructured data; the rapid growth of digital platforms accentuates the need for treating user posts, comments, opinions and feedback as primary data sources for requirements mining and modeling. However, research conducted on unstructured Egyptian Arabic text is largely underexplored, due to its inherent linguistic challenges. This paper fills a critical gap in Arabic NLP, by providing an automated feature extraction and UML modeling pipeline utilizing state of the art embedding models, density-based clustering algorithm, and Large Language Models (LLMs), specifically engineered for handling Egyptian-Arabic text. The proposed pipeline demonstrates strong positive results, producing quality features and meaningful use case diagrams, and further confirms the feasibility of applying this advanced NLP-based pipeline to real world Egyptian dialectal data.

<p dir="ltr">Recent advancements in Natural Language Processing (NLP), specifically Large Language Models (LLMs), have demonstrated their transformative power in various domains, including software engineering. This thesis focuses on addressing the critical problem of conflict and duplicate requirement identification in requirement engineering, leveraging the potential of structured and non-structured data in software engineering for automation. For this purpose, first, a diverse set of requirements are annotated for entity recognition tasks, enabling the creation of machine learning and transformer-based models which provides 95.8% accuracy in identifying software-specific entities from requirement texts present in DOORS dataset. Second, building upon this foundation, a two-phase algorithm is developed for conflict detection in requirement documents. The algorithm utilizes supervised learning on requirement data to identify potential conflicts and subsequently validates them semantically using entity extraction techniques. Third, to further refine the conflict and duplicate identification process, the problem is formulated as a sentence pair classification task. Transfer learning techniques, such as sequential transfer learning and cross-domain transfer learning, are employed, and a novel architecture called Software-Requirement Bi-directional Encoder Representation Transformer (SR-BERT) is proposed for requirement pair classification. SR-BERT achieves 95.3% F1-score in determining conflicts, duplicate, and neutral requirement pairs. Additionally, misclassifications flagged by cross-domain transfer learning are reevaluated using Actor-Action (AA) extraction, Part of Speech (POS) tagging, and Semantic Role Labeling (SRL) techniques. The requirement pair classification approach demonstrates great potential in conflict detection tasks, and its performance is enhanced by integrating bi-directional encoders with a contrastive learning framework and cross-encoders. Fourth, inspired by recent advancements in prompt-based learning, a Prompt-based Text-to-Text Models for Software Requirement Pairs (PT2-SRP) framework is also developed, leveraging specially crafted prompts for requirement pair datasets and training encoder-decoder models to identify conflicts in requirement pairs. Lastly, the impact of Data Augmentation (DA) on requirement conflict identification is examined. Specif ically, during the curation of datasets for requirement pair classification task, a scarcity of conflict requirement pairs is noted. To overcome this limitation, novel data augmentation techniques are devised, which ensure the generation of meaningful data instances while preserving the class labels in requirement pair datasets.</p>

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1

Semantical text understanding holds significant importance in natural language processing (NLP). Numerous datasets, such as Quora Question Pairs (QQP), have been devised for this purpose. In our previous study, we developed a Siamese Convolutional Neural Network (S-CNN) that achieved an F1 score of 82.02% (95% C.I.: 81.83%-82.20%). Given the growing attention toward large language models (LLMs) like ChatGPT, we aimed to explore their effectiveness in text similarity tasks. In this research, we leveraged 5 pretrained LLMs, conducted various fine-tuning approaches (prompt engineering, n-shot learning, and supervised learning using the low-rank adaptation [LoRA]), and compared their performance using F1 score. To ensure a fair comparison, we followed our previous study's design and dataset by employing a 10-fold cross-validation for supervised model training and evaluation. Additionally, we conducted a secondary study by introducing a recent larger LLM with 70B parameters and comparing it with the 7B model using the GLUE benchmark, and both models were finetuned with the corpus. The fine-tuned LLaMA model with 7B parameters (qLLaMA_LoRA-7B) using 100,000 QQP corpus yielded the best results, achieving an F1 score of 84.9% (95% C.I.: 84.13%-85.67%), which outperformed the Alpaca_LoRA-65B (finetuned based on LLaMA-65B) (F1: 64.98% [64.72%-65.25%]; P<0.01) and had a 3% improvement compared to our previously published best model, S-CNN. The finetuned LLaMA3.1-70B (qLLaMA3.1_LoRA-70B) with 70B parameters (F1: 74.4%) outperformed the qLLaMA_LoRA-7B (F1: 71.9%) using the GLUE benchmark. The study demonstrated an effective LLM finetuning framework, which highlights the importance of finetuning LLMs for improved performance. Our task-specific supervised finetuning demonstrated improved LLM performance compared to larger pretrained models with or without n-shot learning; moreover, finetuning a larger LLM further improved performance compared to finetuning a smaller LLM. Our LLM-based finetuning framework may potentially improve various document similarity tasks, such as matching resumes with job descriptions, recommending subject-matter experts, or identifying potential reviewers for grant proposals or manuscript submissions.