System Design for Automating Incident Update Responses Using Large Language Models

We introduce a novel writing method called Probing Chain-of-Thought, which potentially prevents students from cheating using a large language model while enhancing their critical thinking. large language models have disrupted education and many other fields. For fear of students cheating, many educationists have resorted to banning their use. We conduct studies in two different courses with 65 students using qualitative research design primarily (i.e. phenomenological) and quantitative methods. The students in each course were asked to prompt a large language model of their choice with one question from a set of four (random) questions and required to affirm or refute statements in the large language model output by using peer-reviewed references as evidence. In addition, the rubric for assessing the students writing included 5 more criteria: focus, logic, content, style and correctness. The average success rate of the writing of students based on the criteria for the two cases is 79.49% (±12.82%). The results of the rubric assessment show two things: (1) Probing Chain-of-Thought stimulates critical thinking and writing of students through engagement with large language models when we compare the large language models-only output to Probing Chain-of-Thought output and (2) Probing Chain-of-Thought may prevent cheating because of clear limitations in the concerned large language models when we compare students’ Probing Chain-of-Thought output to large language models’ Probing Chain-of-Thought output. In quantitative analysis, we also discover that most students prefer to give answers in fewer words than large language models, which are typically verbose. The average word counts for students in the first course, ChatGPT 3.5, and Phind (v8) are 208, 391 and 383, respectively, while it is 405, 356, and 315 for students, ChatGPT 3.5, and BingAI, respectively, in the second course, where we enforced a minimum word-count of 300 for the students. We provide access to the outputs for possible assessments (available after review).

Large Language Models (LLMs) have demonstrated remarkable capabilities, leading to a significant increase in user demand for LLM services. However, cloud-based LLM services often suffer from high latency, unstable responsiveness, and privacy concerns. Therefore, multiple LLMs are usually deployed at the network edge to boost real-time responsiveness and protect data privacy, particularly for many emerging smart mobile and IoT applications. Given the varying response quality and latency of LLM services, a critical issue is how to route user requests from mobile and IoT devices to an appropriate LLM service (i.e., edge LLM expert) to ensure acceptable quality-of-service (QoS). Existing routing algorithms fail to simultaneously address the heterogeneity of LLM services, the interference among requests, and the dynamic workloads necessary for maintaining long-term stable QoS. To meet these challenges, in this paper we propose a novel deep reinforcement learning (DRL)-based QoS-aware LLM routing framework for sustained high-quality LLM services. Due to the dynamic nature of the global state, we propose a dynamic state abstraction technique to compactly represent global state features with a heterogeneous graph attention network (HAN). Additionally, we introduce an action impact estimator and a tailored reward function to guide the DRL agent in maximizing QoS and preventing latency violations. Extensive experiments on both Poisson and real-world workloads demonstrate that our proposed algorithm significantly improves average QoS and computing resource efficiency compared to existing baselines.

Large language models (LLMs) are rapidly advancing natural language processing and driving increasing interest across biomedical research, clinical care, and healthcare operations. This systematic review synthesizes current empirical evidence on LLM applications in healthcare and biomedical informatics, focusing on their capabilities, evaluation practices, and reported outcomes. Existing studies highlight promising uses in research, where LLMs assist literature-informed genomic interpretation, functional annotation, and biological hypothesis generation, as well as support tasks related to protein and multiomics analysis. In clinical contexts, LLMs are primarily evaluated for natural language-driven tasks, including electronic health record summarization, clinical documentation support, medical question answering, and information extraction, rather than autonomous diagnostic or therapeutic decision-making. Early investigations also describe potential value in healthcare operations, patient communication, clinician education, and drug discovery workflows largely through knowledge retrieval, text generation, and semantic search. However, current evidence remains preliminary: most studies are retrospective, benchmark-based, simulation-driven, or limited to controlled research settings. Key challenges include hallucination and unreliable reasoning, bias and inequitable performance across populations, data privacy and security constraints, reproducibility limitations, and lack of prospective clinical validation and regulatory guidance. Emerging strategies, such as domain-specific pretraining, retrieval augmentation, multi-modal architectures, and privacy-preserving learning, aim to improve reliability, safety, and real-world applicability. This review concludes by outlining methodological, infrastructural, and governance requirements needed to responsibly integrate LLMs into biomedical workflows, emphasizing that clinical deployment remains exploratory and must be rigorously evaluated.

This work presents a Retrieval-Augmented Generation (RAG) pipeline that integrates document preprocessing, embedding-based retrieval, and large language model (LLM) generation into a unified framework. The pipeline begins with the ingestion of PDF documents, followed by text cleaning, sentence segmentation, and chunking to ensure compatibility with embedding model constraints. High-dimensional vector representations are generated using transformer-based embedding models and stored for downstream use. Semantic similarity search, implemented via dot product and cosine similarity, enables efficient retrieval of contextually relevant text. For scalability, the framework is designed to accommodate vector indexing methods such as Faiss. On the generation side, locally hosted LLM (Gemma-7B) is employed with optional quantization for reduced resource consump- tion. Retrieved context is integrated with user queries to enhance the accuracy and relevance of generated responses. This pipeline demonstrates a practical approach for building domain-specific, retrieval-augmented applications that balance efficiency, scalability, and adaptability to local com- pute environments.

Reverse engineering is a cybersecurity process that focuses on understanding the underlying functionality of software or malware. This is an arduous process that demands large amounts of time and effort from cybersecurity practitioners. Large Language Models (LLMs) offer a potential solution to this problem. LLMs have worked their way into various fields of cybersecurity in recent years, including incident response and malware classification. However, LLMs have historically struggled with low-level code comprehension: a necessary part of reverse engineering. While LLMs can generate code and explain its function on the surface level, they struggle to grasp the wider context. In this paper, we utilize parameter-efficient fine-tuning to train LLMs to generate contextual comments for x86 assembly code in an effort to expedite reverse engineering. We choose LLMs from several parameter classes in each of the Qwen2.5-Coder, CodeLlama, and CodeGemma families to fine-tune on a dataset of x86 assembly code. We evaluate each model's performance on cross entropy loss and cosine similarity before and after fine-tuning. We observe promising results, with a significant boost in similarity score for five out of the seven LLMs selected. This is particularly evident in the 0.11 increase in simialrity score for Qwne2.5-Coder-7B and the 0.18 increase in similarity score for CodeLlama-7B after fine-tuning.

Large language models (LLMs) can potentially help with verification using proof assistants by automating proofs. However, it is unclear how effective LLMs are in this task. In this paper, we perform a case study based on two mature Rocq projects: the hs-to-coq tool and Verdi. We evaluate the effectiveness of LLMs in generating proofs by both quantitative and qualitative analysis. Our study finds that: (1) external dependencies and context in the same source file can significantly help proof generation; (2) LLMs perform great on small proofs but can also generate large proofs; (3) LLMs perform differently on different verification projects; and (4) LLMs can generate concise and smart proofs, apply classical techniques to new definitions, but can also make odd mistakes.

The use of large language models (LLMs) has exploded since November 2022, but there is sparse evidence regarding LLM use in health, medical, and research contexts. We aimed to summarise the current uses of and attitudes towards LLMs across our campus’ clinical, research, and teaching sites. We administered a survey about LLM uses and attitudes. We conducted summary quantitative analysis and inductive qualitative analysis of free text responses. In August–September 2023, we circulated the survey amongst all staff and students across our three campus sites (approximately n = 7500), comprising a paediatric academic hospital, research institute, and paediatric university department. We received 281 anonymous survey responses. We asked about participants’ knowledge of LLMs, their current use of LLMs in professional or learning contexts, and perspectives on possible future uses, opportunities, and risks of LLM use. Over 90% of respondents have heard of LLM tools and about two-thirds have used them in their work on our campus. Respondents reported using LLMs for various uses, including generating or editing text and exploring ideas. Many, but not necessarily all, respondents seem aware of the limitations and potential risks of LLMs, including privacy and security risks. Various respondents expressed enthusiasm about the opportunities of LLM use, including increased efficiency. Our findings show LLM tools are already widely used on our campus. Guidelines and governance are needed to keep up with practice. Insights from this survey were used to develop recommendations for the use of LLMs on our campus.

Integrating Large language models into radiology workflow: Impact of generating personalized report templates from summary.

Aim/Purpose: This paper focuses on app review analysis techniques, driven by the rapid advancement of the mobile app market and NLP techniques in optimizing mobile app user experiences. Background: Owing to technological advancements, app review analysis has rapidly evolved. This study examines both conventional and emerging techniques, including current advancements such as large language models (LLMs) in app review analysis. It provides an overview of the various methods used across different categories of app review analysis, comparing effective strategies for identifying user concerns and enhancing app functionality. Methodology: A systematic review was utilized based on two major standard guidelines, PRISMA and Kitchenham’s guidelines, for the period of 2014 to 2024. After defining the review protocol, papers were identified through keyword-based searches on six major online databases: Scopus, Web of Science, IEEE Xplore, ACM Digital Library, Science Direct, and Springer. Following screening and excluding papers based on defined quality criteria, 53 papers were considered for this study. The use of PRISMA ensures a transparent and reproducible review process, while Kitchenham’s guidelines provide a structured and rigorous approach for evaluating and synthesizing the literature. Contribution: This review study aims to evaluate the current state of knowledge on app review analysis techniques to improve mobile app user experiences. This study categorized the existing state-of-the-art papers into eight different categories, such as sentiment analysis, review classification, summarization, and prioritization, and examined challenges related to app review analysis. Furthermore, the study emphasizes the potential of LLMs for optimizing and automating app review analysis and provides future directions to address gaps in user-centric app development. Findings: Among the eight main categories defined in app review analysis, sentiment analysis is the most prevalent, followed by review classification and information extraction. Most studies use a combination of these categories to achieve a comprehensive goal. Prioritization techniques such as risk matrices, thumbs-up count-based approach, and anomaly detection are widely used to identify emerging issues. Extracting meaningful information and evaluating the proposed approach are the most common challenges identified. Novel LLMs, like Chat-GPT, significantly enhance review analysis by automating the process, improving feature extraction, and enabling context-aware review classification. Recommendations for Practitioners: The combination of conventional approaches and novel LLM-based methods can enhance both the efficiency and accuracy in identifying and addressing critical issues raised through mobile app user reviews. It effectively prioritizes user concerns by leveraging the strengths of both traditional preprocessing techniques and advanced LLMs. Recommendation for Researchers: Researchers are encouraged to explore the integration of emerging technologies like LLMs to enhance the of app review analysis, particularly in feature-specific sentiment analysis. Impact on Society: The results of this study contribute to enhancing the mobile app user experience through effective app review analysis, which improves user satisfaction and supports user-centered app development. This ultimately leads to a better mobile app ecosystem, benefiting both users and developers. Future Research: In the future, this research can be extended in multiple directions. Researchers can address the existing research gaps that LLMs have yet to address, particularly in prioritizing user concerns. Additionally, there is potential for further research on tool implementations focusing on identifying persistent issues through time series analysis by considering the app version and date of the app reviews. Moreover, there is a need to develop comprehensive frameworks that are more generalizable across different apps and categories, with a focus on identifying user concerns related to specific features.

Forgetting is the complementary process to memory, enabling both biological and artificial systems to prioritize relevant information while discarding outdated or redundant data. In large language models (LLMs), forgetting occurs through computational mechanisms such as attention bottlenecks, catastrophic interference, and optimization dynamics, mirroring neurobiological processes such as synaptic decay and memory consolidation. While existing research has extensively studied memory mechanisms in LLMs, there remains a critical gap in systematically analyzing forgetting phenomena and their implications for AI system design. This study presents a comprehensive review of forgetting mechanisms in LLMs through a neuroscience-inspired lens. This study first establishes parallels between human forgetting processes and computational forgetting patterns in LLMs. Building on these foundations, this study proposes a novel taxonomy that classifies LLM forgetting across four key dimensions. This taxonomy is further supported by the introduction of quantitative evaluation metrics to provide additional evidence. Furthermore, this study demonstrates how controlled forgetting can be transformed from a limitation into a design feature through bio-inspired optimization strategies. These approaches enable critical applications in privacy-compliant AI, adaptive education systems, and sustainable computing. The study concludes by identifying open challenges in generalization, ethical governance of machine forgetting, and computational efficiency, while outlining future directions for developing LLMs with human-like memory adaptability.

The aim of this study is to assess the role of psycholinguistics and large language models (LLMs), such as GPT-3, in profiling, specifically their ability to expand the possibilities for behavioral analysis based on speech patterns. Psycholinguistics is an effective tool for creating psychological profiles of individuals through language analysis. The integration of LLMs into this process significantly enhances the accuracy and speed of analyzing large volumes of text, uncovering emotional, cognitive, and social behavioral markers. The research methods employed include a mixed approach combining both qualitative and quantitative speech analysis. The data sample consisted of 200 text fragments sourced from social media, emails, and interviews. For analysis, both LLMs (GPT-3) and traditional psycholinguistic methods (LIWC) were used. GPT-3 was applied for the automated analysis of speech patterns, and the results were compared with those obtained through LIWC. Statistical testing was also conducted to examine the differences in marker frequencies. The results indicated that GPT-3 identified a higher frequency of emotional, cognitive, and social markers compared to LIWC, although these differences were not statistically significant (p > 0.05). LLMs demonstrate a significant advantage in processing speed, making them an effective tool for analyzing large amounts of data. Additionally, GPT-3 exhibited a higher level of accuracy in identifying emotional and cognitive markers, particularly in situations where traditional methods may not provide precise results due to limitations in contextual analysis. The key findings of the study suggest that GPT-3 is a powerful tool for psycholinguistic analysis and profiling, capable of enhancing the accuracy of psychological profile creation in fields such as criminal investigations and human resource management. However, ethical considerations regarding data privacy and potential limitations in the accuracy of LLMs must be taken into account. Future research should focus on integrating LLMs with other analytical methods to develop more effective and ethical profiling solutions.

Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks.However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored.In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types.Evaluation is conducted using an commercial model, , and an open-source model, LLaMA-2.Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning.Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model.Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions.

Cybersecurity and artificial intelligence (AI) increasingly intersect as organizations grapple with sophisticated cyber threats and expanding digital landscapes. Incident response teams traditionally rely on structured procedures to identify, manage, and mitigate cyber incidents. Our work explores the effectiveness of generative AI, specifically Large Language Models (LLMs), within cybersecurity, focusing primarily on incident response processes. Experimental evaluations demonstrate that specific LLMs exhibit distinct strengths suitable for different stages of incident management. GPT-4o and GPT-3.5 show high clarity, consistency and coherence, making them appropriate for real-time containment, isolation, eradication and recovery tasks. Conversely, models such as GPT-o1 and GPT-4 offer superior reasoning capabilities and conciseness, better supporting incident preparation, post-incident analysis, vulnerability assessment and training development. Key limitations pertaining to current LLM implementations are identified, particularly token context constraints in addition to a discussion about ethical considerations regarding reliance on AI responses, including potential impacts on workforce skills and organizational security posture.

The rapid deployment of large language models (LLMs) on mobile devices has introduced significant privacy concerns, particularly regarding data collection, user profiling, and compliance with evolving AI regulations such as the GDPR and the AI Act. While these on-device LLMs promise improved latency and user experience, their potential to inadvertently leak sensitive information remains understudied. This paper proposes a red teaming framework to systematically assess the privacy risks of phone-based LLMs, simulating adversarial attacks to identify vulnerabilities in model behavior, data storage, and inference processes. We evaluate popular mobile LLMs under scenarios such as prompt injection, side-channel exploitation, and unintended memorization, measuring their compliance with strict privacy-by-design principles. Our findings reveal critical gaps in current safeguards, including susceptibility to context-aware deanonymization and insufficient data minimization. We further discuss regulatory implications, advocating for adaptive red teaming as a mandatory evaluation step in AI governance. By integrating adversarial testing into the development lifecycle, stakeholders can preemptively align phone-based AI systems with legal and ethical privacy standards while maintaining functional utility.

Background of study: The widespread adoption of Large Language Models (LLMs) raises concerns about biases that affect fairness and credibility. As LLMs affect areas such as recruitment and customer service, systematic quantitative analysis is essential to identify and mitigate these biases.Aims and scope of paper: This research investigates demographic bias in LLM quantitatively by analyzing sentiment polarity scores across different demographic categories. The goal is to provide a statistically confirmed analysis of sentiment bias and propose mitigation methods, focusing on GPT-4, LLaMA-2, Claude, and BLOOM.Methods: Quantitative analysis was performed on GPT-4, LLaMA-2, Claude, and BLOOM using sentiment and demographic data. Sentiment polarity assessments for gender and racial/ethnic groups were obtained with VADER and TextBlob. Demographic Disparity Score, ANOVA, and Cohen's Kappa assessed the significance and appropriateness of bias. Inter-rater reliability between automated tools and human annotators was also evaluated.Result: Sentiment bias was found in all models, varying by gender and race, particularly in GPT-4 and Claude. Sentiment scores were consistently higher for queries pertaining to females than those pertaining to males across all models, with GPT-4 and Claude showing the largest differences. Claude also showed racial sentiment alignment, favoring queries relating to white people over black people. ANOVA confirmed statistically significant sentiment variation by demographics across all models. High inter-rater reliability validated the sentiment analysis.Conclusion: This study shows demographic bias in GPT-4, LLaMA-2, Claude, and BLOOM, with different sentiment trends across demographic classifications. The models showed more positive sentiment for female questions and a trend towards certain racial groups. These findings indicate an embedded bias in the training data, which raises ethical concerns. Identifying and addressing these biases is critical to ensuring fairness and credibility in real-world LLM applications.