• Home
  • Search
  • Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation.
  • Cite Icon27
  • https://doi.org/10.1093/jamia/ocaf059Copy DOI Icon

Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation.

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

This study aims to develop and evaluate an approach using large language models (LLMs) and a knowledge graph to triage patient messages that need emergency care. The goal is to notify patients when their messages indicate an emergency, guiding them to seek immediate help rather than using the patient portal, to improve patient safety. We selected 1020 messages sent to Vanderbilt University Medical Center providers between January 1, 2022 and March 7, 2023. We developed four models to triage these messages for emergencies: (1) Prompt-Only: the patient message was input with a prompt directly into the LLM; (2) Naïve Retrieval Augmented Generation (RAG): provided retrieved information as context to the LLM; (3) RAG from Knowledge Graph with Local Search: a knowledge graph was used to retrieve locally relevant information based on semantic similarities; (4) RAG from Knowledge Graph with Global Search: a knowledge graph was used to retrieve globally relevant information through hierarchical community detection. The knowledge base was a triage book covering 225 protocols. The RAG from Knowledge Graph model with global search outperformed other models, achieving an accuracy of 0.99, a sensitivity of 0.98, and a specificity of 0.99. It demonstrated significant improvements in triaging emergency messages compared to LLM without RAG and naïve RAG. The traditional LLM without any retrieval mechanism underperformed compared to models with RAG, which aligns with the expected benefits of augmenting LLMs with domain-specific knowledge sources. Our results suggest that providing external knowledge, especially in a structured manner and in community summaries, can improve LLM performance in triaging patient portal messages. LLMs can effectively assist in triaging emergency patient messages after integrating with a knowledge graph about a nurse triage book. Future research should focus on expanding the knowledge graph and deploying the system to evaluate its impact on patient outcomes.

Similar Papers
  • Research Article

Construction and evaluation of the knowledge graph and large model question-answering system for Jin San Zhen therapy: a tool study for primary care and general practice.

  • Jan 01, 2026
  • Frontiers in medicine
  • Junjie Chen +8
  • Research Article
  • Citations975

Unifying Large Language Models and Knowledge Graphs: A Roadmap

  • Jul 01, 2024
  • IEEE Transactions on Knowledge and Data Engineering
  • Shirui Pan +5
  • Research Article
  • Citations1

LLM-Based Natural Language to SPARQL Translation over Domain-Specific Knowledge Graph

  • Dec 15, 2025
  • Knowledge Organization
  • Mohammed H Rasheed +1
  • Research Article
  • Citations26

Knowledge Graphs and Their Reciprocal Relationship with Large Language Models

  • Apr 21, 2025
  • Machine Learning and Knowledge Extraction
  • Ramandeep Singh Dehal +2
  • Supplementary Content
  • Citations2

A Review of Fault Diagnosis Methods: From Traditional Machine Learning to Large Language Model Fusion Paradigm

  • Jan 21, 2026
  • Sensors (Basel, Switzerland)
  • Qingwei Nie +2
  • Research Article

Knowledge Graph and Large Language Model Co-learning via Structure-oriented Retrieval Augmented Generation.

  • Dec 01, 2024
  • Bulletin of the Technical Committee on Data Engineering
  • Carl Yang +3
  • Book Chapter
  • Citations5

Enhancing Manufacturing Knowledge Access with LLMs and Context-Aware Prompting

  • Oct 16, 2024
  • Frontiers in artificial intelligence and applications
  • Sebastian Monka +6
  • Conference Article
  • Citations1

Filling the Gap: LLMs as Scaffolds for Competency Question Instantiation

  • Mar 22, 2026
  • Claire Mcnamara +2
  • Research Article
  • Citations7

A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Shiyu Tian +7
  • Front Matter
  • Citations1

Editorial: Large language models in work and business.

  • Nov 29, 2024
  • Frontiers in artificial intelligence
  • Şadi Evren Şeker
  • Conference Article
  • Citations10

SymAgent: A Neural-Symbolic Self-Learning Agent Framework for Complex Reasoning over Knowledge Graphs

  • Apr 22, 2025
  • Ben Liu +5
  • Research Article

Knowledge Graph and Large Language Model for Metabolomics

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Yuxing Lu
  • PDF
  • Research Article
  • Citations17

Knowledge graph-extended retrieval augmented generation for question answering

  • Nov 01, 2025
  • Applied Intelligence
  • Jasper Linders +1
  • Conference Article
  • Citations3

Integrating Large Language Models and Knowledge Graphs for Next-level AGI

  • May 08, 2025
  • Linhao Luo +3
  • Research Article
  • Citations14

LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Tu Ao +9
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.