• Home
  • Search
  • Implementation of Retrieval-Augmented Generation Method on Large Language Model for Development of Campus Service and Information Chatbot
  • https://doi.org/10.35314/3y9hy151Copy DOI Icon

Implementation of Retrieval-Augmented Generation Method on Large Language Model for Development of Campus Service and Information Chatbot

Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Large Language Models (LLMs) have the potential to improve the quality of information services in higher education environments through responsive and natural interactions. However, LLMs are prone to generating answers that are not supported by valid knowledge sources due to knowledge cut-off limitations. This study implements Retrieval-Augmented Generation (RAG) on LLMs to build an information service chatbot for the Universitas Sains dan Teknologi Indonesia (USTI). RAG is built using a hybrid retrieval mechanism that combines dense retrieval (FAISS) and sparse retrieval (BM25) through Reciprocal Rank Fusion (RRF) and is equipped with cross-encoder reranking. The knowledge base is compiled from official and public documents obtained through the USTI website. The evaluation was conducted using 13 test queries by comparing several configurations to analyze the contribution of each component. The evaluation results show that the hybrid retrieval configuration produces the best retrieval performance with Precision@3 of 71.7%, Recall@3 of 87.5%, and NDCG@3 of 96.3%. In addition, the application of RAG improved the quality of answers compared to LLM without retrieval, as shown by an increase in BERTScore-F1 from 84.8% to 89.4% and a faithfulness score of 88.8%. These findings indicate that RAG integration improves the relevance of LLM answers to source documents, with the hybrid configuration providing an optimal balance between retrieval quality and faithfulness.

Similar Papers
  • Research Article

Modern Approaches to Using Knowledge Bases to Address the Challenges of Large Language Models

  • May 12, 2025
  • NaUKMA Research Papers. Computer Science
  • Maksym Androshchuk
  • Research Article
  • Citations18

DracoGPT: Extracting Visualization Design Preferences from Large Language Models.

  • Jan 01, 2025
  • IEEE transactions on visualization and computer graphics
  • Huichen Will Wang +3
  • Research Article
  • Citations2

Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Jianqi Gao +5
  • Conference Article
  • Citations1

Intuitive or Dependent? Investigating LLMs’ Behavior Style to Conflicting Prompts

  • Jan 01, 2024
  • Proceedings of the Annual Meeting of the Association for Computational Linguistics
  • Jiahao Ying +5
  • Research Article
  • Citations3

Quality-of-Service Aware LLM Routing for Edge Computing With Multiple Experts

  • Dec 01, 2025
  • IEEE Transactions on Mobile Computing
  • Jin Yang +5
  • Research Article
  • Citations87

Potential of Large Language Models in Health Care: Delphi Study

  • May 13, 2024
  • Journal of Medical Internet Research
  • Kerstin Denecke +2
  • Research Article

Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study

  • Jan 23, 2024
  • ArXiv
  • Zhe He +8
  • Preprint Article

Augmenting Local LLMs with Specialized Tools for Scientific Workflows

  • Mar 18, 2025
  • Mirko Mälicke +2
  • PDF
  • Conference Article
  • Citations13

Unleashing the Retrieval Potential of Large Language Models in Conversational Recommender Systems

  • Oct 08, 2024
  • Ting Yang +1
  • PDF
  • Research Article
  • Citations8

Rapid Integration of LLMs in Healthcare Raises Ethical Concerns: An Investigation into Deceptive Patterns in Social Robots

  • Feb 18, 2025
  • Digital Society
  • Robert Ranisch +1
  • Conference Article

Game-Oriented ASR Error Correction via RAG-Enhanced LLM

  • Aug 19, 2025
  • Yan Jiang +3
  • Research Article

Enhancing LLM-based medical decision-making by test-time knowledge acquisition.

  • Dec 01, 2026
  • Health information science and systems
  • Shipeng Li +3
  • Research Article
  • Citations69

CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

  • Jan 21, 2025
  • ACM Transactions on Information Systems
  • Yuanjie Lyu +9
  • Research Article

Retrieval-augmented generation in healthcare: A narrative review of methods, contributions, and future directions

  • Feb 24, 2026
  • Digital Medicine
  • Hao Kechun +1
  • Discussion
  • Citations75

Information Retrieval meets Large Language Models: A strategic report from Chinese IR community

  • Jan 01, 2023
  • AI Open
  • Qingyao Ai +32
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.