• Home
  • Search
  • Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation
  • Cite Icon12
  • https://doi.org/10.1038/s41598-025-08031-0Copy DOI Icon

Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation

Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

The substantial increase in mental health disorders globally necessitates scalable, accurate tools for detecting and classifying these conditions in digital environments. This study addresses the critical challenge of automated mental health classification by comparing three distinct computational approaches: (1) Traditional Natural Language Processing (NLP) with advanced feature engineering, (2) Prompt-engineered large language models (LLMs), and (3) Fine-tuned LLMs. The dataset consisted of over 51,000 publicly available text statements from social media platforms, tagged with seven mental health conditions: Normal, Depression, Suicidal, Anxiety, Stress, Bipolar Disorder, and Personality Disorder. The dataset was stratified into training, validation, and test sets for model evaluation. The primary outcome was classification accuracy across these seven mental health conditions. Additional metrics like precision, recall, and F1-score were analyzed. We compared the results of the three computational approaches and overfitting was monitored through validation loss across epochs for the fine-tuned LLM. The NLP model with advanced feature engineering achieved an overall accuracy of 95%, surpassing both the prompt-engineered LLM (65%) and the fine-tuned LLM (91%). This model performed exceptionally well in terms of accuracy and precision. While fine-tuning for three epochs yielded optimal results, further training led to overfitting and decreased performance. This study demonstrates the significant benefits of applying advanced text preprocessing and feature engineering techniques to traditional NLP models, alongside fine-tuning LLMs, such as GPT-4o-mini, for mental health classification tasks. The results clearly indicate that off-the-shelf LLM chatbots using prompt engineering are inadequate for mental health classification, performing 30% points below specialized NLP approaches. Despite the popularity of general-purpose LLMs, specialized approaches remain superior for critical healthcare applications like mental health classification.

Loading PDF

Similar Papers
  • Research Article

#2924 Comparison of large language models and traditional natural language processing techniques in predicting arteriovenous fistula failure

  • May 23, 2024
  • Nephrology Dialysis Transplantation
  • Suman Lama +6
  • Research Article
  • Citations7

Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify Arrhythmia Recurrence.

  • Dec 16, 2024
  • Circulation. Arrhythmia and electrophysiology
  • Ruibin Feng +16
  • Research Article
  • Citations8

Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models.

  • Oct 31, 2024
  • European radiology
  • Marcel C Langenbach +7
  • Research Article

Integrating NLP into Radiation Oncology: A Practical Guide to Transformer Architecture and Large Language Models

  • Aug 13, 2025
  • BJR|Artificial Intelligence
  • Reza Khanmohammadi +10
  • Research Article
  • Citations51

The Applications of Large Language Models in Mental Health: Scoping Review.

  • May 05, 2025
  • Journal of medical Internet research
  • Yu Jin +10
  • Research Article
  • Citations15

Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review

  • Jan 22, 2025
  • Frontiers in Medicine
  • Mahmud Omar +5
  • PDF
  • Research Article
  • Citations36

LLMs and NLP Models in Cryptocurrency Sentiment Analysis: A Comparative Classification Study

  • Jun 05, 2024
  • Big Data and Cognitive Computing
  • Konstantinos I Roumeliotis +2
  • Research Article
  • Citations2

Assessment of a zero-shot large language model in measuring documented goals-of-care discussions

  • Sep 26, 2025
  • medRxiv
  • Robert Y Lee +6
  • Research Article
  • Citations1

Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions.

  • Jan 01, 2026
  • Journal of pain and symptom management
  • Robert Y Lee +6
  • Research Article

Large language models outperform traditional natural language processing methods in extracting patient-reported outcomes in IBD

  • Sep 06, 2024
  • medRxiv
  • Perseus V Patel +9
  • Book Chapter

A Large Language Model Algorithm for Language Processing Technology Deviation Detection

  • Aug 28, 2025
  • Junze Li
  • Research Article

Identification of muscle-invasion status in bladder cancer patients using natural language processing and machine learning.

  • Feb 20, 2022
  • Journal of Clinical Oncology
  • Ruixin Yang +6
  • Dissertation

Designing artifact representation and automated pipeline for machine learning based Software Engineering

  • Jan 01, 2024
  • Hung Dang Phan
  • Research Article
  • Citations3

Improving drug identification in overdose death surveillance by using clinical natural language processing models.

  • Feb 08, 2026
  • Journal of forensic sciences
  • Arthur J Funnell +8
  • Conference Article
  • Citations15

NLP-Fast: A Fast, Scalable, and Flexible System to Accelerate Large-Scale Heterogeneous NLP Models

  • Sep 01, 2021
  • Joonsung Kim +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.