Verifying Outputs in Scopus AI to Promote Critical AI Literacy in Education and Research.
To evaluate the factual accuracy and citation fidelity of Scopus AI's outputs in response to a single health care-related research question about the importance of human trafficking prevention education for professionals. This study employed a mixed-methods content verification approach. A single health care-related research question was entered into Scopus AI (Elsevier), which generated a summary, expanded summary, and concept map. Quantitative data were collected by classifying each statement in the Scopus AI output as accurate, misleading, or incorrect. Qualitative analysis provided contextual insights into citation use, source type, and interpretation of content. Of the 30 statements analyzed from the Scopus AI output, 27 (90.0%) were rated as accurate, and 3 (10.0%) were categorized as misleading. No incorrect or hallucinated content was detected. Qualitative analysis revealed that Scopus AI consistently cited legitimate, peer-reviewed sources. However, in 2 cases, the tool referenced secondary sources without clarification, raising questions about source hierarchy. Though Scopus AI produced largely reliable academic content, this study underscores the need for user verification and scholarly judgment, particularly regarding secondary sources and citation transparency. The findings highlight the importance of teaching students to critically evaluate artificial intelligence (AI)-generated material. In response to the findings, a classroom activity titled "Fact-Check the Bot" was developed to promote critical AI literacy. This activity guides learners in assessing AI-generated claims using a verification matrix and original literature and can be adapted for use with other AI tools. This study demonstrates the potential and the limitations of generative AI in academic research and offers a model for integrating verification practices into educational settings to enhance students' critical engagement with AI tools.
Read more