Multilingual Language Models Predict Human Reading Behavior

Large language models (LLMs) are increasingly used to model human social behavior, with recent research exploring their ability to simulate social dynamics. Here, we test whether LLMs mirror human behavior in social dilemmas, where individual and collective interests conflict. Humans generally cooperate more than expected in laboratory settings, showing less cooperation in well-mixed populations but more in fixed networks. In contrast, LLMs tend to exhibit greater cooperation in well-mixed settings. This raises a key question: Are LLMs about to emulate human behavior in cooperative dilemmas on networks? In this study, we examine networked interactions where agents repeatedly engage in the Prisoner's Dilemma within both well-mixed and structured network configurations, aiming to identify parallels in cooperative behavior between LLMs and humans. Our findings indicate critical distinctions: while humans tend to cooperate more within structured networks, LLMs display increased cooperation mainly in well-mixed environments, with limited adjustment to networked contexts. Notably, LLM cooperation also varies across model types, illustrating the complexities of replicating human-like social adaptability in artificial agents. These results highlight a crucial gap: LLMs struggle to emulate the nuanced, adaptive social strategies humans deploy in fixed networks. Unlike human participants, LLMs do not alter their cooperative behavior in response to network structures or evolving social contexts, missing the reciprocity norms that humans adaptively employ. This limitation points to a fundamental need in future LLM design-to integrate a deeper comprehension of social norms, enabling more authentic modeling of human-like cooperation and adaptability in networked environments.

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning.However, it is unclear to what extent their capabilities are similar to human ones.We here study the processing of an event B in a script-based story, which causally depends on a previous event A. In our manipulation, event A is stated, negated, or omitted in an earlier section of the text.We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist (A B) than under logical conditions (A B).However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge.We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior.Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition.2) Despite this correlation, all models still fail to predict that nil B is less surprising than A B, indicating that LLMs still have difficulties integrating script knowledge.Code and data are available at https://github. com/tony-hong/causal-script.* These authors contributed equally to this work. .0.47 3.50 ** 0.09 5.25 *** Perceiver 201 -0.02 -0.51 n.s.0.04 0.79 n.s.0.01 1.29 n.s.Bigbird-roberta-base 167 0.05 0.34 n.s.-0.03 -0.13 n.s.0.03 2.62 ** BERT: base-uncased 133 0.14 1.71 n.s.0.21 2.00 .-0.00 -0.00 n.s.Nystromformer-512 132 0.06 1.50 n.s.0.04 0.80 n.s.-0.01 -0.46 n.s.ConvBERT: base 130 0.01 1.66 n.s.-0.00 -0.72 n.s.-0.00 -0.33 n.s.FNet-base 108 0.01 0.14 n.s.0.02 0.41 n.s.-0.01 -0.80 n.s.DistilBERT: base-uncased 90 0.12 2.08 .0.16 2.43 * -0.00 -0.01 n.s.Electra-large-generator 83 0.12 1.15 n.s.0.01 0.13 n.s.-0.01 -0.16 n.s.SqueezeBERT: uncased 75 0.13 1.63 n.s.0.21 2.40 * -0.04 -1.09 n.s.Electra-base-generator 57 0.12 2.69 * 0.08 1.25 n.s.-0.04 -1.15 n.s.Electra-small-generator 17 0.18 2.66 * 0.09 1.33 n.s.-0.03 -0.82 n.s.

This Mental Wellness Chatbot is a software-based solution created for the purpose of solving the incomplete alleviation of depression symptoms in a person. This mental wellness Companion is a chatbot that act as a normal chatbot while mainly focus on the mental health of the user. To accomplish this, we are going to be use various techniques and algorithms to create this chatbot. We are going to use API token of a pre-trained hugging face model in order to create this chatbot rather than using the Large Language Model (LLM) locally since executing the chatbot locally requires a lot of GPU power in the device in order to create and embed the data that we are providing as a dataset. Mainly we are going to be using a pre-trained model from the open-source hugging face model and we are not going to be using OpenAI for completing it. For the database we are going to be using the vector database Facebook AI Similarity Search (FAISS) instead of Mangodb. The mental health Chatbot app is like a reliable friend in the pocket, always ready to lend an ear and offer support whenever you need it. Through gentle questions and personalized suggestions, it helps to navigate the emotions and find ways to feel better. By tracking persons progress over time, it creates a valuable record of emotional journey, empowering to understand oneself better. With 24/7 availability and a commitment to privacy and confidentiality, it provides a safe space for a person to express openly and honestly. Whether the person looking for coping strategies, educational insights, or connections to additional resources, the Chatbot app is there to guide a person on their path to emotional well-being.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.

Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction

The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.

Artificial intelligence can detect suicidal ideation manifestations in texts. Studies demonstrate that BERT-based models achieve better performance in text classification problems. Large language models (LLMs) answer free-text queries without being specifically trained. This work aims to compare the performance of three variations of BERT models and LLMs (Google Bard, Microsoft Bing/GPT-4, and OpenAI ChatGPT-3.5) for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese. A dataset labeled by psychologists consisted of 2,691 sentences without suicidal ideation and 1,097 with suicidal ideation, of which 100 sentences were selected for testing. We applied data preprocessing techniques, hyperparameter optimization, and hold-out cross-validation for training and testing BERT models. When evaluating LLMs, we used zero-shot prompting engineering. Each test sentence was labeled if it contained suicidal ideation, according to the chatbot's response. Bing/GPT-4 achieved the best performance, with 98% across all metrics. Fine-tuned BERT models outperformed the other LLMs: BERTimbau-Large performed the best with a 96% accuracy, followed by BERTimbau-Base with 94%, and BERT-Multilingual with 87%. Bard performed the worst with 62% accuracy, whereas ChatGPT-3.5 achieved 81%. The high recall capacity of the models suggests a low misclassification rate of at-risk patients, which is crucial to prevent missed interventions by professionals. However, despite their potential in supporting suicidal ideation detection, these models have not been validated in a patient monitoring clinical setting. Therefore, caution is advised when using the evaluated models as tools to assist healthcare professionals in detecting suicidal ideation.

Large language models (LLMs) continue to exhibit noteworthy capabilities across a spectrum of areas, including emerging proficiencies across the health care continuum. Successful LLM implementation and adoption depend on digital readiness, modern infrastructure, a trained workforce, privacy, and an ethical regulatory landscape. These factors can vary significantly across health care ecosystems, dictating the choice of a particular LLM implementation pathway. This perspective discusses 3 LLM implementation pathways—training from scratch pathway (TSP), fine-tuned pathway (FTP), and out-of-the-box pathway (OBP)—as potential onboarding points for health systems while facilitating equitable adoption. The choice of a particular pathway is governed by needs as well as affordability. Therefore, the risks, benefits, and economics of these pathways across 4 major cloud service providers (Amazon, Microsoft, Google, and Oracle) are presented. While cost comparisons, such as on-demand and spot pricing across the cloud service providers for the 3 pathways, are presented for completeness, the usefulness of managed services and cloud enterprise tools is elucidated. Managed services can complement the traditional workforce and expertise, while enterprise tools, such as federated learning, can overcome sample size challenges when implementing LLMs using health care data. Of the 3 pathways, TSP is expected to be the most resource-intensive regarding infrastructure and workforce while providing maximum customization, enhanced transparency, and performance. Because TSP trains the LLM using enterprise health care data, it is expected to harness the digital signatures of the population served by the health care system with the potential to impact outcomes. The use of pretrained models in FTP is a limitation. It may impact its performance because the training data used in the pretrained model may have hidden bias and may not necessarily be health care–related. However, FTP provides a balance between customization, cost, and performance. While OBP can be rapidly deployed, it provides minimal customization and transparency without guaranteeing long-term availability. OBP may also present challenges in interfacing seamlessly with downstream applications in health care settings with variations in pricing and use over time. Lack of customization in OBP can significantly limit its ability to impact outcomes. Finally, potential applications of LLMs in health care, including conversational artificial intelligence, chatbots, summarization, and machine translation, are highlighted. While the 3 implementation pathways discussed in this perspective have the potential to facilitate equitable adoption and democratization of LLMs, transitions between them may be necessary as the needs of health systems evolve. Understanding the economics and trade-offs of these onboarding pathways can guide their strategic adoption and demonstrate value while impacting health care outcomes favorably.

Building and delivering high-quality software is critical in software engineering and requires verification and validation processes for end-to-end testing that are reliable, robust, and deliver correct results fast. Manual testing of LLM models, while feasible, is very time-consuming and inefficient and has scalability issues depending on how big the model under test (MUT) is. Recent research and cutting-edge technology innovations in LLM models have deeply influenced software engineering. We need to integrate its impact robustly in areas of model analysis, test automation, model execution, debugging, and report generation.This paper focuses on a framework approach for automated software testing of the LLM models to reduce human interactions and has improved results in a fast, cost-efficient, and time-efficient manner for automated testing methods for industries. The proposed Automation Framework (LLMAutoE2E) leverages and integrates LLMs for testing of different LLM models (like BERT, BART, Hugging Face, and multiple models available to test) to automate the end-to-end execution lifecycle of the LLM models. By leveraging LLMs, companies and industries can generate automated test cases, automated unit test codes, automated integration and end-to-end tests, and automated reporting of the LLM model's execution results. This research emphasizes the potential of the Automation Framework (LLMAutoE2E) for LLM to automate and streamline the overall execution and result generation of the LLM models and the overall testing workflows while addressing challenges in current LLM models testing, its accuracy and scalability for deployments, and reporting. The proposed Automation Framework (LLMAutoE2E) can also automate defect analysis, which improves the software reliability by a manifold and reduces the development cycles for companies. This Research paper details the role of Automation Frameworks for LLMs and how it is transforming QA processes, key methodologies, improving reliability and efficiency, addressing current challenges like model safety, bias detection, and continuous monitoring, and future trends in AI-driven software testing.

In this article, we explore the application of Large Language Models (LLMs) in assessing the content validity of psychometric instruments, focusing on the Big Five Questionnaire (BFQ) and Big Five Inventory (BFI). Content validity, a cornerstone of test construction, ensures that psychological measures adequately cover their intended constructs. Using both human expert evaluations and advanced LLMs, we compared the accuracy of semantic item-construct alignment. Graduate psychology students employed the Content Validity Ratio to rate test items, forming the human baseline. In parallel, state-of-the-art LLMs, including multilingual and fine-tuned models, analyzed item embeddings to predict construct mappings. The results reveal distinct strengths and limitations of human and AI approaches. Human validators excelled in aligning the behaviorally rich BFQ items, while LLMs performed better with the linguistically concise BFI items. Training strategies significantly influenced LLM performance, with models tailored for lexical relationships outperforming general-purpose LLMs. Here we highlight the complementary potential of hybrid validation systems that integrate human expertise and AI precision. The findings underscore the transformative role of LLMs in psychological assessment, paving the way for scalable, objective, and robust test development methodologies.

The rapid advancement of Large Language Models (LLMs) has opened new possibilities for analyzing Web server access logs, particularly in the e-commerce domain. Web server logs provide a rich source of data for system monitoring, security assessment, and user behavior analysis. LLMs may facilitate automated log parsing, anomaly detection, and user session analysis, reducing manual effort, typically required for this kind of tasks. The paper investigates the applicability of a popular LLM, ChatGPT-4o, for Web server log data analysis. A series of experiments on a real e-commerce server log was performed, focusing on six key areas: general statistics and automated reporting, user session reconstruction and analysis, error diagnosis and anomaly detection, Web bot detection, generating synthetic user sessions, and analysis of e-customer activities. The results demonstrate the efficiency of LLMs in extracting meaningful insights from raw log data without extensive pre-processing, making advanced log analysis more accessible to researchers and practitioners. However, challenges such as necessity of precise prompt formulation, the risk of LLM-generated hallucinations, and generalized interpretations require careful validation. The study provides practical guidelines for integrating LLMs into e-commerce server log analysis, emphasizing their advantages, limitations, and potential business applications.

Continuous professional development of science teachers is closely related to their ability to reflect on their science lessons. For teachers to effectively reflect on their lesson, they need opportunities to make meaningful teaching-related experiences and receive guiding feedback for their reflections during initial teacher education. A sensible means to provide reflection opportunities for teachers is through written reflection assignments. However, assessing written reflections is conceptually challenging and resource-intensive. Recent developments suggest that artificial intelligence (AI) might provide novel opportunities for science teacher education and assessing written reflections. Consequently, we investigate ways in which AI in the form of a large language model can be utilized in a specific science teacher training program. The potential of AI for the use in higher education have been highlighted in recent years. However, it is noted that large language models oftentimes are too generic to be useful in specific science teacher education contexts. To address this issue, we used a pre-trained large language model and fine-tuned it to provide individualized feedback for pre-service physics teachers’ written reflection and investigated to which degree contextual factors impact the classification accuracy. Our results show that the pre-trained language model allowed a sufficient classification and that the classification accuracy can be improved by context specification. Our study highlights opportunities and challenges when utilizing AI, in particular large language models, for reflective writing analytics in science education.

Large language models (LLMs), and generative AI generally, raise significant concerns regarding human rights. Their promise in finding insights in patterns of data have to be weighed against potential risks to individuals and societies. The typical perspective, which emphasizes accuracy, capability, or scope of such systems, overlooks the fact that generative AI technologies exploit massive collections of data about human behaviors, thoughts, and ideas. The datafication of human life should be examined through the lens of human rights, specifically with regard to autonomy, dignity, equality, and community. This piece argues that discussions about LLMs and generative AI are inherently linked to data originated from individuals, whose information are embedded in the training data. Data are human rights issues because information about individuals are buried in the data. Technical solutions alone are insufficient to address the human rights distortions produced by LLMs. Policy should focus instead on the fact that data are collected on rights-bearing individuals and groups who have been given very little leeway to discuss the implications of or choose to be in the enterprise of creating generative AI.

Biases in language and their reflection in language models have attracted researchers' attention, particularly with the growth of large language models (LLMs). However, many questions on the links between language models and people’s biased attitudes remain unanswered. In the current study we focus on gender–career bias to examine the extent to which language models can be used to model behavioural responses in the Gender–Career Implicit Association Test (IAT). We provide a systematic evaluation of a range of language models, including n-gram, count vector, predict (word2vec), and Large Language Models (LLMs), to determine how well they capture people’s behaviour in the IAT. We compared response time data from over 800,000 participants against 25 language models, with a total of 675 model variants. We find that many language models, including large language models (LLMs), correlated well with human behavior. While results support previous findings for both predict and count model families, we observed that performance of LLMs was consistently different from that of simpler predict models, particularly in terms of the direction and strength of correlations with reaction time and bias. This divergence may indicate successful attempts to mitigate bias in LLMs while preserving other aspects of linguistic information. Our findings reinforce the idea that societal biases are generally encoded in language, but that large language models can exhibit behaviors different to classical language models.