A tutorial on fine-tuning pretrained language models: Applications in social and behavioral science research.

BackgroundLarge language models have had a huge impact on natural language processing (NLP) in recent years. However, their application in epidemiological research is still limited to the analysis of electronic health records and social media data.ObjectivesTo demonstrate the potential of NLP beyond these domains, we aimed to develop prediction models based on texts collected from an epidemiological cohort and compare their performance to classical regression methods.MethodsWe used data from the British National Child Development Study, where 10,567 children aged 11 years wrote essays about how they imagined themselves as 25-year-olds. Overall, 15% of the data set was set aside as a test set for performance evaluation. Pretrained language models were fine-tuned using AutoTrain (Hugging Face) to predict current reading comprehension score (range: 0-35) and future BMI and physical activity (active vs inactive) at the age of 33 years. We then compared their predictive performance (accuracy or discrimination) with linear and logistic regression models, including demographic and lifestyle factors of the parents and children from birth to the age of 11 years as predictors.ResultsNLP clearly outperformed linear regression when predicting reading comprehension scores (root mean square error: 3.89, 95% CI 3.74-4.05 for NLP vs 4.14, 95% CI 3.98-4.30 and 5.41, 95% CI 5.23-5.58 for regression models with and without general ability score as a predictor, respectively). Predictive performance for physical activity was similarly poor for the 2 methods (area under the receiver operating characteristic curve: 0.55, 95% CI 0.52-0.60 for both) but was slightly better than random assignment, whereas linear regression clearly outperformed the NLP approach when predicting BMI (root mean square error: 4.38, 95% CI 4.02-4.74 for NLP vs 3.85, 95% CI 3.54-4.16 for regression). The NLP approach did not perform better than simply assigning the mean BMI from the training set as a predictor.ConclusionsOur study demonstrated the potential of using large language models on text collected from epidemiological studies. The performance of the approach appeared to depend on how directly the topic of the text was related to the outcome. Open-ended questions specifically designed to capture certain health concepts and lived experiences in combination with NLP methods should receive more attention in future epidemiological studies.

Qualitative and Mixed Methods Research in Cross-Cultural Psychology

Continuous professional development of science teachers is closely related to their ability to reflect on their science lessons. For teachers to effectively reflect on their lesson, they need opportunities to make meaningful teaching-related experiences and receive guiding feedback for their reflections during initial teacher education. A sensible means to provide reflection opportunities for teachers is through written reflection assignments. However, assessing written reflections is conceptually challenging and resource-intensive. Recent developments suggest that artificial intelligence (AI) might provide novel opportunities for science teacher education and assessing written reflections. Consequently, we investigate ways in which AI in the form of a large language model can be utilized in a specific science teacher training program. The potential of AI for the use in higher education have been highlighted in recent years. However, it is noted that large language models oftentimes are too generic to be useful in specific science teacher education contexts. To address this issue, we used a pre-trained large language model and fine-tuned it to provide individualized feedback for pre-service physics teachers’ written reflection and investigated to which degree contextual factors impact the classification accuracy. Our results show that the pre-trained language model allowed a sufficient classification and that the classification accuracy can be improved by context specification. Our study highlights opportunities and challenges when utilizing AI, in particular large language models, for reflective writing analytics in science education.

Demographic biases and social stereotypes are common in pretrained language models (PLMs), and a burgeoning body of literature focuses on removing the unwanted stereotypical associations from PLMs. However, when fine-tuning these bias-mitigated PLMs in downstream natural language processing (NLP) applications, such as sentiment classification, the unwanted stereotypical associations resurface or even get amplified. Since pretrain&fine-tune is a major paradigm in NLP applications, separating the debiasing procedure of PLMs from fine-tuning would eventually harm the actual downstream utility. In this paper, we propose a unified debiasing framework Causal-Debias to remove unwanted stereotypical associations in PLMs during fine-tuning. Specifically, Causal-Debias mitigates bias from a causal invariant perspective by leveraging the specific downstream task to identify bias-relevant and label-relevant factors. We propose that bias-relevant factors are non-causal as they should have little impact on downstream tasks, while label-relevant factors are causal. We perform interventions on non-causal factors in different demographic groups and design an invariant risk minimization loss to mitigate bias while maintaining task performance. Experimental results on three downstream tasks show that our proposed method can remarkably reduce unwanted stereotypical associations after PLMs are fine-tuned, while simultaneously minimizing the impact on PLMs and downstream applications. © 2023 Association for Computational Linguistics.

Pre-Trained Language Models and Their Applications

Ethical issues in the conduct of behavioral and social science research with human participants involve considerations that are as diverse as the range of disciplines and fields that constitute these sciences. The methods, study populations, and issues being examined in studying behavioral and social processes all come into play in designing and implementing ethically sound research. This chapter provides an overview of the history of ethical considerations in the conduct of behavioral and social science research; addresses underlying ethical considerations that animate these fields of inquiry; unpacks the complexity of making ethical determinations, especially in the context of challenging circumstances, study populations, or methods; and raises issues—some unresolved—that merit consideration in planning for and reviewing research.

As the largest generation in U.S. history — the population born in the two decades immediately following World War II — enters the age of risk for cognitive impairment, growing numbers of people will experience dementia (including Alzheimer's disease and related dementias). By one estimate, nearly 14 million people in the United States will be living with dementia by 2060. Like other hardships, the experience of living with dementia can bring unexpected moments of intimacy, growth, and compassion, but these diseases also affect people's capacity to work and carry out other activities and alter their relationships with loved ones, friends, and coworkers. Those who live with and care for individuals experiencing these diseases face challenges that include physical and emotional stress, difficult changes and losses in their relationships with life partners, loss of income, and interrupted connections to other activities and friends. From a societal perspective, these diseases place substantial demands on communities and on the institutions and government entities that support people living with dementia and their families, including the health care system, the providers of direct care, and others.Nevertheless, research in the social and behavioral sciences points to possibilities for preventing or slowing the development of dementia and for substantially reducing its social and economic impacts. At the request of the National Institute on Aging of the U.S. Department of Health and Human Services, Reducing the Impact of Dementia in America assesses the contributions of research in the social and behavioral sciences and identifies a research agenda for the coming decade. This report offers a blueprint for the next decade of behavioral and social science research to reduce the negative impact of dementia for America's diverse population. Reducing the Impact of Dementia in America calls for research that addresses the causes and solutions for disparities in both developing dementia and receiving adequate treatment and support. It calls for research that sets goals meaningful not just for scientists but for people living with dementia and those who support them as well.By 2030, an estimated 8.5 million Americans will have Alzheimer's disease and many more will have other forms of dementia. Through identifying priorities social and behavioral science research and recommending ways in which they can be pursued in a coordinated fashion, Reducing the Impact of Dementia in America will help produce research that improves the lives of all those affected by dementia.

Homonymy can easily raise lexical ambiguity due to the misunderstanding of its multiple senses.Correct recognition of homonym sense greatly relies on its surrounding context.This ambiguous nature makes homonyms an appropriate testbed for examining the contextualization capability of pre-trained (PLM) and large language models (LLMs).Considering the impact of part of speech (POS) on homonym disambiguation and the prevalence of English-focused studies in word embedding research, this study extends to Chinese and provides a comprehensive layer-wise analysis of homonym representations in both languages, spanning same and different POS categories, across four families of PLMs/LLMs (BERT, GPT-2, Llama 3, Qwen 2.5).Through the creation of a synthetic dataset and computation of disambiguation score (D-Score), we found that: (1) no universal layer depth excels in differentiating homonym representations; (2) bidirectional models produce better contextualized homonym representations compared to much larger autoregressive models; (3) most importantly, POS affects homonym representations in models in ways that differ from human research findings.The individual differences between LLMs uncovered in our study challenge the simplistic understanding of their inner workings.This reveals a compelling research frontier: conducting controlled experiments with purposefully manipulated inputs to enhance the interpretability of LLMs.We have made our dataset and codes available publicly at https://github.com/neurothew/ exploring-homonym-rep-in-llm.

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Large language models (LLMs) have revolutionized natural language processing by learning rich representations from massive and diverse text data. While pre-trained LLMs have been widely customized for a variety of text-related tasks, their potential in wireless communications remains largely untapped. In this paper, we propose a novel channel state information (CSI) feedback framework for massive multiple-input multiple-output (MIMO) systems that leverages pre-trained LLMs to bridge the gap between natural language processing and wireless communication. To adapt LLMs for the CSI feedback task, we design specialized pre-processing, embedding, and post-processing modules that convert CSI data into token representations compatible with LLMs. Moreover, to address the limitations of a single model and enhance performance in diverse environments and under various compression ratios, we introduce a mixture-of-agents (MoA)based framework that integrates the complementary strengths of multiple LLMs. Simulation results demonstrate that our approach achieves superior CSI reconstruction performance, robust generalization to unseen scenarios, and significant training cost reduction even with limited data. These findings confirm that the transferable knowledge acquired from large-scale natural language pre-training can effectively enhance CSI feedback, offering a promising direction for intelligent and efficient nextgeneration wireless communication systems.

The advent of pre-train techniques and large language models has significantly leveraged the performance of many natural language processing (NLP) tasks. However, pre-trained language models for neural machine translation remain a challenge as little information about the interaction of the language pair is learned. In this paper, we explore several studies trying to define a training scheme to pre-train the cross-attention module between the encoder and the decoder by using the large-scale monolingual corpora independently. The experiments show promising results, proving the effectiveness of using pre-trained language models in neural machine translation.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.

In addressing challenges within the field of Natural Language Processing (NLP), supervised fine-tuning is an efficient technique that allows pre-trained Large Language Models to adapt to specific tasks. This is especially crucial for low-resource languages, such as Tibetan, where the demand for high-quality fine-tuning datasets is particularly pronounced. This paper introduces the Tibetan Instruction-Following Dataset (TIFD), comprising 11,535 JSON objects, each with four attributes: a unique identifier, instructions, input, and output. These attributes correspond to the task’s unique ID, the instruction describing the task, supplementary input for the task instruction, and the response generated by GPT-4. In the data preprocessing phase, unsuitable data for Tibetan fine-tuning were first filtered out. To ensure the diversity of instructions, the Tibetan instructions were effectively vectorized using LaBSE, followed by the selection of diverse Tibetan instruction data using a method based on the K-Center-Greedy algorithm. After preliminary English-Tibetan machine translation of responses generated by GPT-4, a web program was developed to facilitate Tibetan proofreaders in referencing the original English and corresponding Chinese translations for rectifying errors in words, grammar, and structure in the Tibetan data. Subsequently, five Tibetan language professionals were invited to manually proofread the data, checking and modifying the data format and Tibetan content, and removing erroneous and meaningless tasks, thereby ensuring the quality and reliability of the TIFD. Using TIFD to supervise the fine-tuning of the Tibetan large language model TiLamb has enhanced TiLamb’s Tibetan dialogue and instruction-following capabilities. The TIFD serves as a vital resource for Tibetan data, playing a significant role in the development of Tibetan NLP technology and in the effective customization of Large Language Models (LLMs).