Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Why are humans so clever? The ‘Social intelligence’ hypothesis explores the idea that this cleverness has evolved through the increasing complexity of social groups. Our ability to understand and control nature is a by-product of our ability to understand the mental states of others and to use this knowledge to co-operate or deceive. These abilities have not emerged out of the blue. They can be found in many social animals that co-operate and compete with one another, birds as well as mammals. This book brings together contributions from an impressive list of authorities in the field, appropriately concluding with a chapter by Nick Humphrey (one of the pioneers in this field). This volume examines social intelligence in many different animal species and explores its development, evolution and the brain systems upon which it depends. Better understanding and further development of social intelligence is critical for the future of the human race and the world that we inhabit. Our problems will not be solved by mere cleverness, but by increased social co-operation.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

Humans live in complex social environments. By the time they reach adulthood, most people have developed highly sophisticated social skills, including the ability to infer the 'invisible' mental states in others, and to act upon those inferences; they have become experts at 'social chess'. This paper draws upon research from developmental and evolutionary psychology, primatology, as well as studies of autistic children, to explore the processes by which children acquire the complex skills underpinning social interaction. It also examines obstacles to social skill acquisition. In light of ethological studies of social play in humans and other primates, it is argued that play is a fundamental medium for acquiring social skills.

The purpose of this chapter is to review the evidence concerning mirror self-recognition as a measure of self-awareness and examine its applicability to schizophrenia. The evidence suggests that the ability to identify yourself correctly in a mirror is not only related to the capacity to conceive of yourself, but may also be related to your ability to take into account what other individuals may know, want or intend to do. This ability to make accurate inferences about mental states in others (known as mental state attribution, theory of mind or social intelligence) begins to emerge during childhood at the same point in time as mirror self-recognition. Species that fail to recognize themselves in mirrors fail to show any evidence that they can infer mental states in one another. Also consistent with the proposition that these phenomena go hand in hand, recent neuropsychological evidence shows that self-awareness and mental state attribution in humans appear to be a byproduct of brain activity that is related to the frontal cortex. As detailed here and elsewhere in this volume, there is growing evidence that both self-awareness and mental state attribution is impaired in schizophrenic patients and that schizophrenia may be related to frontal lobe dysfunction. Mirror self-recognition Mirrors have a number of unique psychological properties. In principle, mirrors represent a means of seeing yourself as you are seen by others. In front of a mirror you are literally an audience to your own behaviour.

This paper describes the models designated for the MEDIQA 2019 shared tasks by the team PANLP. We take advantages of the recent advances in pre-trained bidirectional transformer language models such as BERT (Devlin et al., 2018) and MT-DNN (Liu et al., 2019b). We find that pre-trained language models can significantly outperform traditional deep learning models. Transfer learning from the NLI task to the RQE task is also experimented, which proves to be useful in improving the results of fine-tuning MT-DNN large. A knowledge distillation process is implemented, to distill the knowledge contained in a set of models and transfer it into an single model, whose performance turns out to be comparable with that obtained by the ensemble of that set of models. Finally, for test submissions, model ensemble and a re-ranking process are implemented to boost the performances. Our models participated in all three tasks and ranked the 1st place for the RQE task, and the 2nd place for the NLI task, and also the 2nd place for the QA task.

Antimicrobial peptides (AMPs) are vital components of innate immunotherapy. Existing approaches mainly rely on either deep learning for the automatic extraction of sequence features or traditional manual amino acid features combined with machine learning. The peptide sequence contains symmetrical sequence motifs or repetitive amino acid patterns, which may be related to the function and structure of the peptide. Recently, the advent of large language models has significantly boosted the representational power of sequence pattern features. In light of this, we present a novel AMP predictor called UniproLcad, which integrates three prominent protein language models—ESM-2, ProtBert, and UniRep—to obtain a more comprehensive representation of protein features. UniproLcad utilizes deep learning networks, encompassing the bidirectional long and short memory network (Bi-LSTM) and one-dimensional convolutional neural networks (1D-CNN), while also integrating an attention mechanism to enhance its capabilities. These deep learning frameworks, coupled with pre-trained language models, efficiently extract multi-view features from antimicrobial peptide sequences and assign attention weights to them. Through ten-fold cross-validation and independent testing, UniproLcad demonstrates competitive performance in the field of antimicrobial peptide identification. This integration of diverse language models and deep learning architectures enhances the accuracy and reliability of predicting antimicrobial peptides, contributing to the advancement of computational methods in this field.

With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get better performance of Chinese NLP tasks. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model named StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.

Oxytocin Improves “Mind-Reading” in Humans

Theory of Mind (ToM), the ability to infer the mental states of others, is integral to facilitating healthy social interactions. People can reason about the mental states of others even with limited or (sometimes) inconsistent information. However, little is known about how people make inferences about the mental states of others under uncertainty, and what features of information are important in aiding mental state reasoning. In the current study, we conducted three unique experiments that alter participant's uncertainty when engaging in ToM tests. In Experiment 1, we simultaneously manipulated both the amount and consistency of information available in social stimuli presented to 59 participants. In Experiments 2 and 3, we aimed to decipher which feature of social stimuli is more conducive to mental state reasoning. Experiment 2 manipulated only the amount of information available to 47 participants, while Experiment 3 manipulated only the consistency of information available to 46 participants. Using both frequentist and Bayesian statistics, results confirmed that manipulating the amount and consistency of information alters ToM performance. Exploratory analysis comparing the effects of the amount and consistency of information suggests that the effects of the consistency of information seem to be stronger than those of the amount of information. Taken together, all three experiments suggest that while both the amount and consistency of information are important features of social stimuli-the consistency of information available is more salient when inferring mental states of others. These findings are discussed in relation to information theory and have important implications for creating enriched social stimuli, which may enhance mental state reasoning in individuals with social deficits.

Pre-trained language models with structured semantic knowledge have demonstrated remarkable performance in a variety of downstream natural language processing tasks. The typical methods of integrating knowledge are designing different pre-training tasks and training from scratch, which requires high-end hardware, massive storage resources, and long computing times. Prompt learning is an effective approach to tuning language models for specific tasks, and it can also be used to infuse knowledge. However, most prompt learning methods accept one token as the answer, instead of multiple tokens. To tackle this problem, we propose the long-answer prompt learning method (KLAPrompt), with three different long-answer strategies, to incorporate semantic knowledge into pre-trained language models, and we compare the performance of these three strategies through experiments. We also explore the effectiveness of the KLAPrompt method in the medical field. Additionally, we generate a word sense prediction dataset (WSP) based on the Xinhua Dictionary and a disease and category prediction dataset (DCP) based on MedicalKG. Experimental results show that discrete answers with the answer space partitioning strategy achieve the best results, and introducing structured semantic information can consistently improve language modeling and downstream tasks.

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.

Structural and functional brain correlates of theory of mind impairment post-stroke

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration