Enhancing the design of voting advice applications with BERT language model

Online voting advice applications (VAAs) have become very popular and may significantly influence voting behaviour. It is therefore important to ask which model of party choice VAAs follow. We establish that VAAs see party choice largely as proximity-based issue congruence, with some elements of the directional and salience models. We then assess how well VAAs follow the proximity model by comparing policy positions extracted from 13 VAAs in seven European countries with established policy measures from expert surveys and party manifestos. Party positions extracted from VAAs show strong convergent validity with left-right and economic positions, but compare less favourably with immigration and environment measures. The voting advice given to users is also inherently limited: VAAs mostly disregard accountability, salience, competence and non-policy factors; they treat policy positions and not outcomes as paramount; and they can be subject to strategic manipulation by political parties. As recommended by their designers, voters should treat these applications as tools and guides rather than as stringent recommendations.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

This paper describes the models designated for the MEDIQA 2019 shared tasks by the team PANLP. We take advantages of the recent advances in pre-trained bidirectional transformer language models such as BERT (Devlin et al., 2018) and MT-DNN (Liu et al., 2019b). We find that pre-trained language models can significantly outperform traditional deep learning models. Transfer learning from the NLI task to the RQE task is also experimented, which proves to be useful in improving the results of fine-tuning MT-DNN large. A knowledge distillation process is implemented, to distill the knowledge contained in a set of models and transfer it into an single model, whose performance turns out to be comparable with that obtained by the ensemble of that set of models. Finally, for test submissions, model ensemble and a re-ranking process are implemented to boost the performances. Our models participated in all three tasks and ranked the 1st place for the RQE task, and the 2nd place for the NLI task, and also the 2nd place for the QA task.

Under the ‘Third Vote’ method, the voters cast no votes but are asked about their preferences on policy issues as declared in the party manifestos (like in voting advice applications, e.g. German Wahl-O-Mat). Then the policy profile of the electorate with the balance of public opinion on every issue is determined. The degree to which the parties match with it is expressed by the parties’ representativeness indices of popularity (the average percentage of electors represented on all the issues) and universality (the percentage of cases when a majority is represented), and the parliament seats are distributed among the parties in proportion to their indices. The voters are no longer swayed by politicians’ charisma and communication skills but are directed to subject matters behind personal images and ideological symbols. The focus on choice properties (e.g., political and economic implications of Brexit) is supposed to make vote more rational and responsible, and representative democracy ‘more representative’ and ‘more democratic’. This method has been approbated during the 2016 and 2017 elections of the Karlsruhe Institute of Technology (KIT) Student Parliament (StuPa). The 2016 experiment showed that the method increased the parliament’s representativeness but also revealed that the critical point was the selection of questions by the election committee. Indeed, they can be favorable for one party and unfavorable for another, or they can poorly discriminate between the parties, finally causing an equalization of sizes of the party factions in the parliament (regarded by some as the method’s malfunction). These problems were tackled in the 2017 experiment. The eligible parties were asked to formulate questions themselves and to answer all of them, including the questions by other parties. The collected 94 questions were reduced to 25 using a model aimed at contrasting as much as possible between the parties by maximizing the total distance between the vectors which characterized their policy profiles. Thereby, the accusation of partiality in the question selection was avoided, the gain in the parliament representativeness was confirmed, but the equalization effect was still persistent. The 2018 experiment has three distinctions. Firstly, we use an advanced model to reduce the list of questions. It enhances the multi-dimensionality of the set of parties’ policy profiles aimed at covering the policy space most evenly. For this purpose, the least squares criterion is applied to principal component variances of the correlation or distance matrices for the parties’ policy profiles. Then the reduced set of questions results in a ball-shaped ‘cloud’ of parties’ policy profiles rather than in a stretched ellipsoid, as in the 2017 experiment. Secondly, we test several variants of the Third Vote, using different optimization models to select questions, and compare their impact on the representativeness of the parliament elected. It turns out that the StuPa is by far most representative if elected by the third votes based on the questions selected using the advanced criterion, and this superiority is observed for all groups of electors considered in the experiment. Thirdly, we tackle the Third Vote’s equalization effect. For this purpose, we reduce the party indices, retaining only their part beyond the threshold between representative and nonrepresentative values. The parliament reallocated in proportion to the reduced indices has a similar faction ratio as the one elected by party name, is still more representative than the latter, but less representative than the one allocated in proportion to the complete indices. This means that the optimal proportional representation of public preferences leads to a certain equalization of party factions. Consequently, the equalization effect should not be regarded harmful; it can be tackled, if desired, but at the price of reducing the gain in the parliament representativeness.

Antimicrobial peptides (AMPs) are vital components of innate immunotherapy. Existing approaches mainly rely on either deep learning for the automatic extraction of sequence features or traditional manual amino acid features combined with machine learning. The peptide sequence contains symmetrical sequence motifs or repetitive amino acid patterns, which may be related to the function and structure of the peptide. Recently, the advent of large language models has significantly boosted the representational power of sequence pattern features. In light of this, we present a novel AMP predictor called UniproLcad, which integrates three prominent protein language models—ESM-2, ProtBert, and UniRep—to obtain a more comprehensive representation of protein features. UniproLcad utilizes deep learning networks, encompassing the bidirectional long and short memory network (Bi-LSTM) and one-dimensional convolutional neural networks (1D-CNN), while also integrating an attention mechanism to enhance its capabilities. These deep learning frameworks, coupled with pre-trained language models, efficiently extract multi-view features from antimicrobial peptide sequences and assign attention weights to them. Through ten-fold cross-validation and independent testing, UniproLcad demonstrates competitive performance in the field of antimicrobial peptide identification. This integration of diverse language models and deep learning architectures enhances the accuracy and reliability of predicting antimicrobial peptides, contributing to the advancement of computational methods in this field.

With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get better performance of Chinese NLP tasks. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model named StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

In recent years, with the great success of pre-trained language models, the pre-trained BERT model has been gradually applied to the field of source code understanding. However, the time cost of training a language model from zero is very high, and how to transfer the pre-trained language model to the field of smart contract vulnerability detection is a hot research direction at present. In this paper, we propose a hybrid model to detect common vulnerabilities in smart contracts based on a lightweight pre-trained language model BERT and connected to a bidirectional gate recurrent unit model. The downstream neural network adopts the bidirectional gate recurrent unit neural network model with a hierarchical attention mechanism to mine more semantic features contained in the source code of smart contracts by using their characteristics. Our experiments show that our proposed hybrid neural network model SolBERT-BiGRU-Attention is fitted by a large number of data samples with smart contract vulnerabilities, and it is found that compared with the existing methods, the accuracy of our model can reach 93.85%, and the Micro-F1 Score is 94.02%.

Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration

Homonymy can easily raise lexical ambiguity due to the misunderstanding of its multiple senses.Correct recognition of homonym sense greatly relies on its surrounding context.This ambiguous nature makes homonyms an appropriate testbed for examining the contextualization capability of pre-trained (PLM) and large language models (LLMs).Considering the impact of part of speech (POS) on homonym disambiguation and the prevalence of English-focused studies in word embedding research, this study extends to Chinese and provides a comprehensive layer-wise analysis of homonym representations in both languages, spanning same and different POS categories, across four families of PLMs/LLMs (BERT, GPT-2, Llama 3, Qwen 2.5).Through the creation of a synthetic dataset and computation of disambiguation score (D-Score), we found that: (1) no universal layer depth excels in differentiating homonym representations; (2) bidirectional models produce better contextualized homonym representations compared to much larger autoregressive models; (3) most importantly, POS affects homonym representations in models in ways that differ from human research findings.The individual differences between LLMs uncovered in our study challenge the simplistic understanding of their inner workings.This reveals a compelling research frontier: conducting controlled experiments with purposefully manipulated inputs to enhance the interpretability of LLMs.We have made our dataset and codes available publicly at https://github.com/neurothew/ exploring-homonym-rep-in-llm.

Depression is a serious mental health disorder that poses a major public health concern in Thailand and have a profound impact on individuals' physical and mental health. In addition, the lack of number to mental health services and limited number of psychiatrists in Thailand make depression particularly challenging to diagnose and treat, leaving many individuals with the condition untreated. Recent studies have explored the use of natural language processing to enable access to the classification of depression, particularly with a trend toward transfer learning from pre-trained language model. In this study, we attempted to evaluate the effectiveness of using XLM-RoBERTa, a pre-trained multi-lingual language model supporting the Thai language, for the classification of depression from a limited set of text transcripts from speech responses. Twelve Thai depression assessment questions were developed to collect text transcripts of speech responses to be used with XLM-RoBERTa in transfer learning. The results of transfer learning with text transcription from speech responses of 80 participants (40 with depression and 40 normal control) showed that when only one question (Q1) of "How are you these days?" was used, the recall, precision, specificity, and accuracy were 82.5%, 84.65, 85.00, and 83.75%, respectively. When utilizing the first three questions from Thai depression assessment tasks (Q1 - Q3), the values increased to 87.50%, 92.11%, 92.50%, and 90.00%, respectively. The local interpretable model explanations were analyzed to determine which words contributed the most to the model's word cloud visualization. Our findings were consistent with previously published literature and provide similar explanation for clinical settings. It was discovered that the classification model for individuals with depression relied heavily on negative terms such as 'not,' 'sad,', 'mood', 'suicide', 'bad', and 'bore' whereas normal control participants used neutral to positive terms such as 'recently,' 'fine,', 'normally', 'work', and 'working'. The findings of the study suggest that screening for depression can be facilitated by eliciting just three questions from patients with depression, making the process more accessible and less time-consuming while reducing the already huge burden on healthcare workers.

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.

Continuous professional development of science teachers is closely related to their ability to reflect on their science lessons. For teachers to effectively reflect on their lesson, they need opportunities to make meaningful teaching-related experiences and receive guiding feedback for their reflections during initial teacher education. A sensible means to provide reflection opportunities for teachers is through written reflection assignments. However, assessing written reflections is conceptually challenging and resource-intensive. Recent developments suggest that artificial intelligence (AI) might provide novel opportunities for science teacher education and assessing written reflections. Consequently, we investigate ways in which AI in the form of a large language model can be utilized in a specific science teacher training program. The potential of AI for the use in higher education have been highlighted in recent years. However, it is noted that large language models oftentimes are too generic to be useful in specific science teacher education contexts. To address this issue, we used a pre-trained large language model and fine-tuned it to provide individualized feedback for pre-service physics teachers’ written reflection and investigated to which degree contextual factors impact the classification accuracy. Our results show that the pre-trained language model allowed a sufficient classification and that the classification accuracy can be improved by context specification. Our study highlights opportunities and challenges when utilizing AI, in particular large language models, for reflective writing analytics in science education.