Poster: Unknown Word Detection for ESL Learners Using Gaze and Pre-trained Language Models

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

The Effect of Contextual Inferencing Strategies on EFL Learners⿿ Attitudes towards Reading

English as a Foreign Language (EFL) has constituted a major part of the curriculum in private and public institutions in Saudi Arabia for over eight decades. More recently, the emergence of English as a global language has made it a socially desirable language in Saudi society, as elsewhere in the world. However, despite the long history of English teaching in Saudi Arabia and the growth of the profession over the years, the overall English proficiency level of the majority of students is low and unsatisfactory. Saudi researchers have reported many reasons for the ineffectiveness of English Language Teaching (ELT) including, but not limited to, low levels of student motivation, negative attitudes toward English, a substandard English as a Foreign Language (EFL) curriculum, and inefficient teacher pedagogical practices. These problems fall into two major categories: problems related to the lwhyr of teaching and learning of EFL, and problems related to the lhowr of teaching and learning EFL. In investigating these issues, much of this research has been carried out within individual/psychological/structuralist/positivist frameworks, which has failed not only to problematize the complexity of such constructs but also to appreciate the fact that all realities are produced/(re)produced via ldiscourser. Hence, the present study investigated EFL teaching and learning practices by employing Foucauldian and poststructuralist frameworks. This study used poststructuralism not only as a theoretical lens per se, but also established it through the implemented analytical tools and textual representations of the collected data. Employing a poststructuralist discursive perspective made it possible to view EFL teaching and learning as complex social practices, interacting with the discourses of English and ELT practices available in the global as well as the local context(s). These discourses create the nexus between power and knowledge and impact on what an EFL teacher/ learner does within a classroom setting by constructing perspectives of reality through which he or she comes to understand the EFL teaching and learning process in certain ways which constitute epistemological spaces and generate particular practices. Drawing on policy documents, EFL textbooks, classroom observations, interviews with two Saudi EFL teachers, and focus group discussions with Saudi EFL students in two rural areas, the study identified seven discourses circulating within and competing across the examined discursive spaces. The first three discourses are related to English and its values, including: English as a universal language; English as the language for better employment opportunities; and English as the language for spreading religion. The remaining four discourses centred around ELT with exams as an important aspect of EFL learning and ELT; CLT as the key to success in ELT; L1 as a barrier to successful EFL learning; and EFL textbooks as an essential aspect of the processes of EFL learning and ELT. In exploring the production and (re)production of these discourses through the examination of how visibilization, normalisation, exclusionary/inclusionary acts, classification, and subject positioning as discursive strategies are employed, it was found that whether a particular discourse was made visible or invisible depended on other discourses operating across the examined discursive spaces: the social context, the policy, the textbook, the classroom, and the teachers and students. The analysis of the connection(s) and misconnection(s) between these discursive spaces demonstrated that the interplay between the different discourses constructed discursive battlegrounds, which impacted the processes of EFL teaching and learning. Through the investigation of this interplay, it was found that there was a disconnect and an ambivalent relationship between the study participantsr espoused discourses of English and their EFL teaching and learning practices, suggesting that EFL teachers and learners should not be considered lstabler figures, but rather as social agents with discursively constructed ever changing subjectivities. Furthermore, the analyses of the constructions of the discourses operating within the examined policy texts demonstrate an intra disparity and fragmentation. This was manifested in the difference between the ideals of the EFL curriculum and the institutionally imposed regulating techniques, that is, examinations and teachersr assessment practices which had negative effects on EFL teaching and learning dynamics. Additionally, as a consequence of the competition between the different sources of discursive constructions, I found that there was a gulf between the EFL policies and the enactment of these policies in EFL teaching and learning dynamics in the observed EFL classrooms. The findings of this study also indicate that if a discourse promoted through the policy is mediated by discourses operating in the larger societal context, it became more visible in teaching and learning practices as compared to other discourses that did not have social moorings. This suggests that taking the prevailing social expectations into account at the stage of developing any policy is of paramount importance.nThe study concludes with a discussion of the implications of the interplay of connections and misconnections between competing discourses across different spaces in Saudi educational settings for improving the EFL teaching/learning environments in Saudi Arabia and other similar contexts.n

This paper describes the models designated for the MEDIQA 2019 shared tasks by the team PANLP. We take advantages of the recent advances in pre-trained bidirectional transformer language models such as BERT (Devlin et al., 2018) and MT-DNN (Liu et al., 2019b). We find that pre-trained language models can significantly outperform traditional deep learning models. Transfer learning from the NLI task to the RQE task is also experimented, which proves to be useful in improving the results of fine-tuning MT-DNN large. A knowledge distillation process is implemented, to distill the knowledge contained in a set of models and transfer it into an single model, whose performance turns out to be comparable with that obtained by the ensemble of that set of models. Finally, for test submissions, model ensemble and a re-ranking process are implemented to boost the performances. Our models participated in all three tasks and ranked the 1st place for the RQE task, and the 2nd place for the NLI task, and also the 2nd place for the QA task.

With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get better performance of Chinese NLP tasks. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model named StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.

Antimicrobial peptides (AMPs) are vital components of innate immunotherapy. Existing approaches mainly rely on either deep learning for the automatic extraction of sequence features or traditional manual amino acid features combined with machine learning. The peptide sequence contains symmetrical sequence motifs or repetitive amino acid patterns, which may be related to the function and structure of the peptide. Recently, the advent of large language models has significantly boosted the representational power of sequence pattern features. In light of this, we present a novel AMP predictor called UniproLcad, which integrates three prominent protein language models—ESM-2, ProtBert, and UniRep—to obtain a more comprehensive representation of protein features. UniproLcad utilizes deep learning networks, encompassing the bidirectional long and short memory network (Bi-LSTM) and one-dimensional convolutional neural networks (1D-CNN), while also integrating an attention mechanism to enhance its capabilities. These deep learning frameworks, coupled with pre-trained language models, efficiently extract multi-view features from antimicrobial peptide sequences and assign attention weights to them. Through ten-fold cross-validation and independent testing, UniproLcad demonstrates competitive performance in the field of antimicrobial peptide identification. This integration of diverse language models and deep learning architectures enhances the accuracy and reliability of predicting antimicrobial peptides, contributing to the advancement of computational methods in this field.

The article focuses on the development of learning strategies during the process of foreign language learning. With the obvious implementation of cognitive learning strategies, the development of other types of strategies, metacognitive and socio-affective, remains an urgent issue. Very often these latter strategies are either disregarded or dismissed during the language learning process. However, recent studies have pointed out the necessity for the development of all types of learning strategies and their implementation at an early stage of learning. Therefore, the aim of this research is to determine the scope of the development of metacognitive and socio-affective learning strategies during the process of foreign language learning. Quantitative and qualitative methods have been used to determine the frequency and mode of the above-mentioned types of learning strategies during the process of teaching and learning English as a foreign language (EFL). Two hypotheses have been raised: (1) Metacognitive strategies are often disregarded in the process of language learning in the classes for adolescents; (2) The potential of implementation of socio-affective strategies in EFL classes is underestimated in the process of language learning in the classes for adolescents. The object of the research consists of 12 selected EFL textbooks and activities included in them. The research results prove the fact that much more attention to metacognitive learning strategy development is needed during EFL classes, as these strategies strongly benefit the overall process of language acquisition. To compare, socio-affective learning strategies are more often implemented during EFL classes; however, their development is rather unsystematic.

Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

The benefits of reading while listening for English as a Foreign Language (EFL) learners have been investigated in different EFL learning contexts. Some studies found that simultaneous reading while listening promotes positive impacts on students’ reading and listening skills. It also helps L2 learners’ fluency and comprehension of reading and listening. Moreover, it can assist L2 learners in acquiring new language input in a new learning environment. Hence, this article reviews some research papers related to the implementation of reading while listening (RWL) program in EFL learning context. Furthermore, to answer the research questions: 1) What is reading while listening?; 2) What are the effects of using reading while listening for EFL learners? How to apply the reading while listening program within the EFL classroom? To answer these questions, this article examines the use of RWL for EFL learners based on the research methodology, subject, instrument used, result and suggestion, including the principles of implementing simultaneous reading and listening within the EFL classroom. 
 HIGHLISGHTS
 Highlights
 
 Reading While Listening (RWL)simultaneously can support students with low reading or listening fluency rate.
 Simultaneous reading while listening affects students reading and listening skills for a long period of time.
 A guideline of RWL program implementation based on ER/EL principles is discussed thoroughly.

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration

Homonymy can easily raise lexical ambiguity due to the misunderstanding of its multiple senses.Correct recognition of homonym sense greatly relies on its surrounding context.This ambiguous nature makes homonyms an appropriate testbed for examining the contextualization capability of pre-trained (PLM) and large language models (LLMs).Considering the impact of part of speech (POS) on homonym disambiguation and the prevalence of English-focused studies in word embedding research, this study extends to Chinese and provides a comprehensive layer-wise analysis of homonym representations in both languages, spanning same and different POS categories, across four families of PLMs/LLMs (BERT, GPT-2, Llama 3, Qwen 2.5).Through the creation of a synthetic dataset and computation of disambiguation score (D-Score), we found that: (1) no universal layer depth excels in differentiating homonym representations; (2) bidirectional models produce better contextualized homonym representations compared to much larger autoregressive models; (3) most importantly, POS affects homonym representations in models in ways that differ from human research findings.The individual differences between LLMs uncovered in our study challenge the simplistic understanding of their inner workings.This reveals a compelling research frontier: conducting controlled experiments with purposefully manipulated inputs to enhance the interpretability of LLMs.We have made our dataset and codes available publicly at https://github.com/neurothew/ exploring-homonym-rep-in-llm.

Depression is a serious mental health disorder that poses a major public health concern in Thailand and have a profound impact on individuals' physical and mental health. In addition, the lack of number to mental health services and limited number of psychiatrists in Thailand make depression particularly challenging to diagnose and treat, leaving many individuals with the condition untreated. Recent studies have explored the use of natural language processing to enable access to the classification of depression, particularly with a trend toward transfer learning from pre-trained language model. In this study, we attempted to evaluate the effectiveness of using XLM-RoBERTa, a pre-trained multi-lingual language model supporting the Thai language, for the classification of depression from a limited set of text transcripts from speech responses. Twelve Thai depression assessment questions were developed to collect text transcripts of speech responses to be used with XLM-RoBERTa in transfer learning. The results of transfer learning with text transcription from speech responses of 80 participants (40 with depression and 40 normal control) showed that when only one question (Q1) of "How are you these days?" was used, the recall, precision, specificity, and accuracy were 82.5%, 84.65, 85.00, and 83.75%, respectively. When utilizing the first three questions from Thai depression assessment tasks (Q1 - Q3), the values increased to 87.50%, 92.11%, 92.50%, and 90.00%, respectively. The local interpretable model explanations were analyzed to determine which words contributed the most to the model's word cloud visualization. Our findings were consistent with previously published literature and provide similar explanation for clinical settings. It was discovered that the classification model for individuals with depression relied heavily on negative terms such as 'not,' 'sad,', 'mood', 'suicide', 'bad', and 'bore' whereas normal control participants used neutral to positive terms such as 'recently,' 'fine,', 'normally', 'work', and 'working'. The findings of the study suggest that screening for depression can be facilitated by eliciting just three questions from patients with depression, making the process more accessible and less time-consuming while reducing the already huge burden on healthcare workers.

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.