CAM: A Large Language Model-based Creative Analogy Mining Framework

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning.However, it is unclear to what extent their capabilities are similar to human ones.We here study the processing of an event B in a script-based story, which causally depends on a previous event A. In our manipulation, event A is stated, negated, or omitted in an earlier section of the text.We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist (A B) than under logical conditions (A B).However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge.We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior.Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition.2) Despite this correlation, all models still fail to predict that nil B is less surprising than A B, indicating that LLMs still have difficulties integrating script knowledge.Code and data are available at https://github. com/tony-hong/causal-script.* These authors contributed equally to this work. .0.47 3.50 ** 0.09 5.25 *** Perceiver 201 -0.02 -0.51 n.s.0.04 0.79 n.s.0.01 1.29 n.s.Bigbird-roberta-base 167 0.05 0.34 n.s.-0.03 -0.13 n.s.0.03 2.62 ** BERT: base-uncased 133 0.14 1.71 n.s.0.21 2.00 .-0.00 -0.00 n.s.Nystromformer-512 132 0.06 1.50 n.s.0.04 0.80 n.s.-0.01 -0.46 n.s.ConvBERT: base 130 0.01 1.66 n.s.-0.00 -0.72 n.s.-0.00 -0.33 n.s.FNet-base 108 0.01 0.14 n.s.0.02 0.41 n.s.-0.01 -0.80 n.s.DistilBERT: base-uncased 90 0.12 2.08 .0.16 2.43 * -0.00 -0.01 n.s.Electra-large-generator 83 0.12 1.15 n.s.0.01 0.13 n.s.-0.01 -0.16 n.s.SqueezeBERT: uncased 75 0.13 1.63 n.s.0.21 2.40 * -0.04 -1.09 n.s.Electra-base-generator 57 0.12 2.69 * 0.08 1.25 n.s.-0.04 -1.15 n.s.Electra-small-generator 17 0.18 2.66 * 0.09 1.33 n.s.-0.03 -0.82 n.s.

Continuous professional development of science teachers is closely related to their ability to reflect on their science lessons. For teachers to effectively reflect on their lesson, they need opportunities to make meaningful teaching-related experiences and receive guiding feedback for their reflections during initial teacher education. A sensible means to provide reflection opportunities for teachers is through written reflection assignments. However, assessing written reflections is conceptually challenging and resource-intensive. Recent developments suggest that artificial intelligence (AI) might provide novel opportunities for science teacher education and assessing written reflections. Consequently, we investigate ways in which AI in the form of a large language model can be utilized in a specific science teacher training program. The potential of AI for the use in higher education have been highlighted in recent years. However, it is noted that large language models oftentimes are too generic to be useful in specific science teacher education contexts. To address this issue, we used a pre-trained large language model and fine-tuned it to provide individualized feedback for pre-service physics teachers’ written reflection and investigated to which degree contextual factors impact the classification accuracy. Our results show that the pre-trained language model allowed a sufficient classification and that the classification accuracy can be improved by context specification. Our study highlights opportunities and challenges when utilizing AI, in particular large language models, for reflective writing analytics in science education.

The limited availability of annotated data often hinders real-world applications of machine learning. To efficiently learn from small quantities of multimodal data, we leverage the linguistic knowledge from a large pre-trained language model (PLM) and quickly adapt it to new domains of image captioning. To effectively utilize a pretrained model, it is critical to balance the visual input and prior linguistic knowledge from pretraining. We propose VisualGPT, which employs a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the PLM with a small amount of in-domain image-text data. The proposed self-resurrecting activation unit produces sparse activations that prevent accidental overwriting of linguistic knowledge. When trained on 0.1%, 0.5% and 1% of the respective training sets, VisualGPT surpasses the best baseline by up to 10.0% CIDEr on MS COCO [43] and 17.9% CIDEr on Conceptual Captions [63]. Furthermore, VisualGPT achieves the state-of-the-art result on IU X-ray [15], a medical report generation dataset. Our code is available at https://github.com/Vision-CAIR/VisualGPT.

As large Pre-trained Language Models (PLMs) trained on large amounts of data in an unsupervised manner become more ubiquitous, identifying various types of bias in the text has come into sharp focus. Existing `Stereotype Detection' datasets mainly adopt a diagnostic approach toward large PLMs. Blodgett et. al. (2021) show that there are significant reliability issues with the existing benchmark datasets. Annotating a reliable dataset requires a precise understanding of the subtle nuances of how stereotypes manifest in text. In this paper, we annotate a focused evaluation set for `Stereotype Detection' that addresses those pitfalls by de-constructing various ways in which stereotypes manifest in text. Further, we present a multi-task model that leverages the abundance of data-rich neighboring tasks such as hate speech detection, offensive language detection, misogyny detection, etc., to improve the empirical performance on `Stereotype Detection'. We then propose a reinforcement-learning agent that guides the multi-task learning model by learning to identify the training examples from the neighboring tasks that help the target task the most. We show that the proposed models achieve significant empirical gains over existing baselines on all the tasks.

Large pre-trained language models (PLMs) have led to great success on various commonsense question answering (QA) tasks in an end-to-end fashion. However, little attention has been paid to what commonsense knowledge is needed to deeply characterize these QA tasks. In this work, we proposed to categorize the semantics needed for these tasks using the SocialIQA as an example. Building upon our labeled social knowledge categories dataset on top of SocialIQA, we further train neural QA models to incorporate such social knowledge categories and relation information from a knowledge base. Unlike previous work, we observe our models with semantic categorizations of social knowledge can achieve comparable performance with a relatively simple model and smaller size compared to other complex approaches.

We survey a current, heated debate in the artificial intelligence (AI) research community on whether large pretrained language models can be said to understand language—and the physical and social situations language encodes—in any humanlike sense. We describe arguments that have been made for and against such understanding and key questions for the broader sciences of intelligence that have arisen in light of these arguments. We contend that an extended science of intelligence can be developed that will provide insight into distinct modes of understanding, their strengths and limitations, and the challenge of integrating diverse forms of cognition.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Homonymy can easily raise lexical ambiguity due to the misunderstanding of its multiple senses.Correct recognition of homonym sense greatly relies on its surrounding context.This ambiguous nature makes homonyms an appropriate testbed for examining the contextualization capability of pre-trained (PLM) and large language models (LLMs).Considering the impact of part of speech (POS) on homonym disambiguation and the prevalence of English-focused studies in word embedding research, this study extends to Chinese and provides a comprehensive layer-wise analysis of homonym representations in both languages, spanning same and different POS categories, across four families of PLMs/LLMs (BERT, GPT-2, Llama 3, Qwen 2.5).Through the creation of a synthetic dataset and computation of disambiguation score (D-Score), we found that: (1) no universal layer depth excels in differentiating homonym representations; (2) bidirectional models produce better contextualized homonym representations compared to much larger autoregressive models; (3) most importantly, POS affects homonym representations in models in ways that differ from human research findings.The individual differences between LLMs uncovered in our study challenge the simplistic understanding of their inner workings.This reveals a compelling research frontier: conducting controlled experiments with purposefully manipulated inputs to enhance the interpretability of LLMs.We have made our dataset and codes available publicly at https://github.com/neurothew/ exploring-homonym-rep-in-llm.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

Knowledge tags on math questions are essential for education as they provide great convenience to both teachers and students in finding desired practicing material during the learning process. Traditionally, pedagogical experts have been responsible for these annotations, as the task requires a profound understanding of how to link problem-solving logic with relevant knowledge concepts. However, with the advancement of sophisticated text encoding algorithms, such as pre-trained language models (PLMs), researchers have developed automated knowledge-tagging systems that leverage deep semantic embeddings. This work explores the automation of this task using Large Language Models (LLMs), addressing the limitations of earlier encoding-based methods—particularly in complex cases that require extensive domain expertise and nuanced concept definitions. By harnessing LLMs' advanced reasoning and tool-using capabilities, we overcome the challenges posed by previous approaches, achieving human-level performance with large-scale LLMs. Our findings underscore the significant potential of LLMs in educational applications.

Large pre-trained language models (LLMs) have demonstrated remarkable ability to encode vast world knowledge in their parameters, but this static, implicit storage poses challenges for keeping models up-to-date and providing provenance for factual claims [1]. Retrieval-Augmented Generation (RAG) methods address these issues by equipping LLMs with external non-parametric memory (typically a vector database of documents) that can be queried at inference time [1][2]. However, in standard RAG pipelines the retrieval module is an external add-on, not a native part of the model architecture. We propose Retrieval-Native Language Models (RLLMs), a new paradigm that treats vector-based memory as a first-class component of the model. RLLM introduces a unified architecture with three channels of knowledge: (1) Parametric memory in the pretrained model weights (static knowledge); (2) a live retrieval channel that shares an embedding space with the generator for continual intake of external vectors; and (3) an internal Bayesian attention router that dynamically adjusts how information flows from retrieval into generation based on the estimated utility of retrieved results (e.g. relevance, correctness, recency). By tightly integrating retrieval into the model's multi-head attention layers and training the retriever and generator jointly, RLLMs align the model's representations with the memory index and enable endto-end learning of when to trust retrieved evidence. We present a theoretical analysis of how unified embedding alignment and multi-channel attention improve efficiency and reduce hallucination. We describe the RLLM architecture in detail, including pseudo-code for the retrieval-augmented attention mechanism and a Bayesian feedback loop that updates routing weights in light of retrieval success or failure signals. In experiments on knowledge-intensive QA benchmarks (NaturalQuestions, HotpotQA) and a streaming enterprise wiki scenario, RLLM outperforms traditional RAG baselines. It achieves higher answer accuracy with 30% fewer parameters than fully-parametric models by leveraging its vector memory, and quickly adapts to new information with minimal latency. This work demonstrates that making retrieval a native component of LLMs yields significant gains in knowledge access, interpretability, and continual learning, suggesting a new direction for building more flexible and powerful language models.

Log statements play a critical role in modern software development, capturing essential runtime information necessary for software maintenance. Recently, new techniques have been developed to automate logging activities, allowing log statements to be injected into code by identifying specific code locations, selecting the appropriate log level, and generating meaningful log messages that describe the behavior being logged. Although automated logging in production code has attracted significant attention, little focus has been given to the injection of logs in test code. To fill this gap, we conduct an empirical study on 5,206,759 Java test methods collected from 6,405 GitHub projects to explore and disclose the effectiveness and limitations of Pre-trained Language Models (PLMs) and Large Language Models (LLMs) for generating and injecting test log statements. Our findings demonstrate that general-purpose LLMs like GPT-3.5-Turbo, when properly instructed to inject logging statements in test methods, performs comparably to the best-performing PLMs on predicting log level. Additionally, GPT-3.5-Turbo substantially outperforms the best in PLMs on predicting log position with a 33.97% improvement while also achieving superior performance in predicting log messages in terms of BLEU and ROUGE . This work takes the first step toward evaluating the capability of PLMs and LLMs to generate test log statement. This work takes the first step toward evaluating the capability of PLMs and LLMs to generate test log statements. To facilitate future research, we have open-sourced all data and source code used in this work.

This paper presents MuCGEC, a multi-reference multi-source evaluation dataset for Chinese Grammatical Error Correction (CGEC), consisting of 7,063 sentences collected from three Chinese-as-a-Second-Language (CSL) learner sources. Each sentence is corrected by three annotators, and their corrections are carefully reviewed by a senior annotator, resulting in 2.3 references per sentence. We conduct experiments with two mainstream CGEC models, i.e., the sequence-to-sequence model and the sequence-to-edit model, both enhanced with large pretrained language models, achieving competitive benchmark performance on previous and our datasets. We also discuss CGEC evaluation methodologies, including the effect of multiple references and using a char-based metric. Our annotation guidelines, data, and code are available at \url{https://github.com/HillZhang1999/MuCGEC}.

This paper describes the models designated for the MEDIQA 2019 shared tasks by the team PANLP. We take advantages of the recent advances in pre-trained bidirectional transformer language models such as BERT (Devlin et al., 2018) and MT-DNN (Liu et al., 2019b). We find that pre-trained language models can significantly outperform traditional deep learning models. Transfer learning from the NLI task to the RQE task is also experimented, which proves to be useful in improving the results of fine-tuning MT-DNN large. A knowledge distillation process is implemented, to distill the knowledge contained in a set of models and transfer it into an single model, whose performance turns out to be comparable with that obtained by the ensemble of that set of models. Finally, for test submissions, model ensemble and a re-ranking process are implemented to boost the performances. Our models participated in all three tasks and ranked the 1st place for the RQE task, and the 2nd place for the NLI task, and also the 2nd place for the QA task.