Masked Language Modeling for Resource Constrained Biological Natural Language Processing.

Introduction: Tumor single-cell data provide unique insights into the cellular dynamics of cancer progression and therapeutic outcomes. Inspired by advancements in natural language processing (NLP), foundation models are increasingly used in single-cell analyses to uncover biological insights and identify clinical biomarkers. However, the performance of these models on cancer-specific tasks remains underexplored. We present an extensive evaluation of foundation models trained on millions of single-cell profiles for cancer-focused investigations. Methods: We evaluated nine predictive models-three traditional methods and six foundation models (four Geneformer and two scGPT models), across various downstream tasks. Traditional models used embeddings from high variable genes (HVG), principal components (PCA), and generative model embedding (scVI), each paired with a random forest classifier for downstream classification. The evaluated foundation models varied in architecture sizes, pretraining dataset sizes (30M to 95M cell profiles), and training datasets, which were either generic or cancer-specific. We assessed the quality of representations by measuring their predictive performance on lung cancer treatment status (TKI treated vs. treatment-naive patients) and breast cancer subtypes (ER+ vs. TNBC). We examined how model characteristics influenced their performance in these domains for all tasks. Results: Zero-shot evaluations of the foundation models showed superior single-cell data clustering, measured by average silhouette width (ASW), compared to traditional methods (HVG, PCA, scVI), suggesting that pre-trained embeddings from millions of single-cell profiles may offer enhanced representations. We thus evaluated the learned representation on downstream predictive tasks, showing that foundation models with a larger pretraining dataset (95M cells), cancer-specific training, larger input size (4096 genes), and deeper model (12 layers) achieved the best performance among foundation models (F1=0.97) for predicting treatment status in lung cancer. In contrast, smaller and generic foundation models had significantly lower performance, however, additional self-supervised training on task-specific data rescued their performance to match larger foundation models. Similar trends were observed in predicting breast cancer subtypes. Notably, the best foundation model’s performance was not significantly different from the best traditional method (PCA) in predicting breast cancer subtypes. Conclusions: These results highlight key factors driving foundation model performance on cancer-specific tasks. While current foundation models show inconsistent contributions to cancer-specific questions, observed performance trends highlight their potential to enhance single-cell data representation and drive significant advancements in cancer research and precision medicine. Citation Format: Haitham Elmarakeby, Ahmed Roman, Shreya Johri, Eliezer M. Van Allen. Evaluation of single-cell foundation models for cancer outcome predictions [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 5084.

Loading models pre-trained on the large-scale corpus in the general domain and fine-tuning them on specific downstream tasks is gradually becoming a paradigm in Natural Language Processing. Previous investigations prove that introducing a further pre-training phase between pre-training and fine-tuning phases to adapt the model on the domain-specific unlabeled data can bring positive effects. However, most of these further pre-training works just keep running the conventional pre-training task, e.g., masked language model, which can be regarded as the domain adaptation to bridge the data distribution gap. After observing diverse downstream tasks, we suggest that different tasks may also need a further pre-training phase with appropriate training tasks to bridge the task formulation gap. To investigate this, we carry out a study for improving multiple task-oriented dialogue downstream tasks through designing various tasks at the further pre-training phase. The experiment shows that different downstream tasks prefer different further pre-training tasks, which have intrinsic correlation and most further pre-training tasks significantly improve certain target tasks rather than all. Our investigation indicates that it is of great importance and effectiveness to design appropriate further pre-training tasks modeling specific information that benefit downstream tasks. Besides, we present multiple constructive empirical conclusions for enhancing task-oriented dialogues.

Traditional approaches for learning representations from somatic mutation data often rely on the distribution of motif counts, which have limitations, including the use of small sequence contexts and the exclusion of critical information, such as mutation location and interactions among mutations. Recent efforts have leveraged advances in natural language processing (NLP) to improve representation learning from somatic mutations by incorporating larger sequence contexts into shallow networks. More recently, studies have demonstrated the feasibility of leveraging attention mechanisms to learn meaningful representations. However, these methods primarily use supervised or weakly supervised learning paradigms, which are inherently task-specific. The development of general-purpose representations, by harnessing the vast amount of somatic mutation data through self-supervised techniques, remains underexplored. Such representations could drive progress in applications including tumor subtype classification, biomarker discovery, patient stratification, drug response prediction, and pathway enrichment analysis. This study aims to develop a deep, unsupervised learning framework that leverages abundant somatic mutation data to generate robust, general-purpose representations. By adapting recent advances in self-supervised learning, the proposed approach seeks to establish a foundation model capable of capturing meaningful patterns in mutation data for diverse downstream tasks. The framework employs novel tokenization and embedding strategies to encode mutation sequences and utilizes transformer-based architectures to model contextual dependencies. The process of inputting mutation data into a transformer architecture, particularly the tokenization step, is critical for this problem, and we explore various tokenization strategies. Benchmarking these representations will include evaluating their utility across tumor subtype classification, biomarker discovery, patient stratification, drug response prediction, identification of mutational signatures. Although the model is under development, experiments from prior work—limited to mutation identifiers without leveraging sequence motifs, mutation location, or other contextual information—have demonstrated the feasibility of encoding mutational data into meaningful representations using transformer-based architectures and self-supervised learning. Building on these findings, this framework will benchmark its performance on multiple downstream tasks, including tumor subtype classification and patient stratification. Citation Format: Marzieh Haghighi, Gustavo Alonso Arango Argoty, Etai Jacob. Self-supervised representation learning of somatic mutational data [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 5059.

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if any, in scientific information extraction tasks. We offer possible explanations for this surprising result.

Prompting techniques unlock valuable insights from pre-trained models, leading to a remarkable performance in Natural Language Processing (NLP). Inspired by NLP advancements, various graph prompt methods have emerged to transfer knowledge from pre-trained graph models to diverse downstream graph tasks. However, designing appropriate prompts for different downstream tasks remains a significant challenge due to substantial differences between pre-training graph tasks and downstream graph tasks. Moreover, existing graph prompt methods prevent transferability across different domains as they independently pre-train a model for each dataset. To address these challenges, we propose Graph Pretraining and Task-Specific Prompting (GTSP). To the best of our knowledge, GTSP is the first unified pre-training framework to handle cross-domain graph data and multiple types of graph tasks simultaneously. The key idea of our framework is to utilize similar label features between pre-training tasks and downstream tasks to seamlessly integrate the downstream task prompt into the pretrained model. We then propose a task-specific prompting architecture and design a label similarity matrix as prompt input. With GTSP, the pre-trained model can adapt to various types of downstream graph tasks, effectively avoiding the challenges of forcibly altering graph task forms. Experimental analysis conducted on benchmark datasets demonstrates the superiority of our method.

The inspection of documents written in natural language with computers has become feasible thanks to the advances in Natural Language Processing (NLP) techniques. However, certain applications require a deeper semantic analysis of the text to produce good results. In this article, we present an exploratory study of semantic-aware NLP techniques for discovering latent concerns in use case specifications. For this purpose, we propose two NLP techniques, namely: semantic clustering and semantically-enriched rules. After evaluating these two techniques and comparing them with a technique developed by other researchers, results have showed that semantic NLP techniques hold great potential for detecting candidate concerns. Particularly, if these techniques are properly configured, they can help to reduce the efforts of requirement analysts and promote better quality in software development.

Software development relies heavily on manual processes for transforming requirements into software artifacts such as models, source code, or test cases. Requirements are the starting point for these transformations, and they are typically written in natural language. However, hardly any automated tools exist that translate natural language texts into software artifacts. We propose to adapt recent advances in natural language processing and semantic technologies to generate UML models, test cases, and perhaps even source code, from natural language input. Though earlier efforts in automatic programming had limited success, we are now in a situation where extracting meaning from text has made substantial progress. For example, encouraging results for generating UML models from textual requirements have been achieved. It might even be possible to generate executable test cases. An intermediate step would be to generate tests from API documentation (which would also be a useful capability in itself). An even greater advance would be to automate rote coding. In all of these cases, entirely new classes of software tools would be needed to extract and process the semantics inherent in natural language texts, augment them with tacit knowledge from ontologies and domain models, and, where necessary, ask humans to clarify ambiguities. Though speculative, a determined, long-term effort in translating text to software could automate and accelerate software development to an unprecedented degree.

Answer set Prolog, or AnsProlog in short, is one of the leading knowledge representation (KR) languages with a large body of theoretical and building block results, several implementations and reasoning and declarative problem solving applications. But it shares the problem associated with knowledge acquisition with all other KR languages; most knowledge is entered manually by people and that is a bottleneck. Recent advances in natural language processing have led to some systems that convert natural language sentences to a logical form. Although these systems are in their infancy, they suggest a direction to overcome the above mentioned knowledge acquisition bottleneck. In this paper we discuss some recent work by us on developing applications that process logical forms of natural language text and use the processed result together with AnsProlog rules to do reasoning and problem solving.

Task-oriented dialogue systems employ third-party APIs to serve end-users via natural language interactions. while existing advances in Natural Language Processing (NLP) and Machine Learning (ML) techniques have produced promising and useful results to recognize user intents, the synthesis of API calls to support a broad range of potentially complex user intents is still largely a manual and costly process. In this paper, we propose a new approach to recognize and realize complex user intents. Our approach relies on a new rule-based technique that leverages both (i) natural language features extracted using existing NLP and ML techniques and (ii) contextual knowledge to capture the different classes of complex intents. We devise a context knowledge service to capture the requisite contextual knowledge.KeywordsTask-oriented conversational botsComplex intent recognitionContext knowledgeSlot value inference

Natural Language Processing (NLP) is an essential element of computational linguistics and artificial ‎intelligence, enabling fluid interactions between humans and machines. Social networking networks ‎produce substantial volumes of user-generated text daily, offering both opportunities and challenges for ‎NLP researchers. Social media discourse's informal, dynamic, and context-dependent characteristics ‎necessitate specific NLP techniques for precise processing and analysis. This study thoroughly examines NLP applications in social media, including essential tasks such as sentiment analysis, topic ‎modeling, misinformation detection, and hate speech identification. It examines the influence of machine ‎learning and deep learning methodologies, particularly transformer models, on the advancement of NLP ‎capabilities. This study also emphasizes the ethical issues related to NLP-driven social media apps, ‎including data privacy, algorithmic bias, and the regulation of misinformation. The paper continues by ‎discussing emerging research paths, highlighting the necessity for adaptable and ethical NLP solutions in ‎the changing social media environment‎.

There has been a recent wave of work assessing the fairness of machine learning models in general, and more specifically, on natural language processing (NLP) models built using machine learning techniques. While much work has highlighted biases embedded in state-of-the-art language models, and more recent efforts have focused on how to debias, research assessing the fairness and performance of biased/debiased models on downstream prediction tasks has been limited. Moreover, most prior work has emphasized bias along a single dimension such as gender or race. In this work, we benchmark multiple NLP models with regards to their fairness and predictive performance across a variety of NLP tasks. In particular, we assess intersectional bias - fairness across multiple demographic dimensions. The results show that while current debiasing strategies fare well in terms of the fairness-accuracy trade-off (generally preserving predictive power in debiased models), they are unable to effectively alleviate bias in downstream tasks. Furthermore, this bias is often amplified across demographic dimensions. We conclude with implications for future NLP debiasing research.

Over the last two decades, electronic health records (EHRs) have evolved as a crucial repository for patient health data, encompassing both structured and unstructured information. The objective of EHR is to enhance patient care, and also to serve as tool for reducing costs, managing population health, and supporting clinical research. Natural language processing (NLP) has emerged as a valuable tool for analyzing narrative EHR data, particularly in named entity recognition (NER) tasks. But traditional NLP methodologies encounter challenges to analyze biomedical text due to variations in word distributions. Recent advancements in NLP, specifically bidirectional encoder representations from transformers (BERT), offer promising solutions. BERT utilizes a masked language model base and bidirectional transformer encoder architecture to learn deep contextual representations of words. The work provides an overview of the BERT algorithm, its architecture, and details of and its varaints like BioBERT and ClinicalBERT for various clinical text classification applications.

The increasing demand for cross-lingual communication in a globally interconnected world has spurred significant advancements in Natural Language Processing (NLP), a branch of artificial intelligence aimed at enabling computers to comprehend, generate, and process human language. Over the past few years, NLP has transitioned from traditional rule-based and statistical approaches to deep learning-based methods, with pre-trained models like BERT and GPT demonstrating remarkable performance across a variety of tasks, including text classification and machine translation. However, the linguistic diversity of the world's languages presents ongoing challenges, as many NLP models are primarily developed for high-resource languages like English, leaving other languages underserved. This paper examines the linguistic features of three representative languagesChinese, Japanese, and Englishand explores NLP methodologies tailored to their distinct grammatical, lexical, and structural properties. Through comparative linguistic analysis, this study discusses the challenges of multilingual NLP and the potential for cross-lingual applications. The findings underscore the importance of adapting NLP models to different language characteristics to enhance the effectiveness and inclusivity of cross-lingual processing.

Research on lifestyle changes during the COVID-19 pandemic often relies on Likert-type scale question surveys. Survey participants respond to questions by selecting one of the numerically ordered choices, "Strongly Disagree"=1, "Disagree"=2, "Neutral"=3, "Agree"=4, and "Strongly Agree"=5. Analyzing Likert-type data requires statistical methods beyond approaches like linear regression. First, it is unclear if the distance between choices are truly equal. For example, are "Agree" and "Strongly Agree" more close than "Neutral" and "Agree? Second, summarizing results using traditional means makes little sense. For example, would a mean of 4.5 imply "Agree and a half"? Finally, participants tend to select more central choices and less extremes.

Tobler's First Law of Geography (TFL) is one of the key reasons why is special. The law, which states that is related to everything else, but near things are more related than distant things, is central to the management, presentation, and analysis of geographic information. However, despite the importance of TFL, we have a limited general understanding of its domain-neutral properties. In this paper, we leverage recent advances in the natural language processing domain of semantic relatedness estimation to, for the first time, robustly evaluate the extent to which relatedness between spatial entities decreases over distance in a domain-neutral fashion. Our results reveal that, in general, TFL can indeed be considered a globally recognized domain-neutral property of geographic information but that there is a distance beyond which being nearer, on average, no longer means being more related.