Implementation of a large-scale language model adaptation in a cloud environment

Background:Understanding complex biological pathways, including gene-gene interactions and gene regulatory networks, is critical for exploring disease mechanisms and drug development. Manual literature curation of biological pathways is useful but cannot keep up with the exponential growth of the literature. Large-scale language models (LLMs), notable for their vast parameter sizes and comprehensive training on extensive text corpora, have great potential in automated text mining of biological pathways.Method:This study assesses the effectiveness of 21 LLMs, including both API-based models and open-source models. The evaluation focused on two key aspects: gene regulatory relations (specifically, ‘activation’, ‘inhibition’, and ‘phosphorylation’) and KEGG pathway component recognition. The performance of these models was analyzed using statistical metrics such as precision, recall, F1 scores, and the Jaccard similarity index.Results:Our results indicated a significant disparity in model performance. Among the API-based models, ChatGPT-4 and Claude-Pro showed superior performance, with an F1 score of 0.4448 and 0.4386 for the gene regulatory relation prediction, and a Jaccard similarity index of 0.2778 and 0.2657 for the KEGG pathway prediction, respectively. Open-source models lagged their API-based counterparts, where Falcon-180b-chat and llama1–7b led with the highest performance in gene regulatory relations (F1 of 0.2787 and 0.1923, respectively) and KEGG pathway recognition (Jaccard similarity index of 0.2237 and 0. 2207, respectively).Conclusion:LLMs are valuable in biomedical research, especially in gene network analysis and pathway mapping. However, their effectiveness varies, necessitating careful model selection. This work also provided a case study and insight into using LLMs as knowledge graphs.

Large-scale language models (LLMs) have basically changed the limits of what can be achieved by natural language processing and this was the case of GPT-3 at its very first demonstration back in 2020. This study outlines in detail their journey. It goes through all the concepts of their architecture. Besides, it describes the training process. The article records the gradual increasing of their volumetry and the consequent differences of their properties along the way. Different methods of testing these models are also indicated. Besides, it refers to their application in various fields. Besides, this research work deals with the advent of multi-modal systems capable of autonomous decision-making. We have centered our work on the landmark research that has been published in the period of 2022–2024. We reckon the self-supervised pre-training, instruction tuning, preference optimization, and reasoning-directed reinforcement learning as four main areas that contributed to the development of LLMs. These approaches have converted LLMs to be no longer just the text generators but rather the most versatile AI systems that they are. Our focus is on their use in the medical field and bioinformatics. We address the challenge of explaining the functioning of large language models (LLMs). We tell the story of in-context learning. We discuss the development of LLMs-based agents. At the end, we are identifying the major limitations of the present-time and, simultaneously, suggesting a few ideas for the future in this fast-evolving area.

Multimodal large-scale language modeling has become the mainstream approach in natural language processing tasks and has been applied to various cross-modal fields such as image description and visual question answering. However, large-scale language modeling has high computational complexity and a large operational scale, which presents significant challenges for deployment in many resource-constrained scenarios. To address such problems, a lightweight multimodal framework, LLaVA-GM, is proposed, based on LLaVA, which can be deployed on devices with low resource requirements and has greatly reduced model parameters. It can also be tested on common VQA tasks and achieves good performance. The main contributions and work are as follows: First, it is found that the backbone of the Vicuna language model in LLaVA is too redundant. When fine-tuning downstream tasks, a very small amount of data sets is difficult to affect the language model. It is replaced with a new Gemma language model, thereby achieving fast task-specific adaptation with fewer parameters and data. Second, in response to the problem of information redundancy, the MoE mixed expert model is introduced. This model can be used in combination with itself, combining the MoE mixed expert model with Gemma to reduce the amount of computation while maintaining performance. Directly training the entire model will lead to a decline in performance. A multi-stage training strategy is adopted to maintain performance. First, the MLP layer is trained for visual adaptation, then the entire Gemma model is trained to improve multimodal capabilities, and finally only the MoE layer is trained for sparsification to ensure a smooth transition from dense models to sparse models. The experiment was tested on multiple VQA datasets and achieved good performance, confirming the potential of this compact model in downstream multimodal applications.

The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.

In this work, we study the impact of Large-scale Language Models (LLM) on Automated Speech Recognition (ASR) of YouTube videos, which we use as a source for long-form ASR. We demonstrate up to 8% relative reduction in Word Error Eate (WER) on US English (en-us) and code-switched Indian English (en-in) long-form ASR test sets and a reduction of up to 30% relative on Salient Term Error Rate (STER) over a strong first-pass baseline that uses a maximum-entropy based language model. Improved lattice processing that results in a lattice with a proper (non-tree) digraph topology and carrying context from the 1-best hypothesis of the previous segment(s) results in significant wins in rescoring with LLMs. We also find that the gains in performance from the combination of LLMs trained on vast quantities of available data (such as C4 [1]) and conventional neural LMs is additive and significantly outperforms a strong first-pass baseline with a maximum entropy LM.

Machine reading comprehension (MRC) has achieved remarkable progress in the open domain in recent years, mainly due to large-scale pre-trained language models. However, it performs much worse in specific domains such as the medical field due to the lack of extensive training data and professional structural knowledge neglect. Besides, the opacity of the large-scale neural language model also limits its practical utility. As an effort, we first collect a medical machine reading comprehension question dataset with experts' annotations for the National Licensed Pharmacist Examination in China. It is a challenging medical examination with a passing rate of less than 14.2% in 2018. Then we propose a novel reading comprehension model eKMQA, which can fully exploit the structural medical knowledge (i.e., medical knowledge graph) and the reference medical plain text (i.e., text snippets retrieved from reference books). Meanwhile, it improves the explainability of the model by leveraging the knowledge entailed in explanations as an additional distillation signal for more efficient learning. The experimental results demonstrate that our method outperforms existing competitive models with a large margin, which improves prediction accuracy on the test set and can generate reasonable free-text rationales for its predictions even with a small-scale training corpus.

This study investigates the potential of artificial intelligence, particularly Natural Language Processing and large-scale language models, to improve resource management and service access for individuals with autism in Alabama. The research aims to explore and evaluate the potential of AI-driven tools to address challenges in navigating complex datasets and supporting social work practices. We designed and tested AI systems, including general language models, domain-specific chatbots powered by advanced language models, and a Retrieval-Augmented Generation framework. A standardized set of queries was used to simulate real-world scenarios encountered by social workers engaged in autism service coordination. System performance was evaluated based on precision, recall, and response accuracy. Results demonstrated that the Retrieval-Augmented Generation (RAG) framework achieved superior performance compared to traditional NLP methods and general-purpose language models, attaining approximately 90–96% precision and recall across evaluated query types. RAG outperformed the domain-specific GPT-4 chatbot by approximately 5–12 percentage points in F1 score, with the largest gains observed for queries requiring geographic specificity, multiple constraints, or complex contextual understanding. The integration of domain-specific retrieval significantly enhanced the accuracy, contextual relevance, and usability of generated responses. The findings highlight the transformative potential of AI-driven tools in improving social work efficiency and enhancing healthcare equity. By streamlining care coordination and delivering accurate, contextually relevant information, these systems offer scalable solutions to improve access to autism-related services. Future research should focus on addressing data quality, minimizing biases, and ensuring ethical deployment to build trust and support widespread adoption of these tools.

Pretrained language models have achieved great success in a wide range of natural language processing (NLP) problems, because they learn language representations from large-scale text corpora and can adapt to downstream tasks by finetuning them on annotated task data. However, such success relies on both large-scale text and annotated data, so the lack of training data is a major practical problem for many languages, especially low-resource languages. In this paper, we explore whether a pretrained English language model can benefit non-English NLP systems in low-resource scenarios, i.e., with limited text corpora or annotated data. To achieve this, we first propose cross-lingual knowledge transfer methods and then validate our methods in low-resource scenarios. Specifically, our cross-lingual knowledge transfer methods are applied in the training stages of language model pretraining or downstream finetuning. At the two stages, the methods are designed for the transfer of upstream general knowledge or downstream task-specific knowledge, respectively. In the experiments, we perform pretraining and finetuning with limited non-English data to simulate the low-resource scenarios. We evaluate our methods on ten downstream tasks over a wide range of languages, and present systematic comparisons among various knowledge transfer methods. Experimental results show that our methods successfully leverage a pretrained English language model to improve task performance in other languages. Besides, we demonstrate the multilinguality of the English language model in various application scenarios. Our findings imply the possibility to improve low-resource-language NLP systems with large-scale English language models.

LLMCL-GEC: Advancing grammatical error correction with LLM-driven curriculum learning

The cinematographic vanguard has historically been important in fostering cinematographic creation and innovation. In the digital age, its study acquires a new dimension of relevance, since the analysis of these vanguard movements provides students with tools to understand how cinema has evolved as a means of artistic expression and social manifestation. This knowledge is essential to developing robust critical thinking, capable of evaluating not only the content of a film, but also its form, its historical context, and its cultural relevance from and in its immediate environment. With the arrival of large-scale language models (LLM) and generative artificial intelligence (IAGen), students face an increasingly complex social landscape where technologies are transforming the production, distribution and analysis of movies. LLMs can generate scripts or critical analyses, while IAGens can create visual effects or even entire film sequences, modifying the cinematic concept. This essay analyzes the importance of cinematic avant-garde in the development of students’ critical thinking, especially in the current context of the proliferation of large-scale language models in the form of generative artificial intelligences.

Multi-microphone speech recognition integrating beamforming, robust feature extraction, and advanced DNN/RNN backend

This paper presents a simple recipe to trainstate-of-the-art multilingual Grammatical Error Correction (GEC) models. We achieve this by first proposing a language-agnostic method to generate a large number of synthetic examples. The second ingredient is to use large-scale multilingual language models (up to 11B parameters). Once fine-tuned on language-specific supervised sets we surpass the previous state-of-the-art results on GEC benchmarks in four languages: English, Czech, German and Russian. Having established a new set of baselines for GEC, we make our results easily reproducible and accessible by releasing a CLANG-8 dataset. It is produced by using our best model, which we call gT5, to clean the targets of a widely used yet noisy Lang-8 dataset. cLang-8 greatly simplifies typical GEC training pipelines composed of multiple fine-tuning stages – we demonstrate that performing a single fine-tuning stepon cLang-8 with the off-the-shelf language models yields further accuracy improvements over an already top-performing gT5 model for English.

Analyzing the potential benefits and use cases of ChatGPT as a tool for improving the efficiency and effectiveness of business operations

Syntactic parsing is one of the areas in Natural Language Processing. The development of large-scale multilingual language models has enabled cross-lingual parsing approaches, which allows us to develop parsers for languages that do not have treebanks available. However, these approaches rely on the assumption that languages share orthographic representations and lexical entries. In this article, we investigate methods for developing a dependency parser for Xibe, a low-resource language that is written in a unique script. We first investigate lexicalized monolingual dependency parsing experiments to examine the effectiveness of word, part-of-speech, and character embeddings as well as pre-trained language models. Results show that character embeddings can significantly improve performance, while pre-trained language models decrease performance since they do not recognize the Xibe script. We also train delexicalized monolingual models, which yield competitive results to the best lexicalized model. Since the monolingual models are trained on a very small training set, we also investigate lexicalized and delexicalized cross-lingual models. We use six closely related languages as source language, which cover a wide range of scripts. In this setting, the delexicalized models achieve higher performance than lexicalized models. A final experiment shows that we can increase performance of the cross-lingual model by combining source languages and selecting the most similar sentences to Xibe as training set. However, all cross-lingual parsing results are still considerably lower than the monolingual model. We attribute the low performance of cross-lingual methods to syntactic and annotation differences as well as to the impoverished input of Universal Dependency Part-of-Speech tags that the delexicalized model has access to.

DeepSeek-R1 demonstrated the highest overall quality in content generation, followed by Claude-4-sonnet. Evaluations using FKGL, SMOG index, and ARI all indicated that Claude-4-sonnet exhibited the best readability. Given that improvements in quality and readability can enhance patient engagement and reduce anxiety, these 2 models should be prioritized for patient education applications. Future efforts should focus on integrating these advantages to develop more reliable large-scale medical language models.