Do You Follow?

Background:Understanding complex biological pathways, including gene-gene interactions and gene regulatory networks, is critical for exploring disease mechanisms and drug development. Manual literature curation of biological pathways is useful but cannot keep up with the exponential growth of the literature. Large-scale language models (LLMs), notable for their vast parameter sizes and comprehensive training on extensive text corpora, have great potential in automated text mining of biological pathways.Method:This study assesses the effectiveness of 21 LLMs, including both API-based models and open-source models. The evaluation focused on two key aspects: gene regulatory relations (specifically, ‘activation’, ‘inhibition’, and ‘phosphorylation’) and KEGG pathway component recognition. The performance of these models was analyzed using statistical metrics such as precision, recall, F1 scores, and the Jaccard similarity index.Results:Our results indicated a significant disparity in model performance. Among the API-based models, ChatGPT-4 and Claude-Pro showed superior performance, with an F1 score of 0.4448 and 0.4386 for the gene regulatory relation prediction, and a Jaccard similarity index of 0.2778 and 0.2657 for the KEGG pathway prediction, respectively. Open-source models lagged their API-based counterparts, where Falcon-180b-chat and llama1–7b led with the highest performance in gene regulatory relations (F1 of 0.2787 and 0.1923, respectively) and KEGG pathway recognition (Jaccard similarity index of 0.2237 and 0. 2207, respectively).Conclusion:LLMs are valuable in biomedical research, especially in gene network analysis and pathway mapping. However, their effectiveness varies, necessitating careful model selection. This work also provided a case study and insight into using LLMs as knowledge graphs.

Language steganography in social networks primarily focuses on embedding secret information into social media text efficiently to achieve covert communication. The misuse of such techniques could pose significant potential threats to public cyberspace, such as the spread of malicious code, commands, or viruses. Existing social text steganalysis techniques mainly focus on the analysis of individual social media texts. However, the information content in a single text is very limited, leading to poor detection performance in practical applications. To address this challenge, this paper proposes a social text steganalysis method that combines large-scale language models with common-sense knowledge graphs (STLC-KG). This method first uses knowledge graphs to expand the knowledge contained in the text under investigation, enriching its linguistic expression, and then utilizes large-scale language models to extract the linguistic features of the social text. The results of tests conducted on three mainstream social media platforms demonstrate that the proposed method significantly improves the performance of social text steganalysis.

Large-scale language models (LLMs) have basically changed the limits of what can be achieved by natural language processing and this was the case of GPT-3 at its very first demonstration back in 2020. This study outlines in detail their journey. It goes through all the concepts of their architecture. Besides, it describes the training process. The article records the gradual increasing of their volumetry and the consequent differences of their properties along the way. Different methods of testing these models are also indicated. Besides, it refers to their application in various fields. Besides, this research work deals with the advent of multi-modal systems capable of autonomous decision-making. We have centered our work on the landmark research that has been published in the period of 2022–2024. We reckon the self-supervised pre-training, instruction tuning, preference optimization, and reasoning-directed reinforcement learning as four main areas that contributed to the development of LLMs. These approaches have converted LLMs to be no longer just the text generators but rather the most versatile AI systems that they are. Our focus is on their use in the medical field and bioinformatics. We address the challenge of explaining the functioning of large language models (LLMs). We tell the story of in-context learning. We discuss the development of LLMs-based agents. At the end, we are identifying the major limitations of the present-time and, simultaneously, suggesting a few ideas for the future in this fast-evolving area.

LLMCL-GEC: Advancing grammatical error correction with LLM-driven curriculum learning

This paper presents a system of large-scale language model adaptation for daily generated big-size text corpus using MapReduce in a cloud environment. Our large-scale trigram language model, consis...

The cinematographic vanguard has historically been important in fostering cinematographic creation and innovation. In the digital age, its study acquires a new dimension of relevance, since the analysis of these vanguard movements provides students with tools to understand how cinema has evolved as a means of artistic expression and social manifestation. This knowledge is essential to developing robust critical thinking, capable of evaluating not only the content of a film, but also its form, its historical context, and its cultural relevance from and in its immediate environment. With the arrival of large-scale language models (LLM) and generative artificial intelligence (IAGen), students face an increasingly complex social landscape where technologies are transforming the production, distribution and analysis of movies. LLMs can generate scripts or critical analyses, while IAGens can create visual effects or even entire film sequences, modifying the cinematic concept. This essay analyzes the importance of cinematic avant-garde in the development of students’ critical thinking, especially in the current context of the proliferation of large-scale language models in the form of generative artificial intelligences.

In this work, we study the impact of Large-scale Language Models (LLM) on Automated Speech Recognition (ASR) of YouTube videos, which we use as a source for long-form ASR. We demonstrate up to 8% relative reduction in Word Error Eate (WER) on US English (en-us) and code-switched Indian English (en-in) long-form ASR test sets and a reduction of up to 30% relative on Salient Term Error Rate (STER) over a strong first-pass baseline that uses a maximum-entropy based language model. Improved lattice processing that results in a lattice with a proper (non-tree) digraph topology and carrying context from the 1-best hypothesis of the previous segment(s) results in significant wins in rescoring with LLMs. We also find that the gains in performance from the combination of LLMs trained on vast quantities of available data (such as C4 [1]) and conventional neural LMs is additive and significantly outperforms a strong first-pass baseline with a maximum entropy LM.

Multimodal large-scale language modeling has become the mainstream approach in natural language processing tasks and has been applied to various cross-modal fields such as image description and visual question answering. However, large-scale language modeling has high computational complexity and a large operational scale, which presents significant challenges for deployment in many resource-constrained scenarios. To address such problems, a lightweight multimodal framework, LLaVA-GM, is proposed, based on LLaVA, which can be deployed on devices with low resource requirements and has greatly reduced model parameters. It can also be tested on common VQA tasks and achieves good performance. The main contributions and work are as follows: First, it is found that the backbone of the Vicuna language model in LLaVA is too redundant. When fine-tuning downstream tasks, a very small amount of data sets is difficult to affect the language model. It is replaced with a new Gemma language model, thereby achieving fast task-specific adaptation with fewer parameters and data. Second, in response to the problem of information redundancy, the MoE mixed expert model is introduced. This model can be used in combination with itself, combining the MoE mixed expert model with Gemma to reduce the amount of computation while maintaining performance. Directly training the entire model will lead to a decline in performance. A multi-stage training strategy is adopted to maintain performance. First, the MLP layer is trained for visual adaptation, then the entire Gemma model is trained to improve multimodal capabilities, and finally only the MoE layer is trained for sparsification to ensure a smooth transition from dense models to sparse models. The experiment was tested on multiple VQA datasets and achieved good performance, confirming the potential of this compact model in downstream multimodal applications.

This study investigates the potential of artificial intelligence, particularly Natural Language Processing and large-scale language models, to improve resource management and service access for individuals with autism in Alabama. The research aims to explore and evaluate the potential of AI-driven tools to address challenges in navigating complex datasets and supporting social work practices. We designed and tested AI systems, including general language models, domain-specific chatbots powered by advanced language models, and a Retrieval-Augmented Generation framework. A standardized set of queries was used to simulate real-world scenarios encountered by social workers engaged in autism service coordination. System performance was evaluated based on precision, recall, and response accuracy. Results demonstrated that the Retrieval-Augmented Generation (RAG) framework achieved superior performance compared to traditional NLP methods and general-purpose language models, attaining approximately 90–96% precision and recall across evaluated query types. RAG outperformed the domain-specific GPT-4 chatbot by approximately 5–12 percentage points in F1 score, with the largest gains observed for queries requiring geographic specificity, multiple constraints, or complex contextual understanding. The integration of domain-specific retrieval significantly enhanced the accuracy, contextual relevance, and usability of generated responses. The findings highlight the transformative potential of AI-driven tools in improving social work efficiency and enhancing healthcare equity. By streamlining care coordination and delivering accurate, contextually relevant information, these systems offer scalable solutions to improve access to autism-related services. Future research should focus on addressing data quality, minimizing biases, and ensuring ethical deployment to build trust and support widespread adoption of these tools.

Commonsense inference poses a unique challenge to reason and generate the physical, social, and causal conditions of a given event. Existing approaches to commonsense inference utilize commonsense transformers, which are large-scale language models that learn commonsense knowledge graphs. However, they suffer from a lack of coverage and expressive diversity of the graphs, resulting in a degradation of the representation quality. In this paper, we focus on addressing missing relations in commonsense knowledge graphs, and propose a novel contrastive learning framework called SOLAR . Our framework contrasts sets of semantically similar and dissimilar events, learning richer inferential knowledge compared to existing approaches. Empirical results demonstrate the efficacy of SOLAR in commonsense inference of diverse commonsense knowledge graphs. Specifically, SOLAR outperforms the state-of-the-art commonsense transformer on commonsense inference with ConceptNet by 1.84% on average among 8 automatic evaluation metrics. In-depth analysis of SOLAR sheds light on the effects of the missing relations utilized in learning commonsense knowledge graphs.

Knowledge enhancement-based large-scale natural language training model is an advanced language model that combines deep learning and knowledge enhancement. By learning from massive unlabeled data and combining with external knowledge such as knowledge graph, it breaks through the limitations of traditional models in interpretability and reasoning ability. Introducing knowledge into data-driven artificial intelligence model is an important way to realize human-machine hybrid intelligence. However, since most pre-trained models are trained on large-scale unstructured corpus data, the defects in certainty and explainability can be remedied to some extent by introducing external knowledge. To solve the above problems, we present a knowledge-enhanced large-scale natural language training model that integrates deep learning with external knowledge sources (e.g., knowledge graphs) to improve interpretability and reasoning ability. This approach addresses the limitations of traditional models trained on unstructured data by incorporating external knowledge to enhance certainty and explainability. We propose a new knowledge enhancement method and demonstrate its effectiveness through a long text representation model. This model processes structured, knowledge-rich long texts by extracting and integrating knowledge and semantic information at the sentence and document levels. It then fuses these representations to generate an enhanced long text representation. Experiments on legal case matching tasks show that our model significantly outperforms existing methods, highlighting its innovation and practical value.

Machine reading comprehension (MRC) has achieved remarkable progress in the open domain in recent years, mainly due to large-scale pre-trained language models. However, it performs much worse in specific domains such as the medical field due to the lack of extensive training data and professional structural knowledge neglect. Besides, the opacity of the large-scale neural language model also limits its practical utility. As an effort, we first collect a medical machine reading comprehension question dataset with experts' annotations for the National Licensed Pharmacist Examination in China. It is a challenging medical examination with a passing rate of less than 14.2% in 2018. Then we propose a novel reading comprehension model eKMQA, which can fully exploit the structural medical knowledge (i.e., medical knowledge graph) and the reference medical plain text (i.e., text snippets retrieved from reference books). Meanwhile, it improves the explainability of the model by leveraging the knowledge entailed in explanations as an additional distillation signal for more efficient learning. The experimental results demonstrate that our method outperforms existing competitive models with a large margin, which improves prediction accuracy on the test set and can generate reasonable free-text rationales for its predictions even with a small-scale training corpus.

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

Large-scale language models such as DNABert and LOGO aim to learn optimal gene representations and are trained on the entire Human Reference Genome. However, standard tokenization schemes involve a simple sliding window of tokens like k-mers that do not leverage any gene-based semantics and thus may lead to (trivial) masking of easily predictable sequences, and subsequently inefficient Masked Language Modeling (MLM) training. Therefore, we propose a novel masking algorithm, GENEMASK, for MLM training of gene sequences, where we randomly identify positions in a gene sequence as mask centers and locally select the span around the mask center with the highest Normalized Pointwise Mutual Information (NPMI) to mask. We observe that in the absence of human-understandable semantics in the genomics domain (in contrast, semantic units like words and phrases are inherently available in NLP), GENEMASK-based models substantially outperform the SOTA models (DNABert and LOGO) over four benchmark gene sequence classification datasets in five few-shot settings (10 to 1000-shot). More significantly, the GENEMASK-based DNABert model is trained for less than one-tenth of the number of epochs of the original SOTA model. We also observe a strong correlation between top-ranked PMI tokens and conserved DNA sequence motifs, which may indicate the incorporation of latent genomic information. The codes (including trained models) and datasets are made publicly available at https://github.com/roysoumya/GeneMask.

Large-scale language models, such as ChatGPT[3] and GPT-4[32], have demonstrated remarkable capabilities in generating human-like text for various applications. In this paper, we focus on two key aspects: (1) adapting open-source large language models (LLMs) for specific use cases like contract drafting using instruction tuning and parameter-efficient fine-tuning, and (2) analyzing the difference in ChatGPT’s behavior in single-role prompts compared to multi-role prompts for synthetic data generation tasks. We present a method for aligning open-source LLMs to follow instructions for customized contract drafting scenarios using parameter-efficient fine-tuning on synthetic data. Furthermore, we investigate the data quality of the synthetically generated instructions data by ChatGPT with single-role vs. multi-role prompts. Our findings reveal that the model performs better when given single-role prompts, highlighting the importance of strategically designing prompting strategy to generate better quality data using LLMs. By combining the insights from these two aspects, we explore potential implications and opportunities for enhancing generative AI solutions for practical implementations. The Contract Drafting model 1 and data 2 are released.