Codon language embeddings provide strong signals for protein engineering

Data science is the study of extracting knowledge and making inferences from data. Various data science techniques have been developed to manage, process, and use big/heterogeneous data. The most successful technique is deep learning using neural networks. This paper first describes the current state of machine learning. It then presents two applications of machine learning are presented. The first is an application of machine learning in molecular biology, and the second is an application of machine learning in biological signal processing. The molecular biology application will mainly explain DNA methylation analysis will be explained. The biological signal processing application will mainly explain the classification and quantification of tremors.

Deep language models, like ELMo, BERT and GPT, have achieved impressive results on several natural language tasks. These models are pretrained on large corpora of unlabeled general domain text and later supervisedly trained on downstream tasks. An optional step consists of finetuning the language model on a large intradomain corpus of unlabeled text, before training it on the final task. This aspect is not well explored in the current literature. In this work, we investigate the impact of this step on named entity recognition (NER) for Portuguese legal documents. We explore different scenarios considering two deep language architectures (ELMo and BERT), four unlabeled corpora and three legal NER tasks for the Portuguese language. Experimental findings show a significant improvement on performance due to language model finetuning on intradomain text. We also evaluate the finetuned models on two general-domain NER tasks, in order to understand whether the aforementioned improvements were really due to domain similarity or simply due to more training data. The achieved results also indicate that finetuning on a legal domain corpus hurts performance on the general-domain NER tasks. Additionally, our BERT model, finetuned on a legal corpus, significantly improves on the state-of-the-art performance on the LeNER-Br corpus, a Portuguese language NER corpus for the legal domain.

Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.

Large Language Models (LLMs) are scaled-up instances of Deep Neural Language Models—a type of Natural Language Processing (NLP) tools trained with Machine Learning (ML). To best understand how LLMs work, we must dive into what technologies they build on top of and what makes them different. To achieve this, an overview of the history of LLMs development, starting from the 1990s, is provided before covering the counterintuitive purely probabilistic nature of the Deep Neural Language Models, continuous token embedding spaces, recurrent neural networks-based models, what self-attention brought to the table, and finally, why scaling Deep Neural Language Models led to a qualitative change, warranting a new name for the technology.

Early risk detection can be useful in different areas, particularly those related to health and safety. Two tasks are proposed at CLEF eRisk-2018 for predicting mental disorder using users posts on Reddit. Depression and anorexia disorders must be detected as early as possible. In this paper, we extend the participation of LIRMM (Laboratoire d’Informatique, de Robotique et de Microelectronique de Montpellier) in both tasks. The proposed model addresses this problem by modeling the temporal mood variation detected from user posts. The proposed architectures use only textual information without any hand-crafted features or dictionaries. The basic architecture uses two learning phases through exploration of state-of-the-art text vectorizations and deep language models. The proposed models perform comparably to other contributions while further experiments shows that attentive based deep language models outperformed the shallow learning text vectorizations.

The increasing use of AI-based approaches such as machine learning (ML) across diverse scientific fields presents challenges for reproducibly disseminating and assessing research. As ML becomes integral to a growing range of computationally intensive applications (e.g. clinical research), there is a critical need for transparent reporting methods to ensure both comprehensibility and the reproducibility of the supporting studies. There are a growing number of standards, checklists and guidelines enabling more standardized reporting of ML research, but the proliferation and complexity of these make them challenging to use. Particularly in assessment and peer review, which has to date, been an ad hoc process that has struggled to throw light on increasingly complicated computational supporting methods that are otherwise unintelligible to other researchers. Taking the publication process beyond these black boxes, GigaScience Press has experimented with integrating many of these ML-standards into the publication process. Having a broad-scope that necessitated looking at more generalist and automated approaches. Here, we map the current landscape of artificial intelligence (AI) standards, and outline our adoption of the DOME recommendations for Machine Learning in biology. We developed a publishing workflow that integrates the DOME Data Stewardship Wizard and DOME Registry tools into the peer-review and publication process. From this case study we provide journal authors, reviewers and Editors examples of approaches, workflows and strategies to more logically disseminate and review ML research. Demonstrating the need for continued dialogue and collaboration among various ML communities to create unified, comprehensive standards, to enhance the credibility, sustainability and impact of ML-based scientific research.

Supervised machine learning (ML) is used extensively in biology and deserves closer scrutiny. The Data Optimization Model Evaluation (DOME) recommendations aim to enhance the validation and reproducibility of ML research by establishing standards for key aspects such as data handling and processing, optimization, evaluation, and model interpretability. The recommendations help to ensure that key details are reported transparently by providing a structured set of questions. Here, we introduce the DOME registry (URL: registry.dome-ml.org), a database that allows scientists to manage and access comprehensive DOME-related information on published ML studies. The registry uses external resources like ORCID, APICURON, and the Data Stewardship Wizard to streamline the annotation process and ensure comprehensive documentation. By assigning unique identifiers and DOME scores to publications, the registry fosters a standardized evaluation of ML methods. Future plans include continuing to grow the registry through community curation, improving the DOME score definition and encouraging publishers to adopt DOME standards, and promoting transparency and reproducibility of ML in the life sciences.

There is a long history between Machine Learning and its application in Biology. Machine Learning has allowed many advancements in the field of Biology. This includes studies for prediction and discovery by different Machine Learning techniques for specified types of biological data. The versatility of techniques and frameworks has helped improve Machine Learning every day, and hopefully, this will help us improve and become more efficient on new things we discover and grow our database on biological data. In this paper, we will discuss the different applications of Machine Learning in Biology, i.e., Synthetic Gene Circuits, Convolutional Neural Networks, Recurrent Neural Networks and Generative Adversarial Networks. As the biological and medical fields have been quickly growing into a data-rich environment, Machine Learning has become a vital tool to sort through all of this data. The application of Machine Learning helps in two main facets. The first way is to help classify and predict tasks that a machine can quickly do. Secondly, it does not allow much human input, which will help minimize human bias or performance issues. Neural Networks go hand-in-hand with Deep Learning as they are one of its techniques. By utilizing Deep Learning and Neural Networks, we can expand our learning in Biology far more than ever before.

Deep learning based general language models have achieved state-of-the-art results in many popular tasks such as sentiment analysis and QA tasks. Text in domains like social media has its own salient characteristics. Domain knowledge should be helpful in domain relevant tasks. In this work, we devise a simple method to obtain domain knowledge and further propose a method to integrate domain knowledge with general knowledge based on deep language models to improve performance of emotion classification. Experiments on Twitter data show that even though a deep language model fine-tuned by a target domain data has attained comparable results to that of previous state-of-the-art models, this fine-tuned model can still benefit from our extracted domain knowledge to obtain more improvement. This highlights the importance of making use of domain knowledge in domain-specific applications.

Chapter 4 - Robust enzymes designing for efficient biocatalysis

Deep language models (deep LMs) are increasingly being used for full text retrieval or within cascade retrieval pipelines as later-stage re-rankers. A problem with using deep LMs is that, at query time, a slow inference step needs to be performed -- this hinders the practical adoption of these powerful retrieval models, or limits sensibly how many documents can be considered for re-ranking.

Deep neural network language models have recently shown impressive results in natural language processing, including the field of text sentiment analysis. However, user confidence in the results of the analysis is low due to the poor interpretability of such models. The paper proposes a method for interpreting language models using an analysis of attention matrices. The experiments were carried out using the ruRoberta language model and the corpus of Russian-language news. The results of the work show that the model focuses on sentiment lexicon, especially negative words.

In recent years, substantial advancements have been made in the development of large language models, achieving remarkable performance across diverse tasks. To evaluate the knowledge ability of language models, previous studies have proposed lots of benchmarks based on question-answering pairs. We argue that it is not reliable and comprehensive to evaluate language models with a fixed question or limited paraphrases as the query, since language models are sensitive to prompt. Therefore, we introduce a novel concept named knowledge boundary to encompass both prompt-agnostic and promptsensitive knowledge within language models. Knowledge boundary avoids prompt sensitivity in language model evaluations, rendering them more dependable and robust. To explore the knowledge boundary for a given model, we propose a projected gradient descent method with semantic constraints, a new algorithm designed to identify the optimal prompt for each piece of knowledge. Experiments demonstrate a superior performance of our algorithm in computing the knowledge boundary compared to existing methods. Furthermore, we evaluate the ability of multiple language models in several domains with knowledge boundary.

Significant progress has been made in sentiment analysis over the past few years, especially due to the application of deep neural language models. However, there is a problem of transferability of trained models from one domain to another, especially for less studied languages such as Russian. We propose an approach to build cross-domain sentiment analysis models based on a two-stage procedure: first, we fine-tune a pre-trained RuBERT language model on a combined non-domain corpus, and then fine-tune this model on a small domain corpus. We conducted large-scale experiments with 30 sentiment annotated corpora across 12 domains. In order to increase the representativeness of news texts with high-quality annotation, we created a novel RuNews corpus, containing 1,823 news articles annotated by sentiment. The results show that fine-tuning the model using a small number (about several hundred) of annotated domain texts can significantly improve the performance of sentiment analysis for a new domain (on average by 4.6 p.p.). We also obtained the state-of-the-art results for 7 out of 14 test corpora.

The increasing complexity and scale of language models have introduced significant challenges related to computational efficiency, inference speed, and the generation of factually accurate responses. Addressing these issues, a novel architectural enhancement known as the Dynamic Sequential Transformer (DST) has been proposed, offering a groundbreaking approach to token processing through dynamic token sequencing and adaptive attention recalibration. By introducing a more flexible and context-aware method for handling tokenized inputs, DST improves both accuracy and speed while reducing the occurrence of hallucinations, which are a common problem in language generation tasks. The experimental results have demonstrated that the DST architecture significantly reduces memory usage, accelerates inference times, and improves model convergence rates, making it an efficient solution for large-scale deployments without sacrificing output quality. Additionally, DST's modular design ensures scalability and robustness across diverse language processing tasks, positioning it as a promising advancement for practical real-world applications. Through extensive evaluation, the study confirms that DST not only optimizes resource utilization but also enhances the linguistic coherence and factual consistency of generated text, establishing a new benchmark for architectural efficiency in modern language models.