Local Alignment of DNA Sequence Based on Deep Reinforcement Learning.

In this paper, we investigate the feasibility of computing DNA patterns on FPGAs by utilising high-performance and parallel computing. In particular, the report analyses and contrasts the FSM -based core implementation of the well-known Smith-Waterman pair-wise sequence alignment algorithm with the dynamic and heuristic alternatives that are currently on the market. In this research project, we investigate the use of parallel and high-performance computing to determine how DNA patterns can be computed on FPGAs. In particular, the report details the findings of a study that compared the FSM -based core implementation of the well-known Smith-Waterman pair-wise sequence alignment algorithm with the algorithm's current dynamic and heuristic programming methods. The study was carried out by comparing these three aspects of the algorithm. The traditional methods of sequence alignment that are software-implied are not adequate to meet the actual data rate requirements. A strategy that is based on the use of hardware makes it possible to process multiple new databases in parallel, which enables a high degree of scalability. A finite state machine serves as the primary data processor in our demonstration of how to organize the sequences of amino acids in a protein (FSM). We also investigate the effectiveness of bit-based sequence alignment algorithms and present the inner stage pipelined FPGA architecture for implementing sequence alignment. Both of these are presented in this paper. Synchronized controllers are utilized in the programming techniques of the algorithm for the purpose of parallel sequence alignment. Inference is the foundation of traditional software-based methods for sequence alignment, but these methods are unable to keep up with the required data rates. A strategy that is based on hardware allows for the processing of parallel tasks with a large number of new databases and provides excellent scalability. Our explanation of the classification of protein sequences is based on a Finite State Machine, which serves as the primary processing unit (FSM). In addition to this, we investigate how well bit-based sequence alignment algorithms work and present the inner stage pipelined Field Programmable Gate Array architecture that can be used to implement sequence alignment. Synchronized controllers are utilized in the process of performing sequence alignment in parallel. The architecture as a whole was made to be scalable and fast, with a small number of computations and the ability to perform parallel processing in hardware using field-programmable gate arrays to compare bits in patterns. Finally, we use FPGA hardware synthesis to show off impressive performance and low resource consumption.

Motivations. A central problem in biological network analysis is the Local Network Alignment. The aim is to detect conserved subnetworks or complexes of proteins, across two or more species, which are involved in processes or functions. This allows to predict either new interactions or the functions of unknown proteins. Since the problem of finding conserved subnetworks in a set of networks is related to subgraph isomorphism, which is known to be NP-Hard, several heuristics have been proposed. These include PathBlast [1] and MaWISh [2] for pairwise local alignment and NetworkBlast-M [3] and Graemlin [4] for the multiple case. Although NetworkBlast-M has been proved to be the most efficient and accurate method, it is able to find significant conserved complexes composed by no more than 15 proteins. Here we introduce GASOLINE (Greedy And Stochastic algorithm for Optimal Local alignment of Interaction Networks) an algorithm based on Gibbs Sampling [5] in connection to a seed-extend approach to search for significant conserved complexes of any size. Methods. The algorithm consists of two main phases. In the first phase, we look for ortholog proteins across the networks. We call these proteins seeds of the suboptimal pattern. In the second phase, called iterative phase, we extend each seed, by adding one adjacent node. Here, through a stochastic process based on Gibbs Sampling, we choose a node among a set of randomly picked adjacent ones. The chosen nodes will be those that maximize similarity among the N extended seeds. We repeat the iterative phase until we obtain a set of N conserved subgraphs each of size W. These N subgraphs represent our final alignment. The topological density and the conservation of a complex are measured through an Index of Density and Structural Conservation (hereafter IDSC). The IDSC score ranges from 0 to 1 and it is dynamically computed during the iterative phase. The iteration will terminate when IDSC is above a fixed threshold. This allows the removal of parameter W producing a set of highly dense and conserved complexes of different sizes. Results. GASOLINE has been tested on 10 microbial PPI networks, taken from Graemlin [4] and on 6 eukaryotic PPI networks, taken from STRING database [6]. The number of proteins in microbial networks ranges from 1,000 to 7,000 with the amount of interactions ranging from 13,000 to 230,000, whereas the size of eukaryotic networks ranges from 6,000 to 12,500 proteins and from 26,000 to 166,000 edges. 2000 of execution of GASOLINE have been performed. As output we consider the best distinct (not overlapping) complexes, with respect to size andIDSC score. In our experiments we selected complexes of at least 5 proteins with IDSC score > 0.7. All tests have been performed on an Intel Core i5-2500 3.30Ghz CPU with 4 GB RAM. The complexes computed by the algorithm have been then validated by annotating their proteins with GO categories. For the microbial networks, annotations have been downloaded from DAVID [7,8], while eukaryotic networks proteins have been annotated using BioDBNet [9]. Significant conserved categories have been obtained by computing a p-value (< 0.0001), based on hypergeometric distribution. A GO category has been considered conserved when it resulted significant in at least N-1 species, where N is the number of aligned networks. The executions of GASOLINE on microbial networks revealed the existence of a big conserved complex of 40 proteins, forming the large and small subunits of ribosome, with IDSC equals to 0.755 (see Tab. 1). As for the 6 eukaryotic networks, 15 conserved complexes have been found by GASOLINE with IDSC greater than 0.75. They are listed in Tab. 2, with their IDSC and the number of GO categories enriched.

The emergence of bioinformatics has led to many new discoveries in living organisms. These discoveries would not have been possible without the developments made in the sequence alignment techniques. Many sequence alignment algorithms were developed to make the alignment process fast and accurate. However, the more precise algorithms take longer than their less precise counterparts. Researchers came with innovative approaches to combat the time consuming constraint. Their aim was to speed up the computational process by using more efficient implementations of the algorithms using state-of-the-art hardware platforms. Smith Waterman (SW) algorithm, being the most accurate in the alignment process, has been implemented on various high performance computing platforms for the same purpose. However, the intrinsic structure of the algorithm has got little attention. In this paper, we present a novel structure of the SW algorithm that takes less number of cycles at the cost of utilizing a minimal amount of extra hardware resources as compared to its existing form. The newly proposed architecture achieves up to 25% performance gain.

The level of conservation between two homologous sequences often varies among sequence regions; functionally important domains are more conserved than the remaining regions. Thus, multiple parameter sets should be used in alignment of homologous sequences with a stringent parameter set for highly conserved regions and a moderate parameter set for weakly conserved regions. We describe an alignment algorithm to allow dynamic use of multiple parameter sets with different levels of stringency in computation of an optimal alignment of two sequences. The algorithm dynamically considers various candidate alignments, partitions each candidate alignment into sections, and determines the most appropriate set of parameter values for each section of the alignment. The algorithm and its local alignment version are implemented in a computer program named GAP4. The local alignment algorithm in GAP4, that in its predecessor GAP3, and an ordinary local alignment program SIM were evaluated on 257 716 pairs of homologous sequences from 100 protein families. On 168 475 of the 257 716 pairs (a rate of 65.4%), alignments from GAP4 were more statistically significant than alignments from GAP3 and SIM.

ESA: An efficient sequence alignment algorithm for biological database search on Sunway TaihuLight

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

Pairwise sequence alignment is a fundamental compute-intensive problem in bioinformatics that has helped researchers analyse biological sequences. The analysis has helped biologists detect pathogens, develop drugs, and identify common genes. The biological sequence database has been growing rapidly due to new sequences being discovered. This has brought many new challenges including sequence database searching and aligning long sequences. To solve these problems, many sequence alignment algorithms have been developed. These algorithms employ various techniques to efficiently find optimal or nearly-optimal alignments. In this paper, we present the popular past and recent work on both local and global pairwise sequence alignment algorithms. In addition to identifying the techniques used, the advantages and limitations of the algorithms are also presented.

A new ensemble deep graph reinforcement learning network for spatio-temporal traffic volume forecasting in a freeway network

Drosha and Dicer are RNase III family members of classes II and III, respectively, which play a major role in the maturation of micro-RNAs. The two proteins share similar domain arrangement and overall fold despite no apparent sequence homology. The overall structural and catalytic reaction similarity of both proteins, on the one hand, and differences in the substrate and its binding mechanisms, on the other, suggest that both proteins also share dynamic similarities and dissimilarities. Since dynamics is essential for protein function, a comparison at their dynamics level is fundamental for a complete understanding of the overall relations between these proteins. In this study, we present a dynamical comparison between human Drosha and Giardia Dicer. Gaussian Network Model and Anisotropic Network Model modes of motion of the proteins are calculated. Dynamical comparison is performed using global and local dynamic programming algorithms for aligning modes of motion. These algorithms were recently developed based on the commonly used Needleman-Wunsch and Smith-Waterman algorithms for global and local sequence alignment. The slowest mode of Drosha is different from that of Dicer due to its more bended posture and allow the motion of the double-stranded RNA-binding domain toward and away from its substrate. Among the five slowest modes dynamics similarity exists only for the second slow mode of motion of Drosha and Dicer. In addition, high local dynamics similarity is observed at the catalytic domains, in the vicinity of the catalytic residues. The results suggest that the proteins exert a similar catalytic mechanism using similar motions, especially at the catalytic sites.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

This paper discusses an autonomous driving control strategy based on Deep Reinforcement Learning (DRL), which aims to improve the decision-making ability of autonomous driving system in complex traffic environments. Deep reinforcement learning has a wide range of applications in many fields, such as robotics and medicine. Autonomous driving has emerged as a significant research focus in recent years. By combining deep learning and reinforcement learning, the model is able to autonomously learn and optimize driving behavior under dynamically changing road conditions. The DRL-based control strategy performs well in vehicle obstacle avoidance, pedestrian recognition, and traffic rule compliance in the face of complex environments such as city streets, intersections, and congested road sections, significantly improving the safety and efficiency of autonomous driving. This article will first introduce deep reinforcement learning. Then, the autonomous driving control strategy based on deep reinforcement learning is introduced. This research provides valuable insights for developing and implementing DRL-based autonomous driving systems.

The Smith-Waterman (SW) algorithm is a typical technique for local sequence alignment in computational biology. However, the SW algorithm does not consider the local behaviours of the amino acids, which may result in loss of some useful information. Inspired by the success of Markov Edit Distance (MED) method, this paper therefore proposes a novel Markov pairwise protein sequence alignment (MPPSA) method that takes the local context dependencies into consideration. The numerical results have shown its superiority to the SW for pairwise protein sequence comparison.