Deep Learning for Drug Discovery: Challenges and Opportunities

Deep (machine) learning in recent years has significantly increased the predictive modeling strengths for applications in the areas of computer vision, speech recognition, and natural language processing. Being heavily data-driven, conventional general-purpose deep learning methods mainly rely on training data to learn task-specific data representations. In this thesis, we focus on leveraging novel learning algorithms in deep learning to tackle visual recognition challenges at the three key levels -- namely pixel, object, and image levels. Throughout the thesis, we demonstrate how each and every of our proposed methods can address a specific visual recognition problem. Firstly, we apply deep learning to improve the saliency detection performance of deep networks. A coarse saliency map of the input image's salient object is initially predicted by a convolutional-deconvolutional network (or equivalently a fully convolutional network). Following that, the coarse saliency map is sequentially refined by a separate network called Recurrent Attentional Convolutional-Deconvolutional Network (RACDNN) that shares its network weights across its sequential steps. At every sequential (temporal) step, the same RACDNN is applied to a newly selected/attended image sub-region and it performs saliency map refinement for that particular sub-region. Despite having many learnable network parameters and no access to additional training data, RADCNN can be trained well and it outperforms the "single-step" (coarse saliency detection) baseline and state-of-the-art saliency detection methods by large margins, on several saliency detection benchmark datasets. Secondly, we introduce auto-encoder weight transfer networks (AE-WTN+), a deep model for scaling up object detection. Conventional WTN transfers object class knowledge from the classification weights of a pretrained large-scale image classification network, to the classification weights of an object detection network. However, they suffer from under-fitting and are not able to preserve the rich object class information of novel classes. In this work, we propose a new WTN method -- AE-WTN+ which makes use of normalization techniques to ease network optimization. Inputs and intermediate features are normalized to improve trainability for reduced under-fitting. More importantly, a novel auto-encoding (reconstruction) loss is proposed to encourage information preservation of all classes in the weight predictions of AE-WTN+. This enables the predicted weights of AE-WTN+ to maintain the class discriminability and neighbourhood relationships contained in the pre-trained large-scale classification weights. Experiments on large-scale detection datasets validate the effectiveness of AE-WTN+ on both seen and novel classes. Thirdly, we improve the efficiency of cross-layer network connections for very deep convolutional network in the context of image classification. This gives rise to a new convolutional network architecture called Deluge Networks (DelugeNets) which pass the output (feature activations) of any preceding composite network layer as direct inputs to all succeeding composite layers. For the incoming feature activations of preceding composite layers, the succeeding composite layers first apply rather lightweight cross-layer depthwise convolutional weights on them for feature aggregation. The heavy lifting is carried out by the last convolutional weight layer in each preceding composite layer that performs cross-channel convolutions. The proposed technique enables unimpeded information flow among many layers in very deep DelugeNets with great efficiency. Experiments on small-scale and large-scale datasets show that DelugeNets perform competitively to state-of-the-art deep convolutional networks at lower computational and parameter costs. Fourthly, we propose Stochastic Downsampling (SDPoint), a method to overcome a major limitation suffered by conventional deep convolutional networks -- their computational costs during inference are fixed and cannot be changed. SDPoint downsamples the feature maps at a randomly selected point (layer index) in the network hierarchy during training. The many downsampling configurations known as SDPoint instances each entail a unique downsampling point and downsampling ratio. The same convolutional network weights are shared across the different SDPoint instances that downsample feature maps differently. During inference, the computational cost of a network trained with SDPoint can be instantaneously adjusted to fit a given computational budget, by manually selecting an appropriate SDPoint instance. Additionally, sharing weights between different feature map scales provides significant regularization benefits, making the networks trained with SDPoint less sensitive to scales and more competent. In image classification experiments, SDPoint demonstrates significant cost-accuracy and computational advantages, over independently-trained (non-weight-shared) baselines with various inference costs.

Artificial Intelligence (AI) has become one of the most transformative foundational technologies of our time cutting across industry and function, while machine learning has become its engine of maturation. The rapid advancement of AI applications in different domains including computer vision, speech recognition, and natural language processing (NLP) has led to a need for highly accurate, efficient, scalable, and deployable systems in the real world. The paper discusses advances needed to advance AI applicability in these three major areas, especially using advanced machine-learning technologies to improve AI efficiency in terms of the performance, energy requirements, and computational scalability of AI systems In order to seek out the accuracy and speed of the computer vision systems, the study’s investigation studies the works related to the modernization of machine learning techniques namely deep learning, reinforcement learning, and transfer learning. Such advances are critical for real-time applications including ampere nous driving, facial recognizance, and medical image diagnosis. In parallel, studies on the architecture optimization for speech recognition models are being conducted to enhance feature extraction, reduce computation and latency while maintaining performance. Such improvements are crucial for voice assistants, transcription services and multilingual communication tools. For NLP, these translations shed light on the latest in transformer-based architectures, better pre-trained language models, and methods for reducing the size and inference time of models, all so that these AI services can be reasonable in chatbots, content summarization, and sentiment analysis, etc. This work presents a detailed insight into how machine learning techniques can offer massive energy savings that can be derived from the analysis of state-of-the-art architectures, finding bottlenecks, and comparing performance. In addition, it underscores recent advances in lightweight models, distributed training, model compression and hardware-aware optimization engage computational burden while preserving or improving predictive performance. This brings these approaches together to maintain that AI technologies can be consolidated more sustainably across edge devices, cloud platforms, and embedded systems.

Chapter 19 - Recent advances in deep learning enabled approaches for identification of molecules of therapeutics relevance

Artificial Intelligence in the American Healthcare Industry: Looking Forward to 2030

The rapidly advancing field of artificial intelligence (AI) has garnered substantial attention for its potential application in drug discovery and development. This opinion review critically examined the feasibility and prospects of integrating AI as a transformative tool in the pharmaceutical industry. AI, encompassing machine learning algorithms, deep learning, and data analytics, offers unprecedented opportunities to streamline and enhance various stages of drug development. This opinion review delved into the current landscape of AI-driven approaches, discussing their utilization in target identification, lead optimization, and predictive modeling of pharmacokinetics and toxicity. We aimed to scrutinize the integration of large-scale omics data, electronic health records, and chemical informatics, highlighting the power of AI in uncovering novel therapeutic targets and accelerating drug repurposing strategies. Despite the considerable potential of AI, the review also addressed inherent challenges, including data privacy concerns, interpretability of AI models, and the need for robust validation in real-world clinical settings. Additionally, we explored ethical considerations surrounding AI-driven decision-making in drug development. This opinion review provided a nuanced perspective on the transformative role of AI in drug discovery by discussing the existing literature and emerging trends, presenting critical insights and addressing potential hurdles. In conclusion, this study aimed to stimulate discourse within the scientific community and guide future endeavors to harness the full potential of AI in drug development.

Microfluidic-based organs-on-chips (OoCs) are a rapidly developing technology in biomedical and chemical research and have emerged as one of the most advanced and promising in vitro models. The miniaturization, stimulated tissue mechanical forces, and microenvironment of OoCs offer unique properties for biomedical applications. However, the large amount of data generated by the high parallelization of OoC systems has grown far beyond the scope of manual analysis by researchers with biomedical backgrounds. Deep learning, an emerging area of research in the field of machine learning, can automatically mine the inherent characteristics and laws of “big data” and has achieved remarkable applications in computer vision, speech recognition, and natural language processing. The integration of deep learning in OoCs is an emerging field that holds enormous potential for drug development, disease modeling, and personalized medicine. This review briefly describes the basic concepts and mechanisms of microfluidics and deep learning and summarizes their successful integration. We then analyze the combination of OoCs and deep learning for image digitization, data analysis, and automation. Finally, the problems faced in current applications are discussed, and future perspectives and suggestions are provided to further strengthen this integration.

Exploring new horizons: Empowering computer-assisted drug design with few-shot learning

Artificial intelligence (AI) has grown widely into diverse areas of research over the last decade. Publications distributed over the years disclose that biochemistry and analytical chemistry are incorporating AI to the utmost level. Enormous complex data sets are produced by analytical instruments comprising a treasure of information which is very helpful to analytical chemists in characterization. Despite this, we lack capabilities in data analysis and algorithms, which restricts our aptitude to discover and employ this data entirely. Machine learning and AI are being instigated to deal with this current challenge by accelerating several other applications in analytical chemistry. A critical valuation of illustrative reports amalgamating AI with analytical chemistry, spectroscopies, and several sensors has been discussed widely. This comprehensive review evaluates the AI advancement in data interpretation, the assistance of AI in analyte identification and quantification in proteomics and metabolomics, AI databases for primary and secondary metabolites information, regulatory and ethical guidelines in analytical chemistry incorporating AI, machine learning applications in drug discovery, agriculture, cosmetic, and food and separation techniques. In summary, this current review puts forward a thorough examination concerning the advancements of AI in different domains of chemistry and strives to facilitate comprehension of its forthcoming trajectories.

Great progress has been made in deep learning over the past few years, which drives the deployment of deep learning–based applications into cyber-physical systems. But the lack of interpretability for deep learning models has led to potential security holes. Recent research has found that deep neural networks are vulnerable to well-designed input examples, called adversarial examples . Such examples are often too small to detect, but they completely fool deep learning models. In practice, adversarial attacks pose a serious threat to the success of deep learning. With the continuous development of deep learning applications, adversarial examples for different fields have also received attention. In this article, we summarize the methods of generating adversarial examples in computer vision, speech recognition, and natural language processing and study the applications of adversarial examples. We also explore emerging research and open problems.

Recently, significant improvements have been made in artificial intelligence. The artificial neural network was introduced in the 1950s. However, because of the low computing power and insufficient datasets available at that time, artificial neural networks suffered from overfitting and vanishing gradient problems for training deep networks. This concept has become more promising owing to the enhanced big data processing capability, improvement in computing power with parallel processing units, and new algorithms for deep neural networks, which are becoming increasingly successful and attracting interest in many domains, including computer vision, speech recognition, and natural language processing. Recent studies in this technology augur well for medical and healthcare applications, especially in endoscopic imaging. This paper provides perspectives on the history, development, applications, and challenges of deep-learning technology.

For the past few years, gender classification has been an active area of study and researchers have been putting a lot of effort to contribute quality research in this area. There is a big potential field of study as it can be used in monitoring, surveillance and human-computer interaction. However, there is still a lack of the performance of existing methods on real live images. The rise of deep learning algorithm has been showing a spectacular increase in performance lately. Many difficult tasks involving computer vision, speech recognition, and natural language processing are easily solved with deep learning. Therefore, the approach to deep learning notably growing and this also happens to be on image classification. Gender classification is an important subject in the face recognition process. This paper shows the results of classifying gender using Convolutional Neural Network based Deep Learning architectures using Tensorflow’s Deep Learning framework. We have used models provided by Keras with weights pre-trained on ImageNet. We have made a comparison of the different type of models which includes VGG16, ResNet-50, and MobileNet. Our own database consists of Asian faces inclusive of Malaysians and some Caucasians. Our trained model on a database consisting of 1000 images shows that VGG-16 delivered the highest recognition accuracy.

Nowadays, deep learning has been widely used in various fields, and great breakthroughs have been made in computer vision, speech recognition and natural language processing. Based on the deep learning algorithm, this paper studies the general optimization technology of OpenCL based on FPGA in detail, and puts forward a targeted hardware design scheme, which has the characteristics of high performance, low power consumption and programmability. By analyzing the algorithm of forward calculation and reverse calculation in convolutional neural network, the hardware acceleration module is designed. Compared with CPU, GPU and other platforms, FPGA has obvious advantages in power consumption, which proves that the scheme proposed in this paper has good feasibility and accuracy.

AI-based visual speech recognition towards realistic avatars and lip-reading applications in the metaverse

TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Its computational model is based on dataflow graphs with mutable state. Graph nodes may be mapped to different machines in a cluster, and within each machine to CPUs, GPUs, and other devices. TensorFlow supports a variety of applications, but it particularly targets training and inference with deep neural networks. It serves as a platform for research and for deploying machine learning systems across many areas, such as speech recognition, computer vision, robotics, information retrieval, and natural language processing. In this talk, we describe TensorFlow and outline some of its applications. We also discuss the question of what TensorFlow and deep learning may have to do with functional programming. Although TensorFlow is not purely functional, many of its uses are concerned with optimizing functions (during training), then with applying those functions (during inference). These functions are defined as compositions of simple primitives (as is common in functional programming), with internal data representations that are learned rather than manually designed. TensorFlow is joint work with many other people in the Google Brain team and elsewhere. More information is available at tensorflow.org.

Learning a similarity function between pairs of objects is at the core of learning to rank approaches. In information retrieval tasks we typically deal with query-document pairs, in question answering -- question-answer pairs. However, before learning can take place, such pairs needs to be mapped from the original space of symbolic words into some feature space encoding various aspects of their relatedness, e.g. lexical, syntactic and semantic. Feature engineering is often a laborious task and may require external knowledge sources that are not always available or difficult to obtain. Recently, deep learning approaches have gained a lot of attention from the research community and industry for their ability to automatically learn optimal feature representation for a given task, while claiming state-of-the-art performance in many tasks in computer vision, speech recognition and natural language processing. In this paper, we present a convolutional neural network architecture for reranking pairs of short texts, where we learn the optimal representation of text pairs and a similarity function to relate them in a supervised way from the available training data. Our network takes only words in the input, thus requiring minimal preprocessing. In particular, we consider the task of reranking short text pairs where elements of the pair are sentences. We test our deep learning system on two popular retrieval tasks from TREC: Question Answering and Microblog Retrieval. Our model demonstrates strong performance on the first task beating previous state-of-the-art systems by about 3\% absolute points in both MAP and MRR and shows comparable results on tweet reranking, while enjoying the benefits of no manual feature engineering and no additional syntactic parsers.