An Improved Approach Based on CNN-RNNs for Mathematical Expression Recognition

Text recognition in natural scene images is a challenging problem in computer vision. Different than the optical character recognition (OCR), text recognition in natural scene images is more complex due to variations in text size, colors, fonts, orientations, complex backgrounds, occlusion, illuminations and uneven lighting conditions. In this paper, we propose a segmentation-free method based on a deep convolutional recurrent neural network to solve the problem of cursive text recognition, particularly focusing on Urdu text in natural scenes. Compared to the non-cursive scripts, Urdu text recognition is more complex due to variations in the writing styles, several shapes of the same character, connected text, ligature overlapping, stretched, diagonal and condensed text. The proposed model gets a whole word image as an input without pre-segmenting into individual characters, and then transforms into the sequence of the relevant features. Our model is based on three components: a deep convolutional neural network (CNN) with shortcut connections to extract and encode the features, a recurrent neural network (RNN) to decode the convolutional features, and a connectionist temporal classification (CTC) to map the predicted sequences into the target labels. To increase the text recognition accuracy further, we explore deeper CNN architectures like VGG-16, VGG-19, ResNet-18 and ResNet-34 to extract more appropriate Urdu text features, and compare the recognition results. To conduct the experiments, a new large-scale benchmark dataset of cropped Urdu word images in natural scenes is developed. The experimental results show that the proposed deep CRNN network with shortcut connections outperform than other network architectures. The dataset is publicly available and can be downloaded from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://data.mendeley.com/datasets/k5fz57zd9z/1</uri> .

Recent experience in applying recurrent neural networks (RNNs) to interpreting Permannt Downhole Gauge (PDG) records has highlighted the potential utility of machine learning algorithms to learn reservoir behavior from data. The power of the RNN resides in its ability to retain information in a form of memory of previous patterns and information contained in the previous behavior of phenomena being modeled. This memory plays a role of informing the decision at the present time by using what happened in the past. This property suggests RNNs as a suitable choice to model sequences of reservoir information, even when the reservoir modeler is faced with incomplete knowledge of the underlying physical system. Convolutional Neural Networks (CNNs) are another variant of machine learning algorithms that have shown promise in sequence modeling domains, such as audio synthesis and machine translation. In this study, RNN and CNN were applied to tasks that traditionally would be modeled by a reservoir simulator. This was achieved by formulating the relationship between physical quantities of interest from subsurface reservoirs as a sequence mapping problem. In addition, the performance of a CNN layer as compared to RNN was evaluated systematically to investigate their capabilities in a variety of tasks of interest to the reservoir engineer. Preliminary results suggest that CNNs, with specific design modifications, are as capable as RNNs in modeling sequences of information and as reliable when making inferences to cases that has not been seen by the algorithm during training. Design details and reasons pertaining to the way these two seemingly different architectures process information, and handle memory are also discussed.

The natural language processing (NLP) area has been magically transformed by the Deep Neural Network (DNN). The two variations of Neural Networks, Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) can handle the different NLP tasks effectively. CNN is based on feature extraction using N-grams at higher levels. The RNN can be effectively used to model the sequential information. Choosing a neural technique for the various NLP tasks is always a challenge. The paper focuses on the evaluating the performance of the two DNN techniques CNN and RNN for the various NLP tasks, so that appropriate technique can be selected.

We present a technique based on stroke order normalization for improving recognition of online handwritten mathematical expressions (ME). The stroke order dependent system has less time complexity than the stroke order free system, but it must incorporate special grammar rules to cope with stroke order variations. The stroke order normalization technique solves this problem and also the problem of unexpected stroke order variations without increasing the time complexity of ME recognition. In order to normalize stroke order, the X–Y cut method is modified since its original form causes problems when structural components in ME overlap. First, vertically ordered strokes are located by detecting vertical symbols and their upper/lower components, which are treated as MEs and reordered recursively. Second, unordered strokes on the left side of the vertical symbols are reordered as horizontally ordered strokes. Third, the remaining strokes are reordered recursively. The horizontally ordered strokes are reordered from left to right, and the vertically ordered strokes are reordered from top to bottom. Finally, the proposed stroke order normalization is combined with the stroke order dependent ME recognition system. The evaluations on the CROHME 2014 database show that the ME recognition system incorporating the stroke order normalization outperforms all other systems that use only CROHME 2014 for training while the processing time is kept low.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

In this paper we propose a novel end-to-end framework for mathematical expression (ME) recognition. The method uses a convolutional neural network (CNN) to perform mathematical symbol detection and recognition simultaneously incorporating spatial context, and can handle multi-part and touching symbols effectively. To evaluate the performance, we provide a benchmark that contains MEs both from real-life and synthetic data. Images in our dataset undergo multiple variations such as viewpoint, illumination and background. For training, we use pure synthetic data for saving human labeling effort. The proposed method achieved 87% accuracy of total correct for clear images and 45% for cluttered ones.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

A hybrid model based on neural networks for biomedical relation extraction

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

In recent years, convolutional neural networks have been widely used in image processing and have shown good results. Particularly useful was their ability to automatically extract image features (textures and shapes of objects). The article proposes a method that improves the accuracy and speed of recognition of an ultra-precise neural network based on image recognition of people’s faces. At first, a recurrent neural network is introduced into the convolutional neural network, thereby studying the characteristics of the image more deeply. Deep image characteristics are studied in parallel using a convolutional and recurrent neural network. In line with the idea of skipping the ResNet convolution layer, a new ShortCut3- ResNet residual module is built. A double optimization model is created to fully optimize the convolution process. A study of the influence of various parameters of a convolutional neural network on network performance is demonstrated, also analyzed using simulation experiments. As a result, the optimal parameters of the convolutional neural network are established. Ex- periments show that the method presented in this paper can study various images of people’s faces regardless of age, gender, and also improves the accuracy of feature extraction and image recognition ability.

Sentiment analysis (SA) detects people’s opinions from text engaging natural language processing (NLP) techniques. Recent research has shown that deep learning models, i.e., Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer-based provide promising results for recognizing sentiment. Nonetheless, CNN has the advantage of extracting high-level features by using convolutional and max-pooling layers; it cannot efficiently learn a sequence of correlations. At the same time, Bidirectional RNN uses two RNN directions to improve extracting long-term dependencies. However, it cannot extract local features in parallel, and Transformer-based like Bidirectional Encoder Representations from Transformers (BERT) are the computational resources needed to fine-tune, facing an overfitting problem on small datasets. This paper proposes a novel attention-based model that utilizes CNNs with LSTM (named ACL-SA). First, it applies a preprocessor to enhance the data quality and employ term frequency-inverse document frequency (TF-IDF) feature weighting and pre-trained Glove word embedding approaches to extract meaningful information from textual data. In addition, it utilizes CNN’s max-pooling to extract contextual features and reduce feature dimensionality. Moreover, it uses an integrated bidirectional LSTM to capture long-term dependencies. Furthermore, it applies the attention mechanism at the CNN’s output layer to emphasize each word’s attention level. To avoid overfitting, the Guasiannoise and GuasianDroupout are adopted as regularization. The model’s robustness is evaluated on four English standard datasets, i.e., Sentiment140, US-airline, Sentiment140-MV, SA4A with various performance matrices, and compared efficiency with existing baseline models and approaches. The experiment results show that the proposed method significantly outperforms the state-of-the-art models.

Automated understanding of mathematical expressions (MEs) is currently a challenging task due to their complex two- dimensional (2D) structure. Recognition of MEs can be online or offline and in either case, the process involves symbol recognition and analysis of 2D structure. This process is more complex for offline or printed MEs as they do not have temporal information. In our present work, we focus on the recognition of printed MEs and assume connected components (ccs) of a given ME image are labelled. Our approach to ME recognition comprises three stages,namely symbol formation, structural analysis and generation of encoding form like LATEX. In this paper, we present symbol formation process, where multi-cc symbols (like =, ≡ etc.) are formed, identity of context-dependent symbols (like a horizontal line can be MINUS, OVERBAR, FRACTION etc.) are resolved using spatial relations. Multi-line MEs like matrices and enumerated functions are also handled in this stage. A rule-based approach is proposed for the purpose, where the heuristics based on spatial relations are represented in the form of rules (knowledge) and those rules are fired depending on input data (labelled ccs). As knowledge is isolated from data like an expert system in our approach, it allows for easy adaptability and extensibility of the process. Proposed approach also handles both single-line and multi-line MEs in an unified manner. Our approach has been tested on around 800 MEs collected from various mathematical documents and experimental results are reported on them.KeywordsMathematical expressionsconnected componentssymbol formationrule-based approach

Deep learning methodologies have had a major impact on performance across a wide variety of machine learning tasks, and speech recognition is no exception. We describe a set of deep learning techniques that proved to be particularly successful in achieving performance gains in word error rate on a popular large vocabulary conversational speech recognition benchmark task (“Switchboard”). We found that the best performance is achieved by combining features from both recurrent and convolutional neural networks. We compare two recurrent architectures: partially unfolded nets with max-out activations and bidirectional long short-term memory nets. In addition, inspired by the success of convolutional networks for image classification, we designed a convolutional net with many convolutional layers and small kernels that create a receptive field with more nonlinearity and fewer parameters than standard configurations. When combined, these neural networks achieve a word error rate of 6.2% on this difficult task; this was the best reported rate at the time of this writing and is even more remarkable given that human performance itself is estimated to be 4% on this data.

Designing of VehiNet Using Convolutional Neural Networks and Deep Learning Techniques

Deep learning such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) have been successfully applied to Natural Language Processing (NLP). Previous research shows that RNN and CNN achieved significant results in extracting long and short-term dependencies. However, there is a lack of research on Persian/Dari due to low resources and with many difficulties in preprocessing. In this research paper, we proposed a novel sentiment analysis model using bidirectional GRU and CNN to address the challenges and improve sentiment knowledge in the Persian/Dari language. Firstly, a preprocessor is utilized to enhance data quality by eliminating noise. Secondly, a bidirectional Gated Recurrent Unit is used to capture long-term dependencies. Thirdly, our model(named PersiSentNet) employs CNN with max pooling to extract contextual features and reduce feature dimensionality. Gaussian Noise and Dropout are applied as regularization techniques to avoid the overfitting problem. Finally, the model’s performance is verified on two standard datasets, and the experimental results indicate that our model significantly outperforms the state-of-the-art models, highlighting its robustness and effectiveness in addressing the challenges of Persian/Dari language sentiment analysis.