Stroke order normalization for improving recognition of online handwritten mathematical expressions

We present a system for recognizing online mathematical expressions (ME). Symbol recognition is based on a template elastic matching distance between pen direction features. The structural analysis of the ME is based on extracting the baseline of the ME and then classifying symbols into levels above and below the baseline. The symbols are then sequentially analyzed using six spatial relations and a respective 2d structure is processed to give the resulting MathML representation of the ME. The system was evaluated on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2011 datasets and demonstrates promising results.

Handwritten mathematical expression recognition (HMER) is a challenging task due to the complex two-dimensional structure of mathematical expressions and the similarity of handwritten texts. Most existing methods for HMER only consider single-scale features while ignoring multi-scale features that are very important to HMER. Few works have explored the fusion of multi-scale features in HMER, but exhibited an extra branch that brings more parameters and computation. In this paper, we propose an end-to-end method to integrate multi-scale features using a unified model. Specifically, we customized the Dense Atrous Spatial Pyramid Pooling (DenseASPP) to our backbone network to capture the multi-scale features of the input image meanwhile expanding the receptive fields. Moreover, we added a symbol classifier using focal loss to better discriminate and recognize similar symbols, to further improve the performance of HMER. Experiments on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016 and 2019 shows that the proposed method achieves superior performance to most state-of-the-art methods, demonstrating the effectiveness of the proposed method.

Detection and recognition of mathematical expressions in document images are two key steps for the development of a mathematical expression retrieval system. So far, many researches have proposed for the recognition of expressions. However, few systems have integrated the detection and recognition of expressions in document images. This paper presents a deep learning based system for mathematical expression detection and recognition. Firstly, mathematical expressions have been detected on document images using the You Only Look Once (YOLO) v3 network. Then, detected expressions have been recognized in an end-to-end way using the advanced neural network that is the Watch, Attend and Parse (WAP). The proposed system has been tested on the Marmot public dataset. The obtained accuracies of the detection of isolated and inline expressions are 93% and 73%, respectively. Meanwhile, accuracies of the recognition for isolated and detected expressions are 51.77% and 45.50%, respectively. The results have shown the promising application of our preliminaiy research.

This paper presents an augmented incremental recognition method for online handwritten mathematical expressions (MEs). If an ME is recognized after all strokes are written (batch recognition), the waiting time increases significantly when the ME becomes longer. On the other hand, the pure incremental recognition method recognizes an ME whenever a new single stroke is input. It shortens the waiting time but degrades the recognition rate due to the limited context. Thus, we propose an augmented incremental recognition method that not only maintains the advantage of the two methods but also reduces their weaknesses. The proposed method has two main features: one is to process the latest stroke, and the other is to find the erroneous segmentations and recognitions in the recent strokes and correct them. In the first process, the segmentation and the recognition by Cocke–Younger–Kasami (CYK) algorithm are only executed for the latest stroke. In the second process, all the previous segmentations are updated if they are significantly changed after the latest stroke is input, and then, all the symbols related to the updated segmentations are updated with their recognition scores. These changes are reflected in the CYK table. In addition, the waiting time is further reduced by employing multi-thread processes. Experiments on our dataset and the CROHME datasets show the effectiveness of this augmented incremental recognition method, which not only maintains recognition rate even compared with the batch recognition method but also reduces the waiting time to a very small level.

Mathematical expression recognition is still a very challenging task for the research community mainly because of the two-dimensional (2d) structure of mathematical expressions (MEs). In this paper, we present a novel approach for the structural analysis between two on-line handwritten mathematical symbols of a ME, based on spatial features of the symbols. We introduce six features to represent the spatial affinity of the symbols and compare two multi-class classification methods that employ support vector machines (SVMs): one based on the “one-against-one” technique and one based on the “one-against-all”, in identifying the relation between a pair of symbols (i.e. subscript, numerator, etc). A dataset containing 1906 spatial relations derived from the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2012 training dataset is constructed to evaluate the classifiers and compare them with the rule-based classifier of the ILSP-1 system participated in the contest. The experimental results give an overall mean error rate of 2.61% for the “one-against-one” SVM approach, 6.57% for the “one-against-all” SVM technique and 12.31% error rate for the ILSP-1 classifier.

Recognition of spatial relations between pairs of subexpressions is a key problem of recognition of handwritten mathematical expressions. Most methods for spatial relation classification are based on handcrafted rules and geometric indices extracted from the subexpression bounding boxes. In this work, we propose new spatial relation features that combine subexpression bounding box and intra-subexpression information, along with prior knowledge about the general position and size of symbols. Instead of handcrafting features, we train artificial neural networks to learn the useful features from two kinds of histograms. The first type captures the relative positions and sizes of the subexpression bounding boxes. The second captures the relative positions and shape of a pair of symbols, called dominant symbols, extracted from the main baselines of the evaluated subexpressions. We evaluate and compare our features with two state-of-the-art features on a benchmark dataset. Experimental results show that our features obtain better accuracy than these two features.

Identifying strong handwritten characters is a difficult task in the field of medical field and it is tedious process on decoding handwritten medicines. Recognition of handwritten mathematical expressions is a complex issue. The distribution and classification of specific characters makes the task more difficult. In our project, handwritten numbers and symbols are read and further addition, subtraction and multiplication operations are performed. The project includes a study about model deployment using convoluted neural networks and flasks. We use the CNN to classify specific characters. Tracking of character string operations are used to solve equations. The maximum accuracy of the proposed model is 99.12% recall is 95%, sensitivity is 89% and specificity is 68%. Effectiveness of our proposed system is helpful for students who want to get handwritten answers. The equation can be extended to more complex equations and more user data can be trained to improve correction and accuracy.

A survey on handwritten mathematical expression recognition: The rise of encoder-decoder and GNN models

A stochastic model of stroke order variation is proposed and applied to the stroke-order free on-line Kanji character recognition.The proposed model is a hidden Markov model (HMM) with a special topology to represent all stroke order variations. A sequence of state transitions from the initial state to the final state of the model represents one stroke order and provides a probability of the stroke order.The distribution of the stroke order probability can be trained automatically by using an EM algorithm from a training set of on-line character patterns. Experimental results on large-scale test patterns showed that the proposed model could represent actual stroke order variations appropriately and improve recognition accuracy by penalizing incorrect stroke orders.

Mathematical expression recognition (MER) in images is a challenging task due to formula symbol recognition and structured analysis. Optical character recognition (OCR) has been used in natural language recognition and many areas. However, it is difficult for OCR to recognize some special formula symbols and accurately confirm their positions in MER. In this paper, an improved end-to-end MER approach based on CNN-RNNs (convolutional neural network - recurrent neural networks) is proposed to optimize the processing for the formula symbol recognition and localization. In our proposed approach, we extract the mathematical expression features in the images by CNN and generate the mathematical expression by RNNs. In order to improve the acquisition of fuzzy or small symbolic feature information and get an accurate symbol position, we double the source images and extract image features with preprocessed images by CNN. And we add a double-attention mechanism between the encoder and decoder so that the symbol position can be obtained accurately. In addition, to prevent overfitting, a dropout layer is introduced to improve the generalization of the MER model. We do the experiment on IM2LATEX-100K dataset and obtain the comparative result. A BLEU accuracy of 88.42% is achieved, which is better than other methods.

Automated understanding of mathematical expressions (MEs) is currently a challenging task due to their complex two- dimensional (2D) structure. Recognition of MEs can be online or offline and in either case, the process involves symbol recognition and analysis of 2D structure. This process is more complex for offline or printed MEs as they do not have temporal information. In our present work, we focus on the recognition of printed MEs and assume connected components (ccs) of a given ME image are labelled. Our approach to ME recognition comprises three stages,namely symbol formation, structural analysis and generation of encoding form like LATEX. In this paper, we present symbol formation process, where multi-cc symbols (like =, ≡ etc.) are formed, identity of context-dependent symbols (like a horizontal line can be MINUS, OVERBAR, FRACTION etc.) are resolved using spatial relations. Multi-line MEs like matrices and enumerated functions are also handled in this stage. A rule-based approach is proposed for the purpose, where the heuristics based on spatial relations are represented in the form of rules (knowledge) and those rules are fired depending on input data (labelled ccs). As knowledge is isolated from data like an expert system in our approach, it allows for easy adaptability and extensibility of the process. Proposed approach also handles both single-line and multi-line MEs in an unified manner. Our approach has been tested on around 800 MEs collected from various mathematical documents and experimental results are reported on them.KeywordsMathematical expressionsconnected componentssymbol formationrule-based approach

Mathematical Expression Recognition (MER) has made significant progress in recognizing simple expressions, but the robust recognition of complex mathematical expressions with many tokens and multiple lines remains a formidable challenge. In this paper, we first introduce CMER-Bench, a carefully constructed benchmark that categorizes expressions into three difficulty levels: easy, moderate, and complex. Leveraging CMER-Bench, we conduct a comprehensive evaluation of existing MER models and general-purpose multimodal large language models (MLLMs). The results reveal that while current methods perform well on easy and moderate expressions, their performance degrades significantly when handling complex mathematical expressions, mainly because existing public training datasets are primarily composed of simple samples. In response, we propose MER-17M and CMER-3M that are large-scale datasets emphasizing the recognition of complex mathematical expressions. The datasets provide rich and diverse samples to support the development of accurate and robust complex MER models. Furthermore, to address the challenges posed by the complicated spatial layout of complex expressions, we introduce a novel expression tokenizer, and a new representation called Structured Mathematical Language, which explicitly models the hierarchical and spatial structure of expressions beyond LaTeX format. Based on these, we propose a specialized model named CMERNet, built upon an encoder-decoder architecture and trained on CMER-3M. Experimental results show that CMERNet, with only 125 million parameters, significantly outperforms existing MER models and MLLMs on CMER-Bench.

The advent of various machine learning methods can add a distinct dimension to the domain of recognition. The realm of pattern recognition has been deeply influenced by the ongoing trend of artificial learning-based methodologies. The two-dimensional structure of mathematical symbols and expressions makes recognition tasks more difficult, particularly for mathematical expressions. This paper delves into recognition approaches based on machine learning and deep learning. The leading recognition algorithms from both categories, SVM, and CNN, have been deployed to recognize the Hasyv2 dataset. The competent accuracies of 62.3% and 76.21% have been given by SVM and CNN, respectively.

In this paper we propose a novel end-to-end framework for mathematical expression (ME) recognition. The method uses a convolutional neural network (CNN) to perform mathematical symbol detection and recognition simultaneously incorporating spatial context, and can handle multi-part and touching symbols effectively. To evaluate the performance, we provide a benchmark that contains MEs both from real-life and synthetic data. Images in our dataset undergo multiple variations such as viewpoint, illumination and background. For training, we use pure synthetic data for saving human labeling effort. The proposed method achieved 87% accuracy of total correct for clear images and 45% for cluttered ones.

In this paper, we propose a new framework for online handwritten mathematical expression recognition. The proposed architecture aims at handling mathematical expression recognition as a simultaneous optimization of symbol segmentation, symbol recognition, and 2D structure recognition under the restriction of a mathematical expression grammar. To achieve this goal, we consider a hypothesis generation mechanism supporting a 2D grouping of elementary strokes, a cost function defining the global likelihood of a solution, and a dynamic programming scheme giving at the end the best global solution according to a 2D grammar and a classifier. As a classifier, a neural network architecture is used; it is trained within the overall architecture allowing rejecting incorrect segmented patterns. The proposed system is trained with a set of synthetic online handwritten mathematical expressions. When tested on a set of real complex expressions, the system achieves promising results at both symbol and expression interpretation levels.