Handwritten word image matching based on Heat Kernel Signature

Keyword Spotting is an alternative method for retrieving query words, without Optical Character Recognition (OCR), by calculating the similarity between features of word images rather than ASCII content. However, because of unconstrained writing styles with large variations, the retrieving results are always not very satisfactory.

Methods for text segmentation from scene images

To realize keyword spotting by means of query-by-example, learning efficient representation for word images is an essential issue. However, the amount of vocabulary at the training stage is often far less than the complete vocabulary of a certain language in various learning based representation approaches. Thus, unseen vocabularies might be taken as query keywords which may not exist in training set. Therefore, out-of-vocabulary (OOV) is frequently occurred in keyword spotting. In this paper, a sequence to sequence model with attention mechanism has been proposed to generate representation vectors of word images for solving the problem of OOV. After that, similarities can be calculated between each word image and a given query keyword image on their representation vectors. And then, a ranking list can be formed in descending order of the similarities for a collection of word images. Experimental results demonstrate that the proposed representation approach can be competent for the task of OOV keyword spotting and outperforms various baseline and state-of-the-art methods.

ABSTRACTSlope and slant correction of offline handwritten word images are two of the major pre-processing steps in document image processing, because these reduce the variations in writing, thereby make further processing of the same much easier. This paper presents novel slope and slant correction methods that are applied in three different script handwritten words namely Devanagari, Bangla and Roman. The language dependency and the computational complexity of state-of-the-art approaches towards the word level slope and slant correction are addressed here. A new technique for approximate core region detection is introduced here for skew detection and then linear regression is recursively applied to de-skew the word image. Whereas, in case of slant correction, a novel cost function over the vertical projection of de-skewed image is designed and optimized to fix the uniform slant angle of text words. A new benchmarked database is developed herein to evaluate the proposed methods both quantitatively and qualitatively. Comparison of the performances by our methods with some existing slope and slant correction methods reveals that our methods are more accurate and faster.

We propose a fast stroke-based method for off-line handwritten word recognition, particularly adapted for use with semantic educational video indexing (blackboard and hand drawn slides taken from low quality videos). We first extract a skeleton from each handwritten word image, then break the skeleton into a sequence of strokes. For each candidate vocabulary word, we use a dynamic programming algorithm on the stroke sequence to incorporate character segmentation and recognition into one procedure, where we find the optimal segmentation and similarity score between word and image simultaneously. Although our method is independent of the implementation of the handwritten character recognition(HCR) module itself, we also propose a stroke-based HCR approach. Stroke matching is based on a set of semantic rules, which are fast and robust to writing styles. We evaluate the overall approach on three different training sets, and the results in this difficult domain are encouraging.

An efficient word spotting framework is proposed to search text in scanned books. The proposed method allows one to search for words when optical character recognition (OCR) fails due to noise or for languages where there is no OCR. Given a query word image, the aim is to retrieve matching words in the book sorted by the similarity. In the offline stage, SIFT descriptors are extracted over the corner points of each word image. Those features are quantized into visual terms (visterms) using hierarchical K-Means algorithm and indexed using an inverted file. In the query resolution stage, the candidate matches are efficiently identified using the inverted index. These word images are then forwarded to the next stage where the configuration of visterms on the image plane are tested. Configuration matching is efficiently performed by projecting the visterms on the horizontal axis and searching for the Longest Common Subsequence (LCS) between the sequences of visterms. The proposed framework is tested on one English and two Telugu books. It is shown that the proposed method resolves a typical user query under 10 milliseconds providing very high retrieval accuracy (Mean Average Precision 0.93). The search accuracy for the English book is comparable to searching text in the high accuracy output of a commercial OCR engine.

There have been many published works on using Dynamic Time Warping (DTW) for word image matching. It has also been shown that constrained DTW works better than classical DTW, due to it's property to limit the warping path and to avoid pathological matching. Moreover, some studies also demonstrate that variants of DTW can perform better, compared to classical DTW. To take advantages from these techniques, we propose in this paper to study how combination of such approaches can even more improve the results. First, we propose to use pseudo Local DTW (LDTW: more efficient than DTW in our experiments) with a constraint band. Next, several other sequence matching techniques are parametrically combined to study their potential advantages. For this, we adapted the approach to the context of word image matching to obtain the value of the parameters. It is shown that, at least on Bentham and George Washington datasets, results can be improved by combining some algorithms. Please note that the goal of this research work is not to propose a competitive word spotting technique but to propose a dynamic programming based learning free approach for word spotting or image matching, which can perform moderately with respect to other existing approaches in this domain.

Optical Character Recognition (OCR) is a technique that generates text from an image. Recognizing the importance of OCR in real-world settings, a plethora of techniques have been developed for Western, as well as Asian languages. Urdu is a prominent South Asian language and a number of different solutions for Urdu OCR have been proposed. However, fewer attempts have been made to develop end-to-end deep learning-based solutions for recognizing printed Urdu text. Furthermore, several benchmark corpora for Urdu OCR have been developed that can be used for training and evaluation of different OCR techniques. However, there are a number of limitations of the existing Urdu corpora: firstly, most of them have either character or word or text images, which are usually rendered using only a single font, Nastaleeq. Secondly, the volume of the existing datasets is so small that it is not suitable for working with the deep-learning techniques that have achieved groundbreaking results for OCRs. To that end, in this study, we have proposed a very large Multi-level and Multi-script Urdu corpus (MMU-OCR-21). It is the largest-ever Urdu corpus of printed text that is effectively suitable to work with deep learning techniques. In total, the corpus is composed of over 602,472 images, including text-line and word images in three prominent fonts, and their respective ground truth. Also, we have performed experiments using multiple state-of-the-art deep learning techniques for text-line and word level images.

We propose novel point descriptors for 3D shapes with the potential to match two shapes representing the same object undergoing natural deformations. These deformations are more general than the often assumed isometries, and we use labeled training data to learn optimal descriptors for such cases. Furthermore, instead of explicitly defining the descriptor, we introduce new Mercer kernels, for which we formally show that their corresponding feature space mapping is a generalization of either the Heat Kernel Signature or the Wave Kernel Signature. I.e. the proposed descriptors are guaranteed to be at least as precise as any Heat Kernel Signature or Wave Kernel Signature of any parameterisation. In experiments, we show that our implicitly defined, infinite-dimensional descriptors can better deal with non-isometric deformations than state-of-the-art methods.

A hierarchical approach to recognition of handwritten Bangla characters

Text recognition in natural scene images is a challenging problem in computer vision. Different than the optical character recognition (OCR), text recognition in natural scene images is more complex due to variations in text size, colors, fonts, orientations, complex backgrounds, occlusion, illuminations and uneven lighting conditions. In this paper, we propose a segmentation-free method based on a deep convolutional recurrent neural network to solve the problem of cursive text recognition, particularly focusing on Urdu text in natural scenes. Compared to the non-cursive scripts, Urdu text recognition is more complex due to variations in the writing styles, several shapes of the same character, connected text, ligature overlapping, stretched, diagonal and condensed text. The proposed model gets a whole word image as an input without pre-segmenting into individual characters, and then transforms into the sequence of the relevant features. Our model is based on three components: a deep convolutional neural network (CNN) with shortcut connections to extract and encode the features, a recurrent neural network (RNN) to decode the convolutional features, and a connectionist temporal classification (CTC) to map the predicted sequences into the target labels. To increase the text recognition accuracy further, we explore deeper CNN architectures like VGG-16, VGG-19, ResNet-18 and ResNet-34 to extract more appropriate Urdu text features, and compare the recognition results. To conduct the experiments, a new large-scale benchmark dataset of cropped Urdu word images in natural scenes is developed. The experimental results show that the proposed deep CRNN network with shortcut connections outperform than other network architectures. The dataset is publicly available and can be downloaded from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://data.mendeley.com/datasets/k5fz57zd9z/1</uri> .

The optical character recognition (OCR) system is implemented for the classification of text and language retrieval systems. Due to the complexity in the text style and the higher number of connected components OCR has failed to classify and retrieve many regional languages like Telugu. Thus, to overcome these problems, retrieval of a word image system is developed using intelligence-based Telugu word image retrieval (TWIR) system. This paper presented a work to retrieve Telugu word images using a DeepCluster-based or multilayered deep learning convolution neural network (DL-CNN). Then, a deep learning-based model is used for extraction of relevant features of Telugu word images using AlexNet and the procedure named as DeepCluster. Thus, by using the DeepCluster method various logy grammar rules of Telugu scripts will be perfectly analysed. Thus, the TWIR system will be effectively used for the retrieval of Telugu words and useful in all real-time applications such as Telugu web browsing and Telugu speech analysis. The proposed DeepCluster model gives outstanding simulation results compared to the state-of-the-art approaches for both average precision (mAP) and mean average recall (mAR) metrics.

The amount of handwritten documents that is digitally available is rapidly increasing. However, we observe a certain lack of accessibility to these documents especially with respect to searching and browsing. This paper aims at closing this gap by means of a novel method for keyword spotting in ancient handwritten documents. The proposed system relies on a keypoint-based graph representation for individual words. Keypoints are characteristic points in a word image that are represented by nodes, while edges are employed to represent strokes between two keypoints. The basic task of keyword spotting is then conducted by a recent approximation algorithm for graph edit distance. The novel framework for graph-based keyword spotting is tested on the George Washington dataset on which a state-of-the-art reference system is clearly outperformed.

Optical Character Recognition is considered one of the fastest methods of data entry. OCR converts the text image representation of x and y coordinates representing pixel information to be converted into text data in a particular language. OCR as a field of pattern recognition and document image understanding, OCR requires a challenging job once a different language text is available on the image. Difference in language script will pose different challenges for OCR which requires entirely different approaches and algorithms. Latin scripts require a different approach whereas the Arabic adopted language scripts require a different approach. In this regard, various solutions have been proposed for different languages. Segmentation is considered one of the important tasks in the process of OCR. A good segmentation will definitely increase the accuracy of an OCR. Segmentation includes the segmentation of text lines from text images which are further divided into words. These segmented words are further divided into characters which are to be recognized. A single segmentation algorithm to segment various scripts of the languages is proposed in this study which checks the script and then segments the text image for the further processing in OCR. The proposed generalized algorithm will check the style, direction and other properties of the script and then adopts the segmentation process to segment text lines, words and characters of the language. The proposed algorithm segments more than ten languages of three scripts and segments for their OCRs. These images can be further processed for feature extraction and classification further. The process of OCR for selected languages will be made easier to recognize. Multiple scripts, languages and images were experimented, and the proposed algorithm successfully segmented 32,833 images of text line, words and character image. The algorithm provides 97% accuracy while segmenting these images and can be extended to further languages as well as scripts.

Word spotting in handwritten document images is a field of immense interest due to its widespread applications. Recognition-free and recognition-based approaches are the two comprehensively studied regimes for the said problem out of which the first one is more realistic for practical applications. In literature, several works have been found that have used contour and distance-based measures for matching of the profiles of two word images. Although this is a prudent choice for printed words, the same often faces bottlenecks for unconstrained handwriting. To this end, this work applies dynamic time warping algorithm on logarithmic profiles of handwritten word images to lessen the uncontrolled profile variation that occurs due to elongation while writing some characters. We have considered both global and local interpretations of a word image by dividing it vertically into a number of sub-parts. This multi-view analysis provides close-up views of different approximations for the same word image. Finally, a voting scheme is evoked to produce the final decision. Besides, we have adopted a pruning method to pre-filter the target word images prior to applying the voting-based word matching scheme. The method has been tested on word images, taken from Qatar University Writer Identification database. We have obtained satisfactory results as compared to many state-of-the-art methods that also include deep learning-based feature extraction models.