Text detection in natural scene images using morphological component analysis and Laplacian dictionary

Reading the text embedded in natural scene images is essential to many applications. In this paper, we propose a method for detecting text in scene images based on multi-level connected component (CC) analysis and learning text component features via convolutional neural networks (CNN), followed by a graph-based grouping of overlapping text boxes. The multi-level CC analysis allows the extraction of redundant text and non-text components at multiple binarization levels to minimize the loss of any potential text candidates. The features of the resulting raw text/non-text components of different granularity levels are learned via a CNN. Those two modules eliminate the need for complex ad-hoc preprocessing steps for finding initial candidates, and the need for hand-designed features to classify such candidates into text or non-text. The components classified as text at different granularity levels, are grouped in a graph based on the overlap of their extended bounding boxes, then, the connected graph components are retained. This eliminates redundant text components and forms words or textlines. When evaluated on the Robust Reading Competition dataset for natural scene images, our method achieved better detection results compared to state-of-the-art methods. In addition to its efficacy, our method can be easily adapted to detect multi-oriented or multi-lingual text as it operates at low level initial components, and it does not require such components to be characters.

Signs are ubiquitous indoors and outdoors, which are used for way finding, finding shops and businesses, accessing variety of services. But the information of signs is inaccessible to many visually impaired people unless they are represented in a non-visual form such as Braille, tactile graphic, and speech. Automatic reading text from signs in natural images becomes a vital application in visually impaired people assistance. However, finding the text in scene images is a great challenge, because it cannot be assumed that the acquired image contains only characters. Natural scene images usually contain diverse complex text of different size, styles and colors with complex backgrounds. Therefore, this paper proposes a novel method for text extraction from scene images. The algorithm is implemented and evaluated using a set of natural scene images. Accuracy, precision and recall rates of the proposed method are analyzed to determine the success and limitation. Recommendations for improvements are given based on the results.

Scene text magnifier aims to magnify text in natural scene images without recognition. It could help the special groups, who have myopia or dyslexia to better understand the scene. In this paper, we design the scene text magnifier through interacted four CNN-based networks: character erasing, character extraction, character magnify, and image synthesis. The architecture of the networks are extended based on the hourglass encoderdecoders. It inputs the original scene text image and outputs the text magnified image while keeps the background unchange. Intermediately, we can get the side-output results of text erasing and text extraction. The four sub-networks are first trained independently and fine-tuned in end-to-end mode. The training samples for each stage are processed through a flow with original image and text annotation in ICDAR2013 and Flickr dataset as input, and corresponding text erased image, magnified text annotation, and text magnified scene image as output. To evaluate the performance of text magnifier, the Structural Similarity is used to measure the regional changes in each character region. The experimental results demonstrate our method can magnify scene text effectively without effecting the background.

Scene text image super-resolution (STISR) aims to improve the resolution and visual quality of low-resolution (LR) scene text images, while simultaneously boost the performance of text recognition. However, most of the existing STISR methods regard text images as natural scene images, ignoring the categorical information of text. In this paper, we make an inspiring attempt to embed text recognition prior into STISR model. Specifically, we adopt the predicted character recognition probability sequence as the text prior, which can be obtained conveniently from a text recognition model. The text prior provides categorical guidance to recover high-resolution (HR) text images. On the other hand, the reconstructed HR image can refine the text prior in return. Finally, we present a multi-stage text prior guided super-resolution (TPGSR) framework for STISR. Our experiments on the benchmark TextZoom dataset show that TPGSR can not only effectively improve the visual quality of scene text images, but also significantly improve the text recognition accuracy over existing STISR methods. Our model trained on TextZoom also demonstrates certain generalization capability to the LR images in other datasets.

This paper proposes a method of identifying character-like regions in order to extract and recognize characters in natural color scene images automatically. After connected component extraction based on a multi-group decomposition scheme, alignment analysis is used to check the block candidates, namely, the character-like regions in each binary image layer and the final composed image. Priority adaptive segmentation (PAS) is implemented to obtain accurate foreground pixels of the character in each block. Then some heuristic meanings such as statistical features, recognition confidence, and alignment properties, are employed to justify the segmented characters. The algorithms are robust for a wide range of character fonts, shooting conditions, and color backgrounds. Results of our experiments are promising for real applications.

Buffer-text: Detecting arbitrary shaped text in natural scene image

Text detection & localization plays an essential role in finding the textual information from natural scene images that can be used in robot navigation, license plate detection, and wearable applications. In this work, we present text detection and localization approach based upon a novel text awareness model that encompasses an improved fast edge preserving and smoothing Maximum Stable Extremal Region (FEPS-MSER) algorithm which uses the fast guided filter to separate the interconnected characters efficiently by removing the mixed pixels around the edges of blurred images. The fast guided filter takes less execution time as compared to other edge-smoothing filters. The combination of five independent and class determining facets namely stroke width deviation, 8-histogram of edge gradients, color variation, occupation ratio, and occupy rate convex area is proposed to differentiate between text and non-text components. The probability of a component to be text is based on Text Awareness Score (TAS) that is calculated by fusing these facets in Naive Bayes using the observation possibility and prior probability of text & non-text components. Naive Bayes classifier helps in accurate and fast determination of the text awareness score and thus helps in the classification of text & non-text components with the help of graph cut algorithm. The text components have been grouped by using the mean-shift clustering algorithm which is a non-parametric technique and does not require the initial knowledge of clusters. The proposed method achieves improved results concerning precision, recall, and f-measure on the ICDAR benchmark datasets for natural scene images.

Text in natural scene image portrays rich semantic information that plays an important role in content analysis. However, apart from Arabic text in documents, the text in natural scene images exhibit much higher diversity and variability, especially in uncontrolled circumstances. In this paper, a hybrid feature extraction approach is presented to detect extremal region of Arabic scene text. The binary image and image mask are considered as a variant of input image and look for concurrent extremal regions in both images. After determination of conjoined extremal points, the scale invariant technique is applied to consider those invariant points which are common in both images based on their coordinate positions. To evaluate the performance, a multidimensional long short term memory (LSTM) network is adapted and obtained 94.21% accuracy for word recognition on unconstrained Arabic scene text recognition (ASTR) dataset.

Information is one of the foremost fact in the prompt world. Within that, text information plays an imperative role and can acquire diverse mold. The natural images that consist of such text information are called scene text images. Semantic information of the image is used for content-based image retrieval, indexing and classification purpose. First stage of text extraction is the text and non-text classification that determines the presence of the text in an image. Compared to English language, determining the presence of Malayalam text in a scene image is more difficult due to its agglutinative nature. In this paper, the proposed work classifies natural scene images into Malayalam text and non-text images using Multiple Instance Learning (MIL). Our own dataset that contains natural scene images with Malayalam text and non-text images are used for the performance evaluation. Analysis is done in terms of precision, recall, F1 score, accuracy rates and has a promising experimental result.

Text in scene images usually contains significant information. Text detection and recognition in the scene is important for a variety of advanced machine vision applications, such as image and video retrieval, automotive assistance, and multilingual translation. In particular, most text recognition systems require texts to be localized in images beforehand and this is a significant demand. The purpose of this study is to provide a method to detect texts in natural images. The proposed approach combines advantages of extremal region, ER, methods and classification of convolutional neural network, CNN. This significantly reduces the false positives and increases the accuracy of detection. The method of sliding windows is employed with different sizes in order to determine text candidates. Extraction of enhanced ERs is performed in three consecutive stages on three distinct color channels, R, G, and B. Then, the results are combined together by an add method. After grouping, the word candidates are classified to two classes of text and non-text sections by a CNN classifier. By applying non-maximum suppression (NMS) algorithm to the same words, words with the highest probability are selected. The average values of accuracy, recall, precision and F-measure of the proposed text detection model on the ICDAR2013 database are 0.893, 0.962, 0.948, and 0.955, respectively. The optimal cut point of the proposed method is 0.648, which has the highest average accuracy, 91.93%. The AUC of ROC and PR diagrams for the proposed model are 0.851 and 0.718, respectively. These results of AUC for ROC and PR curves showed an outstanding enhancement in comparison with the best detection rate of previous methods. Experimental results on the ICDAR2011, ICDAR2013 and ICDAR2015 databases also demonstrate that our algorithm outperforms the state-of-the-art scene text detection methods.

Text extraction is one of the most intuitive and natural ways in understanding of natural scene images. Localization is the first and one of the most important steps in this problem. In this paper, we propose a novel method for localizing text in natural scene images. We introduce a new operator so called Edge Color Transform (ECT) to solve this problem. After extracting the edge map of the input image, this operator follows the gradient direction (and the opposite direction) to assign the nearest color to each of the edge pixels. In the next step, each channel (Red, Green and Blue) of these color pixels are normalized to make them more robust to intensity changes. In the following step, these colored pixels are grouped together using a modified version of region growing algorithm. In this customized version of the algorithm, pixels of a region do not have to be spatially connected and their connectivity is decided based on a predefined oval-shaped neighborhood. When the regions are forged, each of them form a text candidate region. Subsequently, all of these candidates are applied to a classifier to extract text regions. Experimental results show that the proposed method performs well on both Farsi and well-known ICDAR 2013 data sets.

Robust and accurate detection of text in natural scene images and document images is a very challenging and common research problem. Over the past few decades, a variety of algorithms for text detection in images have been developed but there is still need for more robust and accurate text detection methods. In this work, we have proposed an accurate and robust text detection framework in which canny edge detection, maximally stable extremal regions and geometric filtering are employed in combination to efficiently collect and filter letter candidates in an image. Subsequently, individual letter patches are grouped to detect text sequences, which are then fragmented into isolated word patches. Finally, optical character recognition is employed to digitize the word patches. The proposed algorithm is tested on images representing different scenarios ranging from documents to natural scenes. Promising results have been reported which prove the accuracy and robustness of the proposed framework and encourage its practical implementation in real world applications.

Recently proposed methods for weakly-supervised semantic segmentation have achieved impressive performance in predicting pixel classes despite being trained with only image labels which lack positional information. Because image annotations are cheaper and quicker to generate, weak supervision is more practical than full supervision for training segmentation algorithms. These methods have been predominantly developed to solve the background separation and partial segmentation problems presented by natural scene images and it is unclear whether they can be simply transferred to other domains with different characteristics, such as histopathology and satellite images, and still perform well. This paper evaluates state-of-the-art weakly-supervised semantic segmentation methods on natural scene, histopathology, and satellite image datasets and analyzes how to determine which method is most suitable for a given dataset. Our experiments indicate that histopathology and satellite images present a different set of problems for weakly-supervised semantic segmentation than natural scene images, such as ambiguous boundaries and class co-occurrence. Methods perform well for datasets they were developed on, but tend to perform poorly on other datasets. We present some practical techniques for these methods on unseen datasets and argue that more work is needed for a generalizable approach to weakly-supervised semantic segmentation. Our full code implementation is available on GitHub: https://github.com/lyndonchan/wsss-analysis.

Segmentation-based methods have achieved great success for arbitrary shape text detection. However, separating neighboring text instances is still one of the most challenging problems due to the complexity of texts in scene images. In this article, we propose an innovative kernel proposal network (dubbed KPN) for arbitrary shape text detection. The proposed KPN can separate neighboring text instances by classifying different texts into instance-independent feature maps, meanwhile avoiding the complex aggregation process existing in segmentation-based arbitrary shape text detection methods. To be concrete, our KPN will predict a Gaussian center map for each text image, which will be used to extract a series of candidate kernel proposals (i.e., dynamic convolution kernel) from the embedding feature maps according to their corresponding keypoint positions. To enforce the independence between kernel proposals, we propose a novel orthogonal learning loss (OLL) via orthogonal constraints. Specifically, our kernel proposals contain important self-information learned by network and location information by position embedding. Finally, kernel proposals will individually convolve all embedding feature maps for generating individual embedded maps of text instances. In this way, our KPN can effectively separate neighboring text instances and improve the robustness against unclear boundaries. To the best of our knowledge, our work is the first to introduce the dynamic convolution kernel strategy to efficiently and effectively tackle the adhesion problem of neighboring text instances in text detection. Experimental results on challenging datasets verify the impressive performance and efficiency of our method. The code and model are available at https://github.com/GXYM/KPN.

Scene text recognition is a challenging task due to the complex backgrounds and diverse variations of text instances. In this paper, we propose a novel Semantic GAN and Balanced Attention Network (SGBANet) to recognize the texts in scene images. The proposed method first generates the simple semantic feature using Semantic GAN and then recognizes the scene text with the Balanced Attention Module. The Semantic GAN aims to align the semantic feature distribution between the support domain and target domain. Different from the conventional image-to-image translation methods that perform at the image level, the Semantic GAN performs the generation and discrimination on the semantic level with the Semantic Generator Module (SGM) and Semantic Discriminator Module (SDM). For target images (scene text images), the Semantic Generator Module generates simple semantic features that share the same feature distribution with support images (clear text images). The Semantic Discriminator Module is used to distinguish the semantic features between the support domain and target domain. In addition, a Balanced Attention Module is designed to alleviate the problem of attention drift. The Balanced Attention Module first learns a balancing parameter based on the visual glimpse vector and semantic glimpse vector, and then performs the balancing operation for obtaining a balanced glimpse vector. Experiments on six benchmarks, including regular datasets, i.e., IIIT5K, SVT, ICDAR2013, and irregular datasets, i.e., ICDAR2015, SVTP, CUTE80, validate the effectiveness of our proposed method.KeywordsSemantic GANSemantic generatorSemantic discriminatorBalanced attentionScene text recognition