A novel approach to detecting scene text in video

Video scene text contains semantic information and thus can contribute significantly to video indexing and summarization. However, most of the previous approaches to detecting scene text from videos experience difficulties in handling texts with various character size and text alignments. In this paper, we propose a novel algorithm of scene text detection and localization in video. Based on our observation that text character strokes show intensive edge details in the fixed orientation no matter what text alignment and size are, a stroke map is first generated. In the scene text detection, we extract the texture feature of stroke map to locate text lines. The detected scene text lines are accurately located by using Harris' corners in the stroke map. Experimental results show that this approach is robust and can be effectively applied to scene text detection and localization in video.

Script identification at the word level is challenging because of complex backgrounds and low resolution of video. The presence of graphics and scene text in video makes the problem more challenging. In this paper, we employ gradient angle segmentation on words from video text lines. This paper presents new Gradient-Angular-Features (GAF) for video script identification, namely, Arabic,Chinese, English, Japanese, Korean and Tamil. This work enables us to select an appropriate OCR when the frame has words of multi-scripts. We employ gradient directional features for segmenting words from video text lines. For each segmented word, we study the gradient information in effective ways to identify text candidates. The skeleton of the text candidates is analyzed to identify Potential Text Candidates (PTC) by filtering out unwanted text candidates. We propose novel GAF for the PTC to study the structure of the components in the form of cursiveness and softness. The histogram operation on the GAF is performed in different ways to obtain discriminative features. The method is evaluated on 760 words of six scripts having low contrast, complex background, different font sizes, etc. in terms of the classification rate and is compared with an existing method to show the effectiveness of the method. We achieve 88.2% average classification rate.

Scene text detection is a crucial step in end-to-end scene text recognition, a greatly challenging problem in computer vision. This paper proposes a novel scene text detection method that involves superpixel-based stroke feature transform (SSFT) and deep learning based region classification (DLRC). The SSFT is developed for candidate character region (CCR) extraction, which consists in partitioning an input image into several regions via superpixel-based clustering, removing most regions based on predefined criteria satisfied by the characters, and refining the remaining regions to obtain CCRs by computing a stroke width map. The character regions are identified from the CCRs using DLRC, in which several hand-crafted low-level features, i.e., color, texture, and geometric features, and some deep convolution neural network (CNN) based high-level features are first extracted from the regions, and then these features are fused by using two fully connected networks (FCNs) for region classification. In the DLRC step, the deep feature extraction CNN and the feature fusion FCNs are jointly trained. Next, the extracted character regions are merged to form candidate text regions, from which the final scene texts are detected. The proposed method is evaluated on three publicly available datasets: ICDAR2011, ICDAR2013, and street view text. It achieves F -measures of 0.876, 0.885, and 0.631, respectively, which demonstrate the effectiveness of the proposed scene text detection method.

Detecting and recognizing text in natural scene videos and images has brought more attention to computer vision researchers due to applications like robotic navigation and traffic sign detection. In addition, Optical Character Recognition (OCR) technology is applied to detect and recognize text on the license plate. It will be used in various commercial applications such as finding stolen cars, calculating parking fees, invoicing tolls, or controlling access to safety zones and aids in detecting fraud and secure data transactions in the banking industry. Much effort is required when scene text videos are in low contrast and motion blur with arbitrary orientations. Presently, text detection and recognition approaches are limited to static images like horizontal or approximately horizontal text. Detecting and recognizing text in videos with data dynamicity is more challenging because of the presence of multiple blurs caused by defocusing, motion, illumination changes, arbitrarily shaped, and occlusion. Thus, we proposed a combined DeepEAST (Deep Efficient and Accurate Scene Text Detector) and Keras OCR model to overcome these challenges in the proffered DEFUSE (Deep Fused) work. This two-combined technique detects the text regions and then deciphers the result into a machine-readable format. The proposed method has experimented with three different video datasets such as ICDAR 2015, Road Text 1K, and own video Datasets. Our results proved to be more effective with precision, recall, and F1-Score.

Scene text detection is an important and challenging task in computer vision. For detecting arbitrarily-shaped texts, most existing methods require heavy data labeling efforts to produce polygon-level text region labels for supervised training. In order to reduce the cost in data labeling, we study mixed-supervised arbitrarily-shaped text detection by combining various weak supervision forms (e.g., image-level tags, coarse, loose and tight bounding boxes), which are far easier to annotate. Whereas the existing weakly-supervised learning methods (such as multiple instance learning) do not promote full object coverage, to approximate the performance of fully-supervised detection, we propose an Expectation-Maximization (EM) based mixed-supervised learning framework to train scene text detector using only a small amount of polygon-level annotated data combined with a large amount of weakly annotated data. The polygon-level labels are treated as latent variables and recovered from the weak labels by the EM algorithm. A new contour-based scene text detector is also proposed to facilitate the use of weak labels in our mixed-supervised learning framework. Extensive experiments on six scene text benchmarks show that (1) using only 10% strongly annotated data and 90% weakly annotated data, our method yields comparable performance to that of fully supervised methods, (2) with 100% strongly annotated data, our method achieves state-of-the-art performance on five scene text benchmarks (CTW1500, Total-Text, ICDAR-ArT, MSRA-TD500, and C-SVT), and competitive results on the ICDAR2015 Dataset. We will make our weakly annotated datasets publicly available.

Multi-oriented and multi-lingual scene text detection plays an important role in computer vision area and is challenging due to the wide variety of text and background. In this paper, firstly we point out the two key tasks when extending CNN based object detection frameworks to scene text detection. The first task is to localize the text region by a down-sampled segmentation based module, and the second task is to regress the boundaries of text region determined by the first task. Secondly, we propose a scene text detection framework based on fully convolutional network (FCN) with a bi-task prediction module in which one is pixel-wise classification between text and non-text, and the other is pixel-wise regression to determine the vertex coordinates of quadrilateral text boundaries. Post-processing for word-level detection is based on Non-Maximum Suppression (NMS), and for line-level detection we design a heuristic line segments grouping method to localize long text lines. We evaluated the proposed framework on various benchmarks including multi-oriented and multi-lingual scene text datasets, and achieved state-of-the-art performance on most of them. We also provide abundant ablation experiments to analyze several key factors in building high performance CNN based scene text detection systems.

The existing deep learning based state-of-theart scene text detection methods treat scene texts a type of general objects, or segment text regions directly. The latter category achieves remarkable detection results on arbitraryorientation and large aspect ratios of scene texts based on instance segmentation algorithms. However, due to the lack of context information with consideration of scene text unique characteristics, directly applying instance segmentation to text detection task is prone to result in low accuracy, especially producing false positive detection results. To ease this problem, we propose a novel text-context-aware scene text detection CNN structure, which appropriately encodes channel and spatial attention information to construct context-aware and discriminative feature map for multi-oriented and multi-language text detection tasks. With high representation ability of textcontext-aware feature map, the proposed instance segmentation based method can not only robustly detect multi-oriented and multi-language text from natural scene images, but also produce better text detection results by greatly reducing false positives. Experiments on ICDAR2015 and ICDAR2017-MLT datasets show that the proposed method has achieved superior performances in precision, recall and F-measure than most of the existing studies.

Scene texts refer to arbitrary text found in images captured by cameras in real-world settings. The tasks of text detection and recognition are critical components of computer vision, with applications spanning scene understanding, information retrieval, robotics, and autonomous driving. Despite significant advancements in deep learning methods, achieving accurate text detection and recognition in complex images remains a formidable challenge for robust real-world applications. Several factors contribute to these challenges. First, the diversity of text shapes, fonts, colors, and styles complicates detection efforts. Second, the myriad combinations of characters, often with unstable attributes, make complete detection difficult, especially when background interruptions obscure character strokes and shapes. Finally, effective coordination of multiple sub-tasks in end-to-end learning is essential for success. This research aimed to tackle these challenges by enhancing text discriminative representation. This study focused on two interconnected problems: Scene Text Recognition (STR), which involves recognizing text from scene images, and Scene Text Detection (STD), which entails simultaneously detecting and recognizing multiple texts within those images. This research focuses on implementing and evaluating the Efficient and Accurate Scene Text Detector (EAST) algorithm for text detection and recognition in natural scene images. The study aims to compare the performance of three prominent Optical Character Recognition (OCR) techniques—TesseractOCR, PaddleOCR, and EasyOCR. The EAST model was applied to a series of sample test images, and the results were visually represented with bounding boxes highlighting the detected text regions. The inference times for each image were recorded, highlighting the algorithm's efficiency, with average times of 0.446, 0.439, and 0.440 seconds for the respective test images. These results indicate that the EAST algorithm is accurate and operates in real-time, making it suitable for applications requiring immediate text recognition.

The generation of text proposals (i.e. local candidate regions most likely containing textual components) is one critical and prerequisite step in scene text detection task. As one popular text proposal algorithm, the Maximally Stable Extremal Region (MSER), has been exploited by many successful text detection methods, while on the other hand has difficulties in handling complicated scene text involving touching characters and characters composed of multiple unconnected parts (e.g. Chinese characters and text in dot matrix fonts). In this paper, we propose a novel text proposal method for localizing text in natural images, which integrates the MSER algorithm with the multi-scale sliding window framework and efficiently extracts Windowed Maximally Stable Extremal Regions (WMSERs) as text proposals. We further present effective proposal filtering and grouping algorithms for exploiting WMSER-based proposals in text detection task. Experiments on public scene text datasets demonstrate the promising aspects of the proposed method in dealing with complicated scene text.

Text and captions in videos provide useful information for content analysis and understanding. In this paper, we present an approach to detecting video text in a coarse-to-fine strategy. In the coarse phase we propose an efficient method to detect multi-scale candidate text regions with high recall. Then the candidate text regions are segmented and sent to the fine phase where a convolutional neural network(CNN) is applied to generate a confidence map for each candidate text region. Finally, the candidate text regions are further refined and partitioned into text lines by projection analysis. The CNN classifier in the fine phase enables feature sharing and robustly identifies text regions. The coarse phase sharply reduce the number of windows needed to be scanned by the CNN. The combination endows the proposed method with both efficiency and robustness when detecting video text. It was verified by experiment results on two publicly testing datasets and a dataset created by us.

Localizing scene texts by fuzzy inference systems and low rank matrix recovery model

Visually impaired people suffer daily from their disability to read textual information. One of the most anticipated blind-assistive devices is a system equipped with a wearable camera capable of finding the textual information in natural scenes and translating it into sound through a speech synthesizer. To avoid duplicate readings, the device should be able to recognize text areas with the same content, and group them to obtain a single result. Scene text detection and tracking methods attract a lot of interest for these purposes. However, this field is still challenging and methods of scene text detection and tracking are yet to be perfected. This paper proposes a scene text tracking system capable of finding text regions and tracking them in video frames captured by a wearable camera. By combining a text detection method with a feature point tracker, we obtain a robust text tracker which produces much less false positive text images at 2.9 times faster speed compared with the conventional method.

We live in a visual world where text cues are abundant in urban environments. The premise for our work is for robots to capitalize on these text features for visual place recognition. A new technique is introduced that uses an end-to-end scene text detection and recognition technique to improve robot localization and mapping through Visual Place Recognition (VPR). This technique addresses several challenges such as arbitrary shaped text, illumination variation, and occlusion. The proposed model captures text strings and associated bounding boxes specifically designed for VPR tasks. The primary contribution of this work is the utilization of an end-to-end scene text spotting framework that can effectively capture irregular and occluded text in diverse environments. We conduct experimental evaluations on the Self-Collected TextPlace (SCTP) benchmark dataset, and our approach outperforms state-of-the-art methods in terms of precision and recall, which validates the effectiveness and potential of our proposed approach for VPR.

At present, text orientation is not diverse enough in the existing scene text datasets. Specifically, curve-orientated text is largely out-numbered by horizontal and multi-oriented text, hence, it has received minimal attention from the community so far. Motivated by this phenomenon, we collected a new scene text dataset, Total-Text, which emphasized on text orientations diversity. It is the first relatively large scale scene text dataset that features three different text orientations: horizontal, multi-oriented, and curve-oriented. In addition, we also study several other important elements such as the practicality and quality of ground truth, evaluation protocol, and the annotation process. We believe that these elements are as important as the images and ground truth to facilitate a new research direction. Secondly, we propose a new scene text detection model as the baseline for Total-Text, namely Polygon-Faster-RCNN, and demonstrated its ability to detect text of all orientations. Images of Total-Text and its annotation are available at https://github.com/cs-chan/Total-Text-Dataset.

Scene text detection is an important step in the scene text reading system. There are still two problems during the existing text detection methods: (1) The small receptive of the convolutional layer in text detection is not sufficiently sensitive to the target area in the image; (2) The deep receptive of the convolutional layer in text detection lose a lot of spatial feature information. Therefore, detecting scene text remains a challenging issue. In this work, we design an effective text detector named Adaptive Multi-Scale HyperNet (AMSHN) to improve texts detection performance. Specifically, AMSHN enhances the sensitivity of target semantics in shallow features with a new attention mechanism to strengthen the region of interest in the image and weaken the region of no interest. In addition, it reduces the loss of spatial feature by fusing features on multiple paths, which significantly improves the detection performance of text. Experimental results on the Robust Reading Challenge on Reading Chinese Text on Signboard (ReCTS) dataset show that the proposed method has achieved the state-of-the-art results, which proves the ability of our detector on both particularity and universality applications.