DEFUSE: Deep Fused End-to-End Video Text Detection and Recognition

Scene texts refer to arbitrary text found in images captured by cameras in real-world settings. The tasks of text detection and recognition are critical components of computer vision, with applications spanning scene understanding, information retrieval, robotics, and autonomous driving. Despite significant advancements in deep learning methods, achieving accurate text detection and recognition in complex images remains a formidable challenge for robust real-world applications. Several factors contribute to these challenges. First, the diversity of text shapes, fonts, colors, and styles complicates detection efforts. Second, the myriad combinations of characters, often with unstable attributes, make complete detection difficult, especially when background interruptions obscure character strokes and shapes. Finally, effective coordination of multiple sub-tasks in end-to-end learning is essential for success. This research aimed to tackle these challenges by enhancing text discriminative representation. This study focused on two interconnected problems: Scene Text Recognition (STR), which involves recognizing text from scene images, and Scene Text Detection (STD), which entails simultaneously detecting and recognizing multiple texts within those images. This research focuses on implementing and evaluating the Efficient and Accurate Scene Text Detector (EAST) algorithm for text detection and recognition in natural scene images. The study aims to compare the performance of three prominent Optical Character Recognition (OCR) techniques—TesseractOCR, PaddleOCR, and EasyOCR. The EAST model was applied to a series of sample test images, and the results were visually represented with bounding boxes highlighting the detected text regions. The inference times for each image were recorded, highlighting the algorithm's efficiency, with average times of 0.446, 0.439, and 0.440 seconds for the respective test images. These results indicate that the EAST algorithm is accurate and operates in real-time, making it suitable for applications requiring immediate text recognition.

Scene text in video contains semantic information and thus can contribute significantly to video retrieval and understanding. However, most methods detect scene text in image or single frame. Videos differ from images in temporal redundancy. In this paper we present a novel approach to detecting video scene text based on the video temporal redundancy. Video scene texts in consecutive frames have arbitrary motion due to camera or object movement. Therefore, first, we perform the motion detection in 30 consecutive frames to synthesize motion image. Second we implement video scene text detection in single frame to retrieve candidate text regions. Finally, the synthesized motion image is used to filter out candidate text regions and only keep the candidate text regions which have motion occurrence as final scene text. Experimental results show that our algorithm performs well for detecting scene text with various color, font-size and text alignment.

Text detection and recognition in scene images and videos attract much attention in computer vision recently. However, most existing text detection and recognition methods only focus on static images. In this paper an end-to-end scene text recognition method based on multi frame tracking is proposed for text in videos, in which temporal information is employed to improve performance. First, an end-to-end text recognition method based on a unified deep neural network is used to detect and recognize text in each frame of the input video. Then, multi frame text tracking is employed through associations of texts in current frame and several previous frames to obtain final results. Experiments on ICDAR datasets demonstrate that the proposed method outperforms the state-of-the-art methods in end-to-end video text recognition.

Video scene text contains semantic information and thus can contribute significantly to video indexing and summarization. However, most of the previous approaches to detecting scene text from videos experience difficulties in handling texts with various character size and text alignments. In this paper, we propose a novel algorithm of scene text detection and localization in video. Based on our observation that text character strokes show intensive edge details in the fixed orientation no matter what text alignment and size are, a stroke map is first generated. In the scene text detection, we extract the texture feature of stroke map to locate text lines. The detected scene text lines are accurately located by using Harris' corners in the stroke map. Experimental results show that this approach is robust and can be effectively applied to scene text detection and localization in video.

Recently, video text detection, tracking, and recognition in natural scenes are becoming very popular in the computer vision community. However, most existing algorithms and benchmarks focus on common text cases (e.g.,normal size, density) and single scenario, while ignoring extreme video text challenges, i.e., dense and small text in various scenarios. In this paper, we establish a video text reading benchmark, named DSText V2, which focuses on Dense and Small text reading challenges in the video with various scenarios. Compared with the previous datasets, the proposed dataset mainly include three new challenges: (1) Dense video texts, a new challenge for video text spotters to track and read. (2) High-proportioned small texts, coupled with the blurriness and distortion in the video, will bring further challenges. (3) Various new scenarios, e.g., ‘Game’, ‘Sports’, etc. The proposed DSText V2 includes 140 video clips from 7 open scenarios, supporting three tasks, i.e., video text detection (Task 1), video text tracking (Task 2), and end-to-end video text spotting (Task 3).In this article, we describe detailed statistical information of the dataset, tasks, evaluation protocols, and the results summaries. Most importantly, a thorough investigation and analysis targeting three unique challenges derived from our dataset are provided, aiming to provide new insights. Moreover, we hope the benchmark will promise video text research in the community. DSText v2 is built upon DSText v1, which was previously introduced to organize the ICDAR 2023 competition for dense and small video text. The dataset website can be found at RRC or Zenodo.

Building robust recognizers for Arabic has always been challenging. We demonstrate the effectiveness of an end-to-end trainable CNN-RNN hybrid architecture in recognizing Arabic text in videos and natural scenes. We outperform previous state-of-the-art on two publicly available video text datasets - ALIF and ACTIV. For the scene text recognition task, we introduce a new Arabic scene text dataset and establish baseline results. For scripts like Arabic, a major challenge in developing robust recognizers is the lack of large quantity of annotated data. We overcome this by synthesising millions of Arabic text images from a large vocabulary of Arabic words and phrases. Our implementation is built on top of the model introduced here [37] which is proven quite effective for English scene text recognition. The model follows a segmentation-free, sequence to sequence transcription approach. The network transcribes a sequence of convolutional features from the input image to a sequence of target labels. This does away with the need for segmenting input image into constituent characters/glyphs, which is often difficult for Arabic script. Further, the ability of RNNs to model contextual dependencies yields superior recognition results.

OCR2SEQ represents an innovative advancement in Optical Character Recognition (OCR) technology, leveraging a multi-modal generative augmentation strategy to overcome traditional limitations in OCR systems. This paper introduces OCR2SEQ’s unique approach, tailored to enhance data quality for sequence-to-sequence models, especially in scenarios characterized by sparse character sets and specialized vocabularies. At the heart of OCR2SEQ lies a set of novel augmentation techniques designed to simulate realistic text extraction errors. These techniques are adept at generating diverse and challenging data scenarios, thereby substantially improving the training efficacy and accuracy of text-to-text transformers. The application of OCR2SEQ has shown notable improvements in data processing accuracy, particularly in sectors heavily dependent on OCR technologies such as healthcare and library sciences. This paper demonstrates the capability of OCR2SEQ to transform OCR systems by enriching them with augmented, domain-specific data, paving the way for more sophisticated and reliable machine learning interpretations. This advancement in OCR technology, as presented in the study, not only enhances the accuracy and reliability of data processing but also sets a new benchmark in the integration of augmented data for refining OCR capabilities.

In this paper, we investigate the issue of class imbalance in scene text detection. Class Balanced Cross Entropy (CBCE) loss is often adopted for addressing this imbalance problem. We find that CBCE excessively restrains the backward gradients of background. Negative samples own extremely small weights which are offered by CBCE during training of text detectors. These tiny weight values lead to insufficient learning of background. As a result, the CBCE-based text detection methods only can achieve sub-optimal performance.We propose a novel loss function, Strong-Background Restrained Cross Entropy (SBRCE), to deal with the disadvantage in CBCE. Specifically, SBRCE effectively down-weights the loss assigned to the strong background which means well-classified negative samples. Our SBRCE can make training focused on all positive samples and weak background(i.e., hard-classified negative samples). Moreover, it can prevent the enormous amount of strong background from overwhelming text detectors during training. Experimental results show that the proposed SBRCE can improve the performance of the efficient and accurate scene text detector (EAST) by F-score of 3.3% on ICDAR2015 dataset and 1.12% on MSRA-TD500 dataset, without sacrificing the training and testing speed of EAST.

Previous approaches for scene text detection have already achieved promising performances across various benchmarks. However, they usually fall short when dealing with challenging scenarios, even when equipped with deep neural network models, because the overall performance is determined by the interplay of multiple stages and components in the pipelines. In this work, we propose a simple yet powerful pipeline that yields fast and accurate text detection in natural scenes. The pipeline directly predicts words or text lines of arbitrary orientations and quadrilateral shapes in full images, eliminating unnecessary intermediate steps (e.g., candidate aggregation and word partitioning), with a single neural network. The simplicity of our pipeline allows concentrating efforts on designing loss functions and neural network architecture. Experiments on standard datasets including ICDAR 2015, COCO-Text and MSRA-TD500 demonstrate that the proposed algorithm significantly outperforms state-of-the-art methods in terms of both accuracy and efficiency. On the ICDAR 2015 dataset, the proposed algorithm achieves an F-score of 0.7820 at 13.2fps at 720p resolution.

License plate recognition is currently used in various situations, such as parking lot vehicle management and the tracking of wanted vehicles. These applications require recognition from all possible camera angles. Therefore, we propose a method for license plate recognition in images captured from various camera angles. First, the license plate area is detected from the input image using the YOLOv5 object detection method. The subsequent process can be performed in two manners: by rotating the license plate to the front using projective transformation and trapezoidal correction, or by using graph matching. The first method performs Hough transforms or linear approximations as preprocessing, and then calculates the rectangle surrounding the license plate from the YOLOv5 output. After performing trapezoidal correction, the maximally stable extremal regions (MSER) are used to detect character candidates, and characters are recognized using 3D rotated character recognition. The second method performs character candidate detection and recognition without trapezoidal correction, followed by graph matching. These methods are versatile because they do not depend on the layout of license plates, which varies among countries and regions. Two types of datasets were used: a set of images containing Japanese license plates collected by ourselves, and a set of publicly available images containing Taiwanese license plates. Two recognition rates were evaluated: the license plate character recognition rate and license plate recognition rate in which all characters in one plate are recognized. The license plate character recognition rates using graph matching were 91.9% for the Japanese license plates and 89.3% for the Taiwanese license plates. The license plate recognition rates were 84.0% and 84.5%, respectively.

Efficiency of container terminals is important for the modern global economy, with the need for automated systems to facilitate cargo handling and logistics management. Optical Character Recognition (OCR) plays a key role in identifying and tracking containers, reducing the possibility of human error and streamlining operations. But OCR technology is plagued with persistent issues, such as environmental factors (poor lighting, bad weather effects, container shape…), text recognition challenges (skewed text, motion blur, segmentation failure), and hardware reliability concerns (camera malfunction, network instability, and delays in real-time processing). These constraints make the errors of OCR vary between 10 and 30%, affecting the efficiency of the terminals and necessitating higher manual interventions. To overcome such challenges, this study explores the application of Genetic Algorithms (GA) as an optimization technique to enhance OCR performance in container terminals. GA-based optimization allows for adaptive learning, parameterization, and real-time adaptation, significantly improving OCR accuracy, reducing error rates, and optimizing system uptime. A mathematical framework comprising OCR system parameters (hardware, software, environment, and operational constraints) was developed to optimize uptime, accuracy, and cost-effectiveness. The research incorporates a hybrid GA-Neural Network (NN) model further, using machine learning to speed up fitness evaluation and provide optimal settings for OCR performance improvement that is applicable in real world. Clinical trial number: not applicable.

Optical character recognition (OCR) technologies have known an effervescent development in last decade. Development was strongly influenced by the development of hardware, advance in image processing, and classification algorithms. There are multiple OCR technologies available, each of them based on different approaches, e.g., geometric processing or cognitive learning based on neural networks. One critical parameter for each of those approaches is execution time. In our opinion a very important percent of text used to be “OCRed” is coming from preprinted documents and forms, which are bounded by various regulations in layout and/or contained information. Based on this observation we argue that most of the characters that must be recognized have a similar layout, thus improvement of the processing performance can be obtained by creating classes of similar characters (blobs) based on geometric similarities, and performing OCR only on the representative blob from each class. In this paper we will present the architecture of an OCR technology based on a multilayer neural network. Performance improvement has been obtained using a blob classifier that groups characters in classes, and then perform OCR only on the representative blob from each class.

A means to naturally recognizing and fetching out the content of video description would possibly make them indexed in considerable and appropriate way for later reference, and would facilitate actions viz. automatic notification and dissemination, to be triggered in real time by the contents of streaming video. Video text recognition, or video OCR, is a constructive tool to characterize the contents of video containing overlay text (text captions superimposed over the video imagery, such as in broadcast news programs) and scene text (text that appears in the real scene of the video, such as text on street signs, nameplates, and billboards). In this paper exhaustive survey is done for text detecting, extraction and recognizing in complex images and video frames. Digital Videos are widely used both professionally and domestically because of the easy availability of camcorders to mobile phones. People are increasingly making videos may be for commercial use or personal use, this is leading to growing content of Video. While we can capture, compress, store, transmit and display video with great facility, editing videos and manipulating them based on their content is still a non-trivial activity. This paper concentrates on extracting text out of the video frames, taken out of the video.

We present a two stage framework for automatic video text removal to detect and remove embedded video texts and fill-in their remaining regions by appropriate data. In the video text detection stage, text locations in each frame are found via an unsupervised clustering performed on the connected components produced by the stroke width transform (SWT). Since SWT needs an accurate edge map, we develop a novel edge detector which benefits from the geometric features revealed by the bandlet transform. Next, the motion patterns of the text objects of each frame are analyzed to localize video texts. The detected video text regions are removed, then the video is restored by an inpainting scheme. The proposed video inpainting approach applies spatio-temporal geometric flows extracted by bandlets to reconstruct the missing data. A 3D volume regularization algorithm, which takes advantage of bandlet bases in exploiting the anisotropic regularities, is introduced to carry out the inpainting task. The method does not need extra processes to satisfy visual consistency. The experimental results demonstrate the effectiveness of both our proposed video text detection approach and the video completion technique, and consequently the entire automatic video text removal and restoration process.

In video indexing and summarization, video text is the very compact and accurate information. Most video text detection and extraction methods deal with the static videotext on video frames. Few methods can handle motion videotext well since motion videotext may be hardly be extracted well. In this paper, we propose a low computation method to detect and localize the scrolling videotexts to provide this information. Besides, we propose an extraction method to extract the videotext. The extraction method consists of adaptive thresholding and our proposed modified seed-fill algorithm. The detection method is carried out by edge detection, and the projection profile method is used to localize the text region well. We have completed the whole system and the experimental results all of the horizontal and vertical scrolling videotexts can be extracted well.