Text Detection, Tracking and Recognition in Video: A Comprehensive Survey

Text detection and recognition in scene images and videos attract much attention in computer vision recently. However, most existing text detection and recognition methods only focus on static images. In this paper an end-to-end scene text recognition method based on multi frame tracking is proposed for text in videos, in which temporal information is employed to improve performance. First, an end-to-end text recognition method based on a unified deep neural network is used to detect and recognize text in each frame of the input video. Then, multi frame text tracking is employed through associations of texts in current frame and several previous frames to obtain final results. Experiments on ICDAR datasets demonstrate that the proposed method outperforms the state-of-the-art methods in end-to-end video text recognition.

A means to naturally recognizing and fetching out the content of video description would possibly make them indexed in considerable and appropriate way for later reference, and would facilitate actions viz. automatic notification and dissemination, to be triggered in real time by the contents of streaming video. Video text recognition, or video OCR, is a constructive tool to characterize the contents of video containing overlay text (text captions superimposed over the video imagery, such as in broadcast news programs) and scene text (text that appears in the real scene of the video, such as text on street signs, nameplates, and billboards). In this paper exhaustive survey is done for text detecting, extraction and recognizing in complex images and video frames. Digital Videos are widely used both professionally and domestically because of the easy availability of camcorders to mobile phones. People are increasingly making videos may be for commercial use or personal use, this is leading to growing content of Video. While we can capture, compress, store, transmit and display video with great facility, editing videos and manipulating them based on their content is still a non-trivial activity. This paper concentrates on extracting text out of the video frames, taken out of the video.

Video text is very important semantic information, which brings precise and meaningful clues for video indexing and retrieval. However, most previous approaches did video text extraction and recognition separately, while the main difficulty of extraction and recognition with complex background wasn’t handled very well. In this paper, these difficulty is investigated by combining text extraction and recognition together as well as using OCR feedback information. The following features are highlighted in our approach: (i) an efficient character image segmentation method is proposed in consideration of most prior knowledge. (ii) text extraction are implemented both on text-row and segmented single character images, since text-row based extraction maintains the color consistency of characters and backgrounds while single character has simpler background. After that, the best binary image is chosen for recognition with OCR feedback. (iii) The K-means algorithm is used for extraction which ensures that the best extraction result is involved, which is the binary image with clear classification of text strokes and background. Finally, extensive experiments and empirical evaluations on several video text images are conducted to demonstrate the satisfying performance of the proposed approach.

We propose an end-to-end system for text detection and recognition in natural scenes and consumer videos. Maximally Stable Extremal Regions which are robust to illumination and viewpoint variations are selected as text candidates. Rich shape descriptors such as Histogram of Oriented Gradients, Gabor filter, corners and geometrical features are used to represent the candidates and classified using a support vector machine. Positively labeled candidates serve as anchor regions for word formation. We then group candidate regions based on geometric and color properties to form word boundaries. To speed up the system for practical applications, we use Partial Least Squares approach for dimensionality reduction. The detected words are binarized, filtered and passed to a hidden Markov model based Optical Character Recognition (OCR) system for recognition. We show significant improvement in text detection and recognition tasks over previous approaches on a large consumer video dataset. Furthermore, the event detection system built upon the OCR output of this approach outperformed multiple other OCR-only based submissions in the recently concluded NIST TRECVID 2013 multimedia event detection evaluations.

In this paper, we propose a new system for text information extraction from news videos. First of all, a method that integrates text detecting and text tracking is developed to locate text areas in the key-frames (images), together with a scheme to evaluate the performance of this approach. To get better recognition results, we then enhance the quality of the detected text blocks by multi-frame averaging. Finally, we use an adaptive thresholding method to binarize the text blocks and recognize the text using an off-the-shelf OCR module. The detection and recognition rate of the proposed system are 94.7% and 67.5% respectively.

Text detection and extraction from scene images is useful in finding certain facts in supporting court of laws for solving crime problems and various forensic science applications. Text detection from scene images is a difficult and challenging problem due to various issues such as; varying font size, style, uneven illumination and other degradations. In this paper, a new method that uses wavelet energy features for text detection and extraction from scene images is presented. Initially, preprocessed binarized image will be divided into 50x50 blocks. Then, wavelet energy features are obtained from every block and potential text blocks are identified using newly defined discriminant functions. Further, the detected text blocks are merged and refined to extract text regions. The proposed method is robust and achieves a detection rate of 97% on a variety of scene images each of size 240x320. KeywordsText detection, scene images, wavelet energy features, discriminant functions. International Journal of Machine Intelligence ISSN: 0975-2927 & E-ISSN: 0975-9166, Volume 4, Issue 1, 2012 Citation: Angadi S.A., et al. (2012) Text detection and extraction from Scene Images using wavelet features. International Journal of Machine Intelligence, ISSN: 0975-2927 & E-ISSN: 0975-9166, Volume 4, Issue 1, pp.-409. Copyright: Copyright©2012 Angadi S.A., et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The lack of training datasets and computational complexity of 3 dimensions make text localization and recognition in point cloud environments challenging tasks, resulting in them being relatively undeveloped topics in the research community. In this paper, we introduce a method to adapt 2D text detection and recognition techniques in panoramic images with appropriate 3D mapping. This combined with heuristic methods such as a virtual camera creates a fast and efficient 3D text localization and recognition system. In real world applications, the objects of interest for a computer vision task may not be captured by the sensor from an ideal perspective; instead skewed imagery or partial occlusions are common. In a virtual 3D environment, we have full control of viewing angles and distances from the object. Therefore, by placing a virtual camera in certain positions, we can generate synthetic imagery of the object that is based on real imagery and avoids skewed views or occlusions. We use this synthetic imagery to improve the performance of text recognition when the object of interest is sufficiently close to the scanner, and hence the point density is high enough to generate quality imagery. The simplistic nature of this system is attractive and computationally inexpensive as it uses 2D data processes instead of natively 3D. The system shows promising results with over 85% accuracy for detection and localization tasks and 80% on the recognition task.

Scene texts refer to arbitrary text found in images captured by cameras in real-world settings. The tasks of text detection and recognition are critical components of computer vision, with applications spanning scene understanding, information retrieval, robotics, and autonomous driving. Despite significant advancements in deep learning methods, achieving accurate text detection and recognition in complex images remains a formidable challenge for robust real-world applications. Several factors contribute to these challenges. First, the diversity of text shapes, fonts, colors, and styles complicates detection efforts. Second, the myriad combinations of characters, often with unstable attributes, make complete detection difficult, especially when background interruptions obscure character strokes and shapes. Finally, effective coordination of multiple sub-tasks in end-to-end learning is essential for success. This research aimed to tackle these challenges by enhancing text discriminative representation. This study focused on two interconnected problems: Scene Text Recognition (STR), which involves recognizing text from scene images, and Scene Text Detection (STD), which entails simultaneously detecting and recognizing multiple texts within those images. This research focuses on implementing and evaluating the Efficient and Accurate Scene Text Detector (EAST) algorithm for text detection and recognition in natural scene images. The study aims to compare the performance of three prominent Optical Character Recognition (OCR) techniques—TesseractOCR, PaddleOCR, and EasyOCR. The EAST model was applied to a series of sample test images, and the results were visually represented with bounding boxes highlighting the detected text regions. The inference times for each image were recorded, highlighting the algorithm's efficiency, with average times of 0.446, 0.439, and 0.440 seconds for the respective test images. These results indicate that the EAST algorithm is accurate and operates in real-time, making it suitable for applications requiring immediate text recognition.

The objective of this research paper is to propose a novel technique for Arabic text extraction and recognition which is a part of research work aimed at developing a system for moving Arabic video text extraction for efficient content based indexing and searching. Numerous techniques were proposed in the past for text extraction but very few of them focus on Arabic text. All the earlier proposed implementations are not successful in attaining 100 % accuracy in text extraction and recognition process. The proposed technique is new and is based on thinning the given sample image containing Arabic text and splitting the resulting image horizontally (X-axis direction) from right to left in equal intervals. Compare each part of the image for equal number of white pixels to those of samples in the dataset. Upon matching, with the help of index value the corresponding character is stored in an array. This process is repeated by varying the splitting interval until all the characters in the sample image are recognized. To our knowledge, our research is the primary to address the above problem and propose a solution with increased retrieval accuracy and reduced computation time for Arabic text extraction and recognition.

Text extraction from scene images by character appearance and structure modeling

Text quality can significantly affect the results of text detection and recognition in digital video. In this paper we address the problem of estimating text quality. The quality of text that appears in video is often much lower than that in document images, and can be degraded by factors such as low resolution, background variation, uneven lighting, motion of the text and camera, and in the case of scene text, projection from 3D. Features based on text resolution, background noise, contrast, illumination and texture are selected to describe the text quality, normalized and fed into a trained RBF network to estimate the text quality. The performance using different training schemes are compared.

The successful analysis of video data is currently in great demand because a video is a major source of data in our lives. The

The proliferation of social media platforms has led to a significant increase in the sharing of memes and other image-based content. However, some of these images may contain inappropriate or offensive text, which can be challenging to detect using traditional methods. Because the open-source existing solutions does not classify the text content in images. This project aims to develop a robust solution for detecting Not Safe for Work (NSFW) text content in images, with a particular focus on social media memes. The objectives of this project are to accurately extract text from images using optical character recognition (OCR) techniques, and to classify the extracted text as either NSFW or Safe for Work (SFW) using a fine-tuned natural language processing (NLP) model. The methodology employed in this project involves several key steps. First, the Keras OCR library is utilized to extract text from input images, as it has demonstrated superior performance compared to other OCR tools for social media images. Next, a preprocessing step is performed to align the extracted text in the correct sequence using coordinates, distance from the image origin, and the Pythagorean theorem. The pre-processed text is then passed to a fine-tuned BERT-BASE-UNCASED language model, which has been trained on a dataset of Reddit posts (NSFW and SFW) to classify the text as either NSFW or SFW. Key Words: OCR, Inappropriate, BERT, NSFW, SFW, Image, Text

Urdu text detection and extraction from videos especially from news feed is a challenging task. The main challenge is to design a robust algorithm that can accurately detect and localize Urdu text regions in videos. Urdu language is originated from oriental languages like Arabic, Persian and Turkish, so alphabets of these languages are very similar. Due to the similarities of basic alphabets and writing styles, this research is equally applicable to Arabic or Persian text. The development of an OCR that gives acceptable results on Urdu is still an ongoing research area. Thus need of a good text detection and extraction system is needed that can reduce the recognition errors as much as possible. It is because if the input image contains only text candidate regions, then possibilities of errors will be reduced for the same OCR. The proposed system is evaluated on a set of videos of news channels that vary along the dimensions of intensity variations, scale and orientation. Accuracy in terms of precision and recall rates is analyzed to determine the success and limitations of the approach. Problems, possible errors and suggestions for improvements are also discussed in the paper.

In video indexing and summarization, video text is the very compact and accurate information. Most video text detection and extraction methods deal with the static videotext on video frames. Few methods can handle motion videotext well since motion videotext may be hardly be extracted well. In this paper, we propose a low computation method to detect and localize the scrolling videotexts to provide this information. Besides, we propose an extraction method to extract the videotext. The extraction method consists of adaptive thresholding and our proposed modified seed-fill algorithm. The detection method is carried out by edge detection, and the projection profile method is used to localize the text region well. We have completed the whole system and the experimental results all of the horizontal and vertical scrolling videotexts can be extracted well.