Text recognition in natural scenes based on deep learning

To improve text detection and text recognition in natural scenes, it is important to improve legibility by removing the effects of illumination on the image. Therefore, in this study, we present a signal enhancement method for the dark regions in the image. In addition, we propose a procedure to reduce computational complexity of the bilateral filter that is used to estimate the illumination in our method. Since our main aim is to provide the preprocessing of text detection and recognition in natural scenes, we have proposed a simple processing in which only the signals of the dark regions in the image are enhanced. Experimental results show that our method is effective in conserving the naturalness of the image and improving the legibility of texts in natural scenes. Furthermore, by using our proposed procedure, bilateral filter computation loads can be reduced by about 30%.

Scene text detection and recognition are vital challenging tasks in computer vision, which are to detect and recognize sequences of texts in natural scenes. Recently, researchers have investigated a lot of state-of-the-art methods to improve the accuracy and efficiency of text detection and recognition. However, there has been little research on text detection and recognition in natural scenes in Vietnam. In this paper, a deep learning-based empirical investigation of Vietnamese scene text detection and recognition is presented. Firstly, four detection models including differentiable binarization network (DBN), pyramid mask text detector (PMTD), pixel aggregation network (PAN), and Fourier contour embedding network (FCEN), are employed to detect text regions from the images. Then, four text recognition models including convolutional recurrent neural network (CRNN), self-attention text recognition network (SATRN), no-recurrence sequence-to-sequence text recognizer (NRTR), and RobustScanner (RS) are also investigated to recognize the texts. Moreover, data augmentation methods are also applied to enrich data for improving the accuracy and enhancing the performance of scene text detection and recognition. To validate the effectiveness of scene text detection and recognition models, the VinText dataset is employed for evaluation. Empirical results show that PMTD and SATRN achieve the highest scores among the others for text detection and recognition, respectively. For knowledge-sharing, our implementation is publicly available at https://github.com/ThorPham/VN_scene_text_detection_recognition.

Deep learning is one machine learning method based on the layers used in artificial neural networks. The breakthrough of deep learning in classification led to its rapid application in speech recognition, natural language understanding, and image processing and recognition. In the field of biodiversity informatics, deep learning efforts are being applied in rapid species identification and counts of individuals identified based on image, audio, video, and other data types. However, deep learning methods hold great potential for application in all aspects of biodiversity informatics. We present a case study illustrating how to increase data collection quality and efficiency using well-established technology such as optical character recognition (OCR) and some image classification. Our goal is to image data from the scanned documents of various butterfly atlases, add species, specimens, collections, photographs and other relevant information, and build a database of butterfly images. Information collection involves image annotation and text-based descritpion input. Although the work of image annotation is simple, this process can be accelerated by deep learning-based target segmentation to make the selection process easier, such as changing box select to a double click. The process of information collection is complicated, involving input of species names, specimen collection, specimen description, and other information. Generally, there are many images in atlases, the text layout is rather messy, and overall OCR success is poor. Therefore, the measures we take are as follows: Step A: select the screenshot of the text and then call the OCR interface to generate the text material; Step B: proceed with NLP- (natural language processing) related processing; Step C: perform manual operations on the results, and introduce the NLP function again to this process; Step D: submit the result. The deep learning applications we integrated in our client tool include: target segmentation of the annotated image for automatic positioning and background removal, etc. to improve the quality of the image used for identification; making a preliminary judgment on various attributes of the labeled image and using the results to assist the automatic filling of relevant information in step B, including species information, specimen attributes (specimen image, nature photo, hand drawing pictures, etc.), insect stage (egg, adult, etc.); OCR in step A. target segmentation of the annotated image for automatic positioning and background removal, etc. to improve the quality of the image used for identification; making a preliminary judgment on various attributes of the labeled image and using the results to assist the automatic filling of relevant information in step B, including species information, specimen attributes (specimen image, nature photo, hand drawing pictures, etc.), insect stage (egg, adult, etc.); OCR in step A. Some simple machine learning methods such as k-nearest neighbor can be used to automatically determine gender, pose, and so on. While complex information such as collection place, time, and collector can be analyzed by deep learning-based NLP methods in the future. In our infomation collection process, ten fields are required to submit one single record. Of those, about 4-5 input fields can be dealt with the AI-assistant. It can thus be seen from the above process that deep learning has reduced the workload of manual information annotation by at least 30%. With improvements in accuracy, the era of using automatic information extraction robots to replace manual information annotation and collection is just around the corner.

Text recognition in the wild is a challenging task in the field of computer vision and machine learning. Existing optical character recognition engines cannot perform well in the natural scene. In this context, deep learning models have emerged as a powerful state-of-the-art technique in the classification and recognition process. This study proposes a new Convolutional Neural Network based system for scene text reading. We investigate how to combine the character recognition module followed by the word recognition module to achieve the overall system goal. The first module analyzes characters within multi-scale images by relaying on the power of the convolutional network and the fully connected network for character recognition. The second module relies on the Viterbi search to find the closest word to a given characters sequence. For the sake of more precision, a bigram based linguistic module is applied. The proposed system achieves the state-of-the-art performance on three standard scene text recognition benchmarks: chars74k, ICDAR 2003 and ICDAR 2013. In particular, this performance is proven on both of character and word recognition accuracy as well as speed aspects via a comparative study between different deep learning architectures.

With the evolving technological era, the optical character recognition systems have substantial execution, considering the widespread use of daily hand-written human transaction. Optical Character Recognition (OCR) is an implementation of Computer Vision that digitizes numerous hand dealt documents for further analysis and formatting. OCR is achieved by various ways such as discriminative analysis and deep learning. This paper focuses on evaluating deep learning models on a hand-written compiled dataset of Urdu digits. The evaluation is performed for deep convolutional learning algorithms; VGGNet16, InceptionV3, ResNet50, and DenseNet121. The convolutional models are pre-trained on the ImageNet. The model weights of fully connected layers have been evaluated, reducing the training time of the convolutional layers. The testing accuracy for the compiled dataset is observed as, ResNet50 with 96%, InceptionV3 with 95%, VGGNet16 with 95% and DenseNet121 with 94%.

License plate recognition is currently used in various situations, such as parking lot vehicle management and the tracking of wanted vehicles. These applications require recognition from all possible camera angles. Therefore, we propose a method for license plate recognition in images captured from various camera angles. First, the license plate area is detected from the input image using the YOLOv5 object detection method. The subsequent process can be performed in two manners: by rotating the license plate to the front using projective transformation and trapezoidal correction, or by using graph matching. The first method performs Hough transforms or linear approximations as preprocessing, and then calculates the rectangle surrounding the license plate from the YOLOv5 output. After performing trapezoidal correction, the maximally stable extremal regions (MSER) are used to detect character candidates, and characters are recognized using 3D rotated character recognition. The second method performs character candidate detection and recognition without trapezoidal correction, followed by graph matching. These methods are versatile because they do not depend on the layout of license plates, which varies among countries and regions. Two types of datasets were used: a set of images containing Japanese license plates collected by ourselves, and a set of publicly available images containing Taiwanese license plates. Two recognition rates were evaluated: the license plate character recognition rate and license plate recognition rate in which all characters in one plate are recognized. The license plate character recognition rates using graph matching were 91.9% for the Japanese license plates and 89.3% for the Taiwanese license plates. The license plate recognition rates were 84.0% and 84.5%, respectively.

At present, the accuracy of text detection from natural scene images based on deep learning is much higher than that of traditional text detection algorithm. However, most of the text detectors based on deep learning should use large-scale neural network models (such as VGG and ResNet) for improving the accuracy. Because the weight files of these models are very large (VGG is about 500MB and ResNet is about 100MB), they are not suitable for the computationally limited platforms in daily life. Based on the small-scale neural network model named MobileNet V2, and U-Net that is used in semantic segmentation commonly, a text detector for natural scene by using neural network with low complexity is proposed in this paper. The weight file of the designed neural network model is about 16MB, which is suitable for running on mobile platforms. The experiment result based on the ICDAR 2013 dataset proves that this model has good performance in text detection.

The application area of unmanned aerial vehicles increases significantly recent years due to progress in hardware and algorithms for data acquisition and processing. Object detection and classification (recognition) in imagery acquired by unmanned aerial vehicle are the key tasks for many applications, and usually in practice an operator solves these tasks. Growing amount of data of different types and of different nature provides the possibility for deep machine learning which nowadays shows high level results for object detection and recognition. Two key problems are to be solved for applying deep learning for object recognition task when dealing with multi-spectral imagery: (a) availability of representative dataset for neural network training and testing and (b) effective way of multi-spectral data fusion during neural network training. The paper proposes the approaches for solving these problems. For creating a representative dataset synthetic infra-red images are generated using several real infra-red images and 3D model of a given object. An technique for realistic infra-red texturing based on accurate infra-red image exterior orientation and 3D model pose estimation is developed. It allows in automated mode to produce datasets of required volume for deep learning and automatically generate ground truth data for neural network training and testing. Two approaches for multi-spectral data fusion for object recognition are developed and evaluated: data level fusion and results level fusion. The results of the evaluation of both techniques on generated multi-spectral dataset are presented and discussed.

With thousands of books across hundreds of disciplines present in a library, manual allocation of books is both time consuming, human intensive, as well as inefficient in terms of monetary costs. The presence of manual labour while providing employment to a section of society tends to slow down the process of lending in addition to increasing the probability of errors. The proposed research work attempts at automating the library book management system there by reducing latency and long queues as well as the potential of making mistakes in the distribution of books. Recent advancements of deep neural networks have improved textual recognition in natural scenes. The system proposed employs a neural network model for text detection and recognition, having the potential to reduce and even eliminate manual labour performed by the staff. This paper attempts to investigate neural networks for the application of library management by proposing our own Optical Character Recognition (OCR) and text matching algorithms. This task further introduces challenges such as getting the exact name of the book as well as dealing with distorted images and varied backgrounds. The system thus developed performs very well with book covers producing results over 60% in accuracy against current models that provide about 40% thereby making the system achieve state of the art performance on book cover information retrieval.

The characters in the text image or video is having more information for recovery and indexing applications. The recognition of characters from the video text images is a difficult task due to the complex backgrounds, grey-scale values, illuminations, and sizes. Hence, various text detection algorithms like feature extracting algorithms including texture and region-based, machine learning, and deep learning approaches are implemented. Optical Character Recognition (OCR) is implemented in detecting the characters from the text frames and gives the text information including the location of the frame for text in the video. The Natural Language Processing (NLP) technique results in high performance in detecting the different text languages from the video frames. Convolutional Neural Network (CNN) algorithms are used in the feature selection process for efficient recognition of text characters from the images. The deep learning algorithms are used in textual regions in the frames of the video for detecting the text. The LSTM combined CNN networks are also implemented for extracting the features from the image for efficient and high-quality output text.

Optical character recognition (OCR) is an important branch in the field of machine vision. It involves pattern recognition, image processing, digital signal processing, artificial intelligence and other disciplines. It is a comprehensive. It has important use value and theoretical significance in high-tech fields such as word information processing, office automation, machine translation and real-time monitoring system. In the 21st century, with the popularity of smart phones with high-definition cameras, OCR has a new pursuit in its development: more and more people pick up their mobile phones to photograph the things and scenes they see and obtain the text information in the pictures. Therefore, the recognition of characters in natural scenes has become a brand-new topic. In the past, text detection and text recognition algorithms were basically based on artificially designed features and traditional image processing methods. These features and algorithms were difficult to design and needed a lot of professional knowledge and experience support, so the accuracy was not high and they were not generalized. In recent years, with the rapid development of deep learning technology, breakthroughs have been made in the fields of computer vision such as image classification, object detection and semantic segmentation. Deep learning algorithm is a data-driven algorithm. The algorithm based on deep learning can automatically discover and learn the hidden feature rules in a large number of data through iterative training, without too much human intervention, so it has better generalization than traditional image processing related algorithms.

Optical Character Recognition (OCR) for Uighur language is a difficult problem and open research field because of the cursive nature script of the Uighur text, even in the printed text. Except for a few differences and modifications, Uighur characters have possessed the same characteristics of Arabic characters, so the obstacles and challenges of the Arabic OCR researches are also available for Uighur OCR researches. The central functional core of Uighur OCR, as well as Arabic OCR, is consisting of two operation stages: segmentation and classification. This paper proposed an obtaining segmentation point approach in the segmentation stage and applied a deep learning classifier with three-block characters as a recognized unit in the classification stage. The experiment results show that the word segmentation method has gained 95.68%, the segmentation point approach has made 94.74%, and the deep learning approach with three-block characters unit has reached 99.33%, while an identical approach with a single character unit has reached 91.98% within five epochs.

Scene texts refer to arbitrary text found in images captured by cameras in real-world settings. The tasks of text detection and recognition are critical components of computer vision, with applications spanning scene understanding, information retrieval, robotics, and autonomous driving. Despite significant advancements in deep learning methods, achieving accurate text detection and recognition in complex images remains a formidable challenge for robust real-world applications. Several factors contribute to these challenges. First, the diversity of text shapes, fonts, colors, and styles complicates detection efforts. Second, the myriad combinations of characters, often with unstable attributes, make complete detection difficult, especially when background interruptions obscure character strokes and shapes. Finally, effective coordination of multiple sub-tasks in end-to-end learning is essential for success. This research aimed to tackle these challenges by enhancing text discriminative representation. This study focused on two interconnected problems: Scene Text Recognition (STR), which involves recognizing text from scene images, and Scene Text Detection (STD), which entails simultaneously detecting and recognizing multiple texts within those images. This research focuses on implementing and evaluating the Efficient and Accurate Scene Text Detector (EAST) algorithm for text detection and recognition in natural scene images. The study aims to compare the performance of three prominent Optical Character Recognition (OCR) techniques—TesseractOCR, PaddleOCR, and EasyOCR. The EAST model was applied to a series of sample test images, and the results were visually represented with bounding boxes highlighting the detected text regions. The inference times for each image were recorded, highlighting the algorithm's efficiency, with average times of 0.446, 0.439, and 0.440 seconds for the respective test images. These results indicate that the EAST algorithm is accurate and operates in real-time, making it suitable for applications requiring immediate text recognition.

The advances in technologies had led to the accessibility to the entire world at hand through the Internet and mobile phones. While computers and cameras have come to pockets as smart phones, a lot of vision-related applications have become easier. What a human visualizes has been able to be visualized by machines as well through advancements in machine vision algorithms. This chapter gives a detailed study about one such computer vision application – character recognition – with the aid of deep learning techniques. In particular, this chapter is about the implementation of widely used convolutional neural networks in offline character recognition which is used in document analysis, document recognition, scene text classification, localization, and recognition.KeywordsCharacter recognitionDeep learningCNNTamil character recognition

Machine learning is currently a prominent issue due to the direct training of machines with minimal human input. In the present period, the scenario of manual machine feeding has changed, it now learns automatically. Machine learning approaches are utilized for a variety of tasks, such as feature extraction, pattern recognition, object recognition, and classification. In computer vision, machine learning is used to extract important information from visual data. Computer vision contributes to a variety of disciplines, including surveillance, robotics, suspect detection, optical character recognition, and numerous other technologies. Computer vision helps in extracting metadata from humans, such as emotions, sentiments, personality, and so on. This paper presents a thorough yet current survey of human profiling based on computer vision. However, we present a taxonomy of data profiling and computer vision as well as how to profile humans from image data. As an experience, we will discuss our program about human faces recognising emotions from a person’s image using deep learning. In any event, the outcomes we have achieved are critical and important.