On benchmarking of document analysis systems

A framework for evaluating the performance of a document analysis system is presented. This framework takes into account the task definition for the document analysis system, a data base on which that system is evaluated, the metrics used to evaluate performance, and the generalization of the results achieved beyond the confines of the test. Several recent significant efforts in evaluating document analysis systems are surveyed. How these efforts fit the general framework is discussed. The specific task that was evaluated, the data base used for the evaluation, and the generalization of the derived performance is presented. Most of these projects were designed for limited applications in which the translation of images of text into ASCII was the primary consideration. However, this is only part of what a document analysis system must often calculate. Other, less easily measured tasks, such as the subdivision of a document image into zones that represent regions of graphics, photographs, and text, must also be performed. Generally accepted solutions for measuring the performance of such tasks often do not exist. Several of them are mentioned as areas for future research. © 1996 John Wiley & Sons, Inc.

It is the pleasure of the organizing committee to welcome all participants to the 2010 IAPR Workshop on Document Analysis Systems (DAS). This year's workshop is being held June 9-11th in Boston, Massachusetts, a location situated in the heart of beautiful New England in the northeastern United States. Boston has a rich history that dates back to the 1600's and was a major focal point in the history of the American Revolution and America's fight for independence. Over the years, Boston has developed into a center for industrial, academic and cultural excellence and draws millions of visitors each year. DAS 2010 is the ninth workshop in a series. The first DAS was held in Kaiserslautern, Germany in 1994, and was followed by Malvern, PA (1996); Nagano, Japan (1998); Rio de Janeiro, Brazil (2000); Princeton, NJ (2002); Florence, Italy (2004); Nelson, New Zealand (2006) and Nara, Japan (2008). The DAS tradition is to bring together industry, academic and government researchers interested in many aspects of document analysis systems and to provide opportunities for fruitful interaction and collaboration. This year's workshop is organized as a three-day, single track event, with oral and poster presentations, as well as working group discussions on the second and third afternoons. Special sessions on contributed datasets and a keynote talk on this same topic provide a compelling theme, and we hope this focus will help push the field toward greater sharing of data and accepted standards for evaluation. This year, we received 91 submissions from 25 countries on six continents. The papers were reviewed by 45 members of our research community and an international program committee representing 16 different countries. The overall quality was excellent and we have chosen 28 full papers for oral presentation and 37 as poster papers, as well as 15 short papers that will be presented either as posters or in a special short oral format. In addition, six groups are scheduled to present live demos during the poster sessions. Full papers underwent the standard peer review process and will appear in the official workshop proceedings to be published in the ACM International Conference Proceedings Series, available online as part of the ACM Digital Library. The short papers are included in the unofficial hardcopy proceedings distributed at the event as well as on the DAS 2010 website.

This paper presents an offline ground truth generation method using LIDAR(Light Detection and Ranging) scans and odometry. The generated ground truth allows quantitative evaluation of the performance of self-localization methods in urban areas where GNSS(Global Navigation Satellite System) cannot be trusted. The proposed method determines the vehicle pose (position and orientation) by aligning the LIDAR input with previously collected point cloud data. However, as alignment convergences are affected by the environment around the vehicle during each LIDAR scan, it can be erroneous. Incorrect estimates are removed and poses are interpolated by relying on odometry; which is locally accurate. A step by step optimization approach is adopted to yield the most accurate result. Experiments performed in a typical urban environment, with many buildings and surrounding obstacles, demonstrated the effectiveness of the proposed method.

The extraction of graph structures in Euclidean vector space is a topic of interest with applications in many fields, such as the analysis of vascular networks in the biomedical domain. While a number of approaches have been proposed to tackle the problem of graph extraction, a quantitative evaluation of those algorithms remains a challenging task: In many cases, manual generation of ground truth for real-world data is time-consuming, error-prone, and thus not feasible. While tools for generating synthetic datasets with corresponding ground truth exist, the resulting data often does not reflect the complexity that real-world scenarios show in morphology and topology. As a complementary or even alternative approach, we propose GERoMe, the graph extraction robustness measure, which provides a means of quantifying the stability of algorithms that extract (multi-)graphs with associated node positions from non-graph structures. Our method takes edge-associated properties into consideration and does not necessarily require ground truth data, although available ground truth information can be incorporated to additionally evaluate the correctness of the graph extraction algorithm. We evaluate the behavior of the proposed graph similarity measure and demonstrate the usefulness and applicability of our method in an exemplary study on both synthetic and real-world data.

A document analysis system which is capable of extracting the semantics of specific text portions of structured documents is presented. The architecture of the analysis system is based on a knowledge representation scheme, a semantic network, called Resco-Frame Representation of Structured Documents. It allows the definition of knowledge about document components as well as knowledge about analysis algorithms in a uniform, simple, but powerful representation formalism. Hence, this architecture enables the analysis system to exploit the specific power of both the algorithmic knowledge describing the properties of algorithms and the declarative knowledge about properties of text objects in documents. The inference engine and the control algorithm show how these two knowledge sources are combined and utilized. The flexibility of the representation formalism Fresco, the recognition results and the computational complexity of the inference algorithm are presented in two different applications.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Skew angle estimation is an important component of an optical character recognition (OCR) and document analysis systems (DAS). In this paper, a novel and efficient (in terms of accuracy and computations) method to estimate skew angle of a scanned document image is proposed. The proposed technique works based on fixing a boundary for connected components and growing boundaries. The technique uses Hough transform (HT) to estimate skew angles. However, the technique works based on the assumption that the space between the two adjacent text lines is greater than the space between the successive characters present in a text line. The proposed method is compared with other existing methods. The experimental results are also presented.

Document image binarization is a central problem in many document analysis systems. Indeed, it represents one of the basic challenges, especially in case of historical documents analysis. In this paper, we propose a novel robust multi stage framework that combines different existing document image thresholding methods for the purpose of getting a better binarization result. CLAHE technique is introduced to significantly enhance contrast in some poor images. The proposed method then uses a hybrid algorithm to partition image into foreground and background. A special procedure is finally applied in order to remove small noise and correct characters morphology. Experimental results prove the accuracy and the efficiency of our approach on document images binarization over three popular datasets compared to some well-known methods in literature.

Document binarization is one of the initial and critical steps for many document analysis systems. Nowadays, with the success and popularity of hand-held devices, large efforts are motivated to convert documents into digital format by using hand-held cameras. In this paper, we propose a Bayesian based maximum a posteriori (MAP) estimation algorithm to binarize the camera-captured document images. A novel adaptive segmentation surface estimation and normalization method is proposed as the preprocessing step in our work and followed by a Markov Random Field based refine procedure to remove noises and smooth binarized result. Experimental results show that our method has better performance than other algorithms on bad or uneven illumination document images.

A document analysis system which understands the incoming fax cover pages can serve as a mail routing system. This paper reports our research as part of an effort to realize such a system and focuses on locating the recipient's name and address on fax cover sheets. The name and address regions will later be extracted from the image and sent to an address interpretation unit for further processing. Fax cover pages are one type of documents which have changeable layout, contain many objects of different nature, and possess compact contents. Previous page segmentation techniques which mainly used geometric features are found not suitable for the current task. Our method uses a scheme which strongly couples text recognition and block segmentation. The system attempts to locate and recognize words which are data field indicators so as to figure out the positions of such keywords as TO and COMPANY. The information regarding the positions of TO and COMPANY indicators is used as reference in block segmentation to segment out their associated data regions. Our system has proved to work reasonably well on a sample set composed of 12 machined-printed images and 115 handwritten images.

We propose to adapt Turing's seminal 1950 test for machine intelligence to evaluating progress in document analysis systems. Our premise is that a problem can be considered solved if automated and human solutions to the underlying task are indistinguishable to a skeptical human judge. For the domain-specific problems of concern here, we reformulate the test to keep the interaction between judges and human/machine participants to graphical user interfaces that do not require natural language processing, a notable difference from Turing's original formulation. Examples of tasks that may lend themselves to such tests include detecting or identifying specific document components such as logos, photographs, tables, as well as writer and language identification. The administration of the test would be facilitated by commercial crowd-sourcing systems such as Amazon Mechanical Turk, as well as research platforms such as the Lehigh Document Analysis Engine (DAE) that accept arbitrary documents for input, record test results, and provide for trusted execution of submitted programs.

Document deskewing is a crucial pre-processing step in any document analysis system. In mass digitization projects, the ability to automatically assess the success of this step enables significant reductions in the amount of work performed by human operators. The current paper extends our generic skew and orientation detection algorithm with the ability to return well-founded confidence values for the detected rotation angles, which in turn may lie anywhere within the range of -180 to 180 degrees. Starting with an in-depth theoretical analysis of all common situations occurring in digitized document images, including large non-text regions as well as noise patterns, we derive a formula for the expected distribution of the global skew angle. Using the results of the analysis, we propose two generic confidence measures which are able to accurately reflect and cover all the aforementioned situations. Finally, the introduced confidence measures are tested on the UW-I data set comprising 979 heterogeneous document scans.

&lt;p&gt;Existing document analysis systems list words in the document using a morpheme analyzer. Such a structural feature is difficult to help users to understand the document. To understand a document, you need to analyze the keyword in the document and extract the paragraphs including the keyword. The proposed system retrieves keywords from documents written in XML format, extracts them, and displays them to the user. In addition, it extracts the paragraphs including the keyword entered by the user and maintains paragraph sequence and delete for duplicate paragraphs. Then, the frequency and weight of the keyword are calculated, and the number of paragraphs is reduced by removing the paragraphs including the keyword having a weight less than other keywords weighed. This method may reduce the time and effort required for the user to understand the document as compared to the existing document analysis systems.&lt;/p&gt;

Object-Process Methodology (OPM) calls for integrated system development, supported by Object-Process CASE Tool - OPCAT - a Computer Aided Software Engineering tool, which supports development and standard representation of industrial systems. OPCAT is designed to provide an Integrated System Engineering Environment (ISEE). Due to their generic nature and comprehensiveness, OPM and OPCAT are suitable for both developing Document Analysis Systems (DAS's) and representing them in a clear and standard way. We present the principles of OPM and show how it can be used as an ISEE for generic DAS development.

The binarization of image is an important stage in any document analysis system such as OCR. It converts the colored or grayscale images into monochromatic form to reduce the computational complexity in the next stages. In old document images in the presence of degradations (ink bleed, stains, smear, non-uniform illumination, low contrast, etc.) the separation of foreground and background becomes a challenging task. Most of the existing binarization techniques can handle only a subset of these degradations. We present a simple binarization method for old document images. The experimental results confirm that the proposed technique gives good binarization results in the presence of various degradations. It computes the Laplacian of an image to separate the foreground. The subtracted Laplacian image is binarized using a global threshold. Finally, the postprocessing using morphological functions is applied. The results are compared in terms of F-measure, PSNR, time complexity, and OCR based evaluations which shows that our method outperforms existing techniques like Niblack, Sauvola, Gatos, Zhou, NICK, Singh, and Bataineh.

The estimation of traffic ground truth has traditionally been accomplished with a trusted reference detector or by human observation, either live or in time-coded video recordings. These approaches are limited by the error characteristics of any single trusted detector, observation error, or the temporal and spatial resolution of the recorded video used by the human observer. For conventional traffic management purposes, the output of a trusted detector is usually adequate. However, for unbiased performance assessment and comparative ranking of vehicle detectors, an accurate ground truth estimate is essential. With this objective, an automated detector testing system was implemented. This system used consensus filtering methods typically employed in the fusion of data from multisensor networks to optimally estimate ground truth. The algorithm continuously adjusted a level of confidence in each detector under test that was used to form a weighted consensus decision for the presence, speed, and length of each vehicle. Individual detectors under test were then assessed by comparison with the estimated ground truth record and by the confidence factors generated by the algorithm. This paper describes the algorithm and presents the testing methodology to determine its ability to estimate ground truth accurately with the use of synthetic traffic data for which absolute ground truth is known, as well as synthetic detector data derived from the ground truth with known injected errors. The application of this algorithm in the advanced traffic management systems detector testbed of the California Department of Transportation is described.