Data Extraction from Web Tables: The Devil is in the Details

Information extraction from semi-structured webpages provides valuable long-tailed facts for augmenting knowledge graph. Relational Web tables are a critical component containing additional entities and attributes of rich and diverse knowledge. However, extracting knowledge from relational tables is challenging because of sparse contextual information. Existing work linearize table cells and heavily rely on modifying deep language models such as BERT which only captures related cells information in the same table. In this work, we propose a novel relational table representation learning approach considering both the intra- and inter-table contextual information. On one hand, the proposed Table Convolutional Network model employs the attention mechanism to adaptively focus on the most informative intra-table cells of the same row or column; and, on the other hand, it aggregates inter-table contextual information from various types of implicit connections between cells across different tables. Specifically, we propose three novel aggregation modules for (i) cells of the same value, (ii) cells of the same schema position, and (iii) cells linked to the same page topic. We further devise a supervised multi-task training objective for jointly predicting column type and pairwise column relation, as well as a table cell recovery objective for pre-training. Experiments on real Web table datasets demonstrate our method can outperform competitive baselines by +4.8% of F1 for column type prediction and by +4.1% of F1 for pairwise column relation prediction.

Relational Web tables, embedded in HTML or published on data platforms, have become an important resource for many applications, including question answering or entity augmentation. To utilize the data, we require some understanding of what the tables are about. Previous research on recovering Web table semantics has largely focused on simple tables, which only describe a single semantic concept. However, there is also a significant number of de-normalized multi-concept tables on the Web. Treating these as single-concept tables results in many incorrect relations being extracted. In this paper, we propose a normalization approach to decompose multi-concept tables into smaller single-concept tables. First, we identify columns that represent keys or identifiers of entities. Then, we utilize the table schema as well as intrinsic data correlations to identify concept boundaries and split the tables accordingly. Experimental results on real Web tables show that our approach is feasible and effectively identifies semantic concepts.

Data model for warehousing historical Web information

With the evolution from a web of documents to a web of data, it is important we change the way we search for data among the huge amounts made available through social media, IoT devices, mobile devices, etc. A user should be able to express what he is searching for, in natural language or even through voice recognition, and be able to get the result he is seeking for, even if his search involves data of different nature and format: web pages, RSS feeds, RDF triples, web services calls, sensors data, etc. Another fact to consider is that Internet access is mainly done nowadays via mobile devices. In this paper, we propose a framework to answer those challenges by enabling the user to make a search from a Google-like simple interface, and thus create from his search keywords, an intelligent personalized mashup, without needing any programming skills, and yet integrating calls to REST web services, RSS feeds, RDF triples or any other data available on the web. In order to achieve this goal, we designed an ontology that can hold the description of any web resource, and implemented an automated service composition engine that takes advantage of this ontology. The engine has two main execution phases: The startup phase, implemented for performance improvement, which scans the semantically defined resources in order to create a HashMap linking each output to one or more chains of potential matching services. Matching services are services having an output linked by an inferred similarity to another service input. The processing phase, where a query analyzer inspects the ontology, and checks with Dbpedia and WordNet, in order to decompose the user request into input and output parameters. A type-based score model is used to decide which chain among the available chains to execute. The usefulness of our approach, was validated by developing a mobile client application which was used to test the implemented engine.

Data on the web page can be available in various formats, such as table. With the growing of web pages, the need to extract data from tables is increasing. Results of the extraction can be used for integration with other web tables or stored in a database. This study discusses the extraction of data from a table on a web page using a Document Object Model (DOM) tree. The initial step of this extraction process is to transform the HTML document into a DOM tree. Then, by applying search methods Depth First Search (DFS), part of the data in the table is extracted and stored in a CSV file. An engine has been developed using Visual Basic. The results show that the engine can automatically extract data from the table that has the following characteristics: the number of rows and columns are not limited, able to handle all of the table orientation layout, and able to handle tables that are merged cells.

With the growing size of academic library Web sites, constant updating, authentication issues, and organization are increasingly difficult for libraries to maintain user-friendly sites. This usability study examines how students use electronic research libraries such as Questia, which has been designed to replace traditional libraries and compare it with large university library Web sites. Students were asked to perform tasks at two electronic research library sites and then at two large university library Web sites. Major implications of this study are that design features incorporated by Web site designers can drastically affect the success of students doing research.

Knowledge bases, massive collections of facts (RDF triples) on diverse topics, support vital modern applications. However, existing knowledge bases contain very little data compared to the wealth of information on the Web. This is because the industry standard in knowledge base creation and augmentation suffers from a serious bottleneck: they rely on domain experts to identify appropriate web sources to extract data from. Efforts to fully automate knowledge extraction have failed to improve this standard: these automated systems are able to retrieve much more data and from a broader range of sources, but they suffer from very low precision and recall. As a result, these large-scale extractions remain unexploited. In this paper, we present MIDAS, a system that harnesses the results of automated knowledge extraction pipelines to repair the bottleneck in industrial knowledge creation and augmentation processes. MIDAS automates the suggestion of good-quality web sources and describes what to extract with respect to augmenting an existing knowledge base. We make three major contributions. First, we introduce a novel concept, web source slices, to describe the contents of a web source. Second, we define a profit function to quantify the value of a web source slice with respect to augmenting an existing knowledge base. Third, we develop effective and highly-scalable algorithms to derive high-profit web source slices. We demonstrate that MIDAS produces high-profit results and outperforms the baselines significantly on both real-world and synthetic datasets.

With the rapid development of technology, the Web has become the largest encyclopedic database. Although users can get information conveniently on the surface web by using applications such as browsers, it is hard to retrieve information in the deep web. Deep web requires a user submit a query to the server to get information from its database to generate the result webpage. Thus methods different from traditional Web surfing are needed to conduct the data extraction in deep web. Most of the existing deep web data extraction methods are based on DOM tree analysis. In this paper, to fully utilize the visual information contained in a webpage, a data region locating method based on convolutional neural network and a visual information based segmentation algorithm are proposed. In order to verify the efficiency of the proposed method, we apply it to real world commercial websites to perform data extraction. Experiments of data region location model, data extraction, and data item alignment verify that our proposed method can effectively improve the accuracy of data region location and the efficiency of data extraction.

Corpora that contain tabular data such as WebTables are a vital resource for the academic community. Essentially, they are the backbone of any modern research in information management. They are used for various tasks of data extraction, knowledge base construction, question answering, column semantic type detection and many other. Such corpora are useful not only as a source of data, but also as a base for building test datasets. So far, there were no such corpora for the Russian language and this seriously hindered research in the aforementioned areas. In this paper, we present the first corpus of Web tables created specifically out of Russian language material. It was built via a special toolkit we have developed to crawl the Russian Wikipedia. Both the corpus and the toolkit are open-source and publicly available. Finally, we present a short study that describes Russian Wikipedia tables and their statistics.

The lack of large amounts of readily available, explicitly represented knowledge has long been recognized as a barrier to applications requiring semantic knowledge such as machine translation and question answering. This problem is analogous to that facing machine translation decades ago, where one proposed solution was to use human translators to post-edit automatically produced, low quality translations rather than expect a computer to independently create high-quality translations. This paper describes an attempt at implementing a semantic parser that takes unrestricted English text, uses publically available computational linguistics tools and lexical resources and as output produces semantic triples which can be used in a variety of tasks such as generating knowledge bases, providing raw material for question answering systems, or creating RDF structures. We describe the TextCap system, detail the semantic triple representation it produces, illustrate step by step how TextCap processes a short text, and use its results on unseen texts to discuss the amount of post-editing that might be realistically required.

Quantum Computing has a potential, if realized, to significantly alter the computing landscape. However, developing efficient quantum algorithms and/or programming quantum computers is challenging. In this paper, we develop quantum algorithms for the database queries, in particular, the SQL/relational database operators: select, project, join, and set manipulation (intersect, union, and difference) operators. This also involves representation of relational database tables using quantum states and develop appropriate mechanism to extract output tuples from quantum states. Developing efficient quantum algorithms for general database queries has the potential to yield a declarative programming paradigm amenable to automatic optimizations.KeywordsQuantum computationDatabase operationsGrover’s algorithm

This study investigates the application of ChatGPT-4 in extracting and classifying behavioral data from scientific literature, focusing on the daily time-activity budget of dairy cows. Accurate analysis of time-activity budgets is crucial for understanding dairy cow welfare and productivity. Traditional methods are time-intensive and prone to bias. This study evaluates the accuracy and reliability of ChatGPT-4 in data extraction and data categorization, considering explicit, inferred, and ambiguous labels for the data, compared with human analysis. A collection of 55 papers on dairy cow behavior were used in the studies. Data extraction for eating, ruminating, and lying behaviors was performed manually and via ChatGPT-4. The artificial intelligence (AI) model's accuracy and labeling performance were assessed through descriptive and statistical analyses. Mixed model analysis was used to compare human and AI outcomes. Artificial intelligence and human time budget data showed significant differences for eating and ruminating but not for lying. ChatGPT-4 estimated daily eating time at 22.3% compared with 23.8% by humans. For ruminating, AI reported 33.4% against 31.7% by humans. Daily lying times were nearly identical, with AI at 44.4% and human analysis at 44.2%. The global accuracy in data extraction was ∼75%, and labeling accuracy reached 67.3%, with significant variability across behavioral categories. In general, the AI model demonstrates moderate accuracy in extracting and categorizing behavioral data, particularly for inferred and ambiguous data. However, explicit data extraction posed challenges, highlighting AI's dependence on input quality and structure. The consistency between AI and human analyses for lying behavior underscores AI's potential for specific applications. ChatGPT-4 offers a promising complementary tool for behavioral research, enabling efficient and scalable data extraction. However, improvements in AI algorithms and standardized reporting in scientific literature are essential for broader applicability. The study advocates for hybrid approaches combining AI capabilities with human oversight to enhance the reliability and accuracy of dairy cow behavioral research.

GPT-4 is a large language model with potential for multiple applications in urology. Our study sought to evaluate GPT-4's performance in data extraction from renal surgery operative notes. GPT-4 was queried to extract information on laterality, surgery, approach, estimated blood loss, and ischemia time from deidentified operative notes. Match rates were determined by the number of "matched" data points between GPT-4 and human-curated extraction. Accuracy rates were calculated after manually reviewing "not matched" data points. Cohen's kappa and the intraclass coefficient were used to evaluate interrater agreement/reliability. Our cohort consisted of 1498 renal surgeries from 2003 to 2023. Match rates were high for laterality (94.4%), surgery (92.5%), and approach (89.4%), but lower for estimated blood loss (77.1%) and ischemia time (25.6%). GPT-4 was more accurate for estimated blood loss (90.3% vs 85.5% human curated) and similarly accurate for laterality (95.2% vs 95.3% human curated). Human-curated accuracy rates were higher for surgery (99.3% vs 93% GPT-4), approach (97.9% vs 90.8% GPT-4), and ischemia time (95.6% vs 30.7% GPT-4). Cohen's kappa was 0.96 for laterality, 0.83 for approach, and 0.71 for surgery. The intraclass coefficient was 0.62 for estimated blood loss and 0.09 for ischemia time. Match and accuracy rates were higher for categorical variables. GPT-4 data extraction was particularly error prone for variables with heterogenous documentation styles. The role of a standard operative template to aid data extraction will be explored in the future. GPT-4 can be utilized as a helpful and efficient data extraction tool with manual feedback.

Traditional manual methods of extracting anesthetic and physiological data from the electronic health record rely upon visual transcription by a human analyst that can be labor-intensive and prone to error. Technical complexity, relative inexperience in computer coding, and decreased access to data warehouses can deter investigators from obtaining valuable electronic health record data for research studies, especially in under-resourced settings. We therefore aimed to develop, pilot, and demonstrate the effectiveness and utility of a pragmatic data extraction methodology. Expired sevoflurane concentration data from the electronic health record transcribed by eye was compared to an intermediate preprocessing method in which the entire anesthetic flowsheet narrative report was selected, copy-pasted, and processed using only Microsoft Word and Excel software to generate a comma-delimited (.csv) file. A step-by-step presentation of this method is presented. Concordance rates, Pearson correlation coefficients, and scatterplots with lines of best fit were used to compare the two methods of data extraction. A total of 1132 datapoints across eight subjects were analyzed, accounting for 18.9 h of anesthesia time. There was a high concordance rate of data extracted using the two methods (median concordance rate 100% range [96%, 100%]). The median time required to complete manual data extraction was significantly longer compared to the time required using the intermediate method (240 IQR [199, 482.5] seconds vs 92.5 IQR [69, 99] seconds, p = .01) and was linearly associated with the number of datapoints (rmanual = .97, p < .0001), whereas time required to complete data extraction using the intermediate approach was independent of the number of datapoints (rintermediate = -.02, p = .99). We describe a pragmatic data extraction methodology that does not require additional software or coding skills intended to enhance the ease, speed, and accuracy of data collection that could assist in clinician investigator-initiated research and quality/process improvement projects.

BackgroundDiverse types of biological data, primary as well as derived, are available in various formats and are stored in heterogeneous resources. Database-specific as well as integrated search engines are available for carrying out efficient searches of databases. These search engines however, do not support extraction of subsets of data with the same level of granularity that exists in typical database entries. In order to extract fine grained subsets of data, users are required to download complete or partial database entries and write scripts for parsing and extraction.ResultsBioDBExtractor (BDE) has been developed to provide 26 customized data extraction utilities for some of the commonly used databases such as ENA (EMBL-Bank), UniprotKB, PDB, and KEGG. BDE eliminates the need for downloading entries and writing scripts. BDE has a simple web interface that enables input of query in the form of accession numbers/ID codes, choice of utilities and selection of fields/subfields of data by the users.ConclusionsBDE thus provides a common data extraction platform for multiple databases and is useful to both, novice and expert users. BDE, however, is not a substitute to basic keyword-based database searches. Desired subsets of data, compiled using BDE can be subsequently used for downstream processing, analyses and knowledge discovery.AvailabilityBDE can be accessed from http://bioinfo.net.in/BioDB/Home.html.