• Home
  • Search
  • Geoscience Language Processing for Exploration
  • Cite Icon3
  • https://doi.org/10.2118/207766-msCopy DOI Icon

Geoscience Language Processing for Exploration

  • Dec 9, 2021
  • Huseyin Denli +4 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Abstract Deep learning has recently been providing step-change capabilities, particularly using transformer models, for natural language processing applications such as question answering, query-based summarization, and language translation for general-purpose context. We have developed a geoscience-specific language processing solution using such models to enable geoscientists to perform rapid, fully-quantitative and automated analysis of large corpuses of data and gain insights. One of the key transformer-based model is BERT (Bidirectional Encoder Representations from Transformers). It is trained with a large amount of general-purpose text (e.g., Common Crawl). Use of such a model for geoscience applications can face a number of challenges. One is due to the insignificant presence of geoscience-specific vocabulary in general-purpose context (e.g. daily language) and the other one is due to the geoscience jargon (domain-specific meaning of words). For example, salt is more likely to be associated with table salt within a daily language but it is used as a subsurface entity within geosciences. To elevate such challenges, we retrained a pre-trained BERT model with our 20M internal geoscientific records. We will refer the retrained model as GeoBERT. We fine-tuned the GeoBERT model for a number of tasks including geoscience question answering and query-based summarization. BERT models are very large in size. For example, BERT-Large has 340M trained parameters. Geoscience language processing with these models, including GeoBERT, could result in a substantial latency when all database is processed at every call of the model. To address this challenge, we developed a retriever-reader engine consisting of an embedding-based similarity search as a context retrieval step, which helps the solution to narrow the context for a given query before processing the context with GeoBERT. We built a solution integrating context-retrieval and GeoBERT models. Benchmarks show that it is effective to help geologists to identify answers and context for given questions. The prototype will also produce a summary to different granularity for a given set of documents. We have also demonstrated that domain-specific GeoBERT outperforms general-purpose BERT for geoscience applications.

Similar Papers
  • Conference Article
  • Citations3

Engineering Document Summarization Using Sentence Representations Generated by Bidirectional Language Model

  • Aug 17, 2021
  • Yunjian Qiu +1
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Research Article

Spam Detection Based on BERT Model: A Descriptive Study

  • Jan 01, 2023
  • Theory and practice of science and technology
  • Hailin Xu
  • Research Article
  • Citations20

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

  • May 05, 2024
  • Artificial Intelligence In Medicine
  • Shoya Wada +6
  • Research Article
  • Citations4

Artificial Intelligence for CT and MRI Protocoling: A Meta-Analysis of Traditional Machine Learning, BERT, and Large Language Models.

  • Oct 29, 2025
  • AJR. American journal of roentgenology
  • Ethan Sacoransky +2
  • Research Article

Cybershield Monitor Using Deep Learning and BERT Model

  • Feb 24, 2025
  • INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
  • A Nandhini
  • PDF
  • Research Article
  • Citations25

Identification of asthma control factor in clinical notes using a hybrid deep learning model

  • Nov 01, 2021
  • BMC Medical Informatics and Decision Making
  • Bhavani Singh Agnikula Kshatriya +6
  • Research Article
  • Citations1

Natural language processing models for predicting treatment outcomes in internet-delivered cognitive behavioral therapy

  • Oct 10, 2025
  • Internet Interventions
  • Nils Hentati Isacsson +4
  • Research Article
  • Citations9

Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation.

  • Oct 30, 2024
  • JMIR medical informatics
  • Kyungmo Kim +11
  • Research Article
  • Citations64

Contextual semantic embeddings based on fine-tuned AraBERT model for Arabic text multi-class categorization

  • Feb 18, 2021
  • Journal of King Saud University - Computer and Information Sciences
  • Fatima-Zahra El-Alami +2
  • Research Article
  • Citations30

Semantic Textual Similarity in Japanese Clinical Domain Texts Using BERT.

  • Jun 01, 2021
  • Methods of Information in Medicine
  • Faith Wavinya Mutinda +3
  • PDF
  • Research Article

Automatic Short Answer Grading for Enhancing Educational Assessment Using BERT and USE Models

  • Feb 22, 2026
  • Indian Journal Of Science And Technology
  • Chandralika Chakraborty +2
  • Research Article
  • Citations2

CACER: Clinical concept Annotations for Cancer Events and Relations.

  • Sep 03, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Yujuan Velvin Fu +7
  • Research Article

T-­BERT:臺灣語言模型–以臺灣在地語言預訓練BERT模型

  • Jan 01, 2020
  • 鍾明諺
  • Research Article

Analisis Sentimen Ulasan iPhone di Amazon Menggunakan Model Deep Learning BERT Berbasis Transformer

  • Dec 22, 2024
  • Elkom: Jurnal Elektronika dan Komputer
  • Arif Fitra Setyawan +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.