• Home
  • Search
  • ANNA”:" Enhanced Language Representation for Question Answering
  • Cite Icon8
  • https://doi.org/10.18653/v1/2022.repl4nlp-1.13Copy DOI Icon

ANNA”:" Enhanced Language Representation for Question Answering

  • Jan 1, 2022
  • Changwook Jun +6 more
Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.

Loading PDF

Similar Papers
  • PDF
  • Research Article
  • Citations10

AGI-P: A Gender Identification Framework for Authorship Analysis Using Customized Fine-Tuning of Multilingual Language Model

  • Jan 01, 2024
  • IEEE Access
  • Raheem Sarwar +6
  • PDF
  • Research Article
  • Citations1

Prompt Learning with Structured Semantic Knowledge Makes Pre-Trained Language Models Better

  • Jul 30, 2023
  • Electronics
  • Hai-Tao Zheng +5
  • Conference Article
  • Citations3

Chinese-Korean Weibo Sentiment Classification Based on Pre-trained Language Model and Transfer Learning

  • May 06, 2022
  • Hengxuan Wang +3
  • Conference Article

GTSP: A Unified Framework for Graph Pretraining and Task-Specific Prompting

  • Sep 19, 2025
  • Fanghua Lu
  • PDF
  • Conference Article
  • Citations31

PANLP at MEDIQA 2019: Pre-trained Language Models, Transfer Learning and Knowledge Distillation

  • Jan 01, 2019
  • Wei Zhu +6
  • Conference Article
  • Citations4

StyleBERT: Chinese Pretraining by Font Style Information

  • Jun 17, 2022
  • Chao Lv +7
  • Conference Article
  • Citations26

ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction

  • May 13, 2024
  • Jianghao Lin +9
  • Research Article
  • Citations338

Pre-Trained Language Models and Their Applications

  • Jun 01, 2023
  • Engineering
  • Haifeng Wang +4
  • Research Article
  • Citations10

UniproLcad: Accurate Identification of Antimicrobial Peptide by Fusing Multiple Pre-Trained Protein Language Models

  • Apr 11, 2024
  • Symmetry
  • Xiao Wang +3
  • Research Article
  • Citations13

Vulnerability Detection of Ethereum Smart Contract Based on SolBERT-BiGRU-Attention Hybrid Neural Model

  • Jan 01, 2023
  • Computer Modeling in Engineering & Sciences
  • Guangxia Xu +2
  • PDF
  • Conference Article
  • Citations4

CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations

  • Jan 01, 2022
  • Changlong Yu +3
  • Conference Article
  • Citations13

Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition

  • Jan 01, 2024
  • Yukiya Hono +5
  • Video Transcripts

Robust Transfer Learning with Pretrained Language Models through Adapters

  • Aug 01, 2021
  • Underline Science Inc.
  • Wenjuan Han +2
  • PDF
  • Research Article
  • Citations5

Keeping Models Consistent between Pretraining and Translation for Low-Resource Neural Machine Translation

  • Nov 27, 2020
  • Future Internet
  • Wenbo Zhang +4
  • Research Article
  • Citations78

Pre-train, Prompt, and Recommendation: A Comprehensive Survey of Language Modeling Paradigm Adaptations in Recommender Systems

  • Dec 14, 2023
  • Transactions of the Association for Computational Linguistics
  • Peng Liu +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.