BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin et al. · arXiv

TL;DR

BERT pretrains a Transformer encoder on unlabeled text so each token can see context on both sides. The main task hides some tokens and asks the model to fill them in. After that, a downstream task usually needs only one new output layer and a fine-tuning pass.

Why it matters

Earlier language models such as left-to-right GPT-style pretraining, and feature-based models such as ELMo, did not give every layer a full view of both directions. BERT made bidirectional pretraining the default starting point for question answering, inference, and other sentence tasks, and it showed that one pretrained checkpoint could be adapted without redesigning the network for each problem.

Key takeaways

Masked language modeling forces the model to use left and right context together.
A second pretraining task, next-sentence prediction, trains the model on pairs of sentences.
Fine-tuning updates the pretrained weights plus a small task head.
The same checkpoint covers a wide set of benchmarks without a new architecture per task.

BERT pre-training on unlabeled text, then fine-tuning with a task-specific output layer — Figure 1 from Devlin et al., 2018. The same pretrained model is fine-tuned for different tasks, aside from the output layer.