Language Models are Few-Shot Learners

Tom B. Brown et al. · arXiv

TL;DR

Brown et al. train GPT-3, an autoregressive language model with 175 billion parameters, and evaluate it without task-specific fine-tuning. Given a natural-language description and a few examples in the prompt, the model attempts the task. Scaling the model is what makes that in-context behavior useful.

Why it matters

The usual path at the time was pretrain, then fine-tune on thousands of labeled examples for each new dataset. That works, but every task needs its own training run and dataset. GPT-3 argued that a sufficiently large model can adapt inside the prompt. That shifted research attention toward scale, prompting, and the limits of what a frozen model will do from examples alone.

Key takeaways

Few-shot, one-shot, and zero-shot settings put the task description in the prompt instead of a gradient update.
Performance on many tasks rises as the model gets larger, which is the paper's main empirical claim.
The model is still a next-token predictor. The "learning" happens in the context window, not in updated weights.
Fine-tuning remains stronger on some tasks. The paper is an argument about what scale unlocks, not a claim that labeled data stopped mattering.

Four panels comparing fine-tuning with zero-shot, one-shot, and few-shot prompting — Figure 2.1 from Brown et al., 2020. Fine-tuning updates weights. Zero-shot, one-shot, and few-shot only use the prompt.