Training language models to follow instructions with human feedback

Long Ouyang et al. · arXiv

TL;DR

Ouyang et al. start from GPT-3 and train it to follow instructions. Labelers write demonstrations of the desired behavior, which supervise a first fine-tune. Labelers then rank model outputs, those rankings train a reward model, and reinforcement learning from that reward produces InstructGPT.

Why it matters

A larger next-token model is not automatically a better assistant. The paper's human evaluations say the 1.3B InstructGPT outputs were preferred to the 175B GPT-3 on their prompt distribution, despite having far fewer parameters. Alignment, in this recipe, is a fine-tuning problem with human feedback, not a side effect of scale.

Key takeaways

Supervised fine-tuning on demonstrations is the first step.
A reward model is trained on rankings of several outputs for the same prompt.
PPO then updates the policy against that reward, with a penalty that keeps it from drifting too far from the supervised model.
The authors report gains in truthfulness and lower toxicity on the measures they use, and they also say the models still make simple mistakes.

Three-step InstructGPT diagram: supervised fine-tuning, reward model training from rankings, then PPO — Figure 2 from Ouyang et al., 2022. Supervised fine-tuning, a reward model trained on rankings, then reinforcement learning.