TL;DR

Lambert et al. release Tulu 3, an open post-training recipe on Llama 3.1: supervised fine-tuning, direct preference optimization, and a final stage they name Reinforcement Learning with Verifiable Rewards (RLVR). In RLVR the reward is not a learned model of human preference. The policy gets a reward when a verifier can tell that the completion is correct.

Why it matters

RLHF needs people, or a model trained on people, to score open-ended answers. Math, exact instruction checks, and other tasks already have a way to say yes or no. Tulu 3 treats that check as the reward and publishes the data, code, and training report. Later reasoning work cites this recipe when it talks about RLVR: reinforce what you can verify.

Key takeaways

• The paper's stack is SFT, then DPO, then RLVR.
• RLVR pays a constant reward for a successful completion, checked against a ground-truth outcome.
• The authors apply it to skills such as math and instruction following, and they report gains on GSM8K, MATH, and IFEval.
• Starting from the DPO checkpoint, rather than from SFT alone, produced their strongest final models.
• They also document methods that did not reliably help, which is part of the open report.