TL;DR

OpenAI's January 25, 2021 post explains how they ran a single Kubernetes cluster out to 7,500 nodes for large training jobs and for smaller research. The interesting constraints are not a typical web service: one pod often takes a whole machine, jobs are MPI programs, and a failed pod can stop the entire run.

Why it matters

Training clusters at this size fail in the scheduler, the network, and the way pods are admitted together, not in request-per-second load balancing. The post is a rare public writeup of those failure modes, written so other Kubernetes operators can reuse the lessons. It is also the infrastructure behind models the lab was training at the time, including GPT-3.

Key takeaways

• Many workloads want the full node so GPUs can talk over NVLink or reach the NIC with GPUDirect. Bin-packing is not the main problem.
• Scheduler load is spiky. A job can create hundreds of pods at once, then go quiet.
• MPI jobs need their pods to stay up together. The lab treated them as semi-stateful and relied on checkpoints.
• Gang scheduling, landing all members of a StatefulSet before training starts, was a real gap. They later used the Coscheduling plugin.
• Pods talk to each other by IP over MPI, not through Kubernetes service load balancing. Data mostly comes from blob storage.