TL;DR

OpenAI's June 9, 2022 post is a map of how to split one training step across many GPUs. Once a model or a batch no longer fits on one device, you have to move pieces of the calculation. The post separates those patterns, including tensor parallelism, so they are choices about communication rather than a pile of framework flags.

Why it matters

Large-model training is a systems problem wearing a research label. People mix data parallelism, pipeline parallelism, and tensor parallelism and then cannot say which bottleneck they hit. A clear taxonomy makes those combinations discussable: what is copied, what is sharded, and what has to be synchronized before the next step.

Key takeaways

• The unit of work is one synchronized calculation across a cluster, not a single GPU kernel.
• Parallelism techniques grew up as clusters and models grew. The post's job is to make that menu legible.
• Tensor parallelism splits a layer's math across devices, which forces communication inside the step.
• The same ideas compose. A real training run usually stacks more than one kind of parallelism.
• Understanding the pattern does not require knowing every kernel. It requires knowing which bytes move, and when.