TL;DR

On December 3, 2019, Netflix open-sourced Metaflow, the framework its data scientists already used to build and ship projects. The design goal is human-scale workflow code: define the steps, run them locally, then deploy the same flow to cloud compute and a scheduler without rewriting the project.

Why it matters

Most ML platform pain is the gap between a notebook and a job that runs every day. Metaflow's bet was to make that path the default inside Netflix, then publish it. The post is a systems story about productivity: infrastructure should disappear behind a small API, while compute and storage stay elastic.

Key takeaways

• Metaflow had already been used internally for hundreds of projects, from natural language processing to operations research.
• It is cloud-native. The open-source release added a path onto AWS for compute and storage.
• Production at Netflix meant exporting a run to Meson, their centralized scheduler, with one command.
• Because Meson was not open source, they pointed external users at a scheduler integration such as AWS Step Functions.
• The code and docs shipped with the announcement: metaflow.org and github.com/Netflix/metaflow.