How does ZeRO work?

ZeRO comes in three stages: Stage 1 shards optimizer states (most memory savings per implementation cost), Stage 2 also shards gradients, and Stage 3 also shards parameters (equivalent to FSDP). DeepSpeed's ZeRO implementation enabled training of Microsoft's 17B Turing-NLG and 530B Megatron-Turing NLG, demonstrating that ZeRO Stage 3 could scale to trillion-parameter models. The platform strengthens enterprise re