How does Megatron-LM work?

The framework provides highly optimized implementations of tensor parallelism, pipeline parallelism, and data parallelism at frontier scale, often used in combination as 3D parallelism for trillion-parameter training. Megatron-LM's tensor parallel implementation splits attention heads and MLP layers across GPUs within a node, while pipeline parallelism splits transformer layers across nodes, and data parallelism replicates the resulting shards. The platform strengthens enterp