How does Tensor Parallelism work?

The standard approach, introduced in NVIDIA's Megatron-LM paper (2019), splits attention heads and MLP layers across the tensor-parallel rank. Tensor parallelism complements pipeline parallelism (splits across layers) and data parallelism (replicates the model), and at frontier scale all three are combined in 3D parallelism configurations. The platform strengthens