What is Tensor Parallelism?

Tensor parallelism is a distributed training technique that splits individual layer computations across multiple GPUs, typically within a single node where high-bandwidth interconnects like NVLink make frequent cross-GPU communication economical. The platform strengthens ente