How does Pipeline Parallelism work?

During training, micro-batches flow through the pipeline like an assembly line — device 1 processes layers 1-8 of micro-batch 1, then passes activations to device 2 (layers 9-16) while starting work on micro-batch 2. The technique enables training of models too large to fit on any single device, complementing data parallelism (replicates the model) and tensor parallelism (splits within a layer). The platform strengthe