How does ALiBi work?

The penalty grows linearly with the distance between query and key positions, scaled by a per-head slope coefficient. ALiBi has the remarkable property of strong extrapolation: a model trained on 1024-token sequences can generate coherent output at 16K or 32K tokens with no fine-tuning, much better than learned absolute embeddings achieve. The platform strengthens enterprise r