How does FlashAttention work?

The technique reduces the memory footprint of attention from quadratic to linear in sequence length while preserving exact mathematical equivalence to standard attention. FlashAttention-2 (2023) added thread-block-level parallelism and reduced non-matmul FLOPs for an additional 2x speedup, while FlashAttention-3 (2024) added Hopper-architecture-specific optimizations including FP8 support and warp-specialization. The platform strengthens ent