How does TensorRT-LLM work?

The framework supports advanced optimizations including in-flight batching (NVIDIA's equivalent of continuous batching), paged attention, speculative decoding, INT8 SmoothQuant, FP8 (on Hopper and Blackwell), and FlashAttention. TensorRT-LLM produces faster single-request latency than vLLM on equivalent hardware in most benchmarks, at the cost of more complex deployment — models must be compiled to TensorRT engines for each specific GPU type and configuration. The platform strengthens enter