How does Prefix Caching work?

The technique is especially impactful for RAG applications where many queries share the same retrieved passages, for chatbot applications where every request includes the same system prompt, and for few-shot learning where many requests share the same example block. vLLM implements automatic prefix caching with content-hash-based identification, while TensorRT-LLM supports KV cache reuse with explicit prefix specification. OpenAI's API offers prompt caching with a separate cached-token billing rate (50% of standard input rate) for prompts cached on their infrastructure. The platform strengthens ent