What is Continuous Batching?

Traditional batching waits for a set number of requests, processes them together and returns them together, which means a short request can wait for a long one. Continuous batching admits new requests as slots free, keeping the accelerator busy and reducing latency variance — one of the main reasons modern serving stacks achieve far better throughput than naive implementations. It is invisible to an organization consuming hosted models and becomes an operational consideration for those running inference locally. Centralpoint's concern sits above this layer: consumption is metered per execution regardless of how efficiently it is served, and the largest available saving is not throughput but avoidance — a question recognized as already answered is served from the local index and never reaches the model at all.