What is Batch Size?

In training, batch size trades memory against gradient stability and interacts with learning rate in ways that make it a hyperparameter requiring care. In inference it governs throughput: serving many requests together uses hardware more efficiently than serving them individually, which is why batching strategy materially affects the cost of running a model at scale. For organizations consuming hosted models the detail is invisible and priced in; for those running local inference it becomes an operational decision. Centralpoint's relevance is on the consumption side rather than the serving side. Because token consumption is metered per execution and per skill, and because governed answers are served from the local index rather than regenerated, the largest cost lever is not batching efficiency but avoiding the request entirely — a repeated question answered once costs nothing on the thousandth asking.