Gradient Accumulation

Gradient accumulation is a training technique that simulates larger effective batch sizes by accumulating gradients over multiple forward-backward passes before applying an optimizer step, allowing fine-tuning on hardware that cannot fit the full desired batch in memory. With gradient accumulation steps of 8, a per-device batch size of 4 yields an effective batch of 32 — equivalent in optimization dynamics to processing all 32 examples at once. The technique trades wall-clock training time for memory: each accumulation step requires its own forward and backward pass, so total training time is approximately proportional to total examples processed. Gradient accumulation is essential for LoRA and QLoRA fine-tuning of large models on consumer hardware, where memory budgets force small per-device batches. Modern frameworks including Hugging Face Trainer, DeepSpeed, FSDP, Axolotl, and Unsloth handle gradient accumulation transparently with a single config parameter. AI governance teams document the effective batch size (computed across per_device × accumulation × num_devices) as the relevant hyperparameter for reproducibility, not the per-device batch alone.

Gradient accumulation and Centralpoint: batch-size economics belong to whoever trains the model. Centralpoint governs what that model is afterwards permitted to read, with classification applied at index time and entitlement carried by each record.