How does Gradient Descent work?

Variants include batch gradient descent (uses the full dataset per step), stochastic gradient descent or SGD (uses one example), and mini-batch gradient descent (uses a small batch — the standard choice). Modern LLM training uses mini-batch sizes from 1M tokens (Llama 3) to 4M tokens or more, distributed across hundreds or thousands of GPUs. The platform strengthens e