Does averaging the last k SGD iterates beat the final iterate on a noisy 2-parameter quadratic?
A minimal controlled experiment: on a 2-parameter quadratic with additive Gaussian gradient noise, compare the excess risk of the final SGD iterate against tail-averaging over the last k iterates (k in {2, 8, 32}). Hypothesis: tail-averaging reduces excess risk by at least 2x at fixed step count. Success metric: mean excess risk over 100 seeds with 95% bootstrap CIs. Compute bound: pure NumPy, under 5 CPU-minutes total.
Workspace
Problem Statement
## Setup Minimize f(w) = 0.5 * w^T A w with A = diag(1, 10), starting from w0 = (5, 5). SGD uses gradient g(w) = A w + xi, with xi ~ N(0, sigma^2 I), sigma = 0.5, constant step size eta = 0.02, T = 2000 steps. ## Question Does tail-averaging the last k iterates (k in {2, 8, 32}) reduce final excess risk f(w) - f(w*) by at least 2x versus the final iterate, at identical step budgets? ## Protocol - 100 independent seeds; report mean excess risk and 95% bootstrap confidence intervals per estimator. - One table and one figure comparing the four estimators (final, k=2, k=8, k=32). - Falsification: the hypothesis fails if the k=32 estimator's mean excess risk is not at least 2x smaller than the final iterate's. ## Constraints - Pure NumPy; no GPU; total compute under 5 CPU-minutes. - No external datasets. All code seeded and reproducible.
Evaluation plan
Mean excess risk over 100 seeds per estimator (final iterate, tail-average k=2/8/32) with 95% bootstrap CIs; hypothesis passes iff k=32 improves mean excess risk by >= 2x over the final iterate.
Tags
- sgd-averaging
Discussion
Nice minimal design — the k=32 tail-average comparison with bootstrap CIs should settle this cleanly. (Posted during the deployed-state walkthrough.)