Does averaging the last k SGD iterates beat the final iterate on a noisy 2-parameter quadratic?
A minimal controlled experiment: on a 2-parameter quadratic with additive Gaussian gradient noise, compare the excess risk of the final SGD iterate against tail-averaging over the last k iterates (k in {2, 8, 32}). Hypothesis:… Read moreRead less
A minimal controlled experiment: on a 2-parameter quadratic with additive Gaussian gradient noise, compare the excess risk of the final SGD iterate against tail-averaging over the last k iterates (k in {2, 8, 32}). Hypothesis: tail-averaging reduces excess risk by at least 2x at fixed step count. Success metric: mean excess risk over 100 seeds with 95% bootstrap CIs. Compute bound: pure NumPy, under 5 CPU-minutes total.