Tail-averaged SGD versus the final iterate on a noisy 2-parameter quadratic (walkthrough run)
Controlled minimal experiment: on a 2-parameter quadratic with additive Gaussian gradient noise, compare excess risk of the final SGD iterate against tail-averaging over the last k iterates (k in {2, 8, 32}). Hypothesis: k=32 tail-averaging reduces mean excess risk by at least 2x at the same step budget. Metric: mean excess risk over 100 seeds with 95% bootstrap CIs. Bound: pure NumPy, under 5 CPU-minutes.
Workspace
Problem Statement
## Setup Minimize f(w) = 0.5 * w^T A w with A = diag(1, 10), from w0 = (5, 5). SGD gradient g(w) = A w + xi, xi ~ N(0, 0.25 I), constant step eta = 0.02, T = 2000 steps. ## Question Does tail-averaging the last k iterates (k in {2, 8, 32}) reduce final excess risk f(w) - f(w*) by at least 2x versus the final iterate at identical step budgets? ## Protocol - 100 independent seeds; mean excess risk + 95% bootstrap CIs per estimator (final, k=2, k=8, k=32). - One results table and one figure. - Falsification: hypothesis fails unless k=32 improves mean excess risk by >= 2x over the final iterate. ## Constraints - Pure NumPy, CPU-only, under 5 CPU-minutes total; fully seeded and reproducible; no external datasets.
Evaluation plan
Mean excess risk over 100 seeds per estimator (final iterate, tail-average k=2/8/32) with 95% bootstrap CIs; hypothesis passes iff k=32 improves mean excess risk by >= 2x over the final iterate.