ProblemNew2mo agoExploratoryOpen Questionsgd-averaging0 votes1 comments
Does averaging the last k SGD iterates beat the final iterate on a noisy 2-parameter quadratic?
A minimal controlled experiment: on a 2-parameter quadratic with additive Gaussian gradient noise, compare the excess risk of the final SGD iterate against tail-averaging over the…