Gradient Descent in 1D

\(L(\beta)\) represents a loss function for parameter \(\beta\). The ball represents our current estimate, \(\beta^{(t)}\) and it repeatedly takes the step \(\beta^{(t+1)} = \beta^{(t)} - \rho\, \frac{dL(\beta^{(t)})}{d\beta}\). Drag anywhere on the plot to move the starting point \(\beta^{(0)}\). Try: crank \(\rho\) past 1 on the smooth bowl; watch the V-shape bounce forever; drag \(\beta^{(0)}\) across the hump of the double well.

The function, the ball, and its path

\(\beta\)

\(L(\beta)\)

\(L(\beta)\) tangent at \(\beta^{(t)}\) (slope \(\frac{dL(\beta^{(t)})}{d\beta}\)) path so far; dashed = the next step ★ local minima

Is it getting anywhere? — \(L(\beta^{(t)})\) against iteration (dashed line = lowest possible value)

iteration \(t\)

\(L(\beta^{(t)})\)

\(t\) = | \(\beta^{(t)}\) = | \(L(\beta^{(t)})\) = | \(\frac{dL(\beta^{(t)})}{d\beta}\) = | step \(-\rho\, \frac{dL(\beta^{(t)})}{d\beta}\) =

Function

Smooth bowl \((\beta-1)^2+2\)

V-shape \(|\beta-1|+2\)

Double well \((\beta^2-1)^2+0.2\beta\)

Wavy bowl \(0.3\beta^2+\sin 3\beta\)

Flat bottom \(\beta^4\)

Step size (log scale)

\(\rho\) 0.0800

changing \(\rho\) restarts from \(\beta^{(0)}\)

Run

Reset returns to \(\beta^{(0)}\)

current function:

update rule: \( \beta^{(t+1)} \;=\; \beta^{(t)} \;-\; \rho\, \frac{dL(\beta^{(t)})}{d\beta} \)