Gradient Descent in 2D

The downhill direction is the negative gradient \( -\nabla L(\beta_1,\beta_2) = -\left( \frac{\partial L(\beta_1,\beta_2)}{\partial \beta_1}, \frac{\partial L(\beta_1,\beta_2)}{\partial \beta_2} \right) \), shown as the green arrow. The plot shows the contour lines of \(L(\beta_1,\beta_2)\); darker rings are higher. Drag anywhere on the plot to move the starting point \(\boldsymbol{\beta}^{(0)}\). Try: on the elliptical bowl the \(\beta_2\)-direction is twice as curved, so it zigzags across the valley and diverges once \(\rho > \tfrac12\) — the steepest direction sets the limit.

Contours of \(L\), the ball, and its path

\(\beta_1\)

\(\beta_2\)

contours of \(L\) \(-\nabla L\) direction path so far; dashed = the next step ★ local minima

Is it getting anywhere? — \(L(\beta_1^{(t)}, \beta_2^{(t)})\) against iteration (dashed line = lowest value on the grid)

iteration \(t\)

\(L(\beta_1^{(t)}, \beta_2^{(t)})\)

\(t\) = | \((\beta_1^{(t)}, \beta_2^{(t)})\) = (, ) | \(L(\beta^{(t)}_1, \beta^{(t)}_2)\) = | \(\frac{\partial L(\beta^{(t)}_1, \beta^{(t)}_2)}{\partial \beta_1}\) = | \(\frac{\partial L(\beta^{(t)}_1, \beta^{(t)}_2)}{\partial \beta_2}\) = | step \(-\rho\, \nabla L(\beta^{(t)}_1, \beta^{(t)}_2)\) = (, )

Function

Elliptical bowl \((\beta_1-1)^2+2(\beta_2+\tfrac12)^2+2\)

V-cone \(|\beta_1-1|+|\beta_2|+2\)

Double well \((\beta_1^2-1)^2+0.2\beta_1+\beta_2^2\)

Wavy bowl \(0.3(\beta_1^2+\beta_2^2)+\sin 3\beta_1+\sin 3\beta_2\)

Flat bottom \(\beta_1^4+\beta_2^4\)

Step size (log scale)

\(\rho\) 0.100

changing \(\rho\) restarts from \(\boldsymbol{\beta}^{(0)}\)

Run

Reset returns to \(\boldsymbol{\beta}^{(0)}\)

current function:

update rule: \( \beta_1^{(t+1)} = \beta_1^{(t)} - \rho\, \frac{\partial L(\beta_1^{(t)}, \beta_2^{(t)})}{\partial \beta_1}, \qquad \beta_2^{(t+1)} = \beta_2^{(t)} - \rho\, \frac{\partial L(\beta_1^{(t)}, \beta_2^{(t)})}{\partial \beta_2} \) (each coordinate takes its own partial-derivative step; at the V-cone's kinks we use the subgradient 0)