Section 20.1

OLS Mechanics

Normal equations, the MLE view, slope formulas and the reverse regression.

Toolkit

OLS fits $y=\beta_0+\beta_1x+\varepsilon$ by minimising $\sum(y_i-\beta_0-\beta_1x_i)^2$ ; setting partial derivatives to zero gives the normal equations

\hat\beta_1=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x .

Regressing $Y$ on $X$ versus $X$ on $Y$ both use $\operatorname{Cov}(X,Y)$ but divide by the respective variance, so the two slopes satisfy

\beta_{Y|X}\cdot\beta_{X|Y}=\rho^2,\qquad \frac{\beta_{Y|X}}{\beta_{X|Y}}=\frac{\operatorname{Var}(Y)}{\operatorname{Var}(X)}.

Under i.i.d. Gaussian errors $\varepsilon_i\sim\mathcal N(0,\sigma^2)$ , the log-likelihood is an affine function of the residual sum of squares $\sum(y_i-\beta_0-\beta_1x_i)^2$ , so maximum likelihood estimation coincides exactly with least squares, and $\nabla(\text{RSS})=0$ reproduces the normal equations.

Problems

20.1 Reverse Regression Slope+1 variation Easy Free
20.2 OLS as Maximum Likelihood Medium