46 Homework: Linear Models and Basis Functions

In the past few meetings, we’ve been thinking about what it means to fit a model and why we use least squares. Here we’ll work with the core ideas hands-on: what a linear model is, how its residuals behave, and what determines whether our predictions are stable.

Linear Models and Basis Functions

Every linear model can be described in terms of a vector of basis functions. \[ \mathcal{M} = \qty{ m(x) = \phi(x)^T \beta : \beta \in \mathbb{R}^{k+1} } \qqtext{for} \phi(x) = \begin{pmatrix} \phi_0(x) \\ \phi_1(x) \\ \vdots \\ \phi_{k}(x) \end{pmatrix}. \] We call the number of basis functions, $k+1$, the dimension of the model.

What is the dimension of the following models?

Lines.
Piecewise-constant functions with breaks at $k$ points $x_1 \ldots x_k$.
Piecewise-cubic functions with breaks at $k$ points $x_1 \ldots x_k$.
Piecewise-cubic splines with knots at $k$ points $x_1 \ldots x_k$.

Residuals and Orthogonality

Fitting a curve breaks down our observations $Y_i$ into two pieces: the part we’ve been able to predict, $\hat \mu(X_i)$, and what’s left over, the residuals $\hat\varepsilon_i = Y_i - \hat\mu(X_i)$.

The residuals from a linear model have an interesting property. They are orthogonal to all curves in the model: \[ \frac{1}{n}\sum_{i=1}^n \hat \varepsilon_i m(X_i) = 0 \qqtext{for all} m \in \mathcal{M}. \]

Suppose we fit the linear model above by minimizing MSE. Show that this orthogonality property holds. In particular, show that it’s implied by the zero-derivative condition $0 = \frac{\partial}{\partial \beta_j}\mid_{\beta = \hat\beta} \text{MSE}(\beta)$ for all $j$.

When we fit a line, the residuals have zero covariance with $X_i$: \[ \frac{1}{n}\sum_{i=1}^n \hat\varepsilon_i (X_i - \bar X) = 0. \] Prove it.

When we fit a piecewise-constant model, the residuals average to zero on each piece. That is, between each pair of breaks, to the left of the leftmost break, and to the right of the rightmost.

Prove it.
Explain what this tells you about the fitted curve $\hat\mu$. Something like “On each piece, the predictions $\hat\mu(X_i)$ have the same ___ as the ___.”

Stability

Suppose we’re using height to predict rebounds. We’ll work with heights rounded to the nearest inch, and at each even height $X_i$ from 60 inches to 84 inches (5’ – 7’2”), we’ll include one player in our dataset. However, at each height there are two players we could include, one having $2$ more rebounds than the other. If $\mu(X_i)$ is the average number of rebounds for the two players with height $X_i$, then we have one observation $(X_i,Y_i)$ for each $X_i \in 60, 62, 64, \ldots 84$ and \[ Y_i = \mu(X_i) + \varepsilon_i \quad \text{ for } \quad \varepsilon_i = \pm 1. \] And we’ll flip a coin to choose between the two players at each height, so \[ Y_i = \mu(X_i) + \varepsilon_i \text{ where } \varepsilon_1 \ldots \varepsilon_n \text{ are independent with } \varepsilon_i = \begin{cases} +1 & \text{with probability $1/2$} \\ -1 & \text{with probability $1/2$} \end{cases}. \] We’re going to do least squares regression using a cubic polynomial model to predict the number of rebounds we’d see from players at other heights, like the odd heights $61$, $73$, and $89$.

Derive the Variance Formula

Calculate the standard deviation of $\hat\mu(x)$ for arbitrary $x$. To do this, take the following steps.

Write out a formula for $Z=\hat \mu(x)$ as a linear function of $Y_1 \ldots Y_n$.
Calculate its expectation $E Z$ and write out $Z - E Z$.
Write out the square $(Z-E Z)^2$. This should be a double sum, as the square of a sum $\sum_{i=1}^n z_i$ is $\sum_{i=1}^n \sum_{j=1}^n z_i z_j$.
Now you should be ready to calculate the expected square $\text{Var}(Z) = E(Z - E Z)^2$. The expectation of a sum is the sum of the expectation of its terms, so all you’ve got to do is compute the expectation of its terms. Are any zero? If so, why?
Try to find a nice way to express $\text{Var}(Z)$ in terms of the feature vector $\phi(x)$ and the covariance matrix $\hat\Sigma = \frac{1}{n}\sum_{i=1}^n \phi(X_i)\phi(X_i)^T$.

Check the Formula by Simulation

Let $\mu(X_i) = X_i/4$. We can generate observations using the following R code.

X = seq(60, 84, by=2)

B = rbinom(length(X), 1, 1/2)
epsilon = 2*B - 1

Y = X/4 + epsilon

Fit a cubic polynomial to this data and record the prediction $\hat\mu(x)$ for $x=61$. Do this 1000 times, so we have $1000$ predictions $\hat \mu_{1}(x) \ldots \hat \mu_{1000}(x)$ based on different random vectors $\varepsilon=[\varepsilon_1 \ldots \varepsilon_n]$ with the same distribution. Compute the sample variance, \[ \widehat{\text{Var}}\{ \hat \mu(x) \} = \frac{1}{1000}\sum_{j=1}^{1000}\qty{\hat \mu_{j}(x) - \frac{1}{1000}\sum_{j=1}^{1000} \hat \mu_j(x) }^2. \] Then compare to the variance $\text{Var}\{ \hat \mu(x) \}$ you get using your formula from the previous exercise. If they’re not similar, there’s either a bug in your code or an error in your derivation. Work out which and fix it.

Then, once you’re done, plot a histogram of your estimates $\hat \mu_1 \ldots \hat \mu_{1000}$. Roughly how many of these thousand observations fall within one standard deviation of the mean? What about two standard deviations?

Other Prediction Points

Repeat the previous exercise for new heights $x=73$ and $x=81$. What changes? What stays the same?

Other Models

Repeat the simulation exercises using two new models. You’ll need to generalize your variance formula to give you $\text{sd}\{ \hat\mu(x) \}$ for any linear model $\mathcal{M}$, i.e. for any feature vector $\phi(x)$.

Piecewise-constant curves with breaks at $66$, $70$, $74$, $78$, and $82$.
Cubic splines with knots at $66$, $70$, $74$, $78$, and $82$. The R formula for a cubic spline with knots at the points $\text{knot}_1, \text{knot}_2, \ldots \text{knot}_k$ collected into a vector knot $\in \mathbb{R}^k$ is Y ~ bs(X, knots=knot).

Describe the differences you see when you use these different models. And try to explain where these differences come from.