19 Homework: Variance of Comparisons
$$ \newcommand{X}{} \newcommand{Y}{}
$$
Summary
This week’s homework addresses two issues we’ve left hanging.
I know what you’re thinking: we had a whole homework assignment on this. But that’s only half-right. In the Week 2 Homework, we focused on showing that we get near-perfect calibration, not really why. But using normal approximation, we can do the why part pretty easily. This’ll be quick, and it’ll involve a bit of calculus, which makes it a good warm-up for what we’ll do next. It’ll give us an opportunity to revisit some of our Week 2 stuff using a more formula-driven perspective, too.
I said, in our Lecture on Comparing Two Groups, that you’d be calculating the variance of a difference in subsample means in this one. You’ll be doing that and a little more: you’ll be calculating,approximately, the variance of a ratio of subsample means, too. Because sometimes that’s closer to what you want to know. People often say, for example, that women in this country earn 78 cents on the dollar for doing the same work as men. That’s a ratio. Tackling this in addition to the difference won’t be too much additional work. After a little bit of calculus, we basically wind up in the same place as we do for the difference.
Calculus Review: Linear Approximation
We’re going to be using linear approximation to simplify some of our calculations. Given a function \(f(x)\), we can approximate it near any point \(x_0\) like this. \[ f(x) \approx f(x_0) + f'(x_0)(x-x_0) \]
Hopefully you remember that from calculus. If you like, you can call this first-order Taylor approximation. And there are a few formulas for the error of this approximation, which is called the remainder in Taylor’s Theorem, in most calculus textbooks.
When we’re thinking about functions of multiple variables, we use the multivariate version, which involves partial derivatives. \[ \begin{aligned} f(x,y) &\approx f(x_0,y_0) + \qty[\frac{\partial f}{\partial x}(x_0,y_0)] (x-x_0) \ + \ \qty[\frac{\partial f}{\partial y}(x_0,y_0)] (y-y_0). \end{aligned} \]
Why We Usually Get Near-Perfect Calibration
Suppose we’ve sampled with replacement from a binary population in which \(\theta\) is the proportion of ones. If we use the sample mean \(\hat\theta\) as our point estimate and calibrate a 95% confidence interval around it using normal approximation, this is the interval we get.
\[ \hat\theta \pm 1.96 \hat\sigma / \sqrt{n} \qfor \hat\sigma^2 = \hat\theta(1-\hat\theta) \]
On the other hand, the interval we’d want—assuming we’re still happy to use normal approximation—uses the actual variance of our sample proportion, \(\sigma^2=\theta(1-\theta)\), instead of the estimate \(\hat\sigma^2\). Figure 20.1 is an attempt to convince ourselves that it doesn’t make much of a difference at all. I think I called the difference ‘a fingernail thick’ in lecture. Now you’re going to quantify this difference. We’ll assume that \(\hat\theta\) is one of the ‘good draws’ from its sampling distribution, which for our purposes will mean that it’s in the interval \(\theta \pm 1.96\sigma / \sqrt{n}\), the middle 95% of the sampling distribution’s normal approximation.
Solution
Locked (Week 5)
Solution
Locked (Week 5)
This should be familiar from the Week 2 Homework.
Variance Calculation for Comparisons
Differences in Means
In our Lecture on Comparing Two Groups, we talked about how to use subsample means to compare two groups. In particular, we talked about the case that we’ve drawn a sample \((X_1,Y_1) \ldots (X_n,Y_n)\) with replacement from a population \((x_1,y_1) \ldots (x_m,y_m)\) in which \(x_j \in \{0,1\}\) indicates membership in one of two groups, e.g. treated and control groups in Figure 23.1. And we talked about using the difference \(\textcolor[RGB]{0,191,196}{\hat\mu(1)}-\textcolor[RGB]{248,118,109}{\hat\mu(0)}\) in the mean of \(Y_i\) for the subsamples in which \(\textcolor[RGB]{0,191,196}{X_i=1}\) and \(\textcolor[RGB]{248,118,109}{X_i=0}\) to estimate the corresponding difference \(\textcolor[RGB]{0,191,196}{\mu(1)}-\textcolor[RGB]{248,118,109}{\mu(0)}\) in the population.
Solution
Locked (Week 5)
We also calculated a formula for the variance of a subsample mean \(\hat\mu(x)\). \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(x)] = \frac{\sigma^2(x)}{N_x} \text{ for } N_x = \sum_{i}1_{=x}(X_i) \qand \sigma^2(x) = \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y_i \mid X_i=x] \]
And I stated without proof a formula for the variance of the difference of two subsample means. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\hat{\mu}(1)-\hat{\mu}(0)] = \mathop{\mathrm{E}}\qty[\frac{1}{N_1}\sigma^2(1)+\frac{1}{N_0}\sigma^2(0)] \text{ for } N_x = \sum_{i}1_{=x}(X_i) \]
It’s a simple formula. The variance of the difference in means is the sum of the variances of the two means. Why is that the case? To start to see why, we can start from definitions and do a bit of arithmetic.
\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\hat{\mu}(1)-\hat{\mu}(0)] &= \mathop{\mathrm{E}}\qty[ \qty(\{\hat{\mu}(1) - \hat{\mu}(0)\} - \{\mu(1)-\mu(0)\})^2 ] \\ &= \mathop{\mathrm{E}}\qty[ \qty(\{\hat{\mu}(1) - \mu(1)\} - \{\hat{\mu}(0) -\mu(0)\})^2 ] \\ &= \mathop{\mathrm{E}}\qty[ \qty(\{\hat{\mu}(1) - \mu(1)\})^2 ] + \mathop{\mathrm{E}}\qty[ \qty(\{\hat{\mu}(0) -\mu(0)\})^2 ] \\ &- 2\mathop{\mathrm{E}}\qty[ \{\hat{\mu}(1) - \mu(1)\}\{\hat{\mu}(0) -\mu(0)\}] \end{aligned} \tag{23.1}\]
The first two terms here are the ones that appear in our formula above: the variances of the two means. For that formula to be correct, the last term has to be zero. It’s up to you to prove that.
Solution
Locked (Week 5)
Ratios of Means
If \(\hat\mu(1)-\hat\mu(0)\) is a good estimator of \(\mu(1)-\mu(0)\), then shouldn’t \(\hat\mu(1)/\hat\mu(0)\) be a good estimator of \(\mu(1)/\mu(0)\)? Let’s look into it. To do this, we’ll think of the ratio as a function of the two means. \[ \frac{\hat\mu(1)}{\hat\mu(0)} - \frac{\mu(1)}{\mu(0)} = f(\hat\mu(1), \hat\mu(0)) - f(\mu(1), \mu(0)) \qfor f(x,y) = \frac{x}{y}. \]
And we’ll use a linear approximation to this function to think about this difference.
\[ \begin{aligned} f(\hat\mu(1), \hat\mu(0)) \approx f(\mu(1), \mu(0)) &+ \qty[\frac{\partial f}{\partial x}(\mu(1), \mu(0))](\hat\mu(1) - \mu(1)) \\ &+ \qty[\frac{\partial f}{\partial y}(\mu(1), \mu(0))](\hat\mu(0) - \mu(0)) \end{aligned} \]
This approximation should be good if \(\hat\mu(1)\) and \(\hat\mu(0)\) are close to \(\mu(1)\) and \(\mu(0)\).
Solution
Locked (Week 5)
Now that we’ve justified the approximation, let’s use it to analyze our ratio estimator.
Solution
Locked (Week 5)
Solution
Locked (Week 5)
All of that ignores the error of our linear approximation as a potential problem. We should, if we like, be able to reason about this error using tools from calculus.
You can find the figure in the ‘Variance Calculations for Comparisons’ tab at the top of this page. Draw in your interval however you like. You can print this, draw it on paper, and photograph it for submission. You can right click on the plot, save it as an image, and draw on that using your favorite image editor. You can sketch what you see in Figure 2 on paper, add your interval, and photograph that. Maybe the easiest thing to do is use the Tldrawe Chrome Extension to draw right on top of this webpage and take a screenshot. Don’t work too hard. A rough sketch is fine.↩︎