$$ \newcommand{X}{} \newcommand{Y}{}
$$
\[
\DeclareMathOperator{\mathop{\mathrm{E}}}{E}
\DeclareMathOperator{\mathop{\mathrm{\mathop{\mathrm{V}}}}}{V}
\DeclareMathOperator{\mathop{\mathrm{sd}}}{sd}
\DeclareMathOperator{\mathop{\mathrm{bias}}}{bias}
\newcommand{\thetaprior}{\theta^{\text{prior}}}
\newcommand{\nprior}{n^{\text{prior}}}
\]
Setup
In an earlier homework, we studied a biased estimator that mixes \(\nprior\) pseudo-observations with prior mean \(\thetaprior\) into the sample mean. When estimating a population proportion \(\theta\) from a sample \(Y_1 \ldots Y_n\) drawn with replacement, this gives us the estimator \[
\tilde{Y}_{\nprior} = \frac{\nprior \thetaprior + n\bar{Y}}{\nprior + n}.
\] We showed that it has bias \(\frac{\nprior(\thetaprior - \theta)}{\nprior + n}\) and standard deviation \(\sqrt{\frac{n\theta(1-\theta)}{(\nprior+n)^2}}\).
Now we’ll do the same thing with two groups. We have a population \((x_1, y_1) \ldots (x_m, y_m)\) where \(x_j \in \{0,1\}\) indicates group membership. We sample \((X_1, Y_1) \ldots (X_n, Y_n)\) with replacement. The subsample means are \(\hat\mu(0)\) and \(\hat\mu(1)\), and we’re interested in the difference \(\hat\mu(1) - \hat\mu(0)\).
Instead of using the subsample means, we’ll mix in \(\nprior\) pseudo-observations to each group separately. For group \(x \in \{0,1\}\), we use \[
\tilde\mu(x) = \frac{\nprior \thetaprior(x) + N_x \hat\mu(x)}{\nprior + N_x}
\] where \(N_x = \sum_{i=1}^n 1_{=x}(X_i)\) is the number of observations in group \(x\) and \(\thetaprior(x)\) is the prior mean for that group.
Problem 1: Bias of Group Means
Exercise 26.1 (Part A)
Conditional on \(X_1 \ldots X_n\), calculate the bias of \(\tilde\mu(x)\) as an estimator of \(\mu(x)\).
Exercise 26.2 (Part B)
Calculate the bias of the difference \(\tilde\mu(1) - \tilde\mu(0)\) as an estimator of \(\mu(1) - \mu(0)\), conditional on \(X_1 \ldots X_n\).
Problem 2: Same Direction vs. Opposite Direction
Exercise 27.1 (Part A)
Suppose \(\mu(0) = 0.6\) and \(\mu(1) = 0.7\), so the true difference is \(0.1\). Consider two choices of prior:
- \(\thetaprior(0) = \thetaprior(1) = 0.5\) (the same prior for both groups, below both means)
- \(\thetaprior(0) = 0.7\) and \(\thetaprior(1) = 0.5\) (different priors — the prior for group 0 is too high, the prior for group 1 is too low)
For each choice, with \(\nprior = 10\) and \(N_0 = N_1 = 200\), compute the bias of each group mean and the bias of the difference \(\tilde\mu(1) - \tilde\mu(0)\).
Exercise 27.2 (Part B)
In choice 1, each group mean is biased. Why is the difference barely biased?
Exercise 27.3 (Part C)
In choice 2, the individual biases are no larger than in choice 1. Why is the bias of the difference so much worse?
Problem 3: Wrong in the Same Direction
Exercise 28.1
Explain, in a sentence or two, the punchline of this homework: when is it OK to use a biased estimator for a comparison?
Problem 4: Simulation
Exercise 29.1
Using the GA turnout data, simulate the coverage of three interval estimators for the difference in turnout between Black and non-Black voters:
- The unregularized difference in subsample means \(\hat\mu(1) - \hat\mu(0)\)
- The regularized difference \(\tilde\mu(1) - \tilde\mu(0)\) with \(\nprior = 10\) and \(\thetaprior = 1/2\) for both groups
- The regularized difference with \(\nprior = 10\) and \(\thetaprior(0) = 0.6\), \(\thetaprior(1) = 0.8\) (i.e. different priors for the two groups)
For each, draw 10,000 samples of size \(n = 625\) with replacement, compute a 95% confidence interval \(\hat\Delta \pm 1.96\hat\sigma\) where \(\hat\sigma\) is the bootstrap standard deviation, and report the coverage.