11 Homework — Random Variables, Moments, and the Standard Deviation
Reminder: put each answer in its answer block. Working on paper is fine—photograph it and include it with  (image in the same folder as this file).
\[ \newcommand{\thetaprior}{\theta^{\text{prior}}} \newcommand{\nprior}{n^{\text{prior}}} \newcommand{\yprior}{y^{\text{prior}}} \]
Warm-up: Computing Moments
Recognition
Same Move, Many Surfaces
Indicator Counting (Cards)
The indicator-counting technique in this section follows Blitzstein and Hwang, Introduction to Probability, 2nd ed. (2019), Chapter 4.
Tell a Story
Bias and Consistency
Using Prior Information
So far, we’ve exclusively talked about unbiased estimators. That is, estimators with the property that their expected value is equal to the estimation target. \[ \hat\theta \qqtext{ is called unbiased if } \E[\hat{\theta}] = \theta. \] We say an estimator is biased if this isn’t true. To get a sense of what that means, let’s consider a simple example of a biased estimator. \[ \tilde{Y}_1 = \frac{1}{n+1}\cdot \{\frac{1}{2} + \sum_{i=1}^{n}Y_i\} \]
This is a specific example of a more general estimator that integrates information from a prior study. Maybe a real study or maybe a study we’re imagining. Suppose we have \(\nprior\) observations \(\yprior_1 \ldots \yprior_{\nprior}\) from this study1 \[ \thetaprior = \frac{1}{\nprior}\sum_{i=1}^{\nprior}\yprior_i. \]
Then averaging all our observations — from our current study and this prior one — gives us the following estimator.
\[ \begin{aligned} \tilde{Y}_{\nprior} &= \frac{1}{\nprior + n} \qty{\sum_{i=1}^{\nprior}\yprior_i + \sum_{i=1}^{n} Y_i} \\ &= \frac{1}{\nprior + n} \qty{ \nprior\thetaprior + n\bar Y } \end{aligned} \]
We treat these prior observations, and therefore their mean \(\thetaprior\), as deterministic. The simple example we started with, \(\tilde{Y}_1\), is a special case where we have a single prior observation \(\yprior_1 = \frac{1}{2}\).
Visualizating the Impact of Prior Information
Lets’s try to get a sense of how using prior information like this impacts our inference. We’ll do it visually. The plot below shows the sampling distributions of three estimators of a population mean \(\theta\) at three sample sizes \(n\): 10, 40, and 160. These are the estimators.
- The estimator \(\hat\theta_1 = \tilde{Y}_{\nprior}\) for \(\thetaprior=3/4\) and \(\nprior=10\) prior observations.
- The estimator \(\hat\theta_2 = \tilde{Y}_{n}\) for \(\thetaprior=3/4\) and \(\nprior=n\) prior observations—the bigger the sample, the more prior observations we incorporate.
- The sample mean, \(\hat\theta_3 = \bar Y\).
As usual, the estimation target \(\theta\) is indicated by a green line, the sampling distribution’s mean by a solid blue line, and its mean plus and minus two standard deviations by dotted blue lines. In our first exercise, we’ll match estimators to pictures.
Convergence in Probability
Now let’s think about another notion of convergence. If an estimator’s sampling distribution ends up in the right location (\(\text{bias} \to 0\)) with arbitrarily little spread (\(\text{variance} \to 0\)), then it makes sense that any draw from it will be close to the estimation target \(\mu\).
Recall that you can visualize your estimator as a single dot drawn from its sampling distribution. Like the black one below. And if you want to think of what happens if you were to repeat your survey 100 times, you can visualize the results by plotting 100 dots. Like the purple ones.
We formalize this idea by talking about convergence in probability. We say an estimator \(\hat\mu\) converges in probability to \(\mu\) if, for anyone’s idea of `sufficiently close’, the probability that \(\hat\mu\) is sufficiently close to \(\mu\) goes to one. Or equivalently, the probability that it isn’t sufficiently close goes to zero. \[ P(\lvert\hat\mu - \mu\rvert \ge \epsilon) \to 0 \qqtext{ as } n \to \infty \]
Convergence in mean square implies convergence in probability. Let’s see why. We’ll use Markov’s inequality.
Markov’s Inequality
Markov’s inequality says that the probability a nonnegative random variable \(X\) exceeds some threshold \(t\) is bounded by the ratio of its expected value and \(t\), i.e., \[ P(X \geq t) \leq \frac{\E[X]}{t}\] Often, instead of using this to bound the random variable we’re interested in directly, e.g. \(X=\lvert\hat\mu - \mu\rvert\), we use it to bound its square. \(X \ge t\) if and only if \(X^2 \ge t^2\), so the probability that \(X \ge t\) is the same as the probability that \(X^2 \ge t^2\). Applying Markov’s inequality to the random variable \(X^2\) gives us a bound in terms of \(X\)’s mean square. \[ P(X \ge t) = P(X^2 \ge t^2) \leq \frac{\E[X^2]}{t^2} \qqtext{ e.g.} P(\lvert\hat\mu - \mu\rvert \ge \epsilon) = P((\hat\mu - \mu)^2 \ge \epsilon^2) \leq \frac{\E[(\hat\mu - \mu)^2]}{\epsilon^2} \]
This tells us, in terms of the mean squared error of \(\hat\mu\), how likely it is that \(\hat\mu\) is far from \(\mu\). And, in particular, it tells us that if the mean squared error goes to zero, then the probability that \(\hat\mu\) is any distance \(\epsilon\) away from \(\mu\) goes to zero.
Now we’re going to do two things. First, we’re going to prove Markov’s inequality. It’s good practice working with expectations and probabilities. Second, we’re going to look at what Markov’s inequality tells us about how close \(\hat\mu\) is to \(\mu\), i.e., we’re going to use it to calculate the coverage of an interval estimate.
First we define the following two functions, \(f\) and \(g\), with \(t, x\ge 0\). \[ f(x,t) = \frac{x}{t} \] \[ g(x,t) = 1_{\ge t}(x) = \begin{cases} 1 & \text{ if } x \ge t \\ 0 & \text{ otherwise } \end{cases} \]
Markov’s Inequality and Interval Estimation
So far, when we’ve calculated the coverage of an interval estimate in terms of its mean and variance, we’ve relied on normal approximation. That is, we’ve been using a different formula for \(P(\lvert\hat\mu - \mu\rvert \ge \epsilon)\) based on the idea that \(\hat\mu-\mu\) has approximately the same distribution as a normally-distributed random variable \(Z\) with the same mean—zero—and variance \(\sigma_{\hat\mu}^2=\Var[\hat\mu-\mu]\). We can use this approximation, in combination with our knowledge of the probability density of the normal approximation, to get a bound on the probability that \(\lvert\hat\mu - \mu\rvert \ge \epsilon\).
\[ P(\abs{\hat\mu - \mu} \ge \epsilon) \approx P(\abs{Z} \ge \epsilon) = 1-P(\abs{Z} < \epsilon) = 1- \int_{-\epsilon}^{\epsilon} f_{\sigma_{\hat\mu}}(z) dz \qfor f_{\sigma}(z) = \frac{1}{\sqrt{2\pi}\sigma} e^{-z^2/(2\sigma^2)} \]
The reason we’re always talking about intervals of the form \(\hat\mu \pm 2 \sigma_{\hat\mu}\) is that, using this approximation, that yields 95% coverage. That is, \(P(\abs{Z} < 2 \sigma_{\hat\mu}) \approx .95\). To see why we tend to use normal approximation instead of Markov’s inequality, let’s see what Markov’s inequality tells us about how close an estimator is to its estimation target and about the coverage of this interval.
Comparing Interval Calibration Methods
Paley–Zygmund: The Other Half
Sometimes we call these pseudo-observations. This particular interpretation, in which we think of them as coming from a prior study, can be used to derive this estimator from Bayesian principles.↩︎