20 Meeting 14: Why the CLT Works — Stein’s Method
Th Oct 15
Overview
Throughout this unit, we’ve used the Central Limit Theorem as a black box: sample means are approximately normal when \(n\) is large enough. But why? And what does “large enough” mean?
This lecture opens up the black box. We’ll prove the CLT using Stein’s method—an approach that:
- Shows why a particular distribution is close to normal (not just that it converges)
- Gives actual bounds on the approximation error
- Uses an intuitive idea: if swapping observations doesn’t change the distribution much, you’re close to normal
You won’t be tested on this. But knowing that the CLT is a real theorem with a real proof—one you could understand if you sat with it—is valuable.
The Setup
Let \(X_1 \ldots X_n\) be independent copies of a random variable with mean \(\mu\) and finite, positive variance \(\sigma^2\). Their standardized mean is \[ W_n = \frac{\bar X_n-\mu}{\sigma/\sqrt n} = \frac{1}{\sqrt n}\sum_{i=1}^n \frac{X_i-\mu}{\sigma}. \]
The Central Limit Theorem says that the distribution of \(W_n\) approaches the standard normal distribution as \(n\) increases.
The finite-variance assumption is enough for this Central Limit Theorem. The quantitative Stein bound proved later in this lecture uses the stronger assumption that the fourth moment is finite.
This statement has two important parts.
- We center by the actual mean \(\mu\) and measure distance in the actual standard error \(\sigma/\sqrt n\).
- For every fixed number \(x\), \(\Pr(W_n\leq x)\) approaches \(\Pr(Z\leq x)\), where \(Z\) is standard normal.
The second part is called convergence in distribution.
The theorem says the error vanishes eventually. It does not tell us whether \(n=30\) or \(n=3{,}000\) is enough. To answer that question, we need a result that measures error.
Stein’s Characterization of the Normal
Start with a fact about a standard normal random variable \(Z\). Its density is \(\phi(z)=\exp(-z^2/2)/\sqrt{2\pi}\), so \(\phi'(z)=-z\phi(z)\). Integration by parts implies \[ \E[Zf(Z)]=\E[f'(Z)] \] for every smooth function \(f\) for which the expectations exist.
The useful part is the converse: this identity characterizes the standard normal distribution. A mean-zero, variance-one random variable \(W\) is standard normal if \[ \E[f'(W)-Wf(W)]=0 \] for every suitable function \(f\).
That turns a shape question into a calculation. If \(\E[f'(W)-Wf(W)]\) is small for a sufficiently rich collection of functions \(f\), then the distribution of \(W\) is close to standard normal. Stein’s method is the practice of finding a useful way to rewrite and bound that difference.
The Exchangeable Pairs Approach
A pair \((W,W')\) is exchangeable when swapping its order does not change its joint distribution: \[ (W,W')\ \stackrel{d}{=}\ (W',W). \]
For a sample mean, there is a natural way to make such a pair. Choose one observation at random and replace it with an independent copy. It does not matter which member of the pair we call the original and which we call the replacement, so the pair is exchangeable.
Exchangeability alone is not enough. The replacement must also pull predictably toward zero. A pair is an \(a\)-Stein pair when \[ \E[W'-W\mid W]=-aW \] for some number \(0<a\leq 1\). The expected change points back toward zero, and its size is proportional to the current value of \(W\).
For a mean-zero, variance-one random variable \(W\), Ross’s exchangeable-pairs theorem bounds the distance to a standard normal random variable \(Z\) by \[ d_W(W,Z) \leq \frac{\sqrt{\Var\!\left(\E[(W'-W)^2\mid W]\right)}}{\sqrt{2\pi}\,a} +\frac{\E|W'-W|^3}{3a}. \]
Here \(d_W\) measures the largest error in expectations of functions that cannot change faster than their input. The two terms say what can go wrong:
- the squared step size can depend too much on the current value of \(W\); or
- one replacement can move \(W\) too far.
“Swapping one observation does not change the mean much” is therefore not the theorem. It is the mechanism that makes both error terms small.
Applying to Sample Means
Two facts about conditional expectation will be used below. Let \(S\) denote the whole sample, and let \(W\) be a function of \(S\).
- Iterated conditioning implies \(\E[\E[U\mid S]\mid W]=\E[U\mid W]\).
- Conditional expectation reduces variance: \(\Var(\E[U\mid W])\leq\Var(\E[U\mid S])\). To see this, set \(V=\E[U\mid S]\). Then \(\E[V\mid W]=\E[U\mid W]\), and the conditional-variance identity gives \(\Var(V)=\Var(\E[V\mid W])+\E[\Var(V\mid W)]\).
To keep the calculation visible, suppose \(X_1 \ldots X_n\) are independent copies with \[ \E[X_i]=0,\qquad \Var(X_i)=1,\qquad \E[X_i^4]<\infty, \] and define \[ W=\frac{1}{\sqrt n}\sum_{i=1}^n X_i. \]
Choose an index \(I\) uniformly from \(1 \ldots n\), independently draw a new copy \(X_I'\), and replace only that observation: \[ W'=W-\frac{X_I}{\sqrt n}+\frac{X_I'}{\sqrt n}. \]
The pair \((W,W')\) is exchangeable. Conditional on the whole original sample, \[ \begin{aligned} \E[W'-W\mid X_1 \ldots X_n] &=\frac{1}{n}\sum_{i=1}^n \frac{\E[X_i'-X_i\mid X_1 \ldots X_n]}{\sqrt n}\\ &=-\frac{1}{n}\frac{1}{\sqrt n}\sum_{i=1}^n X_i\\ &=-\frac{W}{n}. \end{aligned} \] Conditioning only on \(W\) leaves the same result, so this is a \(1/n\)-Stein pair.
Now look at the two errors in the theorem.
One replacement changes \(W\) by \((X_I'-X_I)/\sqrt n\), so \[ \frac{\E|W'-W|^3}{3a} =\frac{\E|X_1'-X_1|^3}{3\sqrt n}. \]
For the squared step, conditioning on the whole sample implies \[ \E[(W'-W)^2\mid X_1 \ldots X_n] =\frac{1}{n^2}\sum_{i=1}^n(1+X_i^2). \] Conditioning on the whole sample can only make its conditional expectation more variable than conditioning on \(W\) alone. Independence therefore implies \[ \Var\!\left(\E[(W'-W)^2\mid W]\right) \leq \frac{\Var(X_1^2)}{n^3}. \]
Substituting these two calculations and \(a=1/n\) into the theorem produces the concrete bound \[ d_W(W,Z) \leq \frac{\sqrt{\Var(X_1^2)}}{\sqrt{2\pi n}} +\frac{\E|X_1'-X_1|^3}{3\sqrt n}. \]
Therefore \[ d_W(W,Z)=O(n^{-1/2}). \]
This is a quantitative Central Limit Theorem for the finite-fourth-moment case. It proves that the approximation improves with \(n\) and identifies the features that govern the constant: large third and fourth moments make the improvement slower.
Takeaways
The normal approximation is not magic and “large \(n\)” is not a universal number.
- The normal distribution is characterized by the identity \(\E[f'(Z)-Zf(Z)]=0\).
- Replacing one summand creates an exchangeable pair whose expected motion points toward zero.
- If the replacement step is small and its squared size is stable, the standardized sum is close to normal.
- For independent copies with a finite fourth moment, the bound above shrinks at rate \(n^{-1/2}\).
This result explains the normal approximation itself. Bootstrap calibration is a separate approximation: it also requires the distribution made by resampling the observed data to track the sampling distribution made by drawing from the population.
References
- Nathan Ross, “Fundamentals of Stein’s method”, especially Section 3.3 and Example 3.6.