42 Imagining Randomization in Observational Studies

Review: Conditional Randomization

A Personalized Dosing Study

We have patients with joint pain.
They report some covariates \(X_i\), e.g. pain level, age, weight, etc.
We want to know how much naproxen to prescribe, \(W_i\).

Study Design.

Propose a distribution of doses, \(p_{X_i}\), as a function of the covariates.
Assign doses \(W_i\) randomly from \(p_{X_i}\).
Ask what happens when people receive doses drawn from another, \(q_{X_i}\).

Causal Question. What would the mean outcome be if patients got treatments drawn from \(q_{X_i}\)? \[ \text{What is } \quad \mathbb{E}[ Y_i(\tilde W_i) ] \quad \text{ for } \quad \tilde W_i \sim q_{X_i}? \]

Identification

\[ \mathbb{E}[ Y_i(w) \mid W_i=w', X_i] = \mathbb{E}[ Y_i(w) \mid W_i=w, X_i] = \mathbb{E}[Y_i \mid W=w, X_i] \]

People with matching covariates who do take different doses would have had the same outcome, on average, if they had taken the same dose.

In other words, the only salient difference between patients with the same covariates who take 500 and 1000mg is what they actually took.
If they’d taken the same thing, they’d have the same average outcome.
This is true if doses are randomized, given covariates, in the same way.

By the tower rule, it implies the equality of two means. \[ \mathbb{E}[Y_i(w) \mid X_i] = \mathbb{E}[\ \mathbb{E}[ Y_i(w) \mid W_i, X_i] \mid X_i] = m(w,X_i) \quad \text{ where } \quad m(w,x) = \mathbb{E}[Y_i \mid W_i=w, X_i=x]. \]

From Experiments to Observational Studies

Why Would We Randomize as a Function of the Covariates?

It’s better for patients.
- We can give them doses close to what we think is best for each.
It’s more informative.
- Ultimately we’re going to dose each patient as we think is best.
- This way we see more data about what happens when we do that.

This is also an approximation to what happens in observational studies.

People choose a dose of naproxen based on …
- how much pain they’re in
- how various doses have worked in the past
- cultural expectations
We can encode these things in covariates.
For this reason, it’s common to act as if treatment were randomized as a function of covariates, but we don’t know the treatment distribution \(p_{X_i}\).

Acting As If Observational Studies Are Randomized

There are a few pitfalls.

If we don’t know the treatment distribution \(p_{X_i}\), we can’t eliminate bias using the weighted least squares trick.
- We can, however, use the trick with an estimate of \(p_{X_i}\).
- This doesn’t eliminate bias entirely, but it often substantially reduces it.
We may not have all the information used to choose treatment.
- As a result, the answer to our statistical question, based on this flawed premise, may differ from the answer to our causal question.
- This is a generalization of omitted variable bias.
- We can often show that it is implausible that this difference is large.
- This is called causal sensitivity analysis.
We may have information that’s downstream of treatment.
- If we act as if treatment were randomized as a function of one of its consequences, we can explain away part of the treatment.
- For example, many anxiety drugs are used as sedatives in higher doses.
- I could compare the efficacy of two doses only when they induce equal levels of sedation by including sedation as a covariate in my regression.
- If I did this, I might find the lower dose more effective on anxiety simply because the higher dose is sedating except when anxiety is most severe.

Confounding and Model Misspecification

Confounded Assignment

Model Misspecification

Two models for \(\mu(w,x) = \mathbb{E}\{Y \mid W=w, X=x\}\):

DIM (constant per group) and linear (\(Y \sim X \cdot W\)).
Neither captures the true conditional means — both are misspecified.
The question is what this does to inference, not just estimation.

Bias Doesn’t Go Away

DIM and linear model: both misspecified, both biased.
On the estimation scale (\(\hat\tau - \tau\)): variance shrinks with \(n\) — looks like convergence.
On the inference scale (\(\sqrt{n}(\hat\tau - \tau)\)): the bias doesn’t shrink, so it dominates. More data makes things worse on the inference scale.

A Problematic Randomized Study

Unknown Treatment Distribution

Goal. Same as before — patients with joint pain, want to know how much naproxen to prescribe.

Study Design.

Assign doses \(W_i\) randomly from one probability distribution, \(p\).
Ask what happens when people receive doses drawn from another, \(q\).
Forget to write down \(p\), leaving whoever’s analyzing it to guess.

Everything is the same except that we don’t know \(p\). This means that when we go to do weighted least squares, we don’t know the weights. We’ll use some guess \(\tilde p\).

How Much Does This Matter?

Forget the statistical part — that’s fine no matter what the weights are. Let’s look at the modeling part. \[ \begin{aligned} \text{modeling part} &= \int \{\tilde\mu(w) - \mu(w)\}q(w)dw \\ &= \int \frac{q(w)}{\tilde p(w)} \{\tilde\mu(w) - \mu(w)\} p(w) dw \\ &+ \int \left\{ \frac{q(w)}{p(w)}-\frac{q(w)}{\tilde p(w)}\right\} \{\tilde\mu(w) - \mu(w)\} p(w) dw \\ &= \text{zero term} + \text{new term}. \end{aligned} \]

The Product Structure

\[ \text{modeling error} = \int \frac{\{\tilde p(w) - p(w)\} \{\tilde\mu(w) - \mu(w)\}}{\tilde p(w)} q(w) dw \]

If we think of \(\tilde p\) as the limit of a fitted curve, then our estimate is good if

Our model for the outcome is good, so \(\tilde \mu \approx \mu\).
Our model for the treatment is good, so \(\tilde p \approx p\).
Neither is good, but the errors are orthogonal to each other.

Imagining Randomization

What’s Random in an Observational Study?

Sampling? Not randomly sampling states from a superpopulation.
Randomization? Nobody randomized which states adopted a policy.
Model? \(Y_{it} = f(u_i, t) + \epsilon_{it}\), but what is \(\epsilon_{it}\)?

Whatever statistical claims we make are based on a random mechanism we’re imagining.

The Proposal

Imagine that treatment is randomized with idiosyncratic probabilities: \[ D_i \sim \text{Bernoulli}(\pi_i) \qquad \text{independently across } i \]

\(\pi_i\) unknown, unit-specific, can depend on unobserved \(\eta_i\).
The design (units, outcomes) is fixed. The assignment is random.
I can imagine a legislator’s plane getting delayed and a vote shifting. I can’t imagine a distribution over hypothetical states.

What the Confidence Interval Means

A 95% CI means: for any \(\pi\) consistent with the data, this interval covers the true effect 95% of the time. \[ \inf_{\pi \in \Pi} \; P_\pi\!\left(\tau \in \text{CI}\right) \geq 0.95 \]

A real probabilistic statement — not a model-based approximation, not a rough guide.
The probabilities come from a specific, stated source: the imagined assignment mechanism.

The Essential Approach to Observational Causal Inference

The approach has three steps. First, learn to predict outcomes for all combinations of treatment and covariates. Find \(\hat\mu(w,x)\) with \(\hat\mu(w,x) \approx Y_i\) when \(X_i=x, W_i=w\).

Second, compare predictions for different treatments to estimate the effect of treatment, \[ \hat\tau(X_i) = \hat \mu(1,X_i) - \hat \mu(0,X_i). \]

Third, average these comparisons to estimate a summary of the effect, \[ \begin{aligned} \text{ATE} &\approx \frac{1}{n} \sum_{i=1}^n \hat\tau(X_i) && \text{ the average over all people in the sample} \\ \text{ATT} &\approx \frac{1}{N_1} \sum_{i:W_i=1} \hat\tau(X_i) && \text{ the average over the treated people in the sample.} \end{aligned} \]

Sensitivity Analysis

Tobacco lobbyists suggested that people who smoke may develop cancer frequently not because smoking causes cancer, but because some factor common among people who smoke does. A hormone, perhaps.

It’s not possible to rule this out entirely without a randomized study.
But because smoking obviously causes lung cancer, that’s unethical.
Ultimately, the issue was settled by the argument that any such common factor would have to be implausibly prevalent among smokers.

If cigarette smokers have 9 times the risk of nonsmokers for developing lung cancer, and this is not because cigarette smoke is a causal agent, but only because cigarette smokers produce hormone X, then the proportion of hormone-X-producers among cigarette smokers must be at least 9 times greater than that of nonsmokers. (Cornfield et al., JNCI 1959).

Exercise. Suppose an observational study finds that treated units have an outcome rate 4 times higher than controls. A skeptic claims a confounder \(H\) explains the difference.

If 10% of controls have the confounder, what’s the minimum prevalence among treated units needed to explain the 4x ratio?
How does your answer change if the observed ratio is 9x instead of 4x?
At what point does the required confounder prevalence become implausible? What makes it implausible?