Sampling distributions and confidence intervals

An estimator is a random variable

Here is the conceptual jump that makes statistics click. You compute the sample mean of 30 annual returns and get 8%. That number feels like a fact. But run the tape again — a parallel history with the same true distribution and different luck — and you'd compute 6%, or 11%.

A statistic computed from a sample is itself a random variable. Your dataset is one draw from the space of datasets that could have happened; every number you compute from it inherits that randomness. The distribution of an estimator across hypothetical repeated samples is its sampling distribution, and it is the central object of this lesson: it tells you how far from the truth your one computed number is likely to sit.

The sampling distribution of the mean

For the sample mean of $N$ independent observations, A1-07's CLT hands us the answer directly. $\bar{X}_N$ is approximately normal, centered on the true mean $\mu$, with spread:

$\text{SE}(\bar{X}_N) = \frac{\sigma}{\sqrt{N}}$

This standard error is just the standard deviation of the estimator's own sampling distribution — that's the formal definition. A2-01 used the formula $\sigma/\sqrt{N}$ as a handed-down fact; you now know where it comes from (CLT) and what it means (the typical distance between your computed mean and the truth). In practice $\sigma$ is unknown, so we plug in the sample standard deviation (ddof=1, as always since A2-02).

Confidence intervals — the precise reading

A normal variable lands within 1.96 standard deviations of its center 95% of the time. Applied to the sampling distribution:

$\text{95\% CI} = \hat{\mu} \pm 1.96 \cdot \text{SE}$

Now the subtle part, and it is genuinely subtle — the pretest misconception survives entire careers. The correct reading:

If you repeatedly drew datasets and built an interval from each, 95% of those intervals would contain the true mean. The confidence is a property of the interval-making procedure, not of any single interval it produces.

Saying "the true mean is 95% likely to be in [2%, 14%]" treats the truth as random. In the frequentist framework it isn't — the true mean is a fixed unknown number, and once the interval is computed, it either contains that number or it doesn't. What was random was your data, and the randomness has already been realized. (Making probability statements about the parameter itself is exactly what Bayesian posteriors from A1-06 do — but they charge you a prior for the privilege.)

Working shorthand that stays honest: "values inside the interval are consistent with my data; values outside would be surprising."

Two footnotes for the audit-minded: with small $N$, the correct multiplier comes from the t-distribution (2.05 rather than 1.96 at $N=30$; negligible beyond $N \approx 100$), and when the estimator is something gnarlier than a mean, the bootstrap (Efron & Tibshirani) builds the sampling distribution by resampling your data instead of by formula. Both refine, neither changes, the logic above.

Worked example: how little 30 years tells you

US equity annual returns run at roughly $\hat{\mu} = 8\%$ with $\hat{\sigma} = 17\%$. With $N = 30$ years:

$\text{SE} = \frac{17\%}{\sqrt{30}} \approx 3.1\%, \qquad \text{95\% CI} = 8\% \pm 6.1\% = [1.9\%,\; 14.1\%]$

Sit with that. Three decades of data cannot distinguish "stocks return 2%" from "stocks return 14%" — the difference between a disappointing asset class and a fortune-making one. This is A2-01's "the mean is hard to estimate" made quantitative: the interval is wide because $\sigma \gg \mu$ and because $\sqrt{N}$ grows so grudgingly. To halve this interval you need 120 years, not 60.

Why you cannot validate a strategy in 2 years

The same arithmetic, pointed at your own backtest. Any performance statistic — mean return, Sharpe — has a standard error shrinking only as $1/\sqrt{N}$, and with two years of data $N$ is small where it counts (independent bets, not rows of daily prices). A2-04 ran the numbers for the Sharpe ratio via Lo (2002): five years of a true-Sharpe-1 strategy gives a 95% CI of roughly $[0.1, 1.9]$ — barely excluding zero. Two years is hopeless: the interval swallows both "genuine edge" and "pure noise."

So when someone shows a 2-year track record with Sharpe 1.5, the sampling-distribution reflex should fire: that number is one draw of a random variable whose spread is about as large as the number itself. Confidence intervals are the antidote to point-estimate worship — always report the interval, and always be suspicious of anyone who doesn't.

Try it

Teach it

Explain, as if to a colleague who just proudly showed you a 2-year backtest: (1) why the Sharpe they computed is a draw from a sampling distribution, (2) what its confidence interval roughly looks like, and (3) the precise — not the sloppy — meaning of that interval. Write it out in 4–5 sentences; the precision of point (3) is the test.