Sampling distributions and confidence intervals

An estimator is a random variable

Here is the conceptual jump that makes statistics click. You compute the sample mean — the plain average — of 30 annual returns and get 8%. That number feels like a fact. But run the tape again — a parallel history with the same true distribution and different luck — and you'd compute 6%, or 11%.

A statistic computed from a sample is itself a random variable. Your dataset is one draw from the space of datasets that could have happened. Every number you compute from it inherits that randomness. A number computed from data in order to estimate some true quantity is called an estimator — the sample mean is the classic example. The distribution an estimator would have across hypothetical repeated samples is its sampling distribution. That distribution is the central object of this lesson: it tells you how far from the truth your one computed number is likely to sit.

The sampling distribution of the mean

Take the sample mean of $N$ independent observations. One new piece of notation first: $\bar{X}_N$ is read "X-bar-N." The bar drawn over the $X$ means "the average of the X's," and the subscript $N$ says the average was built from $N$ observations. A1-07's central limit theorem (the CLT) hands us the answer directly: $\bar{X}_N$ is approximately normal — bell-shaped — centered on the true mean $\mu$ (mu), with spread:

$\text{SE}(\bar{X}_N) = \frac{\sigma}{\sqrt{N}}$

In words: the standard error of X-bar-N equals sigma divided by the square root of N. Take the spread of one observation ($\sigma$), divide by the square root of how many observations you averaged, and you get the typical error of the average itself.

This standard error (SE) is just the standard deviation of the estimator's own sampling distribution — that's the formal definition. Note what the square root does to you: quadruple your data and the error only halves, because the square root of 4 is 2. A2-01 used the formula $\sigma/\sqrt{N}$ as a handed-down fact. You now know where it comes from (the CLT) and what it means (the typical distance between your computed mean and the truth). In practice $\sigma$ is unknown, so we plug in the sample standard deviation (ddof=1, as always since A2-02).

Confidence intervals — the precise reading

A fact about the normal (bell-curve) distribution: a normal variable lands within 1.96 standard deviations of its center 95% of the time. That 1.96 is called a z-value — the number of standard deviations you must move out from the center to capture a chosen share of outcomes. Apply this to the sampling distribution and you get a confidence interval (CI) — a range of values that your data is consistent with:

$\text{95\% CI} = \hat{\mu} \pm 1.96 \cdot \text{SE}$

In words: the 95 percent confidence interval is mu-hat, plus or minus 1.96 times the standard error. Two new symbols. First, $\hat{\mu}$ is read "mu-hat" — a hat drawn over a Greek letter always means "our estimate of this quantity, computed from data," as opposed to the unknown true value $\mu$. Second, $\pm$ is read "plus or minus": compute the value both ways, and you get the low end and the high end of the range.

Now the subtle part, and it is genuinely subtle — the pretest misconception survives entire careers. The correct reading:

If you repeatedly drew datasets and built an interval from each, 95% of those intervals would contain the true mean. The confidence is a property of the interval-making procedure, not of any single interval it produces.

Saying "the true mean is 95% likely to be in [2%, 14%]" treats the truth as random. In the frequentist framework — the school of statistics where probability describes only repeatable random draws, never fixed unknowns — it isn't random. The true mean is a fixed unknown number. Once the interval is computed, it either contains that number or it doesn't. What was random was your data, and that randomness has already been realized. (Making probability statements about the unknown quantity itself is exactly what Bayesian posteriors from A1-06 do — the rival school, in which a posterior is a probability statement about the unknown — but they charge you a prior, an assumed starting belief, for the privilege.)

Working shorthand that stays honest: "values inside the interval are consistent with my data; values outside would be surprising."

Two footnotes for the audit-minded. First: with small $N$, the correct multiplier is not 1.96 but comes from the t-distribution — a slightly fatter-tailed cousin of the normal curve that accounts for having to estimate $\sigma$ from the same data. At $N=30$ the multiplier is 2.05 rather than 1.96; the difference is negligible beyond $N \approx 100$ (the symbol $\approx$ is read "approximately equal to"). Second: when the estimator is something gnarlier than a mean, the bootstrap (Efron & Tibshirani) builds the sampling distribution by repeatedly resampling your own data instead of by formula. Both refine, neither changes, the logic above.

Worked example: how little 30 years tells you

US equity annual returns run at roughly $\hat{\mu} = 8\%$ with $\hat{\sigma} = 17\%$ ($\hat{\sigma}$ is read "sigma-hat" — the spread, as estimated from data). With $N = 30$ years:

$\text{SE} = \frac{17\%}{\sqrt{30}} \approx 3.1\%, \qquad \text{95\% CI} = 8\% \pm 6.1\% = [1.9\%,\; 14.1\%]$

In words: the standard error is 17% divided by the square root of 30, which is about 3.1%. Multiply that by 1.96 and you get about 6.1%. So the 95% confidence interval is 8% plus or minus 6.1% — a range running from 1.9% up to 14.1%.

Sit with that. Three decades of data cannot distinguish "stocks return 2%" from "stocks return 14%" — the difference between a disappointing asset class and a fortune-making one. This is A2-01's "the mean is hard to estimate" made quantitative. The interval is wide for two reasons. First, $\sigma \gg \mu$ — read "sigma is much greater than mu": the year-to-year spread (17%) dwarfs the average itself (8%). Second, $\sqrt{N}$ grows grudgingly. To halve this interval you need 120 years, not 60.

Why you cannot validate a strategy in 2 years

The same arithmetic, pointed at your own backtest — a simulation of a trading strategy on historical data. Any performance statistic has a standard error shrinking only as $1/\sqrt{N}$. That includes the mean return and the Sharpe ratio — a strategy's average return divided by its volatility, a return-per-unit-of-risk score. And with two years of data, $N$ is small where it counts: what matters is the number of independent bets, not the number of rows of daily prices. A2-04 ran the numbers for the Sharpe ratio via Lo (2002): five years of a strategy whose true Sharpe is 1 gives a 95% CI of roughly $[0.1, 1.9]$ — barely excluding zero. Two years is hopeless: the interval swallows both "genuine edge" and "pure noise."

So when someone shows a 2-year track record with Sharpe 1.5, the sampling-distribution reflex should fire: that number is one draw of a random variable whose spread is about as large as the number itself. Confidence intervals are the antidote to point-estimate worship (a point estimate is a single number reported with no measure of its uncertainty). Always report the interval, and always be suspicious of anyone who doesn't.

Try it

Teach it

Explain, as if to a colleague who just proudly showed you a 2-year backtest: (1) why the Sharpe they computed is a draw from a sampling distribution, (2) what its confidence interval roughly looks like, and (3) the precise — not the sloppy — meaning of that interval. Write it out in 4–5 sentences; the precision of point (3) is the test.