The Law of Large Numbers and the Central Limit Theorem

The Law of Large Numbers

Take $N$ independent draws — draws where no outcome influences any other — from any distribution with true mean $\mu$. The Law of Large Numbers (LLN) says the sample mean — the plain average of the draws you actually got — converges to the truth:

$\bar{X}_N = \frac{1}{N}\sum_{i=1}^{N} X_i \;\longrightarrow\; \mu \quad \text{as } N \to \infty$

In words: X-bar-N (the bar over the X marks "average of") is the sum of all N draws, divided by N. The long arrow reads "approaches". The $\infty$ symbol reads "infinity". So: as the number of draws N grows without limit, the running average homes in on the true mean mu.

Concrete example: average 10 rolls of a fair die and you might get 4.1; average 10,000 rolls and you will sit within a whisker of the true mean, 3.5.

This is why casinos are not gambling. On American roulette, a one-dollar bet on red wins with probability 18/38 — 18 winning slots out of 38. The house's expected take is therefore $2/38 \approx 5.3$ cents per dollar (the $\approx$ sign reads "approximately equals"). Any single spin is nearly a coin flip, and any single gambler can walk away rich. Over a million spins, though, the average take per spin locks onto 5.3 cents with almost no wiggle. The casino doesn't need to win each bet; it needs volume.

Trading translation: a strategy's true edge only reveals itself over many independent trades. Ten winning trades tell you almost nothing — that's the gambler walking away rich. A thousand trades start to pin down whether $\mu > 0$, i.e. whether the true average profit per trade is actually positive. This is also the honest reading of A2-01's warning that mean returns are hard to estimate. The LLN guarantees convergence, but says nothing about it being fast.

The Central Limit Theorem, stated properly

The LLN says where the sample mean goes. The Central Limit Theorem (CLT) says how it fluctuates on the way. In plain terms:

Take $N$ independent draws from any distribution with mean $\mu$ and finite standard deviation $\sigma$. For large $N$, the sample mean $\bar{X}_N$ is approximately normally distributed, centered at $\mu$, with standard deviation $\sigma / \sqrt{N}$.

In words: average enough draws from any source, and the averages themselves pile up in the bell-curve shape (that is what "normally distributed" means). The bell sits on top of the true mean $\mu$. Its width is $\sigma / \sqrt{N}$, read "sigma divided by the square root of N" — the original spread, shrunk by root-N. "Finite standard deviation" means the spread is some definite number, not infinite.

Every clause earns its keep:

• Any distribution. Uniform, coin flips, skewed (lopsided) payoffs — the average comes out bell-shaped regardless. This is why the normal distribution — the bell curve — is everywhere: it's the universal shape of aggregated noise.
• Centered at $\mu$ — that's the LLN inside the CLT.
• Spread $\sigma/\sqrt{N}$ — quadruple the data, halve the noise, because the square root of 4 is 2. This $\sqrt{N}$ is the same one behind A2-01's standard error and A2-02's annualization. It is arguably the most important fact in all of statistics: precision grows only as the square root of effort.
• Finite $\sigma$, independent draws — the fine print. Both clauses get violated in finance, which is where the next section lives.

Paying off two IOUs

Two earlier lessons name-dropped the CLT and promised an explanation. Debts now settled:

• A1-03 claimed sample averages and derived statistics — Sharpe estimates (a return-per-unit-of-risk ratio) and regression coefficients (fitted slopes; both formally later) — are approximately normal even when individual returns aren't. That's the CLT: those statistics are all averages in disguise, and averaging manufactures normality.
• A2-03 claimed monthly returns look more normal than daily returns. A monthly log-return — a return measured with logarithms, from C1-01 — is the sum of ~21 daily ones. That is a partial CLT aggregation. More aggregation (yearly), more normal.

What the CLT does not do

The CLT is about averages of many draws. Tomorrow's return is one draw. No averaging happens inside a single day. So the CLT does nothing to tame the fat tails from A1-03 — "fat tails" meaning extreme moves are far more common than the bell curve predicts. A 5-sigma daily move — one landing five standard deviations away from the average — remains vastly more likely than the normal distribution says, and no theorem rescues you from it.

Two further honest caveats:

• Convergence is slowest in the tails — the extreme ends of the distribution. Even for monthly returns, the center of the distribution normalizes long before the extremes do. The CLT approximation is best exactly where you need it least.
• It assumes independence and finite variance. (Variance — the standard deviation squared, another measure of spread.) Volatility clustering — wild days bunching with wild days, calm with calm — weakens the independence assumption. And very heavy tails, where variance is barely finite, slow convergence dramatically.

Rule of thumb: trust CLT-based reasoning for estimates and averages — standard errors (the typical wobble of an estimate) and confidence intervals (ranges likely to contain the true value), both coming in A2-09. Never use it to reason about single-period risk.

See it, don't take my word

The CLT is one of the few theorems you can watch happen. A uniform distribution on $[0, 1]$ — every number between 0 and 1 equally likely — is flat. Nothing bell-shaped about it. Its mean is $1/2$, and its variance is $1/12$. So the CLT predicts that means of $N$ draws should pile up around $0.5$ with standard deviation $\sqrt{1/(12N)}$.

In words: take the variance, 1/12, divide it by the number of draws N, then take the square root. That works out to about $0.0289$ for $N=100$ and $0.0144$ for $N=400$ — quadruple the draws, halve the spread.

While the code runs, also plot a histogram of the batch means in your head: flat input, bell-shaped output. That transformation is the whole theorem.

Draw it

From memory, sketch three histograms side by side: (1) single draws from uniform(0,1), (2) means of 10 draws, (3) means of 100 draws. Label the center of each and annotate the widths so they shrink in the right proportion. Then add one sentence under the sketch: why doesn't panel (3)'s bell shape apply to a single day's stock return?