Deflated Sharpe and the probability of backtest overfitting

Selection bias, quantified

A2-05 warned that testing many hypotheses and keeping the best inflates results. Here is the size of the inflation.

Each backtest Sharpe is an estimate: the true value plus random noise. The typical size of that noise is the standard error, written $\text{SE}$ — how far an estimate usually lands from the truth (Lo 2002, from A2-04). Now suppose you run $N$ backtests ($N$ is your number of trials — every strategy or parameter set you tried), and every one of them has zero true edge. The more strategies you tried, the higher the best Sharpe you'd expect by pure luck — and this formula quantifies exactly that. It comes from extreme-value theory, the branch of statistics that describes the largest of many random draws:

$E[\max \widehat{SR}] \approx \text{SE} \cdot \sqrt{2 \ln N}$

Read it aloud: "the expected value of the maximum estimated Sharpe ratio is approximately the standard error, times the square root of two times the natural log of N." Three symbols here go beyond the A1-00 basics. The hat on $\widehat{SR}$ means "our estimate of SR from data," not the true value. $\max$ means "take the largest" — the best backtest of the batch. And $\ln$ is the natural logarithm — the log button on a calculator (base $e$); all you need here is that it grows very slowly as $N$ grows.

In words: the best Sharpe among N zero-edge strategies sits, on average, about square-root-of-(2 times ln N) standard errors above zero — the more strategies you tried, the higher the best one looks by luck alone.

Worked example — 100 strategies, 5 years of daily data each, so $\text{SE} \approx 0.45$. First compute the luck multiplier:

$\ln 100 = 4.6052, \quad 2 \ln 100 = 9.2103, \quad \sqrt{9.2103} = 3.0349$

So the best of 100 sits about 3 standard errors above zero, purely from selection. Multiply by the SE:

$E[\max \widehat{SR}] \approx 0.45 \times 3.0349 \approx 1.37$

The best of 100 nothing-strategies looks like a Sharpe-1.37 strategy. The growth is slow — at this SE of 0.45, the expected best is about 0.97 at N = 10, 1.37 at 100, and 1.67 at 1000, so ten times more trials buys surprisingly little extra — but so is the payoff of honesty. And note what this means in reverse: an observed Sharpe of 1.37 after 100 trials is exactly what zero edge predicts.

The Deflated Sharpe Ratio

Bailey and López de Prado (2014) turn this into a test. The Deflated Sharpe Ratio (DSR) asks one question: what is the probability that your observed Sharpe beats the Sharpe you'd expect from the luckiest of $N$ zero-edge trials? Conceptually, three steps:

1. Compute the benchmark your own trials imply, written $SR_0$ (read "S-R-zero" — the hurdle that luck alone sets). It comes from the number of trials and the variance of Sharpe across them. It is a sharper version of the $\sqrt{2 \ln N}$ formula above.
2. Compare the observed Sharpe to $SR_0$, measured in units of its standard error. That comparison is adjusted for the skewness — the lopsidedness of the return distribution — and kurtosis — how fat its tails are, meaning how often extreme days occur — of the return series. Fat tails widen Sharpe's sampling noise, so they must widen the test's error bars too.
3. Read the output like a probability. DSR near 1 means the result survives its own selection process. DSR near 0.5 or below means "consistent with the luckiest of your trials" — no evidence of real edge.

We use it conceptually here; the full formula is in the 2014 paper. The key input it demands is $N$ — which most researchers never recorded.

PBO — probability of backtest overfitting

A complementary diagnostic from Bailey, Borwein, López de Prado, and Zhu. The recipe: split the return history into many blocks. Form all combinatorially symmetric train/test partitions — every possible way of using half the blocks as in-sample (IS) and the other half as out-of-sample (OOS). For each partition, pick the best strategy in-sample. Then look up that winner's rank out-of-sample — its position when all the strategies are sorted by OOS performance. PBO is the fraction of partitions where the IS winner lands in the bottom half OOS. Why this works: if your selection process picks strategies that are merely lucky, the luck does not carry over, so the IS winner ranks near the middle or worse OOS. Then PBO climbs toward and past 0.5 — a coin flip or worse. A robust process keeps PBO low.

You can't deflate what you didn't count

Both corrections need the trial count $N$. That makes the research log — every idea, every parameter set, every discard, written down as you go — a statistical instrument, not bureaucracy. It is D4-01's "log every strategy idea" checkbox and A2-05's multiple-testing recipe fused into one habit. Without the log, $N$ is unknowable. And without $N$, the honest answer to "is this Sharpe real?" becomes "no one can say."

Try it

Summarize

Your generative activity: in four or five written sentences, summarize for a colleague why the best backtest from a research process is biased, roughly how big the bias is as a function of trials and sample length, and what DSR and PBO each do about it. No peeking; then check yourself against the lesson.

Predict before the next lesson

D4-06 closes the track with robustness checks. Predict:

• If a strategy's Sharpe is 1.5 at a 20-day lookback but 0.2 at both 15 and 25 days, what does that tell you?
• Name two perturbations, other than parameters, you could apply to a backtest to see whether the edge is real.