Deflated Sharpe and the probability of backtest overfitting

Selection bias, quantified

A2-05 warned that testing many hypotheses and keeping the best inflates results. Here is the size of the inflation. Each backtest Sharpe is an estimate: true value plus noise with standard error $\text{SE}$ (Lo 2002, from A2-04). For $N$ independent trials with zero true edge, extreme-value theory gives the expected best:

$E[\max \widehat{SR}] \approx \text{SE} \cdot \sqrt{2 \ln N}$

Worked example — 100 strategies, 5 years of daily data each, so $\text{SE} \approx 0.45$:

$\ln 100 = 4.6052, \quad 2 \ln 100 = 9.2103, \quad \sqrt{9.2103} = 3.0349$

$E[\max \widehat{SR}] \approx 0.45 \times 3.0349 \approx 1.37$

The best of 100 nothing-strategies looks like a Sharpe-1.37 strategy. The growth is slow ($\sqrt{2\ln N}$: about 0.97 SE at N = 10, 1.37 at 100, 1.67 at 1000) — but so is the payoff of honesty, and note what this means in reverse: an observed Sharpe of 1.37 after 100 trials is exactly what zero edge predicts.

The Deflated Sharpe Ratio

Bailey and López de Prado (2014) turn this into a test. The Deflated Sharpe Ratio (DSR) asks: what is the probability that the observed Sharpe exceeds the benchmark you'd expect from the best of $N$ zero-edge trials? Conceptually:

1. Compute the trials-implied benchmark $SR_0$ from the number of trials and the variance of Sharpe across them (a sharper version of the $\sqrt{2 \ln N}$ formula above).
2. Compare the observed Sharpe to $SR_0$, in units of its standard error — adjusted for the skewness and kurtosis of the return series, since fat tails widen Sharpe's sampling noise.
3. DSR near 1 means the result survives its own selection process; DSR near 0.5 or below means "consistent with the luckiest of your trials."

We use it conceptually here; the full formula is in the 2014 paper. The key input it demands is $N$ — which most researchers never recorded.

PBO — probability of backtest overfitting

A complementary diagnostic from Bailey, Borwein, López de Prado, and Zhu: split the return history into many blocks and form all combinatorially symmetric train/test partitions (each uses half the blocks as IS, half as OOS). For each partition, pick the best strategy in-sample, then look up its rank out-of-sample. PBO is the fraction of partitions where the IS winner lands in the bottom half OOS. If your selection process picks strategies that are merely lucky, the IS winner will rank near the middle or worse OOS, and PBO climbs toward and past 0.5 — a coin flip or worse. A robust process keeps PBO low.

You can't deflate what you didn't count

Both corrections need the trial count. That makes the research log — every idea, every parameter set, every discard, written down as you go — a statistical instrument, not bureaucracy. It is D4-01's "log every strategy idea" checkbox and A2-05's multiple-testing recipe fused: without the log, $N$ is unknowable, and the honest answer to "is this Sharpe real?" becomes "no one can say."

Try it

Summarize

Your generative activity: in four or five written sentences, summarize for a colleague why the best backtest from a research process is biased, roughly how big the bias is as a function of trials and sample length, and what DSR and PBO each do about it. No peeking; then check yourself against the lesson.

Predict before the next lesson

D4-06 closes the track with robustness checks. Predict:

• If a strategy's Sharpe is 1.5 at a 20-day lookback but 0.2 at both 15 and 25 days, what does that tell you?
• Name two perturbations, other than parameters, you could apply to a backtest to see whether the edge is real.