Hypothesis testing — is this strategy real or luck?

The setup

Hypothesis testing is a framework for asking: could this observation be explained by random chance alone?

The structure:

• Null hypothesis $H_0$: there is no effect (e.g., the strategy has zero true expected return).
• Alternative hypothesis $H_1$: there is an effect.
• Compute a test statistic from your data.
• Compute the p-value: the probability of seeing data this extreme (or more) if $H_0$ were true.
• Reject $H_0$ if p < some threshold (commonly 0.05).

For a strategy with mean return $\bar{r}$ and standard error $\text{SE}(\bar{r})$:

$t = \frac{\bar{r}}{\text{SE}(\bar{r})} = \frac{\bar{r}}{\sigma / \sqrt{N}}$

Under $H_0$ (true mean is zero), $t$ is approximately t-distributed with $N-1$ degrees of freedom. For large $N$, it's approximately standard normal.

So $|t| > 1.96$ corresponds (roughly) to p < 0.05.

Translating to Sharpe

Notice the t-statistic for "mean return ≠ 0" is closely related to Sharpe:

$t = \frac{\bar{r}}{\sigma / \sqrt{N}} = \widehat{\text{Sharpe}} \cdot \sqrt{N} / \sqrt{T_\text{annual}}$

where $T_\text{annual} = 252$ for daily data. The point is: a higher Sharpe and a longer backtest both increase t. A Sharpe of 1 over 5 years has $t \approx \sqrt{5} \approx 2.24$, just barely "significant." Over 10 years, $t \approx 3.16$ — clearly significant.

This is why long backtests are powerful — not because the Sharpe is better, but because the statistical evidence is stronger for the same Sharpe.

The p-value, honestly

A p-value is not:

• The probability that the null hypothesis is true.
• A measure of effect size.
• A measure of practical significance.

A p-value is:

• The probability of observing data at least this extreme if the null were true.

This distinction matters. A study can have p = 0.001 and tiny effect size (millions of observations, trivially small mean). Or p = 0.10 and a huge effect (few observations, large mean). The p-value is one number among many.

The multiple-testing problem

If you test one strategy honestly and get p = 0.03, you have moderate evidence.

If you tested 100 strategies and report the best one with p = 0.03 — you have no evidence. By chance alone, 5 out of 100 strategies would show p < 0.05 even if all of them had zero true edge.

This is the most common way retail quants fool themselves. Tuning parameters, trying different indicators, varying lookback windows — every variation is a new test. The published backtest you settle on is the survivor of many quiet failures, and its statistics are dramatically inflated.

Corrections

Several adjustments exist:

• Bonferroni: divide your p-value threshold by the number of tests. Test 100 strategies → require p < 0.0005. Very conservative.
• Benjamini-Hochberg FDR: controls the expected fraction of false discoveries. Less conservative than Bonferroni.
• Deflated Sharpe Ratio (López de Prado 2018): the right tool for quant. Adjusts Sharpe for the number of trials you ran, the variance of Sharpe across trials, and the skewness/kurtosis of returns. We'll see this in detail when we cover backtest methodology (D4).

Harvey-Liu-Zhu (2016) "factor zoo"

A landmark paper: surveyed hundreds of "factors" published in finance journals claiming to explain stock returns. Argued that with proper multiple-testing corrections, the published t-statistic threshold for a "real" factor should be ~3.0, not the standard 2.0. Many published factors don't survive this bar.

The implication for retail: most "edges" you find online don't survive honest multiple-testing scrutiny. Be very skeptical.

A practical recipe for evaluating your own strategies

1. State your hypothesis in writing before you backtest. "Mean reversion in S&P 500 stocks over 1-day horizons makes money." Specific.
2. Backtest once, with parameters chosen a priori from theory or prior literature (not optimized on the same data).
3. Report the actual t-statistic and p-value of mean returns.
4. Multiply expected effect size by realistic transaction costs (5–20 bps per round trip for retail). Many "edges" vanish under realistic costs.
5. If you iterate, keep a log of every variant tested. Apply a multiple-testing correction at the end.
6. Out-of-sample: hold back the last 20-30% of data and only test the final strategy there once.

This is the discipline that separates serious quants from people who fool themselves.

Try it

Compute the t-statistic for "is this strategy's mean return different from zero?":

Predict before the next lesson

Tomorrow we shift back to finance with alpha (excess return vs. a benchmark). Predict:

• If a strategy has a positive alpha of 5% but volatility much higher than the market, is it actually a good idea?
• What's the cleanest way to define "alpha" so you don't accidentally double-count benchmark exposure?