Hypothesis testing — is this strategy real or luck?

The setup

Hypothesis testing is a framework for asking one question: could this observation be explained by random chance alone?

The structure:

• Null hypothesis — written $H_0$, read "H naught" — the boring explanation: there is no effect. For a strategy, that means its true expected return is zero.
• Alternative hypothesis — written $H_1$, read "H one" — there is an effect.
• Compute a test statistic from your data — a single number that summarizes how far the data sits from what $H_0$ predicts.
• Compute the p-value — the probability of seeing data this extreme (or more extreme) if $H_0$ were true.
• Reject $H_0$ if p is below some threshold. The common convention is 0.05.

The test statistic for a strategy's mean return uses two new symbols. A bar over a symbol, as in $\bar{r}$ (read "r bar"), means "the average of r." And $\text{SE}(\bar{r})$ — the standard error of the mean — is the typical distance between your sample average and the true mean:

$t = \frac{\bar{r}}{\text{SE}(\bar{r})} = \frac{\bar{r}}{\sigma / \sqrt{N}}$

In words: the t-statistic is the average return divided by its standard error. Equivalently: divide the average return by the standard deviation over the square root of the sample size. It answers: "how many standard errors is the average away from zero?"

Concrete feel: if your average daily return is 2 standard errors above zero, t = 2. That's hard to get by luck alone.

Under $H_0$ (true mean is zero), $t$ approximately follows the t-distribution — a bell curve slightly wider than the normal, with "$N-1$ degrees of freedom" (a parameter that grows with sample size and controls how wide). For large $N$, it's approximately the standard normal bell curve you've already met.

Consequence: $|t| > 1.96$ — read "the absolute value of t is bigger than 1.96," meaning t is beyond ±1.96 in either direction — corresponds (roughly) to p < 0.05. That's where the famous "2 standard errors" rule comes from.

Translating to Sharpe

Notice the t-statistic for "mean return ≠ 0" (the ≠ symbol reads "is not equal to") is closely related to the Sharpe ratio from A2-04:

$t = \frac{\bar{r}}{\sigma / \sqrt{N}} = \widehat{\text{Sharpe}} \cdot \sqrt{N} / \sqrt{T_\text{annual}}$

In words: the t-statistic equals the estimated Sharpe ratio times the square root of the number of observations, divided by the square root of the periods per year. (The hat on $\widehat{\text{Sharpe}}$ again marks an estimate from data; $T_\text{annual}$, read "T sub annual," is 252 for daily data.)

The point: a higher Sharpe and a longer backtest both increase t. Simplified for annual units, t ≈ Sharpe × √years. A Sharpe of 1 over 5 years has $t \approx \sqrt{5} \approx 2.24$ — just barely past the 1.96 bar, so barely "significant." Over 10 years, $t \approx 3.16$ — clearly significant.

This is why long backtests are powerful — not because the Sharpe is better, but because the statistical evidence is stronger for the same Sharpe.

The p-value, honestly

A p-value is not:

• The probability that the null hypothesis is true.
• A measure of effect size.
• A measure of practical significance.

A p-value is:

• The probability of observing data at least this extreme if the null were true.

This distinction matters. Effect size — how big the thing you measured actually is — is separate from how statistically detectable it is. A study can have p = 0.001 and a tiny effect size: with millions of observations, even a trivially small mean return becomes "significant." Or p = 0.10 with a huge effect: few observations, large mean. The p-value is one number among many.

The multiple-testing problem

If you test one strategy honestly and get p = 0.03, you have moderate evidence.

If you tested 100 strategies and report the best one with p = 0.03 — you have no evidence. By chance alone, 5 out of 100 strategies would show p < 0.05 even if all of them had zero true edge.

This is the most common way retail quants fool themselves. Tuning parameters, trying different indicators, varying lookback windows — every variation is a new test. The published backtest you settle on is the survivor of many quiet failures, and its statistics are dramatically inflated.

Corrections

Several adjustments exist:

• Bonferroni correction: divide your p-value threshold by the number of tests. Test 100 strategies → require p < 0.0005 instead of 0.05. Very conservative — it rarely lets anything through.
• Benjamini-Hochberg FDR (false discovery rate): instead of trying to avoid any false positive, it controls the expected fraction of your "discoveries" that are false. Less conservative than Bonferroni.
• Deflated Sharpe Ratio (López de Prado 2018): the right tool for quant work. It adjusts the Sharpe for the number of trials you ran, for how much Sharpe varied across those trials, and for the skewness and kurtosis (lopsidedness and tail-heaviness) of the returns. We'll see this in detail when we cover backtest methodology (D4).

Harvey-Liu-Zhu (2016) "factor zoo"

A landmark paper. It surveyed hundreds of "factors" — measurable characteristics, like company size or recent momentum, claimed to explain stock returns — published in finance journals. The argument: academia as a whole has run thousands of tests, so the field has a giant multiple-testing problem. With proper corrections, the t-statistic threshold for calling a factor "real" should be ~3.0, not the standard 2.0. Many published factors don't survive this bar.

The implication for retail: most "edges" you find online don't survive honest multiple-testing scrutiny. Be very skeptical.

A practical recipe for evaluating your own strategies

1. State your hypothesis in writing before you backtest. "Mean reversion in S&P 500 stocks over 1-day horizons makes money." Specific.
2. Backtest once, with parameters chosen a priori — meaning in advance, from theory or prior literature — not optimized on the same data you're testing.
3. Report the actual t-statistic and p-value of mean returns.
4. Check the edge against realistic transaction costs — for retail, 5–20 bps per round trip (a bps, or basis point, is 0.01%; a round trip is one buy plus one sell). Many "edges" vanish under realistic costs.
5. If you iterate, keep a log of every variant tested. Apply a multiple-testing correction at the end.
6. Out-of-sample test — evaluation on data your development process never touched: hold back the last 20-30% of data and test the final strategy there exactly once.

This is the discipline that separates serious quants from people who fool themselves.

Try it

Compute the t-statistic for "is this strategy's mean return different from zero?":

Predict before the next lesson

Tomorrow we shift back to finance with alpha (excess return vs. a benchmark). Predict:

• If a strategy has a positive alpha of 5% but volatility much higher than the market, is it actually a good idea?
• What's the cleanest way to define "alpha" so you don't accidentally double-count benchmark exposure?