Robustness checks — does the edge degrade gracefully?

The robustness mindset

Everything in D4 so far measured a strategy at one point: one parameter set, one period, one universe — the set of assets it trades — and one cost assumption. The last discipline is to poke it. A real edge degrades gracefully under perturbation — a small deliberate change to the setup. An overfit one collapses. Real effects are properties of markets, not of your settings. So they should survive small changes to everything that was your choice rather than the market's.

The four standard checks

1. Parameter perturbation — jiggle every knob you chose. This is the test of parameter sensitivity: how much performance depends on your exact settings. Vary each parameter ±20–50% around your choice and recompute Sharpe. For a 20-day lookback, that means rerunning at everything from roughly 10 to 30 days. Plot the surface. A plateau — broadly similar performance across the whole neighborhood — supports a real effect. A cliff — a sharp peak that collapses one step away — is the signature of a parameter fit to noise.

2. Subperiod analysis — does the edge show up across eras, or only one? Split the history into eras (for example thirds, or pre/post 2015, or crisis vs calm) and evaluate each separately. A real edge should be positive in at least 2 of 3 eras. This is regime robustness — the edge should not depend on one persistent market environment. A strategy whose entire P&L — profit and loss — comes from one regime is a bet that the regime returns.

3. Universe perturbation — does the edge depend on specific tickers? Randomly drop 20% of the assets and rerun, several times. If the result depends on a handful of specific names, you did not find a strategy. You found those names' histories.

4. Cost stress — what if trading costs more than you assumed? Double your cost assumption from D4-03 and rerun. An edge that dies at 2× assumed costs is a bet on your cost model being exactly right — and D4-03 showed how wide that uncertainty is.

The plateau principle

When you must choose a parameter, pick the middle of a stable region, never the peak. The peak is partly selection bias — D4-05's effect operating inside a single strategy, across its own parameter values. The plateau's center is the point most likely to still work when the noise re-rolls. A crude way to score stability across a neighborhood of Sharpes:

$\text{stability} = \frac{\max - \min}{\max}$

In words: take the biggest Sharpe in the neighborhood, subtract the smallest, and divide by the biggest — the fraction of the best performance that vanishes as you move around. Near 0 is a plateau; near 1 is a cliff. Example: neighboring Sharpes between 0.9 and 1.1 give (1.1 − 0.9) / 1.1 ≈ 0.18 — a plateau. A peak of 1.5 whose neighbors fall to 0.1 gives (1.5 − 0.1) / 1.5 ≈ 0.93 — a cliff. You'll implement exactly this below.

The full D4 pre-deployment checklist

D4-01's checklist, extended by everything since:

• Signals lagged ≥1 period; universe point-in-time; dividends/splits handled (D4-01, D4-02)
• Equity curve, max drawdown, and turnover computed and reported (D4-02)
• Costs documented, justified, and shown as a sensitivity table (D4-03)
• Walk-forward with purging/embargo; final holdout touched exactly once (D4-04)
• Research log kept; trial count known; DSR or equivalent deflation applied (D4-05)
• Parameter plateau, not peak; works in ≥2/3 of subperiods; survives universe drops and 2× costs (D4-06)
• Paper-traded before live capital, with divergence from backtest monitored (D4-04)

The honest close

The goal of backtesting is not to make a strategy look good. It is to give the strategy every chance to reveal itself as noise — and to deploy only what refuses to. Most ideas will die on this checklist. That is the checklist working. The few that pass are the only ones worth your capital, and even they get paper-traded first.

Try it

Teach it

Your generative activity: a friend shows you a backtest — Sharpe 1.4, lookback 20, tested on 2015–2025 tech stocks, 5 bps assumed costs (bps — basis points; one bp is 0.01%). Teach them, out loud or in writing, the four robustness checks as a review of their strategy: name each check, say exactly what you would rerun, and state what result would make you trust or reject the edge. Finish by explaining the plateau principle in your own words.

Predict before the next lesson

Track D4 is complete — you now own the methodology. Track D5 begins the payoff: the classic strategy archetypes — momentum, mean reversion, carry, value, and friends. Predict:

• Momentum (buy what went up) and mean reversion (buy what went down) sound like opposites. On what timescales does each tend to appear?
• For each archetype, who is on the other side of the trade — and why might they knowingly pay you?