Walk-forward testing and out-of-sample discipline

In-sample vs out-of-sample

In-sample (IS) — the data you used to make any choice: parameters, features, even which strategy idea to keep. Out-of-sample (OOS) — data that influenced no decision at all. Only OOS performance estimates what happens next. Why? IS performance is contaminated by selection — you kept whatever looked good on that data, so of course it looks good on that data. The IS number measures your picking process, not the strategy.

The subtlety from D4-01's failure mode #6: "choices" include soft ones. If you glanced at the full history before designing the strategy, the design is partly in-sample everywhere — your eyes already tuned it.

The walk-forward loop

Walk-forward analysis — fit on one slice of history, test on the slice right after it, then slide both slices forward and repeat — manufactures a sequence of honest OOS windows:

1. Take the first training window; fit/tune there.
2. Apply those frozen parameters to the test window immediately after.
3. Slide forward by one test-window length; refit; repeat.
4. Concatenate all test-window returns. That concatenation is your OOS track record.

With 10 periods, train width 4, test width 2 (rolling) — capital T marks a training period, lowercase t a test period:

index:    0  1  2  3  4  5  6  7  8  9
split 1: [T  T  T  T][t  t]
split 2:       [T  T  T  T][t  t]
split 3:             [T  T  T  T][t  t]

Rolling windows (shown above) keep the training width fixed. Old data falls out the back, so the fit adapts when the market changes character — what traders call regime change, a shift in the market's persistent behavior. Anchored windows always start at index 0, so the training set grows with each split — more data, less adaptivity. Both are defensible. Pick one before looking at results, so the choice can't become one more thing you tuned.

Purging and the embargo

Adjacent train/test windows still leak — information from the test period seeps into training. Two reasons. First, returns are autocorrelated — nearby days are statistically related, so the last training days carry a whiff of the first test days. Second, labels often span several bars: a 5-day forward return computed on the last training day literally uses days inside the test window. López de Prado's fix has two parts. Purge — delete the training samples whose information overlaps the test window. Embargo — leave a buffer gap of a few bars after the test window before training data resumes. Without them, the test set quietly grades the training set's homework.

The one-shot rule

Beyond walk-forward, keep a final holdout — a reserved slice of data, typically the most recent year or two, that nothing touches until you are completely done. The rule: you get to open it once. Evaluate, then accept the verdict. Suppose you peek, dislike the result, tweak the strategy, and peek again. Now you have selected against the holdout — it has become training data, and A2-05's multiple-testing arithmetic applies to it. Touch it twice and it's contaminated.

Calibrate your expectations

Empirically, strategies that survive honest evaluation typically deliver OOS Sharpe around 30–70% of in-sample. Bailey and López de Prado's work on backtest overfitting shows this haircut — the shrinkage from backtest number to real-world number — is the norm, not the exception. Concretely: an in-sample Sharpe of 1.0 should make you plan for something like 0.3 to 0.7 live. And the final OOS test is one you can't simulate: paper trading — running the strategy with fake money in real time — and then live trading. Divergence from the backtest there is the market grading your methodology. If your plan requires the IS number to hold up, you don't have a plan.

Try it

The bookkeeping of walk-forward is index arithmetic — get it exactly right once. Pure Python is fine here; no numpy needed.

Map the concepts

Your generative activity: draw a concept map connecting these eight nodes — in-sample, out-of-sample, walk-forward, rolling window, anchored window, purging, embargo, final holdout. Label each edge with the relationship (for example, "protects against leakage into"). Add one edge back to A2-05 multiple testing showing where its bias enters.

Predict before the next lesson

D4-05 quantifies selection bias. Predict:

• You test 100 strategies with zero true edge on 5 years of data (Sharpe SE about 0.45 from A2-04). Roughly what Sharpe will the best of them show by luck alone?
• What information would you need to record while researching in order to correct for this afterward?