Walk-forward testing and out-of-sample discipline

In-sample vs out-of-sample

In-sample (IS): data used to make any choice — parameters, features, even which strategy idea to keep. Out-of-sample (OOS): data that influenced no decision. Only OOS performance estimates what happens next, because IS performance is contaminated by selection: you kept what looked good on it.

The subtlety from D4-01's failure mode #6: "choices" include soft ones. If you glanced at the full history before designing the strategy, the design is partly in-sample everywhere.

The walk-forward loop

Walk-forward analysis manufactures a sequence of honest OOS windows:

1. Take the first training window; fit/tune there.
2. Apply those frozen parameters to the test window immediately after.
3. Slide forward by one test-window length; refit; repeat.
4. Concatenate all test-window returns. That concatenation is your OOS track record.

With 10 periods, train width 4, test width 2 (rolling):

index:    0  1  2  3  4  5  6  7  8  9
split 1: [T  T  T  T][t  t]
split 2:       [T  T  T  T][t  t]
split 3:             [T  T  T  T][t  t]

Rolling windows (shown) keep the training width fixed — the fit adapts to regime change. Anchored windows always start at index 0, so training grows — more data, less adaptivity. Both are defensible; pick one before looking at results.

Purging and the embargo

Adjacent train/test windows still leak. Returns are autocorrelated, and labels often span several bars (a 5-day forward return computed at the end of training uses days inside the test window). López de Prado's fix: purge training samples whose information overlaps the test window, and add an embargo — a buffer gap of a few bars after the test window before training data resumes. Without it, the test set quietly grades the training set's homework.

The one-shot rule

Beyond walk-forward, keep a final holdout — the most recent year or two — that nothing touches until you are done. The rule: you get to open it once. Evaluate, accept the verdict. If you peek, dislike the result, tweak, and peek again, the holdout has become training data — you selected against it, and A2-05's multiple-testing arithmetic applies to it now. Touch it twice and it's contaminated.

Calibrate your expectations

Empirically, strategies that survive honest evaluation typically deliver OOS Sharpe around 30–70% of in-sample — Bailey and López de Prado's work on backtest overfitting shows the haircut is the norm, not the exception. And the final OOS test is one you can't simulate: paper trading and then live trading, where divergence from the backtest is the market grading your methodology. If your plan requires the IS number to hold up, you don't have a plan.

Try it

The bookkeeping of walk-forward is index arithmetic — get it exactly right once. Pure Python is fine here; no numpy needed.

Map the concepts

Your generative activity: draw a concept map connecting these eight nodes — in-sample, out-of-sample, walk-forward, rolling window, anchored window, purging, embargo, final holdout. Label each edge with the relationship (for example, "protects against leakage into"). Add one edge back to A2-05 multiple testing showing where its bias enters.

Predict before the next lesson

D4-05 quantifies selection bias. Predict:

• You test 100 strategies with zero true edge on 5 years of data (Sharpe SE about 0.45 from A2-04). Roughly what Sharpe will the best of them show by luck alone?
• What information would you need to record while researching in order to correct for this afterward?