Backtesting — first principles and how to fool yourself

The fundamental problem

A backtest is a simulation of how a strategy would have performed in the past. It is easy to write a backtest that earns 30% per year. It is extremely hard to write one whose conclusions match what would happen with real money.

The gap between a backtest and reality is closed by being relentlessly honest about every place the backtest could lie.

The Big Six failure modes

1. Look-ahead bias

Using information that wouldn't have been available at the moment the trade was placed. Examples:

• Using today's closing price to make a decision and assuming you traded at that close.
• Using a stock's future fundamentals (e.g., this year's earnings) reported in your dataset before the actual filing date.
• Computing technical indicators using future data (some libraries do this; check them).
• Using the adjusted close with today's split adjustment when historical prices wouldn't have had it.

The fix: lag every signal by one period. If a signal is "buy when 20-day MA crosses above 50-day MA," the trade must execute the next day's open using yesterday's MA values.

2. Survivorship bias

Backtesting on a current index (S&P 500 today) gives you only stocks that survived. Lehman Brothers, WaMu, Enron — gone from the dataset. A "buy everything in the S&P 500" strategy backtested with current constituents systematically excludes the failures.

The fix: use a point-in-time index constituent dataset. Bloomberg and CRSP have this; affordable paid options include Norgate Data and Nasdaq Data Link (formerly Quandl). There is no good free point-in-time source — with free yfinance data you're limited to currently-listed stocks, and your results are biased. Know this limitation and be conservative in interpreting results.

3. Optimistic transaction costs

Most retail backtests assume zero or laughably low costs. Real costs for retail:

• Spread: 1–10 bps for liquid US large-caps, much wider for small-caps and crypto.
• Slippage: 5–20 bps for medium-size orders in liquid names.
• Commissions: $0 for most retail brokers now (used to be 1–10 bps).
• Borrow fees: 30–300 bps annualized for shorting (and you have to be able to find borrow).
• Tax drag: 10–25% of profit for short-term trades in non-retirement accounts.

A strategy that turns over 5× per year and trades stocks with 8 bps round-trip cost loses 40 bps/year to costs alone. Many "edges" don't survive.

4. Parameter overfitting

The most common quant-trading sin. You try 50 lookback windows, 20 entry thresholds, and 10 exit rules. You find the combination that produces the best historical Sharpe and declare it the winner. You haven't found an edge; you've found the best of 10,000 noise patterns.

The deflated Sharpe ratio (Bailey & López de Prado 2014) corrects this. The cleanest workaround is to choose parameters from theory or prior literature, not optimization, and test only the final parametrization.

5. Multiple testing

You backtest 100 strategies. By chance alone, ~5 will have p < 0.05. You report the best.

Same disease as parameter overfitting, often worse because the strategies look superficially different and the multiplicity is hidden in the number of ideas tried, not parameters within one idea.

The fix: keep a written log of every strategy idea tested. At the end, apply a multiple-testing correction (Bonferroni, FDR, or DSR). The pruned best-of-100 is dramatically less impressive than the unadjusted best.

6. Backtest contamination from current information

This one's subtle. Your backtest period ends today. You set up the strategy today with full knowledge of what happened in the last 15 years. Your choice of strategy is conditioned on knowledge of which strategies worked.

The fix is walk-forward analysis (re-fit periodically) and a true out-of-sample period that you don't peek at until you're committed.

Walk-forward analysis

The right way to evaluate a strategy that needs any parameter tuning:

1. Split history into rolling windows (e.g., 3 years training, 1 year test).
2. Fit / tune parameters on the training window.
3. Apply those exact parameters to the test window.
4. Roll forward.
5. Concatenate test-window returns. This is your honest performance.

Important: the test windows must be non-overlapping with the training data. López de Prado's "purged" CV adds a buffer to avoid information leakage from autocorrelated returns.

The honest checklist

Before you trust any backtest, ask:

• Are all signals lagged by ≥1 period relative to the prices being predicted?
• Is the universe point-in-time, including delisted stocks?
• Is Adj Close used for returns (or are dividends/splits correctly handled)?
• Are realistic transaction costs deducted?
• Was the parameter choice made before seeing the data, or after?
• Did I keep a log of every strategy idea I tried?
• Is there a held-out out-of-sample period I've never touched?
• Have I computed Sharpe's standard error, or applied DSR?
• Is the result robust to small perturbations in parameters?

A backtest that passes all of these is worth taking seriously. A backtest that fails any one is suspect. A backtest that fails three is probably noise.

We'll cover each correction technique in detail in the rest of Track D4.

Try it

See look-ahead bias with your own hands. The signal array below "predicts" the returns perfectly when they line up — signal 1 on every positive return day. Sum signals * returns unshifted and you get a fantastic +0.08. But in real life yesterday's signal trades into today's return, and the same signal earns −0.02:

Predict before the next lesson

Tomorrow we start linear regression (A2-06) — the workhorse of factor analysis. Predict:

• What does it mean for two return series to be "correlated"?
• What's the difference between correlation and the slope of a regression?