Backtesting — first principles and how to fool yourself

The fundamental problem

A backtest — a simulation of how a strategy would have performed on past data — is the core research tool of quant trading. It is easy to write a backtest that earns 30% per year. It is extremely hard to write one whose conclusions match what would happen with real money.

The gap between a backtest and reality is closed by being relentlessly honest about every place the backtest could lie.

The Big Six failure modes

1. Look-ahead bias

Look-ahead bias — using information in the simulation that you could not actually have had at the moment the trade was placed. It is the single most common backtest bug. Examples:

• Using today's closing price to make a decision and assuming you traded at that close.
• Using a stock's future fundamentals (e.g., this year's earnings) reported in your dataset before the actual filing date.
• Computing technical indicators using future data (some libraries do this; check them).
• Using the adjusted close with today's split adjustment when historical prices wouldn't have had it.

The fix: lag every signal by one period — delay it, so the trade happens strictly after the data that triggered it. Say the signal is "buy when the 20-day MA crosses above the 50-day MA" (MA — moving average, the average of the last N closing prices). Then the trade must execute at the next day's open, using yesterday's MA values.

2. Survivorship bias

Survivorship bias — testing only on companies that survived to today, so the failures silently vanish from your data. Backtest on the current S&P 500 member list and you only get stocks that made it. Lehman Brothers, WaMu, Enron — gone from the dataset. A "buy everything in the S&P 500" strategy backtested with current constituents (the stocks in the index now) systematically excludes the failures.

The fix: use a point-in-time index constituent dataset — one that records which stocks were in the index on each historical date, including companies that later died. Bloomberg and CRSP have this; affordable paid options include Norgate Data and Nasdaq Data Link (formerly Quandl). There is no good free point-in-time source — with free yfinance data you're limited to currently-listed stocks, and your results are biased. Know this limitation and be conservative in interpreting results.

3. Optimistic transaction costs

Most retail backtests assume zero or laughably low trading costs. Costs are quoted in bps (basis points — 1 bp = 0.01%, so 10 bps means one tenth of one percent). Real costs for retail:

• Spread — the gap between the price you can buy at and the price you can sell at, right now: 1–10 bps for liquid US large-caps, much wider for small-caps and crypto.
• Slippage — the extra cost from the price moving between your decision and your actual execution: 5–20 bps for medium-size orders in liquid names.
• Commissions — the broker's explicit fee per trade: $0 for most retail brokers now (used to be 1–10 bps).
• Borrow fees — rent you pay to borrow shares so you can short them: 30–300 bps annualized (and you have to be able to find shares to borrow at all).
• Tax drag: 10–25% of profit for short-term trades in non-retirement accounts.

Now the arithmetic. Say a strategy turns over 5× per year — meaning it replaces its entire portfolio five times a year — and pays an 8 bps round-trip cost (a round trip is one buy plus the matching sell). It loses 40 bps/year to costs alone. Many "edges" don't survive.

4. Parameter overfitting

Parameter overfitting — tuning your strategy's settings until they fit the random noise of the past instead of a real, repeatable pattern. This is the most common quant-trading sin. You try 50 lookback windows, 20 entry thresholds, and 10 exit rules. You find the combination that produces the best historical Sharpe and declare it the winner. You haven't found an edge; you've found the best of 10,000 noise patterns.

The deflated Sharpe ratio (Bailey & López de Prado 2014) — a Sharpe ratio marked down for how many combinations you tried — corrects for this. The cleanest workaround: choose parameters from theory or prior literature, not optimization, and test only that one final set of parameter values.

5. Multiple testing

Multiple testing — running many separate experiments and reporting only the winner. You backtest 100 strategies. By chance alone, ~5 will have p < 0.05 (read: a p-value below 0.05 — a result that pure luck would produce less than 5% of the time in any single test. Run 100 tests and luck hands you about 5 of them for free). You report the best.

This is the same disease as parameter overfitting, and often worse. The strategies look superficially different, so the multiplicity — how many things you actually tried — hides in the number of ideas tested, not in the parameters within one idea.

The fix: keep a written log of every strategy idea tested. At the end, apply a multiple-testing correction — Bonferroni (demand a p-value threshold divided by the number of tests), FDR (false discovery rate control, which caps the share of reported wins that are flukes), or the DSR from above. The pruned best-of-100 is dramatically less impressive than the unadjusted best.

6. Backtest contamination from current information

This one's subtle. Your backtest period ends today. You set up the strategy today, with full knowledge of what happened in the last 15 years. So your very choice of which strategy to test is already shaped by knowing which strategies worked.

The fix is walk-forward analysis — re-fitting the strategy periodically on rolling windows, described next — plus a true out-of-sample period: a stretch of data you never look at while designing, and don't peek at until you're committed.

Walk-forward analysis

The right way to evaluate a strategy that needs any parameter tuning:

1. Split history into rolling windows (e.g., 3 years training, 1 year test).
2. Fit / tune parameters on the training window.
3. Apply those exact parameters to the test window.
4. Roll forward.
5. Concatenate test-window returns. This is your honest performance.

Important: the test windows must be non-overlapping with the training data. López de Prado's "purged" CV (cross-validation) goes further: it adds a time buffer between the training and test windows. The buffer matters because returns are autocorrelated — correlated with their own recent past — so data right at the boundary can leak information across it.

The honest checklist

Before you trust any backtest, ask:

• Are all signals lagged by ≥1 period relative to the prices being predicted?
• Is the universe point-in-time, including delisted stocks?
• Is Adj Close used for returns (or are dividends/splits correctly handled)?
• Are realistic transaction costs deducted?
• Was the parameter choice made before seeing the data, or after?
• Did I keep a log of every strategy idea I tried?
• Is there a held-out out-of-sample period I've never touched?
• Have I computed Sharpe's standard error, or applied DSR?
• Is the result robust to small perturbations in parameters?

A backtest that passes all of these is worth taking seriously. A backtest that fails any one is suspect. A backtest that fails three is probably noise.

We'll cover each correction technique in detail in the rest of Track D4.

Try it

See look-ahead bias with your own hands. The signal array below "predicts" the returns perfectly when the two arrays line up — the signal is 1 on every positive-return day. Multiply signals by returns with no shift, sum them up, and you get a fantastic +0.08. But in real life, yesterday's signal trades into today's return. Shifted honestly, the same signal earns −0.02:

Predict before the next lesson

Tomorrow we start linear regression (A2-06) — the workhorse of factor analysis. Predict:

• What does it mean for two return series to be "correlated"?
• What's the difference between correlation and the slope of a regression?