Model selection and forecasting — the humbling benchmark

The overfitting ratchet

AR(2) contains AR(1) as a special case: set the second coefficient to zero and AR(2) is AR(1). Models related this way are called nested — the smaller one lives inside the bigger one. That containment has a hard consequence. The best-fitting AR(2) can never do worse in-sample — that is, measured on the same data used to fit it — than the best AR(1), because it could always just copy the AR(1) answer. With noisy data it does strictly better. The extra coefficient bends toward whatever noise pattern this particular sample happens to contain. Same for AR(3) over AR(2), and for every ARMA extension from D1-05.

This is the time-series face of the disease you met in A2-05 and D4-01: maximizing in-sample fit selects for luck. Fit a rich enough model and you can reproduce history exactly — and forecast nothing. So we need scorekeeping that either charges for parameters or hides data from the fit.

AIC and BIC — paying for parameters

Information criteria charge for parameters directly. Two symbols first: $L$ is the model's likelihood — how probable the observed data is under the fitted model, so higher means better fit — and $\ln$ is the natural logarithm (np.log), which just compresses $L$ onto a manageable scale. For a model with $k$ parameters fit on $n$ points:

$\text{AIC} = 2k - 2\ln L \qquad\qquad \text{BIC} = k\ln n - 2\ln L$

In words: AIC equals 2 times the parameter count minus 2 times the log-likelihood; BIC swaps the flat 2 for $\ln n$, the log of the sample size. In both, the $-2\ln L$ term rewards fit (it falls as fit improves), the first term is the fine per parameter, and lower is better. The difference is the fine schedule: AIC charges a flat 2 per parameter; BIC charges $\ln n$, which is already ~5.5 at 250 daily observations and grows with the sample. So BIC penalizes harder and picks smaller models. For daily financial data, where true signals are faint and extra parameters mostly harvest noise, the stingier BIC is usually the safer default. Statsmodels prints both for any ARMA fit; you just have to resist the urge to ignore them.

Information criteria are cheap and use all the data — but they still score in-sample likelihood, just with a handicap. The sterner test hides data.

Splitting time: train/test without shuffling

The machine-learning reflex is to shuffle the data and split randomly. For time series this is forbidden. Shuffling puts 2023 observations in the training set for "predicting" 2019 — look-ahead bias (D4-01) manufactured by your own evaluation harness. Autocorrelation makes it worse: a test point randomly sandwiched between two training points is partly memorized, not forecast.

The rule: temporal order is sacred. Train on the first chunk, test on the later chunk, and let information flow only forward:

split = int(len(x) * 0.7)
train, test = x[:split], x[split:]   # fit on train ONLY, forecast test

One split gives one verdict; rolling the split forward gives walk-forward analysis, the full treatment in Track D4.

The naive benchmark

Score forecasts on the test set with mean squared error or mean absolute error:

$\text{MSE} = \frac{1}{n}\sum_t (x_t - \hat{x}_t)^2 \qquad \text{MAE} = \frac{1}{n}\sum_t |x_t - \hat{x}_t|$

In words: $\hat{x}_t$ ("x-hat sub t" — the hat marks a forecast) is your prediction for time t, so $x_t - \hat{x}_t$ is the forecast error. MSE averages the squared errors; MAE averages their absolute values. Tiny example: errors of +2, −1, +1 give MSE = (4 + 1 + 1) / 3 = 2 and MAE = (2 + 1 + 1) / 3 ≈ 1.33. MSE punishes big misses quadratically; MAE is more forgiving of outliers. But a raw error number means nothing alone — you need a baseline, and time series has a brutally effective one: the naive forecast, tomorrow equals today ($\hat{x}_{t+1} = x_t$). For a random walk (D1-02), the naive forecast is optimal. So "beat naive out-of-sample" is the minimum bar for claiming your model found structure — and for prices, most models fail it. For returns, the analogous humbling benchmark is the constant forecast (tomorrow's return equals the historical mean, or just zero): decades of academic horse races show sophisticated return forecasts struggling to beat even that.

The honest scoreboard

Where do linear time-series models actually land, out-of-sample, on financial data?

• Return direction and magnitude: rarely beat naive. Daily return autocorrelations are ~0.01–0.05 (D1-01) — the predictable fraction of variance is tiny, and transaction costs eat most of what survives. Any backtest saying otherwise should trigger the D4-01 checklist, not celebration.
• Volatility: routinely beat naive. Squared returns are strongly autocorrelated — vol clusters. Models that forecast risk rather than return have genuine, repeatable out-of-sample skill.

That asymmetry is the doorway to the rest of this track: the next two lessons are about the forecasts that actually work.

Try it

Race the two classic baselines. The naive forecast predicts each point with the previous point; the mean forecast predicts every point with the training mean (here: the mean of all points before the last step, a miniature train/test split):

Predict before the next lesson

Next up is D1-07: volatility, ARCH and GARCH — the models built for the one thing that is forecastable. Before reading, predict:

• If today's return is a huge shock (say −5%), what should your forecast of tomorrow's variance do?
• Design instinct: how would you write a recursion where today's variance depends on yesterday's squared shock and yesterday's variance?