Regression diagnostics — when OLS lies

The four diagnostic questions

After running any regression, ask four questions. (Reminder from A2-06: the residuals are the leftover misses between each data point and the fitted line.)

1. Are the predictors collinear? — do they overlap with each other? (Multicollinearity)
2. Are the residuals heteroscedastic? — is their variance non-constant, calm in some stretches and wild in others?
3. Are the residuals autocorrelated? — does today's miss predict tomorrow's? (Independence violated)
4. Are the residuals normal? — do the misses follow a bell curve? (Affects hypothesis tests, not point estimates)

We'll cover each. The fix is usually either a robust standard error — a corrected formula for the uncertainty — or a different model.

1. Multicollinearity

When predictors are highly correlated with each other:

• Point estimates remain consistent — with infinite data they'd still converge to the truth.
• Standard errors blow up, because the regression can't tell which of two near-identical predictors deserves the credit.
• Coefficient signs can flip for small changes in the data.

Diagnostic — the Variance Inflation Factor (VIF). To measure how redundant predictor $j$ is, regress it on all the other predictors and record that regression's $R^2$ — call it $R_j^2$ (read "R squared sub j"). If the others can predict $j$ well, $j$ adds little new information. Then:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

In words: the VIF of predictor j is 1 divided by (1 minus the R-squared from predicting j using the other predictors). Example: if the other predictors explain 90% of predictor j ($R_j^2 = 0.9$), its VIF is 1 / 0.1 = 10.

Rules of thumb:

• VIF < 5: fine
• VIF 5–10: be cautious
• VIF > 10: serious problem; consider dropping a predictor

Fix: drop one of the correlated predictors; combine them into one (e.g., with PCA — principal component analysis, a technique for merging overlapping variables, coming in Track A3); or use a regularized regression such as Ridge, which shrinks correlated coefficients toward each other instead of letting them fight.

2. Heteroscedasticity

OLS assumes the residuals are equally noisy across the whole predictor range. Financial residuals usually aren't:

• Volatility clusters (A2-02): high-volatility periods produce larger residuals than calm ones.
• Bigger predictor magnitudes produce bigger residuals — the noise is proportional to the size of the move.

Diagnostic: plot the residuals against the predicted values. A fan-shaped pattern — misses growing wider as predictions grow — means heteroscedasticity. The formal version is the Breusch-Pagan test.

Why it matters: OLS point estimates are still unbiased — not systematically too high or too low. But the standard errors are wrong, so your t-stats and p-values — the numbers that tell you whether a coefficient looks statistically significant — are wrong too.

Fix: use White / robust / heteroscedasticity-consistent (HC) standard errors. In statsmodels:

model.fit(cov_type='HC3')   # one of several heteroscedasticity corrections

3. Autocorrelation

If residuals at time $t$ are correlated with residuals at $t-1$, you've violated independence. Common in:

• Time series of returns (typically only weak autocorrelation in returns themselves).
• Strategies with persistent positions (your daily PnL — profit and loss — is correlated with yesterday's).

Diagnostic: Durbin-Watson statistic (close to 2 = no autocorrelation; close to 0 or 4 = strong autocorrelation). Ljung-Box test for multiple lags — checking several past periods at once. ACF (autocorrelation function) plot.

Fix: use Newey-West / HAC (heteroscedasticity and autocorrelation consistent) standard errors:

model.fit(cov_type='HAC', cov_kwds={'maxlags': 5})

maxlags is roughly how many periods of autocorrelation you suspect. Rule of thumb: $\sim N^{1/4}$ — read "about N to the one-quarter power," i.e. the fourth root of N — for $N$ observations. For 252 daily observations that's about 4.

4. Non-normal residuals

Residuals don't need to be normal for OLS to give consistent point estimates. They do need to be roughly normal for:

• Small-sample t-statistics and confidence intervals to be valid.
• F-tests — significance tests of several coefficients at once — to be valid.

For large samples (say N > 100), the CLT covers the test statistics regardless of residual distribution. For small samples or extreme fat tails, you might prefer non-parametric methods — methods that don't assume any particular distribution shape — such as the bootstrap (resampling your own data to measure uncertainty), or quantile regression, which fits quantiles instead of the mean.

Diagnostic: Q-Q plot of residuals against normal. Should be roughly diagonal. Strong tails curving away = problem.

Outliers and influence

A single extreme observation can drive the regression. Two related concepts:

• Outlier: large residual (the data point is far from where the regression predicts).
• High-leverage point: extreme predictor value (large $x_t$ even if its residual is small).
• Influential observation: combination of both — actually changes the regression coefficients meaningfully.

Diagnostic: Cook's distance for influence. statsmodels' model.get_influence() gives a full diagnostic suite.

In finance, the 2008 financial crisis and March 2020 are often hugely influential observations. A regression run on 2010–2024 daily data is qualitatively different from one run on 2008–2024.

Robust strategy: examine the regression results with and without crisis periods. Big differences = your model isn't stable across regimes.

A practical checklist

Before publishing any regression result, run through:

• Compute VIFs. Any > 10? Reconsider predictor choice.
• Plot residuals vs. fitted. Any pattern? Reconsider model form.
• Plot residuals over time. Autocorrelation? Use HAC.
• Plot residual histogram and Q-Q. Reasonable? Mention any deviations.
• Compute Cook's distance. Anything > 4/n? Inspect those observations.
• Re-run with HC3 robust standard errors at minimum.
• If time series, re-run with HAC standard errors.
• Re-run on a subsample (e.g., excluding 2008). Coefficients stable?

This is a 20-minute exercise that catches 90% of "the regression said it was significant but actually isn't" problems.

Explain it back

In your own words: why do OLS point estimates often survive violations of OLS assumptions, while OLS standard errors do not? What's the practical implication for backtest hypothesis tests?