Regression diagnostics — when OLS lies

The four diagnostic questions

After running any regression, ask:

1. Are the predictors collinear? (Multicollinearity)
2. Are the residuals heteroscedastic? (Non-constant variance)
3. Are the residuals autocorrelated? (Independence violated)
4. Are the residuals normal? (Affects hypothesis tests, not point estimates)

We'll cover each. The fix is usually either a robust standard error or a different model.

1. Multicollinearity

When predictors are highly correlated with each other:

• Point estimates remain consistent (with infinite data they'd converge to the truth).
• Standard errors blow up because the regression can't tell which predictor deserves credit.
• Coefficient signs can flip for small changes in the data.

Diagnostic — Variance Inflation Factor (VIF):

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

where $R_j^2$ is the $R^2$ from regressing predictor $j$ on all the others.

Rules of thumb:

• VIF < 5: fine
• VIF 5–10: be cautious
• VIF > 10: serious problem; consider dropping a predictor

Fix: drop one of the correlated predictors, combine them (e.g., PCA), or use a regularized regression (Ridge — shrinks correlated coefficients toward each other).

2. Heteroscedasticity

OLS assumes residuals have constant variance across the predictor range. Financial residuals usually don't:

• Volatility clusters → high-vol periods produce larger residuals.
• Bigger predictor magnitudes → bigger residuals (proportional noise).

Diagnostic: plot residuals vs. predicted values. Fan-shaped pattern = heteroscedasticity. Formal test: Breusch-Pagan.

Why it matters: OLS point estimates are still unbiased, but the standard errors are wrong → your t-stats and p-values are wrong.

Fix: use White / robust / heteroscedasticity-consistent (HC) standard errors. In statsmodels:

model.fit(cov_type='HC3')   # one of several heteroscedasticity corrections

3. Autocorrelation

If residuals at time $t$ are correlated with residuals at $t-1$, you've violated independence. Common in:

• Time series of returns (typically only weak autocorrelation in returns themselves).
• Strategies with persistent positions (your daily PnL is correlated with yesterday's).

Diagnostic: Durbin-Watson statistic (close to 2 = no autocorrelation; close to 0 or 4 = strong autocorrelation). Ljung-Box test for multiple lags. ACF plot.

Fix: use Newey-West / HAC (heteroscedasticity and autocorrelation consistent) standard errors:

model.fit(cov_type='HAC', cov_kwds={'maxlags': 5})

maxlags is roughly how many periods of autocorrelation you suspect. Rule of thumb: $\sim N^{1/4}$ for $N$ observations.

4. Non-normal residuals

Residuals don't need to be normal for OLS to give consistent point estimates. They do need to be roughly normal for:

• Small-sample t-statistics and confidence intervals to be valid.
• F-tests to be valid.

For large samples (say N > 100), the CLT covers the test statistics regardless of residual distribution. For small samples or extreme fat tails, you might prefer non-parametric methods (bootstrap) or quantile regression.

Diagnostic: Q-Q plot of residuals against normal. Should be roughly diagonal. Strong tails curving away = problem.

Outliers and influence

A single extreme observation can drive the regression. Two related concepts:

• Outlier: large residual (the data point is far from where the regression predicts).
• High-leverage point: extreme predictor value (large $x_t$ even if its residual is small).
• Influential observation: combination of both — actually changes the regression coefficients meaningfully.

Diagnostic: Cook's distance for influence. statsmodels' model.get_influence() gives a full diagnostic suite.

In finance, the 2008 financial crisis and March 2020 are often hugely influential observations. A regression run on 2010–2024 daily data is qualitatively different from one run on 2008–2024.

Robust strategy: examine the regression results with and without crisis periods. Big differences = your model isn't stable across regimes.

A practical checklist

Before publishing any regression result, run through:

• Compute VIFs. Any > 10? Reconsider predictor choice.
• Plot residuals vs. fitted. Any pattern? Reconsider model form.
• Plot residuals over time. Autocorrelation? Use HAC.
• Plot residual histogram and Q-Q. Reasonable? Mention any deviations.
• Compute Cook's distance. Anything > 4/n? Inspect those observations.
• Re-run with HC3 robust standard errors at minimum.
• If time series, re-run with HAC standard errors.
• Re-run on a subsample (e.g., excluding 2008). Coefficients stable?

This is a 20-minute exercise that catches 90% of "the regression said it was significant but actually isn't" problems.

Explain it back

In your own words: why do OLS point estimates often survive violations of OLS assumptions, while OLS standard errors do not? What's the practical implication for backtest hypothesis tests?