Overfitting vs. Underfitting in Quantitative Methods & Econometrics

📌 "A good model captures the signal, not the noise." The twin dangers of overfitting and underfitting are the most common reasons why a statistical model fails. Understanding the difference is the first step towards building robust, reliable models for prediction and inference.

In quantitative methods and econometrics, we build models to understand relationships and make predictions. The goal is to find a model that generalizes well to new, unseen data. Two fundamental problems stand in the way: overfitting and underfitting. Overfitting occurs when a model learns the training data too well, including its random fluctuations. Underfitting happens when a model is too simple to capture the underlying pattern in the data. Both lead to poor performance on new data.

What is Overfitting?

Overfitting is when a statistical model is excessively complex. It fits the training data nearly perfectly but performs poorly on any new data because it has essentially "memorized" the noise and specific quirks of the training set.

Example 1 Polynomial Regression

Imagine we are modeling the relationship between a country's GDP growth (X) and its stock market returns (Y) with 10 years of data. A simple linear model (degree 1) shows a weak positive trend. However, if we fit a 9th-degree polynomial, the curve will pass through every single data point perfectly.

🔍 Explanation: The 9th-degree polynomial has enough flexibility to wiggle and curve to hit every training data point, including those caused by one-time events or measurement errors. This complex model will have near-zero error on the training data but will make wildly inaccurate predictions for future years because it learned the noise, not the true economic relationship.

Example 2 Too Many Variables

An economist tries to predict house prices. The training data includes 20 houses. They build a model using 15 different predictors: square footage, number of bedrooms, distance to school, age of roof, color of the front door, owner's zodiac sign, etc.

🔍 Explanation: With so many variables relative to the number of observations, the model will find spurious correlations (e.g., blue doors might coincidentally be in more expensive houses in this small sample). It "overfits" to this specific set of 20 houses. When tested on new houses, predictors like the door color or zodiac sign provide no real predictive power, causing the model to fail.

⚠️ How to Spot Overfitting

Key Sign: Very high accuracy on training data but very low accuracy on testing/validation data.
Model Behavior: The model has extremely complex terms (high-degree polynomials, many interaction effects) with large coefficients that seem to "chase" individual data points.
Result: Poor generalization and unreliable predictions.

What is Underfitting?

Underfitting is the opposite problem. It occurs when a model is too simple to capture the underlying trend or relationship in the data. It fails to learn enough from the training data, resulting in poor performance on both the training data and new data.

Example 1 Linear Model on Nonlinear Data

We are analyzing the diminishing returns of advertising spend on sales. The true relationship is curved: initial spending boosts sales a lot, but extra spending has less and less effect. If we force a simple straight line (linear model) through this curved data, the line will be a poor fit.

🔍 Explanation: The linear model is too rigid. It cannot bend to capture the curvature of the true relationship. It will have high error on the training data because it misses the pattern. It will also have high error on new data because its fundamental assumption (a straight-line relationship) is wrong. The model is biased towards an incorrect form.

Example 2 Omitting Key Variables

A model predicts an individual's wage based only on years of education, ignoring crucial factors like work experience, job type, and geographic location.

🔍 Explanation: This model is under-specified. It lacks the necessary complexity to explain wage variation. Education alone cannot account for why a senior engineer earns more than a new graduate with the same degree. The model will systematically underpredict high wages and overpredict low wages, showing high error everywhere because it misses the core drivers.

⚠️ How to Spot Underfitting

Key Sign: Poor performance on both training data and testing data. The model just doesn't fit well anywhere.
Model Behavior: The model is overly simplistic (e.g., using a mean, a simple line, or too few variables when more are needed).
Result: High bias and an inability to capture important patterns, leading to consistently inaccurate predictions.

The Bias-Variance Tradeoff

Overfitting and underfitting are two sides of the bias-variance tradeoff, a fundamental concept in machine learning and econometrics.

The Bias-Variance Tradeoff Explained
Problem	Bias	Variance	Model Complexity	Performance
Underfitting	High	Low	Too Low	Bad on training & test data
Overfitting	Low	High	Too High	Good on training, bad on test
Good Fit	Balanced	Balanced	Just Right	Good on training & test data

Bias is the error from erroneous assumptions (an underfit model has high bias). Variance is the error from sensitivity to small fluctuations in the training set (an overfit model has high variance). The goal is to find the sweet spot with optimal complexity that minimizes total error.

Practical Solutions

To Fix Overfitting:

Simplify the Model: Use fewer variables, lower polynomial degrees, or apply regularization techniques like LASSO or Ridge Regression that penalize complexity.
Get More Data: More training data can help the model learn the true pattern rather than the noise.
Use Cross-Validation: Rigorously test your model on held-out data (validation sets) to ensure it generalizes.

To Fix Underfitting:

Increase Model Complexity: Add more relevant variables, include interaction terms, or try nonlinear models (like polynomials, decision trees).
Improve Features: Create better input variables (feature engineering) that more accurately capture the underlying phenomenon.
Reduce Regularization: If you are using regularization, you might be penalizing complexity too much; reduce the penalty term.