Regression Analysis

Regression models the relationship between a dependent variable y and one or more independent variables x. Simple linear regression fits a straight line y = β₀ + β₁x + ε, where β₀ is the intercept, β₁ is the slope, and ε represents the random error. The slope tells you the expected change in y for a one-unit increase in x. Ordinary Least Squares (OLS) finds the β values that minimize the sum of squared vertical distances from the data points to the line. OLS has a beautiful closed-form matrix solution: β̂ = (X^TX)^{-1}X^Ty.

The coefficient of determination R² measures the proportion of variance in y explained by the model. R² = 1: perfect prediction. R² = 0: no better than just predicting the mean. However, R² ALWAYS increases when you add more predictors, even random noise variables. The adjusted R² penalizes model complexity to enable fair comparison. R² also says nothing about causality or model correctness. Hypothesis testing on coefficients provides complementary evidence.

With many predictors, OLS can overfit: the model memorizes noise patterns that do not generalize. Regularization adds a penalty to the OLS objective. Ridge (L2) adds λΣβ_i², shrinking all coefficients toward zero but never to exactly zero — good when all features matter. Lasso (L1) adds λΣ|β_i|, driving some coefficients to EXACTLY zero — automatic feature selection. Elastic Net combines both: λ₁Σ|β_i| + λ₂Σβ_i². The hyperparameter λ controls strength and is chosen by cross-validation.

The OLS line y = β₀ + β₁x passes through (x̄,ȳ) — the center of mass of the data. Use the function plotter to explore how changing slope and intercept affects predictions.