Linear Models, Regression, and Semiparametric Methods

Manual: General · Subject: Statistics

Analyze the geometry, inference, and extensions of linear regression and related semiparametric estimators.

Linear regression as projection

Model specification

The linear model is $Y=X\beta+\varepsilon$ , where $X$ is the design matrix and $\varepsilon$ has mean zero. Ordinary least squares solves $\hat\beta=(X^TX)^{-1}X^TY$ when $X^TX$ is invertible.

Geometry of least squares

OLS is the orthogonal projection of $Y$ onto the column space of $X$ . This yields the normal equations $X^T(Y-X\hat\beta)=0$ and clarifies how collinearity affects estimation.

💡

Hat matrix

The fitted values are $\hat Y=HY$ with $H=X(X^TX)^{-1}X^T$ . The diagonal entries of $H$ measure leverage.

Inference and diagnostics

Standard errors and residuals

If $\varepsilon\sim N(0,\sigma^2 I)$ , then $\hat\beta$ is exactly normal. Heteroskedasticity, autocorrelation, and influential observations require robust variance estimators and diagnostic checks.

What does the hat matrix represent?

Projection onto the fitted subspace Covariance of errors Prior precision A hypothesis test

Why are robust standard errors needed?

Semiparametric and high-dimensional regression

Modern extensions

Semiparametric methods combine parametric structure with nonparametric nuisance components. High-dimensional regression uses penalization such as the lasso, ridge, elastic net, debiased lasso, and double machine learning to handle $p\gg n$ settings.

Regularization toolkit

Ridge

Shrinks coefficients via an

\ell_2

penalty.

Lasso

Encourages sparsity via an

\ell_1

penalty.

Elastic net

Combines

\ell_1

and

\ell_2

penalties.

Double machine learning

Orthogonalizes nuisance estimation to enable valid inference.

Which penalty tends to produce sparse coefficient estimates?

Lasso Ridge No penalty Principal components

What is one advantage of double machine learning?

Neuroanatomy Explorer

Memory Deck

Match the Pairs

Concept Constellation