Machine Learning Theory and Generalization

Manual: General · Subject: Artificial Intelligence

Studies supervised learning, statistical risk, bias-variance trade-offs, and theoretical foundations of generalization.

Learning from Data

Statistical learning view

Machine learning seeks functions that generalize from finite data to unseen inputs. The central objects are a hypothesis class, a loss function, and a data-generating distribution.

Risk minimization

The population risk is $R(f)=\mathbb{E}_{(x,y)\sim \mathcal{D}}[\ell(f(x),y)]$ , while empirical risk uses a sample average. Learning theory asks when minimizing empirical risk also reduces population risk.

Foundational terms

Bias

Error from overly restrictive assumptions.

Variance

Sensitivity to data fluctuations.

Overfitting

Excellent training performance but poor generalization.

Regularization

Penalization or constraint to improve generalization.

Major learning paradigms

Paradigm	Objective	Typical challenge
Supervised	Predict labels or targets	Label scarcity and noise
Unsupervised	Discover structure	Evaluation ambiguity
Self-supervised	Learn from surrogate tasks	Objective design
Semi-supervised	Use few labels with many unlabeled examples	Confirmation bias

What does regularization primarily help control?

Training set size Model complexity and overfitting Label noise only Hardware latency

Why can a model with lower training loss still have worse test performance?

Bias-variance perspectives

High-bias model

Simpler decision boundary
May underfit
Often stable

High-variance model

More flexible boundary
May overfit
Sensitive to data

🔑

Open problem

Explaining why overparameterized models generalize so well remains a major theoretical challenge in modern AI.

Neuroanatomy Explorer

Memory Deck

Match the Pairs

Concept Constellation

Machine Learning Theory and Generalization

Learning from Data

Statistical learning view

Risk minimization

Bias-variance perspectives