Machine Learning Theory and Generalization
Learning from data
Supervised learning
In supervised learning, a model maps inputs to outputs by minimizing empirical loss over a dataset. The core question is not only fit to data but also performance on unseen examples.
Generalization terms
Which phenomenon describes strong training performance but weak test performance?
Overfitting occurs when a model captures noise or idiosyncrasies of the training data.
Correct answer: Overfitting
Define inductive bias in machine learning.
Without inductive bias, generalization from finite samples is impossible.
Correct answer: It is the set of assumptions that guides a model to generalize beyond observed data.
Bias-variance tradeoff
A central idea in generalization is the tradeoff between approximation bias and variance. Increasing model flexibility can reduce bias but may increase sensitivity to sampled data, raising variance.
Regularization approaches
L2 regularization
- Penalizes large weights
- Encourages smooth solutions
Early stopping
- Halts training before overfitting
- Acts as implicit regularization
Statistical learning theory
Bounds on generalization often depend on model complexity, sample size, and confidence. Frameworks such as VC dimension, Rademacher complexity, and margin theory formalize these ideas.
What does regularization primarily aim to improve?
Regularization constrains the hypothesis space or optimization path to improve test performance.
Correct answer: Generalization to unseen data
What is the difference between empirical and population risk?
This distinction lies at the heart of generalization analysis.
Correct answer: Empirical risk is measured on the sample; population risk is the expected loss on the true distribution.