Machine Learning Theory and Modern Statistical Learning
Learning as Optimization and Inference
Core Viewpoint
Machine learning combines empirical risk minimization, statistical inference, optimization, and representation learning. At the research frontier, the central questions concern generalization in overparameterized regimes, inductive bias, robustness, and the geometry of optimization landscapes.
Notation and Objectives
A Modern Paradox
Deep networks can generalize well despite being massively overparameterized, so classical capacity control alone does not fully explain performance.
Major Theoretical Lenses
Classical statistical learning theory
- Uses VC dimension, Rademacher complexity, and uniform convergence
- Provides worst-case guarantees
Optimization-centric view
- Studies gradient descent, SGD, and implicit bias
- Explains why particular solutions are selected
What is the main distinction between empirical risk and population risk?
Empirical risk averages over observed data, while population risk averages over the true distribution.
Correct answer: Empirical risk is defined on the training sample; population risk is defined over the underlying data distribution
Why is generalization in deep learning a major open problem?
Understanding the mechanisms behind generalization remains a central theoretical challenge.
Correct answer: Because models can fit training data extremely well yet still generalize unexpectedly, beyond classical capacity explanations
Robustness and Distribution Shift
Modern ML must contend with adversarial examples, noisy labels, covariate shift, and out-of-distribution inputs. Research increasingly studies robustness as a property of optimization, geometry, and data distribution rather than a single metric.
Which quantity is most associated with controlling model complexity in linear or neural models?
Regularization strength directly penalizes complexity and can improve generalization.
Correct answer: Regularization strength
What is one common interpretation of SGD's implicit bias?
Stochastic gradient methods can select among many interpolating solutions through their dynamics.
Correct answer: That the optimization dynamics favor certain solutions, often simpler or flatter ones, even without explicit regularization