Regularization, Generalization, and Model Tuning
Controlling Complexity
What Regularization Does
Regularization reduces the gap between training and validation performance by constraining model complexity or discouraging overconfident fitting. Common approaches include dropout, weight decay, early stopping, and data augmentation.
Regularization Techniques
Dropout
- Randomly disables units during training
- Encourages redundant representations
- Usually turned off at inference
$L_2$ weight decay
- Penalizes large weights
- Encourages smoother solutions
- Implemented as a parameter penalty
Early Stopping
Stop training when validation performance stops improving for several epochs to avoid wasting compute and overfitting.
Hyperparameters vs Parameters
Parameters are learned from data, while hyperparameters are set by you. Examples of hyperparameters include learning rate, batch size, layer width, dropout rate, and number of epochs.
Practical Tuning Strategy
-
1
1. Start with a simple baseline.
-
2
2. Adjust one or two hyperparameters at a time.
-
3
3. Track validation metrics, not just training loss.
-
4
4. Use early stopping and checkpoints.
-
5
5. Save the best configuration and document it.
Which method randomly disables neurons during training?
Dropout randomly zeroes activations during training to reduce co-adaptation and overfitting.
Correct answer: Dropout
What is the purpose of early stopping?
Early stopping monitors validation metrics and ends training before the model over-specializes.
Correct answer: Prevent overfitting by stopping when validation performance stops improving
Which of the following is a hyperparameter?
Learning rate is chosen by the practitioner, not learned from data.
Correct answer: Learning rate