Training Deep Networks: Initialization, Regularization, and Generalization
Training dynamics
Why training fails
Training instability often comes from poor initialization, mismatched learning rates, exploding or vanishing gradients, bad normalization, inadequate regularization, or dataset shift. TensorFlow enables custom control over these factors through initializers, optimizers, constraints, and callbacks.
Regularization toolkit
Which practice most directly helps prevent overfitting when training on limited data?
Augmentation and early stopping are standard methods for reducing overfitting.
Correct answer: Using data augmentation and early stopping
Initialization and signal propagation
Initialization should keep activations and gradients within a reasonable scale. Techniques such as Xavier/Glorot and He initialization are designed to preserve variance across layers, thereby reducing pathological signal collapse or explosion.
What is the purpose of batch normalization or layer normalization?
Normalization can improve conditioning and make optimization more robust.
Correct answer: To stabilize and speed up training by normalizing activations.
Bias-variance and regularization intuition
Low regularization
- Can fit training data tightly
- May overfit noisy data
- Useful when data are abundant
High regularization
- Encourages simpler solutions
- May underfit if too strong
- Often improves robustness
Generalization caution
A lower training loss does not guarantee a better model; validation and test performance remain the decisive criteria for scientific claims.
A practical training checklist
-
1
Choose an optimizer and a learning-rate schedule.
-
2
Verify that gradients are finite and not collapsing to zero.
-
3
Monitor training and validation curves separately.
-
4
Add regularization only after confirming the baseline is stable.
-
5
Run multiple seeds to estimate variance in reported metrics.
Why is running multiple random seeds important in research evaluations?
Stochastic optimization, initialization, and data order can cause variation, so multiple seeds help assess robustness.
Correct answer: It estimates variability due to stochastic training
Name one common symptom of an excessively large learning rate.
If the learning rate is too large, optimization can overshoot minima and become unstable.
Correct answer: Diverging loss