Transfer Learning and Fine-Tuning
Learning from Existing Models
Why Transfer Learning Works
Pretrained models capture general-purpose features from large datasets. Instead of starting from scratch, you can reuse these learned representations and adapt them to a smaller or domain-specific task, often improving performance and reducing training time.
Feature Extraction vs Fine-Tuning
Feature extraction
- Freeze most pretrained layers
- Train a new task-specific head
- Faster and safer for small datasets
Fine-tuning
- Unfreeze selected layers
- Adapt pretrained features to your dataset
- Can achieve higher accuracy with careful optimization
Catastrophic Forgetting
If you fine-tune too aggressively, the model can lose useful pretrained knowledge; use small learning rates and unfreeze layers gradually.
Transfer Learning Recipe
-
1
1. Choose a pretrained backbone suited to the task.
-
2
2. Freeze or partially freeze layers.
-
3
3. Replace the final classifier or regressor head.
-
4
4. Train the new head first.
-
5
5. Unfreeze selected layers and fine-tune with a small learning rate.
When to Prefer Transfer Learning
Transfer learning is especially valuable when labeled data is limited, training compute is constrained, or the task is related to a large benchmark domain such as image classification or language understanding.
What is the first common step in feature extraction transfer learning?
Feature extraction usually keeps the pretrained layers fixed while training a new task-specific head.
Correct answer: Freeze the pretrained backbone
Why use a smaller learning rate during fine-tuning?
A small learning rate makes updates gentle enough to adapt the model without erasing useful learned representations.
Correct answer: To avoid destroying pretrained features
Transfer learning is especially helpful when
Pretrained models are often the best choice when labeled data is limited.
Correct answer: You have a small dataset