Model Evaluation, Metrics, and Validation Strategy
Evaluating What Your Model Actually Learned
Why Evaluation Is Separate from Training
A model can minimize training loss while still performing poorly on unseen data. Validation and test sets estimate generalization, which is the ability to perform well on new examples rather than memorized training samples.
Common Metrics by Task
| Task | Typical Metrics | Interpretation |
|---|---|---|
| Binary classification | Accuracy, precision, recall, AUC | How well classes are separated |
| Regression | MAE, MSE, RMSE | How close predictions are to numeric targets |
| Multi-class classification | Accuracy, top- accuracy | How often the correct class is ranked highly |
Data Splitting
A standard split uses training data for learning, validation data for tuning, and test data for final reporting.
Diagnosing Failure Modes
High training performance with low validation performance usually indicates overfitting. Low performance on both sets suggests underfitting, weak features, or an unsuitable model architecture.
Evaluation Checklist
-
1
1. Confirm the train/validation/test split is clean.
-
2
2. Choose metrics that match the business objective.
-
3
3. Inspect confusion matrices or error distributions when appropriate.
-
4
4. Compare performance against a baseline.
-
5
5. Re-evaluate after hyperparameter changes.
What does a validation set primarily help you do?
Validation data is used for model selection and tuning while preserving the test set for final evaluation.
Correct answer: Tune choices without using the test set
What is overfitting?
Overfitting means the model has learned training-specific patterns or noise that do not generalize.
Correct answer: When a model performs well on training data but poorly on new data
Which metric is most appropriate for regression?
MSE measures average squared prediction error for numeric outputs.
Correct answer: MSE