Neural Networks and Deep Learning
Learning Representations
Basic Neural Network Idea
A neural network is a composition of simple units arranged in layers. Each layer transforms its input, allowing the network to learn increasingly abstract representations.
Activation Functions
Nonlinear activation functions such as ReLU allow networks to represent complex patterns. Without nonlinearity, stacked layers collapse into a single linear transformation.
Backpropagation
Backpropagation computes gradients of the loss with respect to network parameters using the chain rule. Those gradients guide updates such as gradient descent.
Why are nonlinear activation functions important?
Nonlinear activations let multilayer networks represent much richer functions.
Correct answer: They allow networks to model complex relationships
What does backpropagation compute?
These gradients are used to update weights and biases.
Correct answer: Gradients of the loss with respect to parameters.
Deep Learning Components
Layers
- Transform representations step by step
- Early layers capture simple patterns
- Later layers capture higher-level features
Loss function
- Measures prediction error
- Provides training signal
- Should match the task
Training Challenges
| Issue | Cause | Typical Response |
|---|---|---|
| Vanishing gradients | Repeated multiplication of small derivatives | Use better activations or normalization |
| Overfitting | Model too flexible for the data | Regularization and more data |
| Slow convergence | Poor learning rate or scaling | Tune optimization settings |
| Underfitting | Model too simple | Increase capacity |
What is the role of a loss function?
The loss quantifies error and drives optimization.
Correct answer: To measure how far predictions are from desired outputs