Automatic Differentiation, Gradients, and Optimization Theory

Manual: General · Subject: TensorFlow & AI Programming

Understand gradient computation, backpropagation, and optimization at the level needed for research and custom model design.

Differentiation as a computational primitive

Why gradients matter

Training neural networks is the problem of minimizing an empirical risk objective $\mathcal{L}(\theta) = \frac{1}{N}\sum_{i=1}^N \ell(f_\theta(x_i), y_i)$ by following gradients and curvature information. TensorFlow's automatic differentiation enables exact derivatives of the implemented computation, subject to floating-point arithmetic.

Gradient vocabulary

Tape

Records operations for reverse-mode automatic differentiation

Variable

Trainable state updated by optimization

Jacobian

Matrix of partial derivatives of outputs with respect to inputs

Hessian

Second-order derivative information

Stopping gradient

Prevents gradient flow through a subgraph

In deep learning, reverse-mode automatic differentiation is especially efficient when the model has:

Many outputs and few parameters Few outputs and many parameters No differentiable operations Only linear layers

Chain rule in deep networks

Backpropagation is repeated application of the chain rule. If $z = f(g(x))$ , then $\frac{dz}{dx} = \frac{dz}{dg}\frac{dg}{dx}$ . In deep nets, TensorFlow composes these local derivatives across layers and control flow.

Using GradientTape correctly

1

Enter a tape context and ensure the watched quantities are TensorFlow variables or explicitly watched tensors.
2

Compute the forward pass and scalar loss.
3

Call tape.gradient(loss, variables) to obtain gradients.
4

Inspect gradient norms to detect vanishing, exploding, or disconnected gradients.
5

Apply gradients using an optimizer or custom update rule.

⚠️

Common pitfall

If a tensor is created outside the tape and is not a variable or explicitly watched, its gradient may be None.

What scalar quantity do we typically differentiate in supervised learning?

First-order vs second-order methods

First-order methods

Use gradient information only
Scale well to large models
Often sufficient for deep learning

Second-order methods

Use curvature information such as the Hessian
Can improve convergence
Are usually more expensive at scale

Why is the Hessian rarely formed explicitly for very large neural networks?

Because it is always singular Because it is too expensive in memory and computation Because TensorFlow cannot compute it Because gradients are unavailable

Neuroanatomy Explorer

Memory Deck

Match the Pairs

Concept Constellation