Training at Scale: Mixed Precision, Distribution, and Performance Engineering
Performance as an Algorithmic Constraint
Scaling concerns
At large scale, the bottleneck may be memory bandwidth, communication, or numerical stability rather than raw FLOPs. TensorFlow programs therefore need careful profiling, efficient batching, mixed precision, and distributed execution strategies. Performance engineering is inseparable from model design in modern deep learning.
Stability caution
Mixed precision can introduce overflow or underflow if loss scaling and numerics are not handled correctly.
Key scaling techniques
| Technique | Purpose |
|---|---|
| Mixed precision | Reduce memory use and improve accelerator throughput |
| Gradient accumulation | Simulate larger batches under memory constraints |
| Data parallelism | Replicate model across devices |
| Model parallelism | Split large models across devices |
What is the main idea behind data parallelism?
Each device processes a shard of the minibatch and gradients are aggregated.
Correct answer: Replicate the model on multiple devices and split batches
Why might gradient accumulation be useful?
Accumulating gradients across microbatches approximates a larger batch update.
Correct answer: It allows effective large-batch training when memory cannot hold a full large batch.
Distributed Strategy Concepts
Replica coordination
Distributed training requires synchronization semantics, communication-efficient gradient aggregation, and careful handling of stateful layers and random seeds. Strategies such as synchronous data parallelism favor reproducibility, while asynchronous methods may offer different throughput/consistency tradeoffs.
Which challenge becomes central in distributed training?
Distributed systems must coordinate updates across replicas to maintain a coherent training process.
Correct answer: Gradient aggregation and synchronization
Name one reason profiling is essential for large models.
Optimizing the wrong subsystem wastes effort and may not improve end-to-end throughput.
Correct answer: To identify the real bottleneck, such as input I/O, communication, or compute.