Model Architectures in TensorFlow: MLPs, CNNs, RNNs, and Transformers
Architectural families and inductive bias
From fully connected to attention-based models
Multilayer perceptrons learn general nonlinear mappings but ignore spatial or temporal structure. Convolutional networks encode locality and translation equivariance. Recurrent networks model sequence order. Transformers use self-attention to capture long-range dependencies and scale effectively with data and compute.
Architecture comparison
| Architecture | Strength | Limitation |
|---|---|---|
| MLP | Flexible universal approximator | Weak inductive bias for structured data |
| CNN | Efficient for images and grids | Limited global context without deeper design |
| RNN/LSTM/GRU | Sequential modeling | Harder to parallelize and train at scale |
| Transformer | Global context and scaling | Quadratic attention cost in sequence length |
Which architecture is most associated with self-attention?
Transformers are built around self-attention mechanisms.
Correct answer: Transformer
Attention mechanism
For queries , keys , and values , scaled dot-product attention computes This allows each token to weigh other tokens adaptively.
Open research direction
Long-context modeling remains an active frontier: sparse attention, recurrence, state-space models, memory-augmented transformers, and hybrid architectures compete on efficiency and quality.
Designing a new architecture
-
1
Identify the structure in the data: spatial, temporal, graph, or set-like.
-
2
Select an inductive bias that matches the structure.
-
3
Choose normalization, activation, and residual pathways.
-
4
Test scaling behavior with width, depth, and context length.
-
5
Measure compute cost, memory footprint, and downstream generalization.
Why are residual connections important in deep networks?
Residual paths make optimization easier and reduce degradation as depth increases.
Correct answer: They improve gradient flow and enable training of deeper models.
What is the major computational drawback of standard self-attention on long sequences?
Standard attention compares all token pairs, leading to quadratic cost in sequence length.
Correct answer: Its cost grows quadratically with sequence length
Name one method to reduce transformer memory cost on long sequences.
Sparse or approximate attention reduces pairwise interactions and lowers compute and memory requirements.
Correct answer: Sparse attention