Generative Models and Self-Supervised Learning
Learning to Generate
The generative objective
Generative models learn the data distribution so they can synthesize new samples, fill in missing information, or support latent-variable inference. They are foundational for language models, image generation, and multimodal systems.
Generative paradigms
Autoregressive models
- Factorize the joint distribution sequentially
- Strong likelihood modeling
- Sampling can be slow
Latent-variable models
- Introduce hidden structure
- Compact representations
- Inference may require approximation
Self-supervised learning
Self-supervision creates training signals from raw data, such as next-token prediction, masked reconstruction, or contrastive objectives. This reduces dependence on labeled data and enables large-scale foundation models.
Core terms
What is a main reason self-supervised learning scales well?
The model learns from the data itself, making it possible to exploit very large unlabeled corpora.
Correct answer: It uses structure in unlabeled data to construct training signals
Why do diffusion models often require many sampling steps?
The generative path is constructed as a gradual refinement process.
Correct answer: Because they iteratively reverse a noising process through a sequence of small denoising steps.
Current frontier
A major research direction is making generative models more controllable, reliable, efficient, and grounded in external knowledge.
Evaluating a generative model
-
1
Step 1: Measure likelihood or proxy likelihood where applicable.
-
2
Step 2: Assess sample quality and diversity.
-
3
Step 3: Test conditional control and editing ability.
-
4
Step 4: Evaluate calibration and hallucination rates.
-
5
Step 5: Check robustness under distribution shift.