Research Design, Measurement, and Validity
The Measurement Problem
Constructs and Indicators
Most social-scientific concepts are latent: trust, legitimacy, polarization, inequality, and identity are not directly observed. Measurement therefore requires a theory of how indicators relate to constructs. Good measurement is not merely precise; it is valid for the concept and context under study.
Types of Validity
Reliability and Error
Reliability concerns consistency across repeated measurement, raters, items, or time. In practice, reliability is a prerequisite for many forms of validity, but a reliable measure can still be invalid. Measurement error can be random or systematic, and both matter for inference.
Common Pitfall
A scale can predict outcomes well and still measure the wrong concept. Predictive power is not the same as construct validity.
Measurement Strategies
Direct survey items
- Simple and scalable
- Sensitive to wording and response bias
- Useful for attitudes and self-reports
Latent variable models
- Combine multiple indicators
- Can estimate measurement error
- Require stronger model assumptions
Which statement is true?
Validity is about whether the measure captures the intended construct; prediction alone is insufficient.
Correct answer: Validity concerns whether a measure captures the intended concept.
Give one example of a latent construct in social science.
Latent constructs are not directly observed and must be inferred from indicators.
Correct answer: Examples include trust, social capital, legitimacy, or political ideology.
A Minimal Measurement Workflow
-
1
Step 1: Define the construct precisely and specify scope conditions.
-
2
Step 2: Propose observable indicators grounded in theory.
-
3
Step 3: Test reliability and compare alternative operationalizations.
-
4
Step 4: Evaluate validity across groups, time, and contexts.
-
5
Step 5: Report sensitivity analyses and known limitations.