Hypothesis Testing and p-values
Testing as a decision problem
Null and alternative hypotheses
A hypothesis test evaluates against using a test statistic and a rejection rule. Classical theory focuses on controlling Type I error and understanding power .
Interpretation caution
A p-value is not the probability that the null hypothesis is true; it is the probability, under , of observing data at least as extreme as what was seen.
Type I error is the probability of what event?
Type I error is rejecting when is true.
Correct answer: Rejecting a true null
Define statistical power.
Power is and measures sensitivity to real effects.
Correct answer: The probability of rejecting the null hypothesis when the alternative is true.
Neyman-Pearson lemma
For simple versus simple hypotheses, the most powerful test of size is based on the likelihood ratio. This result motivates likelihood ratio tests and many optimality arguments.
The likelihood ratio test is primarily based on comparing what?
Likelihood ratio tests compare the maximized likelihood under the null and alternative models.
Correct answer: Nested model likelihoods
What does a smaller p-value generally indicate?
It is evidence against under the test's assumptions, not a posterior probability.
Correct answer: Stronger incompatibility between the observed data and the null model.
Error types
| Error type | Meaning |
|---|---|
| Type I | Rejecting a true null |
| Type II | Failing to reject a false null |
| Power | Correctly rejecting a false null |