Hypothesis Testing, p-Values, and Multiple Comparisons
Classical Testing Theory
Null and alternative hypotheses
A test considers against . A rejection region is chosen to control Type I error , while power is .
Neyman-Pearson lemma
For testing simple versus simple , the likelihood ratio test is uniformly most powerful among level- tests. This result underpins much of classical testing theory.
Interpretation warning
A p-value is not the probability that is true; it is the probability of observing data at least as extreme as the data observed, under .
p-Values and evidence
Calibration
A p-value is a tail-area probability computed under the null model. Small p-values indicate incompatibility with , but evidence strength depends on sample size, model quality, and multiplicity.
Which quantity is controlled by the significance level ?
The significance level is the maximum allowable Type I error rate under the null hypothesis.
Correct answer: Probability of Type I error
Why can very small p-values still be misleading in large samples?
Statistical significance is not the same as practical significance.
Correct answer: Because even tiny, practically irrelevant departures from the null can become statistically significant with enough data.
Multiple testing and false discovery
Multiplicity
When many hypotheses are tested, naive thresholding inflates false positives. Modern methods control the family-wise error rate or the false discovery rate, often via Bonferroni, Holm, Benjamini-Hochberg, and empirical Bayes approaches.
Error rates
What does the Benjamini-Hochberg procedure primarily control?
Benjamini-Hochberg is designed to control the false discovery rate under broad conditions.
Correct answer: False discovery rate
Give one reason to prefer FDR control over family-wise error control in large-scale screening problems.
In high-dimensional settings, strict family-wise control can be too restrictive.
Correct answer: FDR control is less conservative and yields more power when many true signals are expected.