Hypothesis Testing and p-Values

Manual: General · Subject: Statistics

Test claims about populations using evidence from sample data and quantify how surprising results are.

Testing Statistical Claims

The logic of a test

Hypothesis testing compares an observed sample result to what would be expected if a null hypothesis were true. The p-value measures how incompatible the observed data are with that null model.

Hypothesis test vocabulary

Null hypothesis

H_0

The default claim, usually no effect or no difference.

Alternative hypothesis

H_a

The competing claim of interest.

Test statistic

A numerical summary that measures departure from

H_0

p-value

Probability, assuming

H_0

is true, of results at least as extreme as the observed result.

A standard hypothesis test

1

Step 1: State $H_0$ and $H_a$ clearly.
2

Step 2: Choose a test statistic and check assumptions.
3

Step 3: Compute the test statistic from the sample.
4

Step 4: Find the p-value from the relevant null distribution.
5

Step 5: Make a conclusion in context using the chosen significance level.

⚠️

Common p-value mistake

A p-value is not the probability that $H_0$ is true. It is computed assuming $H_0$ is true.

p-value and significance level

p-value

Measures data compatibility with $H_0$
Smaller values mean more surprising data under $H_0$
Computed from the sample

Significance level $\alpha$

Chosen before analysis
Threshold for rejecting $H_0$
Controls the long-run Type I error rate

If the p-value is smaller than the significance level $\alpha$ , what is the usual decision?

Fail to reject

H_0

Reject

H_0

H_0

as proven Increase the sample size

Why can statistically significant results still have little practical importance?

Neuroanatomy Explorer

Memory Deck

Match the Pairs

Concept Constellation

Hypothesis Testing and p-Values

Testing Statistical Claims

The logic of a test

A standard hypothesis test

p-value and significance level