Hypothesis Tests and the p-Value
If the null hypothesis $H_0: p = p_0$ is true, the sample proportion $\hat{p}$ is approximately normal with mean $p_0$ and standard deviation $\sqrt{\dfrac{p_0(1-p_0)}{n}}$. The test statistic is $z = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$. The p-value is the probability, if $H_0$ is true, of a result as extreme as or more extreme than the one observed: the red area on the plot. The dashed line is the critical value for your $\alpha$.
What to notice
- The p-value is not "the probability that $H_0$ is true". It is computed assuming $H_0$ is true and measures how surprising the data are.
- If the p-value is below $\alpha$, $H_0$ is rejected; otherwise we "fail to reject $H_0$". We never "accept $H_0$": failing to find evidence is not evidence of absence.
- Keep $\hat{p}$ fixed and increase $n$: the same difference becomes significant in a large sample. Statistical significance does not mean the difference is large or important.
- On the same data the two-sided p-value is twice the one-sided one. The alternative hypothesis must be chosen before looking at the data.
- The decision flips exactly when $\hat{p}$ crosses the dashed line: the critical-value method and the p-value method always agree.