Statistical Hypothesis Testing

What is a Hypothesis Test?

A hypothesis test starts with a claim (or assumption) about a population parameter — typically a probability $p$ or a mean $\mu$. We collect data and ask: "Is this data consistent with the claim, or does it provide evidence against it?"

The test follows a structured process:

1. Define the hypotheses.
2. Choose a significance level.
3. Collect data and calculate the test statistic.
4. Compare with the critical value (or calculate the $p$-value).
5. Draw a conclusion in context.

Null and Alternative Hypotheses

The null hypothesis $H_0$ is the default assumption — typically that nothing has changed or that a parameter takes a specified value. For example:

$H_0: p = 0.3$

The alternative hypothesis $H_1$ specifies what we suspect might be true instead. It can take one of three forms:

• One-tailed (upper): $H_1: p > 0.3$ (we suspect $p$ is larger)
• One-tailed (lower): $H_1: p < 0.3$ (we suspect $p$ is smaller)
• Two-tailed: $H_1: p \neq 0.3$ (we suspect $p$ is different, but don't specify the direction)

The choice of $H_1$ depends on the context of the problem and must be decided before looking at the data.

Significance Level

The significance level $\alpha$ is the probability of incorrectly rejecting $H_0$ when it is actually true (a Type I error). Common significance levels are:

• $\alpha = 0.05$ (5%) — the most common
• $\alpha = 0.01$ (1%) — more stringent
• $\alpha = 0.10$ (10%) — more lenient

The smaller the significance level, the stronger the evidence needed to reject $H_0$.

The Binomial Test

At A-Level, many hypothesis tests involve a binomial distribution. If we observe $X$ successes in $n$ independent trials, each with probability $p$ of success, then under $H_0$:

$X \sim B(n, p_0)$

where $p_0$ is the value of $p$ specified in $H_0$.

To carry out the test, we calculate the probability of obtaining a result as extreme as (or more extreme than) the observed value, assuming $H_0$ is true.

For a one-tailed test ($H_1: p < p_0$):

Calculate $P(X \leq x_{\text{obs}})$ under $H_0$. If this probability is less than $\alpha$, reject $H_0$.

For a one-tailed test ($H_1: p > p_0$):

Calculate $P(X \geq x_{\text{obs}})$ under $H_0$. If this probability is less than $\alpha$, reject $H_0$.

For a two-tailed test ($H_1: p \neq p_0$):

Calculate the probability in the relevant tail and compare with $\frac{\alpha}{2}$ (since the significance level is split between both tails).

Critical Regions and Critical Values

The critical region is the set of values of the test statistic that lead to rejection of $H_0$. The critical value is the boundary of this region.

For a binomial test with $H_1: p < p_0$ at the 5% level, the critical region consists of all values $x$ such that $P(X \leq x) < 0.05$. The largest such $x$ is the critical value.

For $H_1: p > p_0$, the critical region is in the upper tail: values $x$ such that $P(X \geq x) < 0.05$.

When the observed value falls in the critical region, we reject $H_0$. When it falls outside, we do not reject $H_0$.

The Actual Significance Level

Because the binomial distribution is discrete, we usually cannot achieve exactly $\alpha = 0.05$. The actual significance level is the probability of the critical region, which is as close to $\alpha$ as possible without exceeding it.

For example, if the critical region is $X \leq 2$ and $P(X \leq 2) = 0.0382$, then the actual significance level is $3.82\%$, not exactly $5\%$.

Type I and Type II Errors

A Type I error occurs when we reject $H_0$ when it is actually true. The probability of a Type I error equals the significance level $\alpha$.

A Type II error occurs when we fail to reject $H_0$ when it is actually false. The probability of a Type II error depends on the true value of the parameter and is harder to calculate.

	$H_0$ true	$H_0$ false
Reject $H_0$	Type I error	Correct decision
Don't reject $H_0$	Correct decision	Type II error

Writing Conclusions

Conclusions must be written in context and with appropriate language:

• Reject $H_0$: "There is sufficient evidence at the $\alpha$ significance level to reject $H_0$ and conclude that [contextual statement about $H_1$]."
• Do not reject $H_0$: "There is insufficient evidence at the $\alpha$ significance level to reject $H_0$. There is no significant evidence that [contextual statement about $H_1$]."

Important: we never say we "accept $H_0$" — we only say we "do not reject" it, because failing to find evidence against $H_0$ is not the same as proving it true.

Tip 1: Read the Context Carefully

The wording of the question tells you which alternative hypothesis to use. Phrases like "believes the proportion has increased" suggest $H_1: p > p_0$; "claims it has changed" suggests $H_1: p \neq p_0$.

Tip 2: Set Up Hypotheses Before Calculating

Always write down $H_0$ and $H_1$ before doing any calculations. This ensures you test the correct tail and use the correct comparison.

Tip 3: Use the Correct Tail Probability

For $H_1: p < p_0$, calculate $P(X \leq x)$ (lower tail). For $H_1: p > p_0$, calculate $P(X \geq x)$ (upper tail). Mixing these up is one of the most common errors.

Tip 4: State the Distribution Under $H_0$

Explicitly write "Under $H_0$, $X \sim B(n, p_0)$". This earns a method mark and shows the examiner you understand the test framework.

Tip 5: Always Conclude in Context

A conclusion that says only "reject $H_0$" without reference to the real-world situation will lose marks. Always relate your answer back to the scenario described in the question.