Hypothesis Testing (CFA): Type I vs Type II Errors and p-Values

Hypothesis testing carries a reliable handful of Level 1 Quantitative Methods questions and produces a disproportionate share of avoidable errors. Almost none are arithmetic. They are setup errors, made before any number is computed.

Setting up the hypotheses

Two rules govern the setup, and getting either wrong loses the question regardless of what follows.

The null hypothesis always contains the equality. It may be stated as =, ≤ or ≥, but the equals sign is always on the null side. The alternative never contains it.

The alternative is what you are trying to establish. The null is the position of no effect, the default you are attempting to overturn with evidence. If a researcher believes a fund manager adds value, "the manager adds value" is the alternative; "the manager adds nothing" is the null.

This runs against intuition. Candidates instinctively put the interesting claim in the null because it is the thing being investigated. The framework works the other way: you assume no effect and require evidence to move away from that.

One-tailed or two-tailed

The wording of the question decides this, and specific phrases signal each.

Two-tailed — you care about a difference in either direction. Signalled by "different from", "not equal to", "has changed". H₀: μ = μ₀ against H_a: μ ≠ μ₀. The significance level is split between both tails, so at 5% each tail holds 2.5%.

One-tailed — you care about a difference in one direction only. Signalled by "greater than", "exceeds", "less than", "at least". The entire 5% sits in one tail.

The consequence is practical: a one-tailed test has a lower critical value and is easier to reject with. The same test statistic can be significant one-tailed and insignificant two-tailed, which is exactly the discrimination the exam builds questions around.

Type I and Type II errors

Four outcomes exist, and two are errors:

• Type I error: rejecting a null that is true. A false positive — concluding an effect exists when it does not. Its probability is the significance level, α, which you choose.
• Type II error: failing to reject a null that is false. A false negative — missing a real effect. Its probability is β, which you do not directly control.

Power of a test = 1 − β, the probability of correctly rejecting a false null.

A memory hook that survives exam pressure: Type I comes first, and it is the error of acting first — concluding too eagerly. Type II is the error of doing nothing when you should have acted.

The trade-off

Reducing α makes rejection harder, which necessarily raises β. Tightening the significance level from 5% to 1% cuts false positives and increases false negatives. You cannot lower both at a fixed sample size.

The only way to reduce both is to increase the sample size. A larger sample shrinks the standard error, tightening the sampling distribution and improving power without loosening the significance level. This is a favourite conceptual question, and the answer is always sample size.

The p-value

The p-value is the smallest level of significance at which the null hypothesis can be rejected. Equivalently, it is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null is true.

The decision rule is one line: reject the null when the p-value is below α.

A p-value of 0.03 means the null is rejected at the 5% level but not at the 1% level. That single sentence answers a large share of p-value questions.

What a p-value is not: it is not the probability that the null is true, and it is not the probability that your conclusion is wrong. Both misreadings appear as distractors. The p-value is computed assuming the null is true, so it cannot also be a statement about the null's probability.

Choosing the test statistic

For a test of a population mean:

• Population variance known, any sample size (normal population) → z-statistic
• Population variance unknown, normal population → t-statistic, with n − 1 degrees of freedom
• Population variance unknown, large sample, non-normal population → t-statistic acceptable; z acceptable by the central limit theorem
• Small sample, non-normal population, variance unknown → no valid test using these methods

In practice the population variance is almost never known, so the t-statistic is the working default. The t-distribution has fatter tails than the normal, so its critical values are further out and rejection is harder — appropriately, since you are estimating one more parameter.

The test statistic itself follows a consistent shape: (sample statistic − hypothesised value) / standard error of the sample statistic.

Worked example

An analyst tests whether a fund's mean monthly return differs from 1%. A sample of 36 months gives a mean of 1.4% and a sample standard deviation of 1.2%. Test at the 5% level.

H₀: μ = 1% against H_a: μ ≠ 1% — two-tailed, from "differs from".

Standard error = 1.2 / √36 = 0.2
t = (1.4 − 1.0) / 0.2 = 2.00, with 35 degrees of freedom

The two-tailed 5% critical value with 35 df is approximately 2.03. Since 2.00 < 2.03, do not reject the null.

Note how close that is. Had the test been one-tailed, the critical value would have been roughly 1.69 and the conclusion would have reversed. The tail choice, made in the setup, decided the answer.

Statistical versus economic significance

A conceptual point the curriculum makes explicitly and questions test.

A result can be statistically significant and economically worthless. With a large enough sample, a strategy generating 2 basis points of excess return can be significant at the 1% level — and be entirely consumed by transaction costs, taxes and the risk taken to obtain it.

Statistical significance says an effect is unlikely to be chance. It says nothing about whether the effect is large enough to act on. Any question presenting a significant result with a tiny effect size is asking about this distinction.

Exam checklist

• The equality always sits in the null
• The alternative is what you are trying to demonstrate
• "Different from" means two-tailed; "greater than" or "less than" means one-tailed
• Type I is a false positive with probability α; Type II is a false negative with probability β
• Power = 1 − β; only a larger sample reduces both error types
• Reject when the p-value is below α — and the p-value is not the probability the null is true
• Unknown population variance means a t-statistic