3.9 Potential Errors When Performing Tests

Type I and Type II Errors

A Type I error occurs when we reject a true null hypothesis (a 'false alarm'). A Type II error occurs when we fail to reject a false null hypothesis (a 'missed detection'). The probability of a Type I error equals the significance level (α) chosen for the test.

Consequences Depend on Context

Errors aren't just statistical—they have real-world consequences. In a medical test, a Type I error might mean telling a healthy person they're sick; a Type II error might mean missing an actual illness. Always interpret errors in context, describing what happens in the real situation, not just abstractly.

Power of a Test

The power of a test is the probability of correctly rejecting a false null hypothesis (1 − P(Type II error)). Power increases with larger sample size, larger effect size, higher α, and less variability. Reducing Type I error risk increases Type II error risk, and vice versa—there's a tradeoff.

Key terms

Type I error

Rejecting a true null hypothesis (false positive); probability = α.

Type II error

Failing to reject a false null hypothesis (false negative); probability = β.

Power of a test

Probability of correctly rejecting a false null hypothesis; equals 1 − β.

Relationship between α and β

Decreasing α (to lower Type I error risk) increases β (Type II error risk), and vice versa, holding other factors constant.

Ways to increase power

Increase sample size, increase effect size/difference from H0, increase α, or reduce variability in the data.

Try an AP-style question

What is a Type I error?

1. Rejecting H₀ when it is false
2. Rejecting H₀ when it is actually true
3. Failing to reject H₀ when it is false
4. Failing to reject H₀ when it is true