4.7 Potential Errors When Performing Tests

Type I and Type II Errors

A Type I error occurs when we reject a true null hypothesis (a false positive) — its probability equals alpha (α). A Type II error occurs when we fail to reject a false null hypothesis (a false negative) — its probability is called beta (β). You can only make a Type I error if H₀ is actually true, and only a Type II error if H₀ is actually false.

The Tradeoff Between Errors

Lowering α (like using 0.01 instead of 0.05) reduces Type I error risk but increases Type II error risk, since it's harder to reject H₀. The consequences of each error type depend on context — sometimes a false positive is worse (unnecessary treatment), sometimes a false negative is worse (missing a real effect). Choose α based on which error is more costly.

Power of a Test

Power is the probability of correctly rejecting a false null hypothesis, equal to 1 − β. Power increases with larger sample size, larger effect size (true parameter farther from H₀ value), higher α, and lower variability in the population. Increasing power reduces the chance of a Type II error but not a Type I error.

Key terms

Type I Error

Rejecting a true null hypothesis (false positive); probability = α.

Type II Error

Failing to reject a false null hypothesis (false negative); probability = β.

Power of a test

The probability of correctly rejecting a false null hypothesis; equals 1 − β.

How to increase power

Increase sample size, increase effect size, increase α, or decrease population variability.

Effect of lowering α

Decreases risk of Type I error but increases risk of Type II error (lowers power).

Try an AP-style question

H₀: μ = 500 mg (the pills are fine). What is a Type II error here?

1. Correctly finding that the mean dose differs from 500 mg
2. Concluding the mean dose differs when it's really 500 mg
3. Failing to detect that the mean dose differs from 500 mg when it does
4. Correctly finding no evidence against the 500 mg claim