Potential Errors When Performing Tests

SECTION 1

Historical Context & Motivation

Statistical hypothesis testing did not arrive fully formed; it evolved through decades of intellectual debate about how scientists should weigh evidence against uncertainty. The fundamental tension—how do we decide when data are surprising enough to reject a claim?—naturally carries the risk of reaching the wrong conclusion. Two pioneers, Jerzy Neyman and Egon Pearson, formalized the idea that hypothesis tests are decision procedures that can fail in exactly two directions: rejecting a true null hypothesis or failing to reject a false one. Their framework gave statisticians the vocabulary—Type I and Type II errors—that remains central to every AP Statistics course and every modern research study.

1900s

Foundations of Significance Testing

Karl Pearson develops the chi-square goodness-of-fit test, introducing the concept of measuring how well observed data conform to a theoretical distribution—an early step toward formalized hypothesis testing.

1925

Fisher's p-value Framework

Ronald Fisher publishes "Statistical Methods for Research Workers," popularizing p-values and the notion of significance levels. However, Fisher's framework does not formally define an alternative hypothesis or error types.

1933

Neyman–Pearson Lemma

Jerzy Neyman and Egon Pearson publish their foundational paper, formalizing hypothesis testing as a decision between two hypotheses and explicitly naming Type I error (α) and Type II error (β). This framework introduces the concept of statistical power.

1945–1960

Adoption in Applied Sciences

Medical trials, quality control in manufacturing, and social science research adopt the Neyman–Pearson decision framework, making error control a practical concern in fields far beyond pure mathematics.

2000s–Present

Replication Crisis & Renewed Focus

Widespread failures to replicate published findings across psychology and medicine prompt a reexamination of how researchers handle Type I and Type II error rates, reinforcing the importance of understanding these concepts.

The central question this lesson addresses is deceptively simple: when we perform a hypothesis test for a population proportion, what can go wrong with our conclusion, how do we quantify those risks, and what levers do we have to control them? Answering this question is not merely an academic exercise—it is the foundation for making responsible inferences from data in medicine, policy, and science.

SECTION 2

Core Principles & Definitions

Every hypothesis test begins with two competing claims about a population parameter: the null hypothesis (H₀), which represents the status quo, and the alternative hypothesis (Hₐ), which represents the researcher's claim. Because we base our decision on sample data rather than a census, there is always the possibility that the data mislead us. Two distinct mistakes can occur, and they are not symmetric in their consequences.

Type I Error (False Positive)

Rejecting H₀ when H₀ is actually true. The probability of committing this error equals α, the significance level chosen before the test is conducted.

Type II Error (False Negative)

Failing to reject H₀ when H₀ is actually false. The probability of this error is denoted β, which depends on the true value of the parameter and the sample size.

Significance Level (α)

The pre-set threshold for how much evidence against H₀ we require before rejecting it. Common choices are α = 0.05 and α = 0.01. Lowering α reduces the Type I error rate but increases the Type II error rate.

Power (1 − β)

The probability of correctly rejecting H₀ when it is false. High power means the test is sensitive enough to detect a real departure from the null. Power increases with larger sample sizes and larger true effect sizes.

✦ KEY TAKEAWAY

Think of a hypothesis test like a smoke detector. A Type I error is a false alarm—the detector goes off when there is no fire. A Type II error is a missed alarm—the detector stays silent while a fire is burning. You can make the detector more sensitive (lower α threshold), but then it is more likely to cry wolf. The engineering challenge—just like the statistical one—is finding the right balance between sensitivity and specificity.

SECTION 3

Visual Explanation: The Decision Matrix

The relationship between reality and our test decision can be summarized in a 2 × 2 decision matrix. The columns represent the true state of the world (which we never know for certain), and the rows represent our decision based on sample data. Two cells are correct outcomes, and two cells are errors. The diagram below illustrates this framework with color coding: green for correct decisions and red/amber for errors.

The decision matrix shows four possible outcomes of a hypothesis test. The upper-left cell (Type I error) and lower-right cell (Type II error) represent incorrect conclusions. The two green cells represent correct decisions. Notice that the probability of a Type I error is entirely set by the researcher's choice of α, while the probability of a Type II error is influenced by multiple factors.

A critical insight from this matrix is that you can never make both errors simultaneously—they arise under mutually exclusive realities. If H₀ is true, the only error possible is Type I. If H₀ is false, the only error possible is Type II. This is why reducing α (moving the rejection threshold) inevitably affects β: when you demand stronger evidence before rejecting H₀, you become more cautious about false positives but more vulnerable to missing real effects.

SECTION 4

Mathematical Framework

When performing a one-proportion z-test, we compute a test statistic and compare it to the standard normal distribution. The rejection region is determined by our chosen significance level α. Understanding the mathematics behind the error probabilities clarifies exactly how they are connected to sample size, the true proportion, and the significance level.

TEST STATISTIC FOR ONE-PROPORTION Z-TEST

z = (p̂ − p₀) / √(p₀(1 − p₀) / n)

Where p̂ is the sample proportion, p₀ is the hypothesized proportion under H₀, and n is the sample size.

TYPE I ERROR PROBABILITY

P(Type I Error) = P(reject H₀ | H₀ is true) = α

This is a direct consequence of our choice: by setting α = 0.05, we accept that 5% of the time we will incorrectly reject a true null hypothesis.

TYPE II ERROR PROBABILITY

β = P(fail to reject H₀ | H₀ is false)

β depends on the true proportion p, the hypothesized p₀, the sample size n, and the significance level α. There is no single formula to memorize; rather, β is computed by finding the probability that the test statistic falls outside the rejection region under the true parameter value.

POWER OF THE TEST

Power = 1 − β = P(reject H₀ | H₀ is false)

Power increases when (1) the sample size n increases, (2) the true proportion is farther from p₀ (larger effect size), or (3) α is increased. Researchers typically aim for power ≥ 0.80.

The interplay between α and β is often described as a tradeoff: for a fixed sample size, decreasing α (demanding more stringent evidence) shifts the critical value farther from the center of the null distribution, making it harder to reach the rejection region even when H₀ is false, thereby increasing β. The only way to decrease both α and β simultaneously is to increase the sample size, which narrows both distributions and separates them more cleanly.

SECTION 5

Factors Affecting Errors & Real-World Consequences

Understanding the factors that influence each type of error is essential for designing responsible studies and interpreting results correctly. The diagram below illustrates how key factors shift the balance between Type I and Type II error rates, with particular attention to the proportions context on the AP Statistics exam.

This diagram shows four factors that influence error rates. The significance level (α) is the only factor that directly controls Type I error. Sample size, effect size, and power all affect Type II error. The dashed line between the two error circles emphasizes the inherent tradeoff.

Contextualizing Errors in Proportion Tests

On the AP Statistics exam, you will often be asked to describe what a Type I or Type II error means in context. This means you must translate the abstract definitions into a specific scenario. For example, suppose a pharmaceutical company tests whether the proportion of patients who experience side effects from a new drug exceeds 0.10. A Type I error would mean concluding that the side-effect rate exceeds 0.10 when it actually does not—potentially pulling a safe drug from the market. A Type II error would mean failing to detect that the side-effect rate exceeds 0.10 when it truly does—allowing a harmful drug to remain available. The consequences of each error are asymmetric, and this asymmetry should inform the choice of α.

Summary of how key factors influence each error type in a one-proportion z-test

Factor	Effect on Type I Error (α)	Effect on Type II Error (β)
Increase α (e.g., 0.05 → 0.10)	Increases (more false positives)	Decreases (fewer missed effects)
Decrease α (e.g., 0.05 → 0.01)	Decreases (fewer false positives)	Increases (more missed effects)
Increase sample size n	No change (still equals α)	Decreases (more power)
Larger true effect \|p − p₀\|	No change (still equals α)	Decreases (easier to detect)

SECTION 6

Worked Example: Errors in a Proportion Test

A school board claims that 60% of parents in the district support a proposed curriculum change. A parent group suspects the true proportion is less than 0.60 and surveys a random sample of 200 parents. They plan to test at the α = 0.05 significance level. Out of the 200 sampled parents, 108 express support (p̂ = 0.54). Let us walk through the hypothesis test and identify what errors would mean in context.

Testing a Claim About Parent Support

Step 1 — State HypothesesH₀: p = 0.60 (the true proportion of parents who support the change is 0.60). Hₐ: p < 0.60 (the true proportion is less than 0.60). This is a one-sided (left-tailed) test.

Step 2 — Check ConditionsRandom: The sample is stated to be random. Independence: n = 200 is almost certainly less than 10% of all parents in the district (10% condition). Normal: np₀ = 200 × 0.60 = 120 ≥ 10 and n(1 − p₀) = 200 × 0.40 = 80 ≥ 10. All conditions are met.

Step 3 — Compute the Test Statisticz = (p̂ − p₀) / √(p₀(1 − p₀) / n) = (0.54 − 0.60) / √(0.60 × 0.40 / 200) = −0.06 / √(0.0012) = −0.06 / 0.03464 ≈ −1.732.

z ≈ −1.732

Step 4 — Find the p-value and Make a DecisionUsing the standard normal table or a calculator, P(Z < −1.732) ≈ 0.0416. Since 0.0416 < 0.05 = α, we reject H₀. There is sufficient evidence at the 5% significance level to conclude that less than 60% of parents support the curriculum change.

p-value ≈ 0.0416 → Reject H₀

Step 5 — Describe Errors in ContextType I Error: We concluded that fewer than 60% of parents support the change, but in reality, 60% (or more) actually do. Consequence: the school board may abandon a popular initiative based on misleading sample data. Type II Error: We failed to reject the claim that 60% support the change, but in reality, fewer than 60% actually support it. Consequence: the school board proceeds with a curriculum change that lacks majority support.

Always describe errors using the specific context of the problem, not just α and β.

📝 AP Exam Tip

On free-response questions, the AP readers require you to describe Type I and Type II errors in context. Simply writing "rejecting H₀ when H₀ is true" will not earn full credit. You must reference the specific parameter and scenario—for instance, "concluding that the proportion of parents who support the change is less than 0.60 when it is actually 0.60."

SECTION 7

Strengths, Limitations & the Error Tradeoff

The Neyman–Pearson framework gives us a powerful and structured way to make decisions under uncertainty, but it comes with inherent limitations. Understanding when the framework works well and when it can mislead you is critical for interpreting results responsibly, both on the AP exam and in professional practice.

Strengths and limitations of the Neyman–Pearson hypothesis testing framework

Strengths	Limitations
Provides a clear, reproducible decision rule (reject or fail to reject) based on pre-specified α.	The binary decision (reject/fail to reject) discards information about the strength of evidence beyond the threshold.
Type I error rate is directly controlled by the researcher's choice of α.	Type II error rate is harder to control and depends on unknown quantities (the true parameter value).
Power analysis allows researchers to plan adequate sample sizes before collecting data.	A statistically significant result does not necessarily imply practical significance; a tiny but real difference can be detected with large n.
The framework is widely understood and accepted, enabling standardized communication across disciplines.	Multiple testing inflates the overall Type I error rate unless corrections (e.g., Bonferroni) are applied.

⚖️ KEY TAKEAWAY

Choosing α is like calibrating a scale of justice. In a criminal trial, we set a very high bar for conviction (small α) because a Type I error (convicting an innocent person) is considered worse than a Type II error (acquitting a guilty person). In contrast, a screening test for a dangerous disease might use a larger α because missing a sick patient (Type II) could be fatal. The choice is always driven by which error has worse consequences in the given context.

SECTION 8

Connection to Advanced Topics

The concepts of Type I error, Type II error, and power extend well beyond one-proportion z-tests. On the AP Statistics exam and in college-level coursework, you will encounter these ideas in the context of two-proportion z-tests, chi-square tests for homogeneity and independence, and t-tests for means. The underlying logic of the decision matrix remains identical—only the test statistic and distribution change.

How this lesson's concepts connect to advanced statistics topics

Topic in This Lesson	Advanced Extension
Type I / Type II errors in a single proportion test	Errors in two-proportion z-tests and chi-square tests (same decision matrix, different test statistics)
Fixed significance level α	Multiple comparison corrections (Bonferroni, Benjamini–Hochberg) that adjust α when performing many tests simultaneously
Power as 1 − β	Formal power analysis and sample size determination formulas used in research design
Consequences of errors described in context	Decision theory and loss functions that assign numeric costs to each type of error for optimal decision-making

Looking ahead, introductory courses in Bayesian statistics offer a fundamentally different perspective on inference: rather than classifying decisions into binary error categories, Bayesian methods update the probability of a hypothesis given the data. However, the frequentist error framework you learn in AP Statistics remains the dominant paradigm in many scientific fields, and a deep understanding of its strengths and pitfalls will serve you well in any future statistics course.

SECTION 9

Practice Problems

A researcher tests H₀: p = 0.40 versus Hₐ: p > 0.40 at the α = 0.05 level. She rejects H₀. Later, it is discovered that the true population proportion is actually 0.40. Which type of error has been committed?

A city health department tests H₀: p = 0.15 versus Hₐ: p ≠ 0.15 at α = 0.01, where p is the proportion of restaurant inspections that reveal critical violations. If the test has a power of 0.72 against the true proportion p = 0.20, what is the probability of a Type II error?

A polling organization tests H₀: p = 0.50 versus Hₐ: p > 0.50 at α = 0.05 using a sample of n = 400 voters. They obtain p̂ = 0.535 and compute a p-value of 0.081. The organization fails to reject H₀. However, the true proportion of voters who support the ballot measure is p = 0.54. Describe the error (if any) that has been made, identify which type of error it is, and describe the consequence of this error in context.

PROBLEM 4 — APPLIED

A quality control engineer at a bottling plant monitors the proportion of defective bottles. The current standard is that no more than 3% of bottles are defective. Each day, the engineer takes a random sample of 500 bottles and tests H₀: p = 0.03 versus Hₐ: p > 0.03 at α = 0.05. (a) Describe what a Type I error and a Type II error would mean in this context, including the consequences of each. (b) The engineer considers lowering α to 0.01. Explain the effect this change would have on the probabilities of Type I and Type II errors, and discuss whether this is a good idea given the consequences you described in part (a). (c) Instead of changing α, the engineer proposes increasing the daily sample size from 500 to 1,000 bottles. Explain how this change affects the probability of each type of error.

PROBLEM 5 — CRITICAL THINKING

A medical researcher is designing a study to test whether a new treatment changes the proportion of patients who achieve remission from a disease. The current remission rate with standard treatment is p₀ = 0.30. She tests H₀: p = 0.30 versus Hₐ: p ≠ 0.30 at α = 0.05. (a) If the true remission rate with the new treatment is p = 0.35, explain why a sample of n = 100 might have low power. Use the standard error formula to support your reasoning. (b) The researcher argues: "If my test is not significant, it means the new treatment has no effect." Critique this statement by referencing the concepts of Type II error and power. (c) A colleague suggests that if the researcher runs the same test on 20 different patient subgroups at α = 0.05, approximately one subgroup will show a significant result by chance alone even if the treatment truly has no effect. Explain why this is the case and name one method for addressing this problem. (d) Given that approving an ineffective treatment wastes resources but failing to approve an effective treatment costs lives, argue for a specific choice of α and justify your reasoning in terms of Type I and Type II error consequences.

SUMMARY

Lesson Summary

Every hypothesis test about a population proportion carries the risk of two types of errors. A Type I error (false positive) occurs when we reject H₀ even though it is true, and its probability equals the significance level α. A Type II error (false negative) occurs when we fail to reject H₀ even though the alternative hypothesis Hₐ is true, and its probability is denoted β. The power of the test (1 − β) measures the test's ability to correctly detect a real effect and increases with larger sample sizes and larger effect sizes.

The fundamental tradeoff is that decreasing α reduces Type I error but increases Type II error for a fixed sample size. The only way to reduce both simultaneously is to increase the sample size. On the AP exam, always describe errors in the specific context of the problem—stating what it would mean to reject or fail to reject H₀ if you are wrong—and consider which error has more serious practical consequences when justifying your choice of significance level.