Feb 18

Fido explains Type I & Type II Errors

by David Harper, CFA, FRM, CIPM

FRM | Quant |

Thelma and her dog Fido are friends of the Bionic Turtle. They offered to help us understand an important statistical concept, Type I and Type II errors.

While driving to the dog park, Thelma and Fido approach an intersection, just as the traffic light turns yellow. Under the circumstance, Thelma has two choices: hit the brakes or hit the accelerator. In her head, Themla quickly formulates a null hypothesis: the traffic light will stay yellow.

 

Four things can happen subsequently: two are good and two are judgement errors.  First, Thelma can reject her null hypothesis, and hit the brakes just as the light turns red. In this outcome, stopping was appropriate and Thelma guessed correctly:

 

Second, Thelma can accept her null hypothesis ("the light will stay yellow"), guess that she is near enough to the intersection, and proceed safely through the intersection. So she hits the accelerator and the light indeed remains yellow. Again, Thelma is justified. Fido is happy, too, because they don't take any longer to get to the dog park!

 

In both outcomes above, Thelma guessed correctly.

Consider a third outcome: Thelma rejects her null, hits the brakes, but the light remains yellow for a long time. This was a safe decision but not correct. Thelma realizes after-the-fact, as she waits, that she could have continued through the intersection. Statistically, she rejected the null but the null was true. Logically, this is an error (and Fido especially thinks this is an error because they take longer to get to the park. One dog minute is apparently quite a long time in human terms).

 

Finally, consider a fourth outcome: Thelma hits the accelerator, to beat the light, but the light turns red before she exits the intersection. Here she accepted a false null. This was clearly an error in judgement (and the police officer, it turns out, does not accept her explanation that she confused her statistical error types). 

The last two "error scenarios" illustrate the necessary trade-off between Type I and Type II errors. Thelma cannot simultaneously minimize the odds of committing both errors. She can opt to be a conservative driver; this implies a bias in favor of hitting the brakes. As a conservative driver, she minimizes the odds of running a red light but increases her odds of unnecessarily stopping on a yellow light. On the other hand, she can adopt an aggressive orientation; this minimizes her odds of unnecessarily stopping on a yellow light but increases her odds of running a red light. She must decide which error is more costly.

In hypothesis testing, the steps are:

  1. Formulate the null hypothesis
  2. Identify a test statistic
  3. Specify the significance level
  4. State the decision rule
  5. Calculate the test statistic
  6. Make a statistical decision
  7. Decide if it's economically significant

In our example, Thelma's null hypothesis was, "the light will remain yellow for at least a minimum period of time." The alternative hypothesis was therefore, "the light will not remain yellow long enough." We didn't articulate a test statistic, but it could have been "the approximate distance to the intersection."

The third step, specify the significance level, is where we express our preferences about which error (Type I or Type II) we want to avoid. Just as Themla decides whether to be conservative (i.e., lower her odds of a Type I error) or aggressive (lower her odds of a Type II error), we specify a significance level that reflects our preference about which error to avoid.

The statistical decision framework looks like the following, each of two decisions (accept the null or the alternative) combine with two possible realities (the null is true or false):

 

Typical significance levels are 5% and 1%. A 5% significance level says we are willing to tolerate a 5% chance of a Type I error. In regard to rejecting a true null, that may be too much risk for us. So we could reduce the significance level to 1% (this is also called the alpha). Now we only have a 1% chance of committing a Type I error. But, what is the price of this? We have simultaneously increased the odds of a Type II error (accepting a false null).

Comments

  1. Ashok D Jain 22 Apr 2007

    I have got confused in the last part of your explanation and need help -
    “Typical significance levels are 5% and 1%. A 5% significance level says we are willing to tolerate a 5% chance of a Type I error. In regard to rejecting a true null, that may be too much risk for us. So we could reduce the significance level to 1% (this is also called the alpha). Now we only have a 1% chance of committing a Type I error. But, what is the price of this? We have simultaneously increased the odds of a Type II error (accepting a false null).”

Leave a Comment