Feb 20

Themes in risk: error avoidance

by David Harper, CFA, FRM, CIPM

FRM | Quant |

In teaching the FRM curriculum, we consolidate disparate elements into building blocks and themes.  A key theme is error avoidance. Risk is uncertainty. To make a decision in the face of uncertainty is to incur risk. Statistics provides elegant tools for coping with uncertainty. Although we cannot know a priori the wisdom of a decision, we do have the power to "hedge our bets."

Put another way, we can't know the outcome, but we can decide which type of error to prefer (or eschew) over another. In a previous post, we explained that the specification of a significance level is just the articulation of, the quantification of, an error bias.  We saw that, in deciding whether to try and beat a red light at an intersection, the driver implicitly weighs the costs of two errors. Which risk does she prefer, to run a red light or to make an unnecessary stop? (it doesn't really matter that she 95% prefers to make an unnecessary stop and only 5% prefers to run the red light, or 99%/1%).

Generic avoidance

Generically, we saw in the previous post, the Type I error is the rejection of a true null (i.e., to accept the alternative when the null was true) and a Type II error is the acceptance of a false null:

 

Errors in Significance Testing

In the Quantitative Module of the FRM, tests of significance are common. For example, we test for the significance of a sample mean ("is it likely that the sample mean is near to, or the same as, the underlying population mean?"). Also, we test whether the coefficient of determination (r2) is significant ("is the linear relationship meaningful?"). In the case of a test of the sample mean, the null hypothesis might be: the population mean is zero. Therefore, the alternative hypothesis is: the population mean is non-zero. Note this refers to a two-tailed test. We could instead formulate a one-tailed test, wherein the null is: the population mean is less than (or greater than) zero. The generic risk avoidance framework in this case looks like this:

 

Errors in credit modeling

Now consider an example in the Credit Module of the FRM, which can be confusing. Credit risk models permit us to generate credit scores ("credit scoring"). But what do we do with the credit scores? Given a certain score, based on a certain methodology, how do we decide if the firm is "bad" (likely to default) or "good" (unlikely to default)? The answer is that we apply one of several decision frameworks. And decision frameworks differ largely in the way that they prefer or eschew error types.

We have several frameworks, which include the Neyman-Pearson Decision Rule. Under de Servigny's treatment of the Neyman-Pearson rule, the null hypothesis is: the firm is bad. So, we can drop the Neyman-Pearson rule into the framework below. We also see why it's the definition of the null that matters, not the coincidence that Type II errors are typically shunned in favor of Type I errors. Under this definition of the null, a Type I error is worse because it represents the bank rejecting a true null (accepting a bad loan).

 

Errors in Basel II

Finally, we can move the Operational Risk Module for yet another example. In Basel II, we learn about the so-called traffic light system of model backtesting. This is a decision rule for deciding whether a risk model is accurate. The Basel framers defined the null as: your (the bank's) risk model is accurate. As defined, you can see why we are back to a situation where the Type II error is arguably worse (i.e., we mistakenly accept a faulty model). Further, you can then understand why the calibration is largely about the tension between avoiding Type I and Type II errors.

Comments

  1. Be the first to leave a comment!

Leave a Comment