Oct 15

Basel II: Backtesting the VaR Model with the traffic light

by David Harper, CFA, FRM, CIPM

FRM |

trafficLightBasel2

Learning Outcome (LO 68.13)

  • Discuss the supervisory backtesting framework used in conjunction with an institution’s internal models, and describe the 3-zone supervisory framework for evaluating backtesting results.

Internal Models Approach (IMA) is the advanced approach to the market risk charge

I previously explained Basel's approach to charging banks for market risk. The advanced market risk approach is called the Internal Models Approach (IMA) and it allows banks to develop the capital charge based on their own internal value at risk (VaR) model. Further, banks can use their own particular flavor of VaR model; i.e., parametric, historical, or Monte Carlo. As with all of Basel II's advanced approaches (i.e., internal-ratings based for credit risk, IMA for market risk, and advanced measurement approach for operational risk), banks must satisfy several qualitative and quantitative standards in order to deploy internal models:

image

To use IMA, banks must stress test and backtest

Under IMA, in addition to stress testing (e.g., scenario analysis), the bank must backtest its value at risk (VaR) model. Backtesting is simply a historical test of the accuracy of the VaR model.

To conduct a backtest, the bank reviews its actual daily value at risk (VaR) over one year (about 250 trading days). It compares actual daily VaR outcome to its VaR estimate. Perfection is not expected. If the VaR model is 95% accurate, then about 12.5 exceptions are expected. That is, we expect the actual daily VaR to exceed the VaR estimate (used, at the beginning of the year, to calculate the market risk charge) because 5% of 250 = 12.5.

The backtest uses probability theory (we make a statistical inference about the VaR model based on a sample of 250 trading days)

Basel wants the bank's model to be at least 99% accurate. But generally backtesting cannot prove a model accurate or inaccurate. Rather, Basel turns to probability. Specifically, they draw a statistical inference based on the sample.

In the case of the Basel backtest, the null hypothesis is "the bank's VaR model is 99% accurate." Given that null, as usual we can commit two errors. A Type I error rejects an accurate model (calls broken a model that is really accurate). A Type II error accepts an inaccurate model (calls accurate a model that is really broken).

thumbTypeErrors

Three zones reflect wanting to avoid either a Type I or Type II error

Basel is smartly uses a three-zone framework, which is the best way to treat a confidence interval. Consider two practical problems:

  • Imagine the framework were to set the cut-off at 5 exceptions and simply reject a model that produced 5 exceptions. This would be a "bar set too low:" good models would be rejected almost 11% of the time (Type I error)
  • Imagine instead the framework were to set the cut-off at 10 exceptions and simply accept a model that produces less than 10 exceptions: This would be a "bar set too high:" a model that is semi-accurate (98% accurate) would be accepted 97% of the time! A model that is inaccurate (95% accurate) would be accepted fully 19.5% of the time. In short, a cut-off of 10 would rigidly accept too many inaccurate models (Type II error).

Against this unavoidable tension, Basel crafts a yellow-zone from five to nine exceptions:

backtestFactors

If the exceptions fall into the red zone, the multiplicative factor (k) is automatically increased from 3.0 to 4.0:

  • Green Zone: Probably a good model. Tiny chance of erroneously accepting an inaccurate model
  • Yellow Zone: Uncertain. Plausible outcome for either accurate or inaccurate model.
  • Red Zone: Probably a bad model. Conceivable accurate model could produce ten exceptions but unlikely. Supervisor should increase scaling (multiplication) factor +1 (from 3.0 to 4.0)

Yellow zone defers to supervisor discretion but "burden of proof" is on the bank

If the exceptions fall into the yellow zone, the "burden of proof...should be on the bank to prove their model is fundamentally sound." The frameworks provides guidance in regard to the increase in the multiplicative factor (k) but it also says that the penalty should be a function of root causes. For example, if model imprecision drives the exceptions, the framework says the supervisor "should" impose the penalty (increase the scaling factor). But if intra-day trading results or unusual events drove the exceptions, the frameworks only says the supervisor "should consider" the penalty.

backtestingDescretion

Primary Source: Basel II: Revised international capital framework

Comments

  1. Be the first to leave a comment!

Leave a Comment