Standard & Poor's last month released their 2007 Annual Global Corporate Default Study And Rating Transitions. This is useful to FRM candidates because it publishes actual credit rating transition matrices (migration tables). I'll post more on these later. The study is also interesting because S&P uses a cumulative accuracy profile (CAP) curve to evaluate the performance of its ratings. In the 2007 exam, one learning outcome asked us to "compare and contrast receiver operating characteristic and cumulative accuracy profile as user-independent performance measures."

In the S&P study, about their own ratings performance they noted:

"Gini ratios attest that ratings continue to be effective indicators of the relative default risk. Among corporate entities rated by Standard & Poor's Rating Services, the average one-year Gini coefficient was 80%; three-year, 76%..."

What is a gini coefficient?

To illustrate the Gini/CAP approach, I constructed the simple EditGrid spreadsheet below. You can open the spreadsheet below into MS Excel by selecting File > Export As > Excel. Or you can click here to open a new editgrid worksheet.

 

A Gini/Cap approach sorts the obligors according to the credit rating model, on the x-axis, starting from the worst (most likely to default, highest PD) to the best (least likely, lowest PD). The y-axis plots the actual fractional defaults. A good model will therefore produce a steep line because it will have correctly sorted the defaulters to the left.

The spreadsheet example

This example assumes 100 obligors, all with a highly exaggerated 10% probability of default (PD) (just to illustrate) . So, the X-axis runs from 0 to 100, one for each obligor. Then there are three plots (blue, red, green). Each plot represents a different credit scoring model, respectively, uninformative, perfect and accurate. The idea of the model is to rank/sort the obligors from worst to best (i.e., starting with the most likely to default and ending with the least likely to default). Actual defaults are plotted on the y-axis (the cumulative fraction of defaults) such that the shape of the line tells about the performance of the credit scoring model.

• The blue line plots a randomized set of defaults. Each of the 100 credits defaults if =RAND()< 0.1. Hitting F9 refreshes into a new random set. In the set above, it happened to simulate that 13 of 100 defaulted. The Y-axis plots the cumulative fraction of defaults. The first simulated default is the 9th obligor, so the line jumps at 9 from y=0 to y = 1/13. The second default is 16th obligor, so the line jumps from 1/13 to 2/13 at x=13. In this way, the random line will tend to approximate a 45 degree angle. The 45 degree angle represents a random or uninformative credit scoring model.
• The red/maroon line charts a set where the defaults are all "perfectly" organized at the start; i.e., X = {1,1,1,1,1,1,1,1,1,1,0,0,0,....} where 1 = default and 0 = no default. This line steeply rises because the first point is {1,1/13}, the second point is {2,2/13}, and the 13th point is {13,13/13}. This is a perfect credit scoring model because the defaulting obligors were perfectly stacked at the beginning of the line.
• The green line isn't random. Rather I just simulated a model that wasn't perfect but did identify the likely defaulters early in the list: X = {1,0,0,1,0,0,1,...}. This line falls between the other two, as any line would

The Gini coefficient is then the fraction of the area under the curve (between accurate and random) relative to the entire area (between perfect and random). A perfect credit scoring model would therefore earn a 1.0; a totally uninformative model would lie on the 45% angle line and therefore score zero.

The weakness

The Gini/Cap approach described is considered inferior to the receiver operating characteristic (ROC) measure because it does not factor in the costs of misclassification. It treats a Type I error (models says no default, but firm defaults) with the same weight as a Type II error (model says default, but firm does not default). In practical terms, the errors rarely have the same cost. Another weakness, of both the Gini/Cap and the ROC, is that it "provides a rank-ordering performance measure of a model and is highly dependent on the sample on which the model is calibrated" (de Servigny). That's two criticisms. In my opinion, the sample size critique is more relevant than the rank-order critique; i.e., PDs are sufficiently volatile that a model with good "ranking" ability is quite a lot to accomplish.

1 Comment  |  Login to add your comment

1. 

   John 11 Dec 2008

   Hello,

   very nice sheet!

   For my thesis I’m looking for a program to estimate the cap curve. Does anybody know a program which does that?

Login or Signup to Discuss