What's new

How to Fix the FRM: David's 2018 Memo to the FRM Committee and GARP's Board of Trustees

David Harper CFA FRM

David Harper CFA FRM
Staff member
Subscriber
Thread starter #1
In May 2018, I sent the attached 18-page memo called "How to Fix the FRM" to GARP's leadership (including the Board of Trustees). As host of this forum for over a decade, I have a front-row seat into candidates' experiences and their feedback; in particular, I've spent several hundred hours debugging syllabus material and practice questions. The primary motivation for the memo was the high error rate we've observed in the source material, in particular sample practice questions. Related, the anthology approach (i.e., differing authors) of the exam has created certain inconsistencies and ambiguities, many of which are detailed in the memo. As I wrote in the memo:
On behalf of my customers and myself, this memo has three goals. One, to warn you of reputational risks that are accruing to the FRM product, at least online. Two, to convince you there is an opportunity for improvement in the FRM product. Three, if those arguments are convincing, to learn that the Committee will take some concrete steps to make improvements. With that agenda, at the end of this memo, after asking important questions, I offer recommendations and suggestions.
The irony of these quality problems is that they are mostly unnoticed by the majority of candidates who are too busy digesting the material in a first pass effort at comprehension. Rather, these issues have been noticed only by the more experienced and proficient candidates, who tend to arrive with prerequisites that enable them to discern defects. What I mean is, most candidates are just trying to learn how to calculate value at risk (VaR), for example, or credit value adjustment (CVA), it's after you master them are you likely to noitce that GARP's own approaches can be inconsistent. A key point that I have repeatedly emphasized to management, with actual examples, is that some inconsistencies ironically bias against better-prepared candidates; that is, some submitted questions have a perceived correct answer that is easier to identify if you know less (candidates who know more often tend to identify an alternative)! In any case, I assume that I do not need to defend high standards especially in the context of how much time and money must be invested by an FRM candidate.

This memo summarizes, on a certain level, my years long effort to contribute to the improvement of syllabus. I did receive a careful, thoughtful, and encouraging reply from Bill May (Global Head of Certifications and Education Program). My reply to his reply is appended, but due to privacy, his reply to me is not included. I'm sharing here now only because this phase has long-since concluded (months ago) and I am very much looking forward to seeing GARP's release of the 2020 syllabus.

The file is attached but same is here at https://www.dropbox.com/s/1vx96sckabn2qbv/howto-fix-frm-v4.pdf?dl=0
 

Attachments

Last edited:
#2
Inspiring to have people of your calibre pushing for these improvements. I doubt there are many people capable of finding errors in garp's material or areas for improvement - let alone take the time to raise these issues and comments . Thank you. I've always admired and appreciated your dedication to those of us trying to better themselves through the FRM programme.
 
#3
So finally someone showed an intent to Bell the CAT and I would like to congratulate David for that. I hope GARP will take it positively and take steps to develop its own study material content (which will be [hopefully] error free and as interesting as CFA institute study material :):) )
 
#5
I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.
 
#7
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?
 

Nicole Seaman

Chief Admin Officer
Staff member
Subscriber
#8
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?
Hello @amit.m.sharma

We are currently working on updating the materials in the study planner for 2020 so there will be limited forum support at this time. The two threads below may help to answer your questions. Please feel free to discuss any further questions that you have in those threads after you have read through the discussions. Thank you.
 

ami44

Active Member
Subscriber
#9
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?
1.+4. is answered by Nicoles links
2. no penalizing. You should guess in case you don‘t know the answer or your time runs out.
3. All questions have the same weight
 
#10
I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.
I don't know Mr.Taleb personally but I've read his books. Although the books are quite decent, (from his writing) Mr.Taleb appears to be an oversmart person. According to him not only Risk professionals but many other professionals (stock Traders, Investment professionals, Dentists) are incompetent. It seems that according to him only he is the most intelligent and most competent professional on this planet earth. Of course, Mr.Taleb has right to hold his opinions and express them freely. But now after listening to Taleb, should I stop seeing a Dentist or Stop Investing in Mutual fund (Because of incompetent dentist and fund managers) ??? I WILL NOT.
As far as FRM exam is concerned, although there a some problems, overall its a good course and do lot of value addition, Provided, you approach the syllabus correctly and use good resources Like Bionic Turtle. Thanks
PS : This is what I feel, I may be wrong/ I may be right and others need not agree with what I said
 
Last edited:
#11
If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.
 
#13
If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.
I agree with some of the points and disagree with some points mentioned in your 2 replies.
First let's speak about the points I agree with 1) David's videos are useful- I agree with that. However, you are saying videos are More important than exam and questions. Sorry, I didn't get your point. I don't know much about coaching industry but in my opinion most of the students are subscribing to David's services (BT) for understanding difficult concepts of risk management and TO PASS FRM EXAM. If there is no FRM exam, then I wonder how many them would join BT just for the sake of enhancement of their knowledge. At least, people from India/China (Major mkt for FRM) cant afford to spend $799 just for intellectual stimulation.
2) Number of Practical/calculation-based questions should be increased ( I assume this is what you mean when you say "Incorporating questions with applied examples "). There are lot of theory questions in actual exam. In my opinion this number should be reduced.

Now points I Disagree with
1) 30% questions are wrong : 30% wrong questions in MOCK. We can not say that about actual exam.

2) Zero transparency : The level of transparency in FRM exam is similar to CFA exam. Even, CFA institute doesn't publish its past exam papers. So now are you going to say that even CFA is scam ?

3) Electronic test : How can you say electronic test is better than paper format test ?? In fact, in case electronic tests there is more possibility of manipulation, hacking and even cyber attack.
 
Last edited:
#14
I've just been reading the letter that David sent to GARP and wonder if the major(?) changes that have been made to the 2020 Part 1 syllabus were the result of David's letter?

I am also wondering if the CFA suffers from the same shortcomings.
 
Top