What's new

How to Fix the FRM: David's 2018 Memo to the FRM Committee and GARP's Board of Trustees

David Harper CFA FRM

David Harper CFA FRM
Staff member
Subscriber
In May 2018, I sent the attached 18-page memo called "How to Fix the FRM" to GARP's leadership (including the Board of Trustees). As host of this forum for over a decade, I have a front-row seat into candidates' experiences and their feedback; in particular, I've spent several hundred hours debugging syllabus material and practice questions. The primary motivation for the memo was the high error rate we've observed in the source material, in particular sample practice questions. Related, the anthology approach (i.e., differing authors) of the exam has created certain inconsistencies and ambiguities, many of which are detailed in the memo. As I wrote in the memo:
On behalf of my customers and myself, this memo has three goals. One, to warn you of reputational risks that are accruing to the FRM product, at least online. Two, to convince you there is an opportunity for improvement in the FRM product. Three, if those arguments are convincing, to learn that the Committee will take some concrete steps to make improvements. With that agenda, at the end of this memo, after asking important questions, I offer recommendations and suggestions.

The irony of these quality problems is that they are mostly unnoticed by the majority of candidates who are too busy digesting the material in a first pass effort at comprehension. Rather, these issues have been noticed only by the more experienced and proficient candidates, who tend to arrive with prerequisites that enable them to discern defects. What I mean is, most candidates are just trying to learn how to calculate value at risk (VaR), for example, or credit value adjustment (CVA), it's after you master them are you likely to noitce that GARP's own approaches can be inconsistent. A key point that I have repeatedly emphasized to management, with actual examples, is that some inconsistencies ironically bias against better-prepared candidates; that is, some submitted questions have a perceived correct answer that is easier to identify if you know less (candidates who know more often tend to identify an alternative)! In any case, I assume that I do not need to defend high standards especially in the context of how much time and money must be invested by an FRM candidate.

This memo summarizes, on a certain level, my years long effort to contribute to the improvement of syllabus. I did receive a careful, thoughtful, and encouraging reply from Bill May (Global Head of Certifications and Education Program). My reply to his reply is appended, but due to privacy, his reply to me is not included. I'm sharing here now only because this phase has long-since concluded (months ago) and I am very much looking forward to seeing GARP's release of the 2020 syllabus.

The file is attached but same is here at https://www.dropbox.com/s/1vx96sckabn2qbv/howto-fix-frm-v4.pdf?dl=0
 

Attachments

  • howto-fix-frm-v4.pdf
    288.1 KB · Views: 212
Last edited:
Inspiring to have people of your calibre pushing for these improvements. I doubt there are many people capable of finding errors in garp's material or areas for improvement - let alone take the time to raise these issues and comments . Thank you. I've always admired and appreciated your dedication to those of us trying to better themselves through the FRM programme.
 

Champaklal Chodoo

New Member
So finally someone showed an intent to Bell the CAT and I would like to congratulate David for that. I hope GARP will take it positively and take steps to develop its own study material content (which will be [hopefully] error free and as interesting as CFA institute study material :):) )
 
I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.
 
Plus, the constant turnover of the syllabus smells like a desperate attempt to unfairly take in fees. Things are not changing that often
 
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?
 

Nicole Seaman

Chief Admin Officer
Staff member
Subscriber
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?
Hello @amit.m.sharma

We are currently working on updating the materials in the study planner for 2020 so there will be limited forum support at this time. The two threads below may help to answer your questions. Please feel free to discuss any further questions that you have in those threads after you have read through the discussions. Thank you.
 

ami44

Well-Known Member
Subscriber
That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?

1.+4. is answered by Nicoles links
2. no penalizing. You should guess in case you don‘t know the answer or your time runs out.
3. All questions have the same weight
 

Champaklal Chodoo

New Member
I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.
I don't know Mr.Taleb personally but I've read his books. Although the books are quite decent, (from his writing) Mr.Taleb appears to be an oversmart person. According to him not only Risk professionals but many other professionals (stock Traders, Investment professionals, Dentists) are incompetent. It seems that according to him only he is the most intelligent and most competent professional on this planet earth. Of course, Mr.Taleb has right to hold his opinions and express them freely. But now after listening to Taleb, should I stop seeing a Dentist or Stop Investing in Mutual fund (Because of incompetent dentist and fund managers) ??? I WILL NOT.
As far as FRM exam is concerned, although there a some problems, overall its a good course and do lot of value addition, Provided, you approach the syllabus correctly and use good resources Like Bionic Turtle. Thanks
PS : This is what I feel, I may be wrong/ I may be right and others need not agree with what I said
 
Last edited:
If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.
 
For example David’s videos are vastly more useful than the exam and questions. Incorporating questions with applied examples is the way to go.
 

Champaklal Chodoo

New Member
If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.

I agree with some of the points and disagree with some points mentioned in your 2 replies.
First let's speak about the points I agree with 1) David's videos are useful- I agree with that. However, you are saying videos are More important than exam and questions. Sorry, I didn't get your point. I don't know much about coaching industry but in my opinion most of the students are subscribing to David's services (BT) for understanding difficult concepts of risk management and TO PASS FRM EXAM. If there is no FRM exam, then I wonder how many them would join BT just for the sake of enhancement of their knowledge. At least, people from India/China (Major mkt for FRM) cant afford to spend $799 just for intellectual stimulation.
2) Number of Practical/calculation-based questions should be increased ( I assume this is what you mean when you say "Incorporating questions with applied examples "). There are lot of theory questions in actual exam. In my opinion this number should be reduced.

Now points I Disagree with
1) 30% questions are wrong : 30% wrong questions in MOCK. We can not say that about actual exam.

2) Zero transparency : The level of transparency in FRM exam is similar to CFA exam. Even, CFA institute doesn't publish its past exam papers. So now are you going to say that even CFA is scam ?

3) Electronic test : How can you say electronic test is better than paper format test ?? In fact, in case electronic tests there is more possibility of manipulation, hacking and even cyber attack.
 
Last edited:
I've just been reading the letter that David sent to GARP and wonder if the major(?) changes that have been made to the 2020 Part 1 syllabus were the result of David's letter?

I am also wondering if the CFA suffers from the same shortcomings.
 

Sjegathe

Member
Subscriber
I've just been reading the letter that David sent to GARP and wonder if the major(?) changes that have been made to the 2020 Part 1 syllabus were the result of David's letter?

I am also wondering if the CFA suffers from the same shortcomings.

I can confidently say no, having gone through all the levels i can attest that the standards are top notch. I believe they've
integrated machine learning and the latest fin tech technologies in to the curriculum as well. There are people who would say level
2 and level 3 questions are tough but what that means is you would need solid understanding of the concepts not by means of flawed questions and answers. This is my opinion though.
 
I don't want to unburry an older thread, but I came here from the October 2020 Exam Feedback Thread and wanted to share some of my thoughts.

I think the lack of actual excercises in the curriculum makes it quite hard to pass the exam just with the GARP provided materials - frankly, it is beyond me how one could pass just with the GARP books, the online questions and the one or one-and-a-half mock exams provided with an acceptable effort.

I do firmly belive in free markets and this way it might be possible that you get a better outcome by letting external providers do the job
and obviously BT does deliver a real good service given the pass rates and feedback by subscribers (I did recomend it on Reddit as well, David also commented in this thread :) - and of course I did not receive any payments for my recommendation).

I also really appreciate Davids effort to cooperate with GARP and give some constructive feedback. While a good reputation of the FRM will help th BT business modell, all the lacks described in the letter are probably even better for BT ;-) So I think it is safe to say that David did take the effort for all the current and future FRM candidates.

I think the points are very valid and as we saw in the process of the exam postponements and the rather chaotic publication of exam results, GARP has also some serious issues with their communication.
A timely, friendly and empthatic communication could take a lot of the edges plus it would look much more professional - a level of professionalism I would expect from an international organisation with that reputation.

The whole curriculum is very extensive and challenging and it takes a lot of effort. As David quoted “lack of quality time spent with my kids while they're still young enough to think I’m cool.” - I totally feel that.

That said, I think the curriculum does cover the field well and I fell that I have learnt a lot of relevant knowledge - so I don't regret to have earned the FRM. I also think, that it is ok to have different opinions and approaches in the literature, as it good to know that there might be divergent opinions amongs professionals if you want to be able to discuss and evaluate the approaches in daily business. However, it would be extermly helpful if that fact would be pointed out by GARP, rather than just put them next to eachother in the sylabus.

Another point is that I am not against challenge or difficulty. In oder to have a real value, the designation should not be given away easily. But the challenge should be fair - if you know the right answer, you should be able to pass. To some extent, red herrings are ok too, if they are used to test the understanding of a candidate. But cheapshots, unnecessary tedious calculations (like adding up 20 random figures to calculate an Expected Shortfall, which is just a test of concentration and how well you can use the calculator) and distraction long descriptions in the questions (Like: John is a Hedge Fund Manger based in the USA. He likes to play Golf and his Grand-Grandmothers Name was .... and in the end they ask something about volatility, just to waste your time) should not be used just to provoke errors or put people under time pressure.

So long story short: Thanks again David, especially the BT question were more than helpful. Whitout them, I would not have passed the exam.
For all future candidates: The mock exam from GARP might help you with the general format of the questions, but BT will prepare you better. And be prepared / watch out for cheap tricks in the exam questions ;-)
 
Addition, some examples how it is not supposed to be:

- as a non native English Speaker, I found in some cases in the exam, they used synonyms for words like "good", "acceptable" or "appropriate" which are not that common - like someone said, "we used "best approach" already twice in the exam, lets get a thesarus and look for the most uncommon word for "best" to replace it".

- They should test, if the candidate understands the problem, not how well he or she can handle the calculator under pressure. E.g. if you have to calulate an Expected Shortfall over a Period of 300 days, 1% confidence is enough. The candidate should know, wether to take the biggest 4, 3 or 2 losses and how to average them. They shouldn't ask for 10% or even 5% confidence, because adding up over 10 figures, at best with completely random number over 1bn has nothing to do with risk skills or knowledge.

- Red Herrings are ok, e.g. there was one case where there was an output of statistical software, were the figure could just be read from the table. It could also be calculated by the number given, but it was more tedious. So fair enough, if you know shortcuts, that's a skill.
But putting in an endless story with all the information hidden in it, which just wastes you time, is just not fair.
I always read the question first before reading the backstory, in order to pick up the right information on the first go.
But in a very time constraint exam, it is unfair to uneccessarily waste the candidates times.
 
Top