Can We Trust the Experts during Risk Assessments?

The nature of being a risk manager requires us to engage extensively with subject-matter experts (SMEs) to understand concepts, processes, and risks. We use this information to understand how the SMEs’ activities fit into the bigger picture of the organization.

Without their input, it’s almost impossible to determine the appropriate risk response unless you have extensive amounts of data, quantitative or qualitative, at your disposal.

A common, introductory method for soliciting input during the risk assessment and analysis processes is to survey — either electronically or during a workshop  — a group of SMEs and ask them to rate the impact, likelihood, and potentially other metrics of a risk on a scale, typically 1-5. (As a side note, this is not my preferred method…)

However, you don’t need me to tell you that opinions and experiences vary widely in all areas of life, and risk assessments are no different.

Although you have a group of SMEs on a given topic providing input on a risk, each of these individuals has their own unique experiences, role, and information. Some score a risk at a 2 while others give that same risk a 3, 4, or even a 5…each of these participants are “experts” in a given area, but their interpretations vary widely.

A situation like this brings up an important question – how do we account for these differences?

(Now I’m going to be a bit blunt for a moment; if you do the following, I apologize, but this is something that really bugs me and can be really dangerous to the organization.)

In situations where SMEs provide a range of scores on a particular risk, many organizations will take the easy road and average out the scores.

Let’s say you have 5 SMEs who provide a range of scores like 1, 2, 3, 5, and 5.

Instead of making the effort to understand and account for this disparity, the risk manager will just take the average and say the impact (…or likelihood) of risk X is a 3.2. This rating bears little to no resemblance to reality, and this [extremely flawed] information will be given to executives and other decision-makers.

How should you handle this situation when you have such disparity in the risk ratings?

Calibration helps you figure out what’s driving differences in assessments and take action(s) to address these differences.

Calibration is a term usually associated with instruments for measuring weight, speed, or some other metric in which you refine or fine-tune a measuring instrument to ensure its accuracy.

Take this embarrassing situation: If you’ve ever been pulled over and written a ticket for speeding, one question you may ask is if the officer’s radar gun is properly calibrated.

To get an accurate picture of a particular risk or issue, risk managers must “calibrate” the inputs they receive, regardless of how experienced and insightful the SMEs are, because as Douglas Hubbard explains in his book The Failure of Risk Management:

No matter how much experience we accumulate and no matter how intelligent we are, we seem to be very inconsistent in our estimates and opinions.

One method for calibrating responses, especially for companies just starting out or who don’t have robust data analysis capabilities, is to ask about the information the SME used to arrive at the conclusion he or she did. Do this with each person who has the disparate ratings. Then give all SMEs in the sample group the same information and then rescore, but be careful to avoid groupthink, conformity and other biases that can negatively impact decision-making.

The key to making this work is to have a team of what Professor Philip Tetlock of the Wharton School at the University of Pennsylvania calls “belief updaters,” or people willing to change their minds when given new information. Otherwise, this will make for a very difficult discussion because you will not get agreement on the risk rating.

If your organization has data and the capacity to understand it, there are other quantitative-based calibration methods you can employ.

Hubbard has written about the topic of calibration mostly from a quantitative perspective in his books and in this webinar from 2020’s Risk Awareness Week (RAW) event.

While many thought leaders in risk management, including Hubbard, extol the benefits of quantitative over qualitative assessment, no method is fool proof, including data-driven models.

The best example of this is a method we’ve discussed before called Monte Carlo simulation. Data or numbers for these models may come from subjective methods, but as Hubbard explains in his RAW 2020 webinar, Monte Carlo simulations appear to measurably improve estimates and decisions of SMEs.

A word of caution about quantitative modeling – many risk thought leaders are very adamant about avoiding qualitative assessments at all costs. However, as I discuss in this article, most companies I’m familiar and/or work with are not ready for this level of risk assessment and analysis.

For example, if you’re new to exercise, you wouldn’t jump right into an intense, athlete-level workout but instead ease your way up to this level.

Risk assessments are the same way – you won’t want to use quantitative methods if your company does not have the data, structure, processes, and skill sets to gather, maintain, and monitor data. Doing so could end up being more destructive than simply taking an average of disparate scores.

What methods do you use to account for differences in risk assessment by subject-matter experts?

If you are able, we would love to hear your thoughts on this important topic, so please feel free to leave a comment below or join the conversation on LinkedIn.

Also, if your company is struggling to get accurate information to properly understand threats and opportunities to achieving strategic objectives, please don’t hesitate to contact us to discuss how your risk management processes can be harnessed to give your company a strategic advantage.

Posted in

Sign Up For Our Newsletter

Sign Up For Our Newsletter


Meet Carol

Helping companies achieve their vision and strategy, and succeeding in today's turbulent world, is something I'm honored to be a part of. Whether you're an occasional blog visitor or a long-term client, thank you for letting us be a part of your journey.

Most Recent Posts

The 12 Days of ERM Christmas

Without a doubt, one of my family’s favorite holidays is Christmas. Part of the fun, especially for our son, is seeing what “Santa” brought, but most importantly, we treasure the spirit of peace and goodwill the season brings. And after what seemed to be a never-ending warm spell, the weather is expected to be good…

Read More

Don’t Let Goals and Initiatives Be Blindsided by External Events

As the end of the year draws near, I think we’d all agree that while it wasn’t without its challenges, this year also wasn’t quite as turbulent as the previous two. While a lot of people are juggling company parties, shopping for friends and family, and special activities for the kids, most companies are putting…

Read More

Going the Distance: Ensuring Successful Execution of Strategic and Annual Initiatives

Strategic planning is a challenge – of all people, I understand… After all the meetings, risk and data analysis, and brainstorming of the preceding months, it’s tempting to think this is the end of the road and you can relax. Contrary to this common perception though, this is exactly not the time to relax, but…

Read More

Avoid Rookie Mistakes and Protect your Internal Reputation

Be honest – have you ever done something that you soon realized was a real rookie mistake? Me raising my hand… Considering the nature of ERM’s role to ask questions and challenge assumptions (often during conversations with executives), it can be argued that, in at least some cases, the expectations bar for risk professionals is…

Read More

ERM at Thanksgiving – An Illustration of Risk Management in Action

On occasion, I like to take some of the concepts we risk professionals think about in our jobs and apply them to different personal situations…take some of the same concepts we use when working with executives to develop corporate strategy and manage risks or uncertainty around that strategy. It’s Thanksgiving week in the U.S. –…

Read More

Why Quantitative Risk Assessment is Not Just the Best But the Only Option – A Conversation

Periodically, I have the pleasure of speaking one-on-one with Hans Læssøe on a variety of topics around ERM, strategic risk, and other issues and trends. As you know from my previous conversations (here, here) and posts featuring his work, Hans was formerly a practitioner at the iconic LEGO Company, but even more notably, is a…

Read More

The Three Lines Model – 3 Reasons Why I Don’t Like It

Everyone likes a clear-cut template that offers an easy way to create or manage something…I mean what’s not to like about a step-by-step process for accomplishing what you want? Sometimes this can work without any issues, such as the case with the Project Management Book of Knowledge (PMBOK), ISO 9001 standard, or a new cooking…

Read More

5 Avenues for Expanding your ERM Knowledge

One thing I was taught to appreciate from a young age was the value of education and knowledge. It didn’t necessarily matter what the subject was, just that I always maintain a learning or growth mindset regardless of my current status in life. This mindset has served me well over the years, and it’s a…

Read More

Storytelling and Risk Management – Developing Skills that Technology Cannot Replace

It’s amazing how technology has developed and changed our working world over time. Imagine trying to run my risk and strategy consulting firm without tools like Zoom, Box, Slack, and other ERM-specific technology tools. There is no way we would be able to serve our clients the way that we do. Just consider how the…

Read More

3 Phases to Creating and Launching an ERM Program Focused on Organizational Success

If you’ve been handed the task of creating an ERM program for your organization, let me first offer my congratulations quickly followed by my empathy for the task ahead of you. I don’t say that to scare you but to provide a small dose of reality. Building, launching, and refining an ERM program that is…

Read More