Are Qualitative Risk Assessments Fatally Flawed?

Early last month, I published an article explaining why I don’t discuss quantitative assessment methods like modeling, Monte Carlo simulation, and others very much and what companies must have in place before they can use methods like this to guide their decision making and strategic planning.

In short, without establishing corporate governance policies, a strategic plan, or clearly defined roles and responsibilities, any sort of advanced ERM methods will fail to be a useful tool for ensuring a company is taking the right amount of risks in pursuit of strategic objectives.

In this article on enterprise risk assessment from two years ago, I explain how qualitative methods use descriptive elements to rank a particular risk. Risk managers can then take this information and develop risk scores based on likelihood, impact and other parameters. In some cases, these scores can be plotted on a heat map, which as COSO explains, is “…a graphical representation of likelihood and impact of one or more risks.”

However, thought leaders at the forefront of ERM are extremely skeptical about the value qualitative risk assessments and heat maps can provide an organization. For example, in response to my article from last month, Hans Læssøe comments:

Relying on qualitative risk assessment is infected by human biases and potentially dangerous…it will not help you run [or] perform better, so waste as little effort on this as possible.

If a company’s goal with ERM is to make better decisions and improve performance, Hans explains:

…there is no avoidance of quantitative approaches – including Monte Carlo simulation. This will require skilled people and effort – but also add the value of enhancing performance.

Other thought leaders are even more critical of qualitative methods, including David Vose, who explains in this article:

Qualitative risk analysis may be appealingly simple, and lots of companies use it, but it’s almost valueless for making risk-based decisions.

David goes onto explain in his article and in a presentation at Risk Awareness Week 2019 how qualitative scores are “…too imprecise and inflexible.” 

If you browse around my site and others like RIMS, NC State’s ERM Initiative, and others on the topic of risk assessments, you will find a lot of information on qualitative methods, but…

Based on comments from Hans, David, and others, should we assume qualitative risk assessments are fatally flawed?

I certainly agree with Hans and David that qualitative methods, especially how they’re currently done by many companies, are fatally flawed due to human bias. 

While I agree with their opinions, the actual move from qualitative-based risk assessments and heat maps to quantitative modeling is an entirely different matter.

For many companies, qualitative risk assessment is all they know…asking them to go from this to more quantitative methods like modeling and Monte Carlo simulation represents a huge shift. If the culture of the organization will not support such a change, it is doomed to fail.

As an example, heat maps are quite popular with not only the general risk management community, but executives and boards as well. This popularity is likely due to their simple color scheme (red/yellow/green) and quick glance comprehension – finding or developing a quantitative reporting style that meets these needs will be challenging.

In the meantime, companies can use different techniques to overcome bias and quantify risks that are normally qualitatively assessed…

Despite the urgency for companies to take informed risks in pursuit of strategic objectives, getting to the ideal situation promoted by Hans and others will take a long time. Certain companies like those in finance may already use quantitative methods, but for many, this is simply out of reach at this time.

However, this doesn’t mean companies shouldn’t be on the lookout for ways to address the shortcomings of qualitative risk assessments.

One way to overcome bias is to ask executives what history tells them. They may consider a risk to be of grave importance, but after careful examination, you may learn that when the risk materialized in the past, it was not that significant after all.

There are numerous books and resources out there for helping reduce bias. 

Many people think a quantitative risk assessment is providing a scale of 1 to 5 with criteria associated with each number. No. But taking a risk that is normally thought of as qualitative or difficult to put a number to it…that is the hard part. 

Let’s walk through an example using a frequently discussed risk: the ability to recruit and retain quality talent. 

Rather than say that the risk is a 4 on a scale of 1 to 5, there needs to be some hard numbers put to it. If your organization cannot recruit and retain quality talent, what will the organization not be able to achieve? Will certain revenue numbers not be met? Will products be designed and manufactured on time? Putting scope and boundaries on a risk will help you identify those numbers. And when all else fails, you can always check out the book “How to Measure Anything” by Douglas Hubbard.

While Hans, David Vose, and others are correct about the shortcomings of qualitative risk assessments and are visionary in how risk management must change in order to help companies create a strategic advantage, getting there is going to be a long road for many. 

But as Hans also explains, the world is changing at such an unprecedented speed that companies may not have a choice if they wish to remain competitive.

Do you think qualitative risk assessments are fatally flawed? 

What methods have you used to overcome bias in your organization when it comes to risk management and decision-making?

Share your thoughts by leaving a comment below or join the conversation on LinkedIn.

And if you are struggling to develop risk assessment methods for your company, please don’t hesitate to send me an email to discuss your specific situation today.

Featured image courtesy of Kevin Ku via Unsplash.com

Posted in

Sign Up For Our Newsletter

Sign Up For Our Newsletter

SDS-Logo
about-sidebar-v2

Meet Carol

Helping companies achieve their vision and strategy, and succeeding in today's turbulent world, is something I'm honored to be a part of. Whether you're an occasional blog visitor or a long-term client, thank you for letting us be a part of your journey.

Most Recent Posts

The 12 Days of ERM Christmas

Without a doubt, one of my family’s favorite holidays is Christmas. Part of the fun, especially for our son, is seeing what “Santa” brought, but most importantly, we treasure the spirit of peace and goodwill the season brings. And after what seemed to be a never-ending warm spell, the weather is expected to be good…

Read More

Don’t Let Goals and Initiatives Be Blindsided by External Events

As the end of the year draws near, I think we’d all agree that while it wasn’t without its challenges, this year also wasn’t quite as turbulent as the previous two. While a lot of people are juggling company parties, shopping for friends and family, and special activities for the kids, most companies are putting…

Read More

Going the Distance: Ensuring Successful Execution of Strategic and Annual Initiatives

Strategic planning is a challenge – of all people, I understand… After all the meetings, risk and data analysis, and brainstorming of the preceding months, it’s tempting to think this is the end of the road and you can relax. Contrary to this common perception though, this is exactly not the time to relax, but…

Read More

Avoid Rookie Mistakes and Protect your Internal Reputation

Be honest – have you ever done something that you soon realized was a real rookie mistake? Me raising my hand… Considering the nature of ERM’s role to ask questions and challenge assumptions (often during conversations with executives), it can be argued that, in at least some cases, the expectations bar for risk professionals is…

Read More

ERM at Thanksgiving – An Illustration of Risk Management in Action

On occasion, I like to take some of the concepts we risk professionals think about in our jobs and apply them to different personal situations…take some of the same concepts we use when working with executives to develop corporate strategy and manage risks or uncertainty around that strategy. It’s Thanksgiving week in the U.S. –…

Read More

Why Quantitative Risk Assessment is Not Just the Best But the Only Option – A Conversation

Periodically, I have the pleasure of speaking one-on-one with Hans Læssøe on a variety of topics around ERM, strategic risk, and other issues and trends. As you know from my previous conversations (here, here) and posts featuring his work, Hans was formerly a practitioner at the iconic LEGO Company, but even more notably, is a…

Read More

The Three Lines Model – 3 Reasons Why I Don’t Like It

Everyone likes a clear-cut template that offers an easy way to create or manage something…I mean what’s not to like about a step-by-step process for accomplishing what you want? Sometimes this can work without any issues, such as the case with the Project Management Book of Knowledge (PMBOK), ISO 9001 standard, or a new cooking…

Read More

5 Avenues for Expanding your ERM Knowledge

One thing I was taught to appreciate from a young age was the value of education and knowledge. It didn’t necessarily matter what the subject was, just that I always maintain a learning or growth mindset regardless of my current status in life. This mindset has served me well over the years, and it’s a…

Read More

Storytelling and Risk Management – Developing Skills that Technology Cannot Replace

It’s amazing how technology has developed and changed our working world over time. Imagine trying to run my risk and strategy consulting firm without tools like Zoom, Box, Slack, and other ERM-specific technology tools. There is no way we would be able to serve our clients the way that we do. Just consider how the…

Read More

3 Phases to Creating and Launching an ERM Program Focused on Organizational Success

If you’ve been handed the task of creating an ERM program for your organization, let me first offer my congratulations quickly followed by my empathy for the task ahead of you. I don’t say that to scare you but to provide a small dose of reality. Building, launching, and refining an ERM program that is…

Read More