Tuesday, June 28, 2016

Making Useless Nails vs. Achieving Your Desired Outcome

Image courtesy of Christina Welsh
Which metrics have you selected to track your customer experience improvement efforts?

I've got a lot of questions about the metric(s) you've chosen to measure customer sentiment about the experience with your company and to track your progress toward improving that experience.

For starters: How did you select the metric? Was it based on a desired business outcome? Was it based on a desired customer outcome? Did a consultant tell you to use that one? Did you read about it in a book or hear about it at a conference? Was there some clear basis for selecting the metric?

And then: Why? What do you do with that metric? Track it? Focus on it? Figure out how to move the needle on the metric? Make it all about the metric?

Have you ever heard of Goodhart’s Law? It states: When a measure becomes a target, it ceases to be a good measure. According to Wikipedia, its origin lies in finance and economics:
The original formulation by Goodhart, a former advisor to the Bank of England and Emeritus Professor at the London School of Economics, is this: “As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.” This is because investors try to anticipate what the effect of the regulation will be, and invest so as to benefit from it.
According to AdExchanger, the most-often cited example of Goodhart's Law is from a nail factory in the Soviet Union:
The goal of central planners was to measure performance of the factories, so factory operators were given targets around the number of nails produced. To meet and exceed the targets, factory operators produced millions of tiny, useless nails. When targets were switched to the total weight of nails produced, operators instead produced several enormous, heavy and useless nails.

The above example is absurd, but illustrates the point: When a measure of performance is the same as the target, it can be abused to the point of no longer being useful in measuring the desired outcome.
My fear is that too many focus on making useless nails in order to simply achieve the target. Unfortunately, this is one of the pitfalls of NPS. The target, not the customer and his desired outcome, becomes the focus. Moving the needle is what it's all about, at all costs.

When it was first introduced, and still today, the beauty of NPS was that it gets executives focused on the customer experience. The problem is, many simply focused on NPS. The directive became: Identify the benchmark! Tell me what our competitors' scores are! Beat them!

While metrics can focus the organization on the customer and the customer experience, that can only happen if the metrics are presented in the right context - and they're not made the focus or the goal but simply a way to measure progress. In order to do that, you'll need to...
  • talk about the score in context, i.e., talk about what it means relative to the customer and the customer experience
  • avoid gaming surveys (selecting certain customers, surveying at a specific time when you know scores will be better, offering incentives a la “the car dealer curse,” etc.) just to get a score
  • model appropriate behavior that leads to improved customer experiences - and, ultimately, an improved score
  • reward and recognize behavior and experiences that your customers want and prefer
  • compensate based on improving the experience, not the score
  • dig deeper and get at what's behind the metric
  • think strategy, not tactics, in order to improve the experience overall
  • have an action plan for improvement - talk about it and own it
Listen to customers. Understand customers, their needs, and what they are trying to achieve. Fix the things that your customers tell you are wrong. Know what their desired outcomes are but also know the business' desired outcomes. Then use the metric to measure progress - each customer's and yours.

Don't measure anything unless the data helps you make a better decision or change your actions. If you're not prepared to change your diet or your workouts, don't get on the scale. -Seth Godin


  1. My favourite example of Goodhart's law came from an Ex Policeman.

    In the UK the police are targeted on arrests / policing hour (or something similar). I live in a large provincial city and as we Brits enjoy a drink or two, 1am on Sunday mornings in the city centre is a sight to behold as people stagger from pub to bar to nightclub.

    The easy way to hit the arrests / hour target is park a police van in the market square and shovel any abusive drunk you find into the back of the van. Then let them sleep it off in the cells and throw them out the following morning.

    It ticks the target box quite nicely, but does nothing to reduce crime.

    1. James,
      I live near the centre of another provincial city where parking space is at a premium. However, I'm pretty sure a few years ago our parking wardens and their overlords fell victim to Goodhart's Law. There are now mechanisms in place to stop them from being over zealous but it doesn't solve the underlying problem.

    2. Great example, James. Numbers made, drunks off the street, root cause not fixed. Sadly.

    3. That's too bad, Adrian. Unfortunately, I think this Law has wide-reaching examples. It's crazy how people want to just keep taking shortcuts to achieve the quota - wouldn't their jobs be easier if they fixed the root cause?

  2. Call centres are another example of too many measurements and targets driving perverse behaviours with measurements such as handle times and hold times etc.both could be argued as dissatisfaction measures not satisfaction measures.

    1. Good point, John. This is quite the pervasive issue. Sadly.