Eliminate Building Any Metric with Simple 'What' Questions

“5x improvement” [FULL STOP]it's meant to make you think, “That's incredible; it should be worth my time and money.“

However, a subjective statement like that is a red flag, and knowing how to investigate vague metrics is a basic skill for anyone looking to separate real value from clever marketing.

That's why I invite you to put on your data analysis glasses and get the context you need to get actionable insights.

The popularity of math has followed a similar path to psychology, moving from professional guidance to self-help books and eventually to blogs, YouTube videos, and Instagram motivational quotes emblazoned on t-shirts. The whole idea was to pour out our pain in words that we can wear with pride in the best piece of clothing there – a hoodie – and finally explain to the world what an analyst (me!) does, thinking that it defines the field of analysis, too.

Yes, for people who catch me on laundry day, I have a simple explanation of the math ready to go, and I share that the math is “What you see is rarely what you get,” while statistics, on the other hand, say “The truth is there” (as mentioned in The X-Files).

Then I added that I am always interested in searching for the “truth,” referring to my experience of six years in the Ivory Tower, but I prefer to swim in analytical waters now.

That's why I stand by the statement:

Glossy dashboards backed by data storytelling often aim to dazzle the untrained eye by presenting carefully selected data.

Meanwhile, the trained eye knows that there is more hidden behind the metrics whenever a statement like this appears:

“5x improvement”

Followed by a full stop/period/full stop/dot or whatever you like to call it. I don't know about you, but it hurts when I look at that sentence knowing that someone once dared to call it a metric of success.

For the sake of common sense I hope still exists in the current AI limbo, let's rewrite the above statement on how the metric should be presented by adding…

#1: Size

I know a few of my friends who are still fans of Simon Sinek would say, “Start with why.” 🙂

No. I started with “what,” and asked, “Development of what?”

I took courage because the first thing that came to my mind after seeing the word “development” hanging like that was:

print(“The improvement is 5x.”)

And that's not a way to get information that anyone can do anything about, is it?

But imagine for a moment the statement is made up of “The improvement in model accuracy is 5x.”

If that were the case, I would imagine something different and imagine a small table with some performance dimensions, such as the accuracy of the model, and their exact measurements.

However, just to make sure the “5x” wasn't pulled out of thin air, the second factor missing from this imaginary table of mine is the date/time of day. Which makes our statement sound like this:

“The monthly model accuracy improvement is 5x.“

Now, that's better, or at least it feels that way, because we can say that we got an improvement by comparing the log accuracy models across all monthly runs.

But, to understand this “5x”, and any development, some important information is missing…

#2: The foundation

That is why I will continue with my “what“ questions by asking, “Progress from what?”

“5x improvement” sounds really impressive until we see the accuracy of the basic model for predicting the exact result of 100 chances was 1% last month, and now it's 5%.

If we saw a raw value of 5%, we would know that it means that the model's predictions are no longer correct 95% of the time, and we would not consider it an improvement that should drive our actions. That's why we usually don't show the raw numbers, but just “5x” because it looks good on the dashboard.

Now that we have this information, we can rewrite our statement again:

“The monthly model accuracy improvement is 5x, growing from a base of 1% to 5%.

That looks better too. Anyway, seeing some time attached brings me to the next missing piece…

#3: It's time to compare

Which leads me to my last “what” question: “The development is comparable to what time?”

Our statement does not tell us what kind of comparison we are making – a change over time, a fixed-cadence comparison (month-over-month, quarter-over-quarter, year-over-year), or a fixed-period comparison?

Retrospective: Is it a direct comparison to the previous month, or to several months in a row? Maybe a year-to-year comparison, where this month is compared to the same month last year? Or is it just an arbitrary comparison between two hand-picked months?

Assuming we've got the answer, we can redo the previous version:

“Compared to the results from May 2026 to April 2026, the monthly model accuracy improvement is 5x, increasing from a base of 1% to 5%.

The best. The sentence at the end tells us what was developed, from what basis, and at what time. And now, with the pleasure of some of my friends, I will ask…

“Why” questions.

To wrap up, two “why” questions I would ask when presented with a statement like that “5x improvement” they are:

#1: Why should this metric concern me?
The metrics that drive a presenter's decisions don't always drive yours, and the same depends on the level of detail. Enough for them to decide is often not enough for you to do. So the next time someone makes a statement like the one above, ask them WHO designed for and what actions it should drive.

#2: Why are the “what” questions left out?
Of course, starting with “why” is important, especially if you're trying to understand a particular implementation problem. But when it comes to understanding the metric in front of you, someone presented theirs with a “why” in mind, you need to interrogate it with “what” questions so you don't get fooled by shiny dashboards and data storytelling.