The Arithmetic of Productivity Improvement: Why “40% Increase in Productivity” Never Really Works?

Introduction: False promises?

as a consultant and manager in the data industry, I've sat through my fair share of slide deck presentations. On both sides. And any slide deck worth its salt promises something, usually in terms of efficiency or productivity. You've probably heard something like this:

• This tool makes your data scientists 40% more productive!
• You'll spend 30% less time fixing bugs by doing this. You can use a 6 hour work day and come out ahead!
• With our solution, you can code two projects in the time it previously took you to do just one. This reduces the amount of production time!

Sometimes promises don't work because the proposed product is bad. But why does this not seem to work even with good products? You may end up switching to a product you genuinely like, but not really seeing the promised improvement. Why? Are the numbers you got false?

My background as a PhD in mathematics has probably scarred me for life in more ways than one. One of the deepest scars is my need to understand precisely what the numbers represent. And the numbers you get from the above statements all point to one thing, while they tell a completely different story when you stop to think about it.

Although lying does happen, what is the most common practice misleading. This type of marketing assumes that you don't think critically when presented with numbers. Let's think together and see what we come up with.

Lies, lies, and marketing

So what's the problem with production statements?

The main problem is that they claim to increase a certain part of the process, while (indirectly) promising global productivity gains.

Let's go through a simple example to understand what this means.

Say you're a big player in AI and you just launched a great product to help data scientists with model parameter selection. Good! Initial surveys show that it has given data scientists a 20% increase in productivity in model parameter selection. He introduced this first:

Our tool has improved model parameter selection productivity for data scientists by 20%.

Happy with this positive result, you send your statement to marketing and they come back with only minor changes:

Our tool improved model parameter selection, making data scientists 20% more productive.

You cringe and wonder a little if these people in advertising really get paid to blurt out a few words. In fact, they've now changed your statement from moderately impressive to insanely impressive.

Why? Marketing adjustments make it appear that the product makes data scientists 20% more productive. as usual. But your survey only addressed productivity when data scientists were choosing model parameters. What is the real difference?

A data scientist does a lot of things, including prototyping, managing stakeholders, coordinating meetings, etc. Although machine learning is often front and center in how they describe themselves, most data scientists only spend about 40% of their time on typical data science tasks. A large portion of this 40% is fixing data quality issues, pipeline management, and data validation. Model parameter selection can take only 10% of their 40% time doing data science tasks. Replication shows us that this is only 4% of their total time.

If a data scientist uses a tool to make model parameter selection 20% more productive, that can make a difference of about 1% of their overall time. You wouldn't notice this during the work week. In fact, with the added complexity of learning a new tool at first, you may see a drop in productivity at first.

The best part? Look at the statement carefully:

Our tool improved model parameter selection, making data scientists 20% more productive.

It certainly seems counterintuitive that data scientists will be 20% more productive, but this is just one interpretation. If pressed, marketing will make a connection between the beginning and the end of the sentence, and it means that it is said that the increase in productivity is only for choosing a parameter of the model.

So you get to say one thing effectively, while reverting to another if a misleading statement is discovered. The marketing charge comes from mixing in the area the words are right!

A better way? Focus on cognitive load instead of productivity

What does this story I just told you really tell you? If you have many different complex tasks (like a data scientist does), then aiming to achieve productivity isn't pushing the needle that much.

Don't get me wrong. If you have an easy 20% productivity opportunity with one of your jobs, do it! But don't expect it to result in more than a percent or two difference in overall productivity.

What can we do instead when we have many different complex tasks? We can use cognitive load as a metric, and try to reduce that instead.

Say that a competing company has developed its own model parameter selection tool. Instead of trying to speed up the process, their tool had the sole purpose of reducing the cognitive load of the data scientist. So the model selection process may take the same amount of time, but the data scientist will feel motivated and ready for another challenge after selecting the model parameters.

Most people, myself included, cannot work 8 hours a day and be at the top of our game all the time. Some days I feel like I have 6 hours working on me. Some days it's like 2 working hours. If one process does not require so much mental load, I can work for a long time effectively. This often results in the same overall productivity of a few percent, but with the added benefit of improved behavior.

So the next time someone introduces “40% increase in productivity“, they asked the following:

• How much does this increase in productivity affect total work time?
• How much cognitive load does this remove or introduce?