Ivory Tower Notes: Method

0 0 5 minutes read

This is post #2 in my Ivory Tower Notes series. In post #1, I wrote about the problem: how every data and AI project starts.

In this case, the topic is the method of operation, and why “fast in, smooth out” is what usually happens when we skip.

Quick, get out

I smiled a little when one of my contacts commented, “You sent me AI Slop” under a random post that had hundreds of likes. The post, which contained a decision matrix, provided guidance on which platform should be used to upload certain data, albeit with questionable criteria. Apart from the quality, it looked really good.

My fun didn't end there as I thought how AIS, that is, “AI Slop”, should be added as a button in all social networks now next to a button like it.

If other YouTubers are reading this, this is a feature comment instead of asking people, “Does this sound like AI going down?”

However, YouTube nailed the “emotion” part because we all tend to make decisions based on emotions, often at the expense of critical thinking.

Why invest in empiricism, rationalism, and skepticism when we have AI now? Deadlines are not on our side, and we have this new tool that brings us results, without the “quick entry, escape” effect.

But let's assume you're genuinely interested in how Platform A compares to Platform B in terms of machine learning (ML) capabilities, because you've noticed two data groups in your company using different platforms for nearly identical ML use cases. Therefore, your goal is to combine the objective overview of both and propose to reduce development costs by keeping only one.

What now? How do you decide whether to combine ML workloads?

Of course not by relying solely on AI, but…

How to ask

So you go back to the days of the Ivory Tower again, where you were taught that all discoveries are put together “The way it works”:

Problem → Hypothesis → Hypothesis testing → Conclusions

Furthermore, he was taught that finding a problem is part of the job, and the art of getting there lies in asking good questions to narrow it down to something specific and testable.

So, he takes a vague question, “Should we merge into one ML domain?”and you keep rewriting it until it's something that can be answered by the test:

Does Platform A use our churn pipeline with the same accuracy and lower cost than Platform B?

Now you've defined the topic, comparison, and measurables, enough to turn a business question into a testable idea.

But first, do your homework and gather more information, such as how much Platform B costs per transaction today, what it achieves in terms of accuracy, and how it's designed (eg, the data, algorithm, and hyperparameters it uses), so you can reproduce the pipeline on Platform A.

Then, before the comments on your question start coming in, you say:

If we use the same churn pipeline on Platform A instead of Platform B, using the same data, algorithm and hyperparameters, the average cost of each activity will decrease by at least 15%, while the accuracy of the estimate remains within 1 point of Platform B.

With this “if-then” construction, you've managed to silence (at least some) opinionated responses, knowing that PoC is coming next. Therefore, to test the stated assumption, you design and implement a PoC, where you only change the independent variable, which is the platform. Along with this, you set the control variables: data set, algorithm, and hyperparameters, and measure cost and accuracy, which are variables that depend on you.

You repeat the run several times to separate the signal from the noise by collecting more data points, considering how one run can avoid local noise (eg, a cache), and you want to avoid that situation. Then take into account many nuances, eg, starting to work at different times of the day (morning, evening, or night), to expose both platforms to the same mix of conditions.

Finally, you collect all the results and test the data against your hypothesis, which leads you to one of these three results:

Result 1: The data supports your hypothesis*. Multiple runs show how Platform A is at least 15% cheaper, and the accuracy remained within the defined limit. (*Just a note: the data will support, but not prove your hypothesis, that is, it will give you a reason to hold on to it, which in science is as close to “yes” as you get.)
Result 2: The data rejects your hypothesis. Multiple runs show how Platform A failed to meet one or both criteria; it is only 5% cheaper, or the cost is reduced, but the accuracy is reduced beyond the defined limit.
Result 3: Your run is too noisy to call either way, and the only answer is to keep checking before making any conclusions.

No matter what situation you end up in, you have a discovery: you may have confirmed your educated guesses, learned something new, or discovered that you need to keep exploring.

And to be clear about this short example: the first two conclusions will not give you the green light to merge the two platforms. The business reality (and overall testing) is worse than that, and there is more data (to be collected and tested) about people and processes than a single-scope PoC can address.

Okay, we can stop here now, because most of you are reading the steps above and wondering…

Photo by Toru Wa on Unsplash

What is the dickens? Where is AI in all this?

I can only imagine how something like: “MCP, agent frameworks, agents,…” was going through your head when you read the steps above. I couldn't agree more, all good stuff, and this is how you can speed up the process.

However, just sending AI results from a prompt like, “Give me an idea of how The platform A comparison The platform B for the ML workload,” is where the slop occurs, and “if you don't make hands, your opinion about it may be completely wrong.“

“If you don't use your hands, your perception of it might be completely wrong.”

Relevance and positive impact don't come from a good AI post or presentation infographics, and can damage professional relationships.

If you already have influence and want to be seen as an authority, it can be more effective to share ideas and findings from real-life research and your proven experience.

Instead of starting your post with “This is where you should use Platform A more The platform B in…”, try something concrete (if it's true, of course):

“When we (I) change the [independent variable] to see how it affects you [dependent variable]while keeping the [control variables] the same, our (my) findings be…”

Then see if the number of your followers is growing, and report your findings.

The inspiration for this post comes from a Croatian paper by Professor Mladen Šolić, “You'll make me happy” (Introduction to Scientific Research, 2005, [LINK]). I first read it as a student, and it remains one of the clearest explanations of the scientific research method I have come across.