what annoys me is the amount of people online, personally, even in my comment section saying “how AI will shut down data scientists.”
I find this frustrating because it often comes from people who don't work in the field, and it discourages potential great data scientists from pursuing this career path.
Not to mention, I strongly disagree with this view and believe that AI will not replace data scientists, at least definitely not within the next decade.
And this is coming from someone who has worked in this industry for 5 years in many companies, and seen what the industry was like before and after AI.
I have no worries about AI taking my job as it is, and in this article, I want to explain exactly why I think that and dispel all this fear.
You Need to Learn AI
Before we get into the actual “meat” of this article, let me start by saying that I am not a complete AI hater.
I use AI every day, and I'm always improving my skill in AI as it's a crazy productivity tool:
- Writing boilerplate code
- Discussing technical ideas
- Creating and writing documents
- Generate data visualizations and graphs quickly
- The perfect smart sparring partner
This technology is here to stay, and you need to learn how to use it; otherwise, you will be left behind.
Proficiency with AI tools will become commonplace, just as everyone is expected to use email these days or know Microsoft Word.
AI will not replace data scientists, but someone with fewer technical skills but greater AI skills can.
As a data scientist, you need to master tools such as:
Many more.
This will be fundamental to our industry, just as Python has become the most widely used machine learning language.
It's inevitable, and you need to get on board as soon as possible.
There will be big problems
Let's break down the skills that AI will need to develop to fully replace data scientists:
- Decompose abstract business problems into structured mathematical systems or algorithms.
- Interact with non-technical stakeholders and explain specific results with live questions.
- Write error-free production code every time to ensure that all important business decisions go smoothly.
- Make both logical and human trade-offs between complex, architectural design, and development processes.
- Build relationships and trust within the team, company, and industry.
If AI mastered all these skills to a level better than the current data scientist, what job would never end?
Many of them will disappear too.
If this happens, we have much bigger problems to worry about, almost level one problems, and your concerns about whether you should pursue a data science career will pale in comparison.
The AI singularity is the theoretical future point at which artificial intelligence surpasses human intelligence, leading to rapid, uncontrolled, and irreversible technological growth.
If data scientists are replaced, there may be bigger fish to fry in our lives than just worrying about our jobs.
Lack of Mathematical Thinking
One thing AI is very lacking in is statistical reasoning.
I'm not talking about the demographics that most people wonder about AI:
- Help me find the gradient for this function.
- Calculate the determinant of this matrix.
- What is the formula for Fibonacci numbers?
What I mean by “mathematical thinking” is the ability to solve intractable mathematical problems.
For example, AI is currently unable to solve the Riemann Hypothesis because it lacks the intelligence and logical reasoning to make major breakthroughs in pure mathematics.
The Riemann Hypothesis is a famous unsolved prediction that suggests that there is a hidden, underlying order to the apparently random distribution of prime numbers. Focusing on the “zeros” of a complex mathematical tool called the Riemann Zeta Function, it suggests that all non-significant zeros lie on a single straight line (the “significant line”).
The Riemann Hypothesis is an extreme example as it is arguably the most difficult problem currently in existence.
However, it shows that AI has not surpassed humans in statistical skills, which is the basis of data science.
Many people forget that these AI models are actually a type of model called large language models (LLMs), which are specifically designed to predict the next word from a pre-calculated probability distribution.
These models can only extrapolate, or support extrapolation, from the data they have observed; they can depart from what exists and not create anything “brand new”.
A data science career requires developing new solutions to abstract problems. In fact, we actually need data scientists and machine learning practitioners to build these AI models from scratch and maintain them!
AI Makes Mistakes
As someone who works with these tools every day in a series of applications, the AI makes so many mistakes it's ridiculous.
These LLMs often “hallucinate”, a term you may have heard and this is when these AI models produce results that seem plausible but are actually very wrong.
This comes from the fact that they are inherently probabilistic models and can “combine” arbitrary terms to meet user demands or expectations.
People also make mistakes, but the difference is that most people know their mistakes when you correct them. They are unsure of their first answer and, depending on the situation.
Although AI is quite stubborn, smart, and sure of the answers it gives you, it tricks us, humans, into thinking it is correct.
Think how difficult this can be in a work environment.
An AI data scientist cannot accurately measure how annoying or funny their product is, so they fail to set expectations when using a given 'solution'.
It misses that lack of imagination and intangibles that we humans have about most data science and machine learning projects.
Limitation of Performance
What's interesting to me is that these types of AI actually don't get much better over time.
The reason is twofold:
- The basic algorithm remains the same; all these LLMs use i The Transformer structures, so each “new” model is not actually “new”.
- There is a limit to the amount of data they can be trained on, as only so much information exists in the world.
For example, OpenAI's GPT models are basically trained all over the Internet to some extent, there isn't much “new” data to use.
There is a cap on how good they can get.
This data also comes from humans, so it cannot exceed human intelligence; that is its roof.
These AI models will not improve unless there is a major scientific breakthrough in the underlying algorithm.
And the fact that they won't get better means that the status quo will remain the same, and AI has not yet replaced data scientists.
Can't Build Relationships
AI does not know relationships, despite how many people are tragically affected by these robots.
Humans are social creatures, and most business interactions in the world are conducted through relationships.
People do business with, hire, and work with people they like, even if they aren't the “best fit”.
It's just how we're wired to act from a biological point of view.
A stakeholder will trust you as a data scientist if you deliver consistent results.
Even if AI comes up with a “better” solution to their problem, stakeholders will prioritize it because of the intangible relationship of the person who built it.
Every job depends on human connection. Some parts will be automatic, but most will not.
In the case of a data scientist, it can be very difficult to automate:
- Discussion of technical problem data issues with a specific stakeholder
- Gathering requirements from the business leader about the problem they want to solve
- Communicating and influencing members of other groups and activities
Any active human part cannot be replaced.
Has Something Really Changed?
One of my former bosses once asked me:
Has anything really changed since the release of AI?
Sure, we now have better tools to solve certain problems, and productivity in some aspects of our jobs has increased, but the role of the data scientist honestly hasn't changed that much.
Take a minute to think about what has changed in your daily life from AI.
I doubt you can say much, if anything.
AI, in its current form, has been around for over 4 years, yet society as a whole has not been affected much from where I stand.
That's all that needs to be said here.
If, after reading this, you really want to go deeper into learning AI, I recommend my previous post, which gives you a full, in-depth roadmap of everything you need to master AI.
You can check it out below!
One thing!
Join my free newsletter where I share weekly tips, insights, and advice from my experience as a practicing data scientist and machine learning engineer. And, as a subscriber, you will receive mine FREE Resume Template!
Data Extraction
Weekly emails help you find your first job in data science or machine learningnewsletter.egorhowell.com
