Loss Work Explained For Noobs (How Models Know They're Wrong)

# Introduction

I know that when beginners start learning machine learning, things seem easy at first. You follow a tutorial that asks you to load a dataset, train a model, and see something like this: loss = "mse" or criterion = nn.CrossEntropyLoss().

And just like that, the tutorial starts talking about equations, gradients, optimization, and Greek letters. If you've ever nodded off without really understanding what a loss function does, you're not alone. Loss functions are often defined backwards. Many studies start with a formula when they should start with an idea. This article is part of my noob series, where I will make things easier for you to understand. So, let's begin.

# What is a Loss Function?

The loss function is how a machine learning model knows how wrong it is. That's literally the whole idea. The model makes a prediction. The loss function compares that prediction with the correct answer. Then it gives the model a number that says, “How bad was your mistake.”

A high loss means that the model was which is very wrong.

A low loss means that the model was shut up.

During training, the model continuously adjusts itself to minimize losses.

Such is learning. If you have ever played a game of darts, it is very similar. He throws an arrow. To improve, you need feedback. You need to know if your dart was a little too far, too far, too high, or too far to the left. Without that feedback, you can't improve. Therefore, the bullseye is the correct answer and the arrow is the prediction. He measures the distance between the dart and the bullseye. The loss function measures how far the dart has traveled. That distance becomes the model's feedback signal. Here's what it might look like if you choose to visualize.

Like the distance from the center, throwing too close is not the same as being too far. Similarly, in models, just knowing that the answer is wrong is not enough. A model needs to know how badly it failed in order to improve.

Now that we have an understanding of what a loss function is and why we need it, let's look at something else common loss functions used in machine learning.

# Mean Squared Error

The most common loss in numerical prediction is the mean squared error (MSE). It is often used when the model predicts numbers such as house prices, temperatures, or delivery times. The idea is very simple.

• Error: For each prediction, take the gap between the prediction and the reality.
• Square: Multiply each gap by itself.
• It means: The average of all those square spaces.

You can write it in Python like this:

def mean_squared_error(predictions, actuals):
    squared_errors = [(p - a) ** 2 for p, a in zip(predictions, actuals)]
    return sum(squared_errors) / len(squared_errors)

Now, I know that taking mistakes and weighing them over predictions makes intuitive sense, but understanding why we owe them can be confusing. This is done for two reasons:

• Squaring makes every error positive. An error of +3 and an error of -3 are equally bad, and squaring turns both into 9, so they stop canceling.
• Squaring penalizes large errors more severely than small ones. This is good for many use cases. For example, when predicting house prices, making a mistake by $1,000 versus $200,000 should be penalized accordingly.

# It means Total Error

Another common loss function is the mean absolute error (MAE). MAE also measures the gap between forecasts and actual values, but does not account for error. Instead, it simply takes an integer value.

Here's a Python function to write it:

def mean_absolute_error(predictions, actuals):
    absolute_errors = [abs(p - a) for p, a in zip(predictions, actuals)]
    return sum(absolute_errors) / len(absolute_errors)

Therefore, it penalizes large errors, but not as badly as MSE does.

• A mistake of 10 is worth 10 and a mistake of 20 is worth 20.
• If your data naturally has some outliers and you don't want your model to overreact, MAE is a good choice.

Let me show you a quick graph comparing the MSE and MAE curves.

# Cross-Entropy Loss

So far, we have talked about predicting numbers. But most machine learning problems are about predicting categories.

Is this email spam or not?

Is this a picture of a cat, dog, or fish?

Is a job fake or not?

For partition functions, models typically output probabilities like these:

Dog: 70%
Cat: 20%
Fish: 10%

If the picture is really a dog, that's a good prediction. But if it is a cat, the model needs to be penalized by giving a lower probability of the correct answer.

Therefore, intuition says:

• Good and confident – low losses
• OK but not sure – medium loss
• Incorrect and confident – high losses

That is why cross-entropy is widely used in classification. It doesn't matter if the model was right. It's also worrying how confident the model was.

# Losing vs. Accuracy

Now that we have gone through the different functions of loss, I also want to clarify the difference between loss and accuracy. They are not the same thing.

Accuracy he tells you how many predictions were correct.

But loss he tells you how bad were the model errors.

If you have two models – Model A and Model B – and they both get 90 out of 100 correct predictions, they will have the same accuracy. But one model may be overconfident on correct answers and underconfident on incorrect ones, while another may be underconfident on many examples and overconfident on incorrect ones.

If so, the accuracy will be the same, but the loss will be different.

# Training Loop

If the model has a number of losses, it can improve. The training loop looks like this:

1. The model makes predictions.
2. The loss function measures the errors.
3. The optimizer updates the model.
4. The model tries again.
5. I hope the loss will be minimal.

When we train the model, we also plot the loss over time. At first, the model makes many mistakes and is poor at making predictions, so the losses are high. But as the training continues, the loss decreases and the model gets better at making predictions.

A healthy training curve usually looks like this:

High loss initially → sharp decline → gradual flattening

as you can see in the image below.

Flatness is normal. It means that the model has learned simple patterns and now makes small improvements. But if training losses decrease while validation losses start to increase, that could be a warning sign to exaggerate – which means the model may be memorizing the training data instead of learning regular patterns.

# Final thoughts

The loss function results in a model error.

It tells the model how wrong its predictions are, and gives the training a clear goal: make that number smaller.

Once you understand loss functions, many other machine learning concepts become easier to understand – including gradient descent, posterior distribution, optimization, superimposition, and evaluation metrics.

You don't have to start with scary math. Start with an idea:

1. The model is predictive.
2. The loss function leads the guesswork.
3. The model updates itself to reduce the score.

That is the heart of machine learning.

Losing is how the model knows it's wrong.

Training is how he learns not to be so good.

This brings us to the end of this article. We will continue to cover interesting concepts throughout our noob series.

Kanwal Mehreen is a machine learning engineer and technical writer with a deep passion for data science and the intersection of AI and medicine. He co-authored the ebook “Increasing Productivity with ChatGPT”. As a Google Generation Scholar 2022 for APAC, he strives for diversity and academic excellence. He has also been recognized as a Teradata Diversity in Tech Scholar, a Mitacs Globalink Research Scholar, and a Harvard WeCode Scholar. Kanwal is a passionate advocate for change, having founded FEMCodes to empower women in STEM fields.