Overriding the vs

Models like cooking: Too little toning and the dish is bland, too much and passable. The goal? That's the perfect balance – just enough difficulty to capture the flavor of the details, but not so much that it's overwhelming.

In this post, we'll go through two of the most common obstacles to model development: to overdo it and Underfing. Whether you're training your first model or planning your hundredth, keeping these concepts in mind is key to building models that work in the real world.

It's overkill

What is excess?

Overfitting is a common problem with data science models. It happens when the model learns best from the trained data, which means it learns from certain patterns in the data and the trained noise. Therefore, it cannot be accurately predicted based on unobserved data.

Why is it more of a problem?

1. Poor performance: The model cannot perform well. Patterns found during training do not apply to all data. You get the impression that the model is working well based on the training errors, when in reality the test or real world errors are not what one hopes for.
2. Predictions with high variability: model performance is unstable and predictions are unreliable. Small adjustments to the data resulted in high variance in the predictions made.
3. Training a complex and expensive model: Training and building a complex model in production is an expensive and technical task. If a simple model works just fine, it works fine to use it instead.
4. Risk of losing Business Trust: Data scientists who are overly optimistic when trying out new models may backfire on business stakeholders. If overfitting is discovered only after the model has been introduced, it can seriously damage reliability and make it difficult to regain trust in the model.

How to Get Skipped

1. Cross validation: During cross validation, the input data is divided into several folders (training sets and test data). Different input data folders should give the same error results. A large gap in performance across portfolios can provide exemplary instability or data leakage, both of which can be signs of overcriticism.
2. Keep track of training, testing and common mistakes. The error when the model is submitted (Generalization error) should not deviate too much from the known errors. If you want to go the extra mile, consider using a warning alert if the performance of the used model deviates significantly from the set error.

How to reduce / prevent overeating

1. Remove Features: Too many features can “steer” the model too much, therefore leading to an underperforming model.
2. Maximize Training Data: By providing more examples to learn from, the model learns to improve and is less sensitive to variables and noise.
3. Increase continuity: General techniques help by penalizing coefficients that are already reduced. This prevents the model from fitting too well to the data.
4. Adjust hyper-parameters: some parameters are over-coupled, it may result in a model that can't perform well.

Underfing

What's down?

It rarely occurs when the nature of the model or features are too simple to capture the underlying data. It also results in poor predictions for unobserved data.

Why is it less of a problem?

1. Poor performance: The model performs poorly on training data, and therefore also poorly in testing with real-world data.
2. Predictions with high biases: The model cannot make reliable predictions.

How to Get Underfing

1. Training and testing errors will be poor.
2. The error of normalization will be high, and probably close to the error of training.

How to fix underfing

1. Improve Features: Introduce new features, or add complex features (eg: Add interaction effects / polynomial terms / annual style terms) that will capture common patterns in the underlying data
2. Maximize Training Data: By providing more examples to learn from, the model learns to improve and is less sensitive to variables and noise.
3. Reduce normalization power: When using a normalization procedure with high power, features become more uniform and the model does not prioritize any feature, preventing it from learning important patterns.
4. Adjust the hyper-parameters: a linguistically complex model with hyper-parameters may not be able to capture all the complexity. Paying more attention to regularization can be valuable (eg add more trees to a random forest).
5. If all other options don't fix the underlying problem, it may be beneficial to drop the model and replace it with someone who can capture common patterns in the data.

To put it briefly

Machine learning isn't magic, it's a balancing act between too much and too little. Overdo your model, and it becomes a perfectionist who can't handle new situations. Don't press him, and it misses the point entirely.

The best models live in a sweet spot: Geneling Well, enough reading, but not too much. By understanding and managing proactively and tirelessly, you are not only improving, but also building trust, reducing risk, and creating solutions that go beyond training.

Resources

[1]

[2]