Lessons from a Machine Learning Engineer – Part 3: Evaluation | by David Martin | January, 2025
Practical ideas for a data-driven approach to model development
In this third part of my series, I will explore the testing process which is a critical piece that will lead to a clean data set and improve the performance of your model. We will see the difference between the tests of a trained model (not yet in production), and test a suspended model (which makes real-world predictions).
In Part 1, I discussed the process of labeling your image data that you use in your image classification project. Show me how to define “good” images and create sub-classes. In part 2, I went through various datasets, beyond the usual test sets for train validation, such as benchmark sets, and how to handle synthetic data and duplicate images.
Evaluation of the trained model
As machine learning engineers we look at accuracy, F1, log loss, and other metrics to determine if a model is ready to go into production. These are all important steps, but in my experience, these points can be tricky especially as the number of classes increases.
Although it can be time consuming, I find it very important to manually review the images that the model receives which is wrongas well as…
