for us we ask for a model.
We built a proof of concept. I got the green light. Model delivered.
Weeks of work… all nothingness.
It's a myth as old as time, and one that plagues data professionals everywhere, from analysts to ML developers.
So, what happened?
Your Model is a Mystery
Our work focuses on modern computer science and technology development. Many of the most powerful solutions at our fingertips are those that would have been computationally expensive decades ago. With reliance on new, more technologically competent developments, comes skepticism.
In data science, we have the ability to create incredibly complex models. My team alone has thousands of standard features in our feature library that we provide for each new model built. We tune dozens of parameters and use powerful algorithms that iterate over hundreds of runs to maximize predictive performance. This technique can create models with incredible accuracy, but it comes at a cost: interpretation.
There's a fine line between a solid model, and a black box that can't even be defined by those who built it.
The tradeoff of detail and accuracy is a big factor in my industry, healthcare, in particular. Clients and stakeholders are often doctors and therapists. These doctors tend to make clinical judgments using their years of experience and deep medical knowledge. Although a prognostic model may be good at predicting a given outcome, if it is not well defined, clinicians will question its reliability. If doctors have to choose between a trusted and proven clinical procedure, or a black box model with hidden features and inexplicable algorithms, they will probably choose the clinical procedure every time.
So, what can you do to avoid this? I find great success by providing clients with an easy-to-digest model brief. This is a set of slides that walk customers through the model. It begins by defining the population of interest, targets, characteristics, and concludes with proof of concept and validation. Along the way, I'm sure to explain the metrics in terms of a business question, putting myself in the customer's shoes. I avoid statistical talk and keep explanations based on the client's goals. If the model is complex, I stick to the top descriptions of the algorithm and make sure to mention why I chose a broad (or simple) feature set. Creating a concise model is an important step in pulling back the curtain and allowing customers to understand the model using terms they are familiar with.
Your Solution Took Too Long
Building effective models takes time. From writing back and forth with clients, to unexpected twists you didn't see coming, designing an effective, useful model is no quick task. Then there's the shipping. That is a whole process in itself.
What can't wait patiently is the real world. Customers live day to day with the tools they already have. The tools were there before they came to you for help. If building a model takes too long, they can abandon the idea altogether, or find creative solutions that don't involve predictive models.
We see this all the time in health care. Participants will request a model. After a few roadblocks (stopped communication from requesters, data access issues, shipping bugs, etc.), weeks of development turned into months. Finally, you are ready to present the findings after everything has been verified and is working as expected. He tries to stop the meeting and gets heartache: “We don't need a model anymore, we got one for ourselves.” The hospital environment is a fast-paced environment. Employees don't have time to sit around waiting for months on end. They can and will come up with creative solutions to improve their patients' care, even if that means sacrificing the use of a shiny predictive model.
There is a saying that I live by at work: “Don't let the perfect get in the way of the good”. Build quickly. Clarify, refine, revise…but always move forward. Completeness can prevent you from providing important information. The world moves fast, and if you get stuck in the construction phase for too long, the world will go on without you. So, press that v1. If you find a better way to do things later, it can be first on your development list for v2. Some solution is almost always better than no solution.
If things are going slower than planned, you'll need to communicate with customers early and often. Keep them updated on your progress, and give them snippets to keep them engaged and excited about the final product. Bid your time while grinding to get v1 working and in their hands.
Your Model Is Not Easy To Use
Building a good predictive model is only half the battle. In most industries, stakeholders are busy. In health care, doctors and nurses are busy caring for patients. If the data science team comes to the low maintenance team to install their new, more accurate model, but accessing the predictions adds complexity to their workflow and slows down the process, the model will not be used. The same can be seen in many industries. Stakeholders are looking for solutions that can increase efficiency, effectiveness, and productivity, not ones that add complexity to their already busy days.
If the predictions present a contradiction, you are making a path to destruction, not discovery.
Providing easy-to-use predictions can be one of the biggest challenges for data scientists. We may have the ability to create precise and accurate models, but integrating the model into the daily lives of customers is less automatic. This part is less about numbers, probability, and math knowledge, and more about practicality, business knowledge, and familiarity with the applicants' daily routines.
In the hospital setting, this looks like integration into Epic, the system-wide electronic health record software. Instead of requiring busy physicians to log into a separate system to see forecasts, they can access it right there, in patient charts, alongside their other clinical tools and patient data. In other industries, the same concept applies. Do not interrupt the current process. Get into it.
Wrapping up
One of the biggest disappointments a data scientist can face throughout their career is when their hard work goes unused. It happens more than one would like to think, and it is easy to blame the customer. After all, it's easy on the ego.
In fact, there may be important things that the data scientist overlooked somewhere along the development line. Knowing common pitfalls can help data scientists get their models on track. I the original finish line: discovery.
