Stop Blaming the Data: A Better Way to Manage Covariance Shift

Despite tabular data being the bread and butter of industry data science, data variability is often overlooked when analyzing model performance.

We've been there: You develop a machine learning model, get good results from your validation set, and then apply it (or test it) to a new, real-world dataset. Suddenly, performance drops.

So, what's the problem?

Often, we point the finger Covariance Shift. The distribution of features in the new data is different from the training data. We use this as a “Get Out of Jail Free” card: “The data has changed, naturally, the performance is poor. It's the data's fault, not the model's.”

But what if we stop using the covariance shift as an excuse and start using it as a tool?

I believe there is a better way to handle this and create a “gold standard” for model performance analysis. That approach will allow us to measure performance accurately, even when the ground shifts beneath our feet.

Problem: Comparing Apples and Oranges

Let's look at a simple example from the medical field.

Assume that we trained the model on elderly patients 40-89. However, in our new target test data, the age range is tight: 50-80.

If we simply apply the model to test data and compare it to our actual validation scores, we are fooling ourselves. To compare “apples to apples,” a good data scientist can go back to the validation set, filter for patients aged 50-80, and recalculate the baseline performance.

But let's make it difficult

Let's say our test data set contains millions of records that are 50-80 years old, and one patient 40 years old.

• Do we compare our results with the 40-80 confidence interval?
• Are we comparing the 50-80 range?

If we ignore certain age distributions (which most standard analyzes do), that one 40-year-old patient conceptually changes the meaning of the group. Actually, we can just remove that extraneous. But what if there were 100 or 1,000 patients under the age of 50? Can we do better? Can we automate this process to handle the differences many variables simultaneously without sorting the data manually? Furthermore, filtering the data is not a good solution. It only calculates the correct range but ignores the change in distribution within that range.

Solution: Inverse Probability Weighting

The solution is to statistically rescale our validation data to look like test data. Instead of binary input/output (maintaining or dropping a row), we provide continuous weight for each record in our validation set. It's like an extension of the simple filter method above to match the same age range.

• Weight = 1: General analysis.
• Weight = 0: Do not enter a record (sorting).
• The weight is non-negative: Low sample or high sample of record influence.

Intuition

In our example (Test: Age 50-80 + one 40yo), the solution is to simulate a test set within our validation set. We want our authentication set to “pretend” has exactly the same age distribution as the test set.

Note: Although it is possible to convert these weights to binary inclusion/exclusion with a small random sample, this usually does not provide a statistical advantage over using the weights directly. Subsampling is especially important for information or if your specific performance analysis tools cannot handle weighted data.

Mathematics

Let's make this legal. We need to define two possibilities:

• P_t(x): Probability of seeing the value of characteristic x (eg, Age) in Objective Test data.
• P_v(x): Probability of seeing the characteristic value x in Confirmation data.

The weight w of any record with factor x is a measure of this probability:

w(x) := P_t(x) / P_v(x)

This is understandable. If 60-year-olds are rare in training (P_v low) but normal in production (P_t is higher), the ratio is larger. We measure these records up in our analysis to match the truth. On the other hand, in our example where the test set is aged 50-80 years, any validation patients outside this range will receive a weight of 0 (since P_t(Years)=0). This is exactly the same as excluding, as needed.

This is a mathematical process often referred to as Importance of Sampling or Inverse Probability Weighting (IPW).

By using these weights when calculating metrics (such as accuracy, AUC, or RMSE) for your validation set, you create a synthetic set that closely matches the test domain. Now you can compare apples to apples without complaining about the change.

Extension: Managing High-Dimensional Shifts

Doing this for one variable (Age) is easy. You can just use histograms/bins. But what if the data changes between many different variables at the same time? We cannot construct a twelve-dimensional histogram. The solution is a clever trick using a binary separator.

We train a new model (“Propensity Model,” let's call it M_p) to distinguish between two datasets.

• Input: Record characteristics (Age, BMI, Blood Pressure, etc.) or variables we want to control.
• Target: 0 if the record is from the validation set, 1 if the record is from the test set.

If this model cannot classify the data easily (AUC > 0.5), then there is a covariate change. The AUC of M_p and serves as a diagnostic tool. It describes how different your test data is from the validation set and how important the accountability was. Most importantly, the potential output of this model gives us what we need to calculate the weights.

Using Bayes theorem, the sample weight x becomes opportunities that the sample belongs to the test set:

w(x) := M_p(x) / (1- M_p(x))

• If M_p(x) ~ 0.5, the data points are indistinguishable, and the weight is 1.
• If M_p(x) -> 1, the model is more confident that this looks like the test data, and the weight increases.

Note: Using these weights does not lead to a decrease in expected performance. In some cases, the test distribution may shift to smaller groups where your model is more accurate. In that case, the method will increase those conditions and your relative performance will reflect that.

Does it work?

Yes, like magic. If you take your validation set, use these weights, and plot the distribution of your variables, they will very well cover the distribution of your target test set.

It's more than that powerful rather: aligns i joint distribution of all varieties, not just their individual distributions. Your weighted validation data becomes indistinguishable from the target test data when the forecast works well.

This is a generalization of the single variable that we saw earlier and gives exactly the same result as the single variable. Intuitively M_p studies the differences between our test and validation datasets. We then use this learned 'insight' to combat the differences statistically.

For example you can look at this code snippet to generate 2 distributions: one uniform (validation set), one normal distribution (directed test set), with our weights.

import pandas as pd
import numpy as np
import plotly.graph_objects as go

df = pd.DataFrame({"Age": np.random.randint(40,89, 10000) })
df2 = pd.DataFrame({"Age": np.random.normal(65, 10, 10000) })
df2["Age"] = df2["Age"].round().astype(int)
df2 = df2[df2["Age"].between(40,89)].reset_index(drop=True)
df3 = df.copy()

def get_fig(df:pd.DataFrame, title:str):
    if 'weight' not in df.columns:
        df["weight"] = 1
    age_count = df.groupby("Age")["weight"].sum().reset_index().sort_values("Age")
    tot = df["weight"].sum()
    age_count["Percentage"] = 100 * age_count["weight"] / tot
    f = go.Bar(x=age_count["Age"], y=age_count["Percentage"], name=title)
    return f, age_count

f1, age_count1 = get_fig(df, "ValidationSet")
f2, age_count2 = get_fig(df2, "TargetTestSet")

age_stats = age_count1[["Age", "Percentage"]].merge(age_count2[["Age", "Percentage"]].rename(columns={"Percentage": "Percentage2"}), on=["Age"])
age_stats["weight"] = age_stats["Percentage2"] / age_stats["Percentage"]

df3 = df3.merge(age_stats[["Age", "weight"]], on=["Age"])
f3, _ = get_fig(df3, "ValidationSet-Weighted")

fig = go.Figure(layout={"title":"Age Distribution"})
fig.add_trace(f1)
fig.add_trace(f2)
fig.add_trace(f3)

fig.update_xaxes(title_text='Age') # Set the x-axis title
fig.update_yaxes(title_text='Percentage') # Set the y-axis title
fig.show()

Limitations

Although this is a powerful method, it does not always work. There are three main statistical limitations:

1. Hidden Conjunctions: If the shift is caused by a variable you did not measure (eg, a genetic marker you do not have in your tabular data), you cannot measure it. However, as model developers, we often try to use more predictive features in our model when possible.
2. Indifference (Emptiness): You cannot divide by zero. If P_v(x) is zero (eg, your training data has no more than 90 patients, but the test set does), the weight explodes endlessly.
   • Repair: Identify these non-overlapping groups. If your validation set actually contains zero information about a particular subpopulation, you should explicitly exclude those subpopulations from the comparison and mark them as “unknown”.
3. Propensity Model Quality: Since we rely on the model (M_p) to measure the weights, any inaccuracy or miscalculation in this model will introduce noise. For low-dimensional transformations (such as the single 'Age' transformation), this is ignored, but for complex high-dimensional transformations, ensuring that M_p well balanced is important.

Although the propensity model is not perfect in practice, using these weights greatly reduces the shift in the distribution. This provides a more accurate representation of real-world performance than doing nothing at all.

Mathematical Powers Note

Note that using weights is variable Effective Sample Size. High variance weights reduce the stability of your measurements.

Bootstrapping: When using bootstrapping, you are safe as long as you include the weights in the sampling process itself.

Power Statistics: Do not use raw number of lines (N). Please refer to Effective Sample Size formula (Kish's ESS) to understand the true power of your weighted analysis.

What about pictures and text?

The propensity model approach works for those domains as well. However, the main issue is from a practical perspective in general ignorance. There is a complete disconnect between our validation and target test set which leads to resistance to change. It does not mean that our model will perform poorly on those datasets. It just means that we can't measure its performance based on your current completely different verification.

Summary

A best practice for evaluating the performance of a model on tabular data is to strictly follow the covariance shift. Instead of using the shift as an excuse for inefficiency, use it Inverse Probability Weighting to estimate how your model should work in the new environment.

This allows you to answer one of the most difficult questions in implementation: “Is the performance degradation due to data changes, or is the model really broken?”

Using this method, you can define the gap between training and production metrics.

If you found this helpful, connect us on LinkedIn