Common Statistical Pitfalls in FAANG Discussions

# Introduction

When you apply for a job at Meta (formerly Facebook), Apple, Amazon, Netflix, or Alphabet (Google) – a group known as FAANG – interviews rarely check if you can repeat textbook definitions. Instead, interviewers want to see if you analyze the data carefully and if you can identify bad analysis before it is sent to production. Statistical traps are one of the most reliable ways to test that.

These pitfalls replicate the kinds of decisions that analysts face every day: a dashboard number that looks good but is actually misleading, or a test result that seems plausible but contains a structural error. The questioner already knows the answer. What they're looking at is your thought process, including whether you're asking the right questions, missing notice, and backtracking on a number that looks good at first glance. Students often stumble over these methods, even those with strong math backgrounds.

We will examine five of the most common pitfalls.

# Understanding Simpson's Paradox

This trap is intended to catch people who unquestioningly trust composite numbers.

Simpson's paradox occurs when a trend appears in different groups of data but disappears or reverses when those groups are combined. A classic example is UC Berkeley's 1973 admissions data: overall admission rates were male-favored, but when broken down by the door, women had equal or better admission rates. The total number was misleading because women applied to highly competitive departments.

Confusion is inevitable whenever groups are of different sizes and at different basic levels. Understanding that is what can separate a superficial response from a deep one.

In interviews, the question might look like this: “We ran an A/B test. Overall, variant B had the highest conversion rate. However, when we break it down by device type, variant A performs better on both mobile and desktop. What happened?” A strong candidate refers to Simpson's paradox, explains its cause (group proportions differ between two variables), and asks to see a disaggregation rather than relying on a composite figure.

Interviewers use this test to automatically ask about subgroup distribution. If you just report the total number, you lose points.

// Showing with A/B Testing Data

In the following demonstration you use Pandaswe can see how the aggregate value can be misleading.

import pandas as pd

# A wins on both devices individually, but B wins in aggregate
# because B gets most traffic from higher-converting mobile.
data = pd.DataFrame({
    'device':   ['mobile', 'mobile', 'desktop', 'desktop'],
    'variant':  ['A', 'B', 'A', 'B'],
    'converts': [40, 765, 90, 10],
    'visitors': [100, 900, 900, 100],
})
data['rate'] = data['converts'] / data['visitors']

print('Per device:')
print(data[['device', 'variant', 'rate']].to_string(index=False))
print('nAggregate (misleading):')
agg = data.groupby('variant')[['converts', 'visitors']].sum()
agg['rate'] = agg['converts'] / agg['visitors']
print(agg['rate'])

Output:

# Identifying Selection Bias

This test allows interviewers to check where you think the data comes from before analyzing it.

Selection occurs when the data you have does not represent the population you are trying to understand. Because the bias is in the data collection process rather than in the analysis, it is easy to overlook.

Consider these possible interview structures:

• We analyzed the survey of our users and found that 80% are satisfied with the product. Does that tell us that our product is good? A strong candidate can show that satisfied users are more likely to answer surveys. The 80% figure is probably more than satisfied as unhappy users probably chose not to participate.
• We surveyed customers who left last quarter and found that they primarily had negative engagement scores. Should our attention be focused on engagement to reduce tension? The problem here is that you only have engagement data for stripped down users. You don't have engagement data for existing users, making it impossible to know if low engagement actually predicts churn or if it's just a characteristic of abused users in general.

The relative difference that you should be able to irrigate to survive: you only see the results that made it through a certain filter. If you only use data from successful products to analyze why they succeeded, you ignore those that failed for the same reasons you consider strengths.

// Simulating Survey Nonresponse

We can model how non-response bias distorts the results using NumPy.

import numpy as np
import pandas as pd

np.random.seed(42)
# Simulate users where satisfied users are more likely to respond
satisfaction = np.random.choice([0, 1], size=1000, p=[0.5, 0.5])
# Response probability: 80% for satisfied, 20% for unsatisfied
response_prob = np.where(satisfaction == 1, 0.8, 0.2)
responded = np.random.rand(1000) < response_prob

print(f"True satisfaction rate: {satisfaction.mean():.2%}")
print(f"Survey satisfaction rate: {satisfaction[responded].mean():.2%}")

Output:

Interviewers use choice questions to see if you can separate “what the data shows” from “what is true about users.”

# To prevent ip-Hacking

Hacking (also called data dredging) occurs when you run multiple tests and report only those with ( p < 0.05 ). The problem is that ( p )-values ​​are only calculated for each test. One false positive can only be expected by chance if 20 tests were performed at a 5% significance level. The false discovery rate increases by fishing for a significant result. The interviewer might ask you the following: "Last quarter, we performed fifteen factor tests. At ( p < 0.05 ), three were found to be significant. Do all three need to be submitted?" The weak answer is yes. A strong response will first ask what the assumptions were before the test was conducted, if the significance threshold was set in advance, and whether the team prepared for multiple comparisons. The follow-up usually involves how to design the test to avoid this. Preregistering hypotheses before data collection is the most straightforward fix, as it removes the option of deciding after the fact which tests were "realistic."

// False Positive Views Accumulate

We can see how false positives occur with luck using SciPy.

import numpy as np
from scipy import stats
np.random.seed(0)

# 20 A/B tests where the null hypothesis is TRUE (no real effect)
n_tests, alpha = 20, 0.05
false_positives = 0

for _ in range(n_tests):
    a = np.random.normal(0, 1, 1000)
    b = np.random.normal(0, 1, 1000)  # identical distribution!
    if stats.ttest_ind(a, b).pvalue < alpha:
        false_positives += 1

print(f'Tests run:                 {n_tests}')
print(f'False positives (p<0.05): {false_positives}')
print(f'Expected by chance alone: {n_tests * alpha:.0f}')

Output:

Even with a true zero effect, ~1 out of 20 tests clear ( p < 0.05 ) by chance. If a group runs 15 tests and only reports significant ones, those results are probably noisy. It is equally important to treat exploratory analysis as a form of hypothesis generation rather than validation. Before anyone can take action based on a test result, a confirmatory test is required.

# Managing Multiple Tests

This test is closely related to p-hacking, but it's worth understanding on your own.

The multiple testing problem is a formal statistical problem: if you do many hypothesis tests at once, the probability that at least one of them is false increases rapidly. Even if the treatment has no effect, you should expect about five false positives if you test 100 metrics in an A/B test and declare anything ( p < 0.05 ) as significant. The remedies for this are well known: Bonferroni correction (divide alpha by the test value) and Benjamin-Hochberg (controls the false discovery rate rather than the family-wise error rate).

Bonferroni is a conservative method: for example, if you test 50 metrics, your test limit drops to 0.001, making it difficult to find real results. Benjamin-Hochberg is best suited if you are willing to accept false positives for more statistical power.

In interviews, this comes up when discussing how the company tracks evaluation metrics. A question might be: “We monitor 50 metrics in each test. How do you decide which ones are important?” A robust response discusses pre-defined primary metrics prior to testing and considers secondary metrics as a test while agreeing on the issue of multiple testing.

Interviewers are trying to find out if you know that taking more tests results in more noise than more information.

# Dealing with Confounding Variables

This trap catches candidates who treat correlation as causal without asking what else might explain the relationship.

A confounding variables it is one that influences both the independent and dependent variables, creating the illusion of a direct relationship where none exists.

A classic example: ice cream sales and drowning rates are related, but the confounder is summer heat; both rise in the warmer months. Making such correlations without accounting for confounders leads to poor conclusions.

Confusion is especially dangerous for observational data. Unlike randomized trials, observational data do not distribute potential confounders equally between groups, so the differences you see may not be due to the differences you study at all.

A typical interview outline is: “We noticed that the users who use our mobile app the most tend to have the highest revenue. Should we push notifications to increase app opens?” A weak candidate says yes. Strict asks what type of user opens the app regularly to begin with: probably highly engaged, high-value users.

Engagement drives both app opens and spend. The application opens does not generate income; they are a sign of the same basic user quality.

Interviewers use paradoxes to test whether you separate from reason before drawing conclusions, and whether you can push a randomized trial or match propensity scores before recommending action.

// Imitating a Confused Relationship

import numpy as np
import pandas as pd
np.random.seed(42)
n = 1000
# Confounder: user quality (0 = low, 1 = high)
user_quality = np.random.binomial(1, 0.5, n)
# App opens driven by user quality, not independent
app_opens = user_quality * 5 + np.random.normal(0, 1, n)
# Revenue also driven by user quality, not app opens
revenue = user_quality * 100 + np.random.normal(0, 10, n)
df = pd.DataFrame({
    'user_quality': user_quality,
    'app_opens': app_opens,
    'revenue': revenue
})
# Naive correlation looks strong — misleading
naive_corr = df['app_opens'].corr(df['revenue'])
# Within-group correlation (controlling for confounder) is near zero
corr_low  = df[df['user_quality']==0]['app_opens'].corr(df[df['user_quality']==0]['revenue'])
corr_high = df[df['user_quality']==1]['app_opens'].corr(df[df['user_quality']==1]['revenue'])
print(f"Naive correlation (app opens vs revenue): {naive_corr:.2f}")
print(f"Correlation controlling for user quality:")
print(f"  Low-quality users:  {corr_low:.2f}")
print(f"  High-quality users: {corr_high:.2f}")

Output:

Naive correlation (app opens vs revenue): 0.91

Correlation controlling for user quality:

Low-quality users:  0.03
High-quality users: -0.07

A naive number looks like a strong signal. Once you control for the confounder, it disappears completely. Interviewers who see a candidate run this type of graded assessment (rather than accepting a composite correlation) know they're talking to someone who won't send a broken recommendation.

# Wrapping up

All five of these traps have something in common: they require you to slow down and question the data before accepting what the numbers seem to show at first. Interviewers use these situations precisely because your first instinct is often wrong, and the depth of your response after that first instinct is what separates a candidate who can't work independently from one who needs guidance throughout the analysis.

None of these concepts are difficult to understand, and interviewers ask about them because they are a common failure mode in real data work. A candidate who sees a Simpson's paradox in a product metric, grabs a choice in a survey, or asks if a test result is successful in multiple comparisons is the one who will post fewer bad decisions.

When you enter FAANG conversations and the reflex to ask the following questions, you are already ahead of the majority of students:

• How is this data collected?
• Are there subgroups that tell a different story?
• How many tests contributed to this result?

Besides aiding in negotiations, these practices can also prevent bad decisions from reaching productivity.

Nate Rosidi he is a data scientist and product strategist. He is also an adjunct professor of statistics, and the founder of StrataScratch, a platform that helps data scientists prepare for their interviews with real interview questions from top companies. Nate writes about the latest trends in the job market, provides interview advice, shares data science projects, and covers all things SQL.