Calculate conditional probability for user retention

Question

Given the following scenario about user conversion, calculate the the p-value.

Codemia · Accepted Answer

Suppose Netflix runs an experiment on a new onboarding/conversion flow. Users are randomly assigned to:

- **Control**: existing signup and onboarding experience
- **Treatment**: new personalized onboarding experience

The business question is whether the treatment increases **retention conditional on conversion**.

Define events:

- $T$: user is assigned to treatment
- $C$: user converts to paid subscription
- $R$: user is retained after 30 days

The estimand is:

$$
P(R \mid C, T) - P(R \mid C, \text{control})
$$

The hypotheses are:

$$
H_0: P(R \mid C, T) = P(R \mid C, \text{control})
$$

$$
H_1: P(R \mid C, T) > P(R \mid C, \text{control})
$$

This is a conditional probability problem because we are not comparing all assigned users. We are comparing retention **among users who converted**. That matters because conditioning on conversion can introduce selection bias if the treatment changes who converts.

Let:

- $n_0$: number of converted users in control
- $x_0$: number of retained converted users in control
- $n_1$: number of converted users in treatment
- $x_1$: number of retained converted users in treatment

The observed conditional retention rates are:

$$
\hat p_0 = \frac{x_0}{n_0}, \qquad \hat p_1 = \frac{x_1}{n_1}
$$

Under the null hypothesis that the conditional retention rates are equal, we use the pooled estimate:

$$
\hat p = \frac{x_0 + x_1}{n_0 + n_1}
$$

By the Central Limit Theorem, for large enough sample sizes:

$$
\hat p_1 - \hat p_0 \approx N\left(0, \hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_0}\right)\right)
$$

So the test statistic is:

$$
z = \frac{\hat p_1 - \hat p_0}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_0}\right)}}
$$

For a one-sided test, the p-value is:

$$
p\text{-value} = P(Z \ge z)
$$

where $Z \sim N(0,1)$.

Assume Netflix observes the following experiment results:

| Group | Converted users | Retained after 30 days |
|---|---:|---:|
| Control | 10,000 | 6,200 |
| Treatment | 9,500 | 6,080 |

The conditional retention rates are:

$$
\hat p_0 = \frac{6200}{10000} = 0.620
$$

$$
\hat p_1 = \frac{6080}{9500} = 0.640
$$

The observed lift is:

$$
0.640 - 0.620 = 0.020
$$

So treatment retention among converted users is 2 percentage points higher.

Now compute the pooled retention rate:

$$
\hat p = \frac{6200 + 6080}{10000 + 9500} = \frac{12280}{19500} \approx 0.6297
$$

Standard error:

$$
SE = \sqrt{0.6297(1-0.6297)\left(\frac{1}{9500}+\frac{1}{10000}\right)}
$$

$$
SE \approx \sqrt{0.2332 \cdot 0.0002053} \approx 0.00692
$$

Test statistic:

$$
z = \frac{0.640 - 0.620}{0.00692} \approx 2.89
$$

For a one-sided test:

$$
p\text{-value} = P(Z \ge 2.89) \approx 0.0019
$$

So the p-value is approximately **0.0019**.

A p-value of **0.0019** means that if the new Netflix onboarding flow truly had no effect on 30-day retention among converted users, the probability of observing a retention lift at least this large purely by random chance is about **0.19%**.

At a 5% significance level, we reject $H_0$. Statistically, the treatment appears to improve retention among users who converted.

However, I would be careful before recommending launch. The metric is conditional on conversion:

$$
P(R \mid C)
$$

If the treatment changes the mix of users who convert, the observed retention lift may not be purely caused by better onboarding. For example, treatment may convert fewer casual users and mostly retain highly engaged users, making conditional retention look better while total business impact is worse.

Netflix should also check the unconditional impact:

$$
P(C \cap R) = P(R \mid C)P(C)
$$

For example, if treatment improves retention among converted users but lowers conversion rate, the net number of retained subscribers could decline. The decision should depend on retained subscribers, revenue, engagement, churn, and long-term customer lifetime value, not only conditional retention.

A strong follow-up is multiple testing. If Netflix tested this onboarding change across 20 countries, devices, or user segments, a raw p-value of 0.0019 may need correction.

Using Bonferroni with 20 tests:

$$
\alpha_{adj} = \frac{0.05}{20} = 0.0025
$$

Since $0.0019 < 0.0025$, this result would still be significant. If the p-value had been 0.03, it would be significant before correction but not after Bonferroni.

Another likely follow-up is Simpson's paradox. Treatment may improve retention overall but hurt retention in every major device category if the treatment group has more users on high-retention devices like smart TVs. I would stratify by platform, country, acquisition channel, and prior engagement.

A Bayesian version would place priors on the two retention probabilities and estimate:

$$
P(p_1 > p_0 \mid \text{data})
$$

That may be easier for product decision-making because stakeholders can ask directly: what is the probability the treatment improves retention by at least 1 percentage point?

For causal validity, I would emphasize that randomization must occur before conversion exposure, and analysis should include both conditional and intent-to-treat metrics. Otherwise, conditioning on conversion can create post-treatment selection bias.

Calculate conditional probability for user retention

Netflix

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

1. Problem formulation for Netflix retention

2. Step-by-step p-value calculation

Submit Your Answer

Netflix Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example