Calculate conditional probability for coin flips

Question

Given the following scenario about user conversion, calculate the the sample size needed.

Codemia · Accepted Answer

To determine the sample size needed for an A/B test on user conversion based on coin flips, we need to define our analytical question: **What is the probability of conversion for users exposed to a new feature (treatment group) compared to those not exposed (control group)?**  
We will need the following data:  
1. **Historical conversion rates** for similar features or campaigns (e.g., 5%).  
2. **Standard deviation of the conversion rate** to understand variability.  
3. **Desired statistical power** (commonly 80% or 90%) and **alpha level** (commonly 0.05).  
4. **Minimum detectable effect size**, which is the smallest change in conversion rate we want to detect (e.g., an increase from 5% to 7%).

In this scenario, we will use **Cohen's sample size formula for proportions** to determine the required sample size for each group. The formula is:  
$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 * (p_1(1-p_1) + p_2(1-p_2))}{(p_1 - p_2)^2} $$  
Where:  
- $ p_1 $ = control group conversion rate (0.05)  
- $ p_2 $ = treatment group conversion rate (0.07)  
- $ Z_{\alpha/2} $ = Z-score for the desired confidence level (1.96 for 95% confidence)  
- $ Z_{\beta} $ = Z-score for desired power (0.84 for 80% power).  
Plugging in these values will allow us to calculate the necessary sample size.

Let's compute the sample size using the provided values:  
- $ p_1 = 0.05 $  
- $ p_2 = 0.07 $  
- $ Z_{\alpha/2} = 1.96 $  
- $ Z_{\beta} = 0.84 $  
First, calculate the variance for the two groups:  
- $ Var_1 = p_1(1-p_1) = 0.05(1-0.05) = 0.0475 $  
- $ Var_2 = p_2(1-p_2) = 0.07(1-0.07) = 0.0651 $  
Next, plug these values into the formula:  
$$ n = \frac{(1.96 + 0.84)^2 * (0.0475 + 0.0651)}{(0.05 - 0.07)^2} $$  
Calculating this gives us:  
$$ n = \frac{(2.8)^2 * (0.1126)}{(-0.02)^2} \approx \frac{7.84 * 0.1126}{0.0004} \approx 22192 $$  
This means we need approximately 22,192 users per group.

The calculated sample size of approximately 22,192 users per group indicates how many users we need to enroll in both the treatment and control groups to detect a statistically significant difference in conversion rates with 80% power at a 5% significance level. In practice, this means we should aim for a total of around 44,384 users to ensure that our test is adequately powered to detect the anticipated increase in conversion from 5% to 7%.

Common mistakes in this type of analysis include:  
1. **Ignoring confounding variables** such as user demographics or prior purchase behavior that could impact conversion rates.  
2. **Not accounting for selection bias**, which can occur if the users in the treatment group differ significantly from those in the control group.  
3. **Overlooking the importance of randomization** in assigning users to groups, which can lead to skewed results if not properly implemented.  
4. **Failing to adjust for multiple testing** if conducting simultaneous analyses, which can inflate the type I error rate.

Calculate conditional probability for coin flips

Shopify

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Shopify Product Manager Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention