Calculate variance for coin flips

Question

Given the following scenario about user conversion, calculate the the sample size needed.

Codemia · Accepted Answer

To determine the sample size needed for an A/B test at Dropbox for user conversion rates, we first need to define our primary analytical question: **What is the minimum sample size required to detect a significant difference in conversion rates between two groups (A and B) with a specified level of statistical power and significance?**

**Data Needed:** We would require:
- Historical conversion rate data for Dropbox users (e.g., previous conversion rates, standard deviation of conversion rates).
- Desired effect size (the minimum difference in conversion rates we want to detect).
- The significance level (typically set at 0.05 for a 95% confidence level).
- Desired power of the test (commonly set at 0.8 or 80%).

For this specific problem, we will utilize the formula for sample size calculation in A/B testing for proportions:

$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2} $$

Where:
- $ n $ = sample size per group (A and B)
- $ Z_{\alpha/2} $ = Z-score for the significance level (1.96 for 0.05)
- $ Z_{\beta} $ = Z-score for power (0.84 for 80%)
- $ p_1 $ = conversion rate for group A
- $ p_2 $ = conversion rate for group B
- The effect size is given as $ p_1 - p_2 $.

Assuming:
- Historical conversion rate $ p_1 = 0.10 $ (10% for control group A)
- Desired effect size is an increase to $ p_2 = 0.15 $ (15% for treatment group B)

We first calculate the pooled proportions:
- $ p_{pooled} = \frac{p_1 + p_2}{2} = \frac{0.10 + 0.15}{2} = 0.125 $

Then calculate the variances:
- $ p_1(1 - p_1) = 0.10(1 - 0.10) = 0.09 $
- $ p_2(1 - p_2) = 0.15(1 - 0.15) = 0.1275 $

Plugging into the sample size formula:
- $ n = \frac{(1.96 + 0.84)^2 \cdot (0.09 + 0.1275)}{(0.10 - 0.15)^2} $
- $ n = \frac{(2.8)^2 \cdot (0.2175)}{(-0.05)^2} $
- $ n = \frac{7.84 \cdot 0.2175}{0.0025} = \frac{1.707}{0.0025} = 682.8 $

Thus, we round up to **683** participants needed per group for the test.

The calculated sample size of 683 per group means that to reliably detect a conversion rate increase from 10% to 15% with 80% power at a 5% significance level, we need at least 1,366 users participating in the experiment (683 in control group A and 683 in treatment group B). This ensures that our results, if statistically significant, can be confidently interpreted as indicative of the true effect of the treatment on user conversion rates.

Common pitfalls in this type of analysis include:
- **Underestimating Variability:** Failing to accurately estimate historical conversion rates and their standard deviations can lead to inappropriate sample size calculations.
- **Ignoring Dropout Rates:** Not accounting for potential user dropouts or non-participation in the test can result in insufficient sample sizes.
- **Overlooking Segmentation Effects:** Different segments of users might respond differently to the treatment. Not conducting a power analysis for each segment could lead to misleading conclusions.
- **Misinterpreting the Effect Size:** Assuming a larger effect size than is realistic can lead to underpowered studies, and consequently invalid results.

Calculate variance for coin flips

Dropbox

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Dropbox Data Scientist Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention