Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

We want to design an A/B test to evaluate the impact of new pricing tiers on user engagement. Our aim is to determine whether the new pricing structure leads to a statistically significant increase in engagement metrics, such as conversion rates or average revenue per user (ARPU). We will formulate the following hypotheses:

- Null Hypothesis (H0): There is no difference in engagement metrics between the control group (current pricing tiers) and the treatment group (new pricing tiers).
- Alternative Hypothesis (H1): There is a significant difference in engagement metrics between the control and treatment groups.

To design the A/B test, we need to:
1. Define our primary metric for measurement (e.g., conversion rate).
2. Determine the expected effect size (the smallest difference in conversion rates we want to detect).
3. Calculate the required sample size using the formula for comparing two proportions:
   
   $$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2} $$
   
   where:
   - $Z_{\alpha/2}$ is the z-score for the significance level (e.g., 1.96 for 95% confidence).
   - $Z_{\beta}$ is the z-score for the desired power (e.g., 0.84 for 80% power).
   - $p_1$ and $p_2$ are the estimated conversion rates for the control and treatment groups.
4. Analyze the results using hypothesis testing to determine if we can reject the null hypothesis.

Let’s assume:
- Current conversion rate (p1) = 0.10 (10%)
- Expected conversion rate with new pricing (p2) = 0.12 (12%)
- We aim for a significance level of 0.05 and power of 0.80.

Next, we calculate:
- $Z_{\alpha/2} \approx 1.96$  (for 95% confidence)
- $Z_{\beta} \approx 0.84$ (for 80% power)

Now, we plug in the values:
$$ n = \frac{(1.96 + 0.84)^2 \cdot (0.10(1 - 0.10) + 0.12(1 - 0.12))}{(0.10 - 0.12)^2} $$
Calculating:
- Variance terms: $0.10(0.90) + 0.12(0.88) = 0.090 + 0.1056 = 0.1956$
- Thus,
$$ n = \frac{(2.8)^2 \cdot 0.1956}{(-0.02)^2} = \frac{7.84 \cdot 0.1956}{0.0004} = 3,068.64 $$
We round up to 3,069 participants per group.

This means we need approximately 3,069 users in both the control and treatment groups to reliably detect a 2% increase in conversion rates with 95% confidence and 80% power. If the test yields a p-value less than 0.05, we can conclude that the new pricing tiers significantly impact user engagement. Conversely, if we fail to reject the null hypothesis, we may conclude that the new pricing structure does not significantly change user behavior.

This A/B testing design connects to various concepts in statistics and causal inference. For instance, if the initial A/B test is successful, we might consider a multivariate test to analyze the impact of price alongside other factors, such as discounts or promotional offers. Additionally, understanding Type I and Type II errors is essential, as we want to minimize false positives and negatives. The results can also feed into Bayesian inference frameworks, where prior beliefs about pricing effects can be updated with observed data.

Design an A/B test for a search ranking change

Citadel

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Citadel Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example