Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To test the impact of new pricing tiers at Grafana Labs, we need to design an A/B test. We will define:
1. **Control Group (A)**: Current pricing tiers.
2. **Treatment Group (B)**: New pricing tiers.

**Metrics**: The primary metric will be the conversion rate (CR), defined as the percentage of users who purchase a subscription after visiting the pricing page. We will also track secondary metrics like average revenue per user (ARPU) and user engagement metrics.

**Hypotheses**:
- Null Hypothesis (H0): There is no difference in the conversion rates between the control and treatment groups.
- Alternative Hypothesis (H1): The conversion rate for the treatment group is higher than that of the control group.

To determine the sample size needed for our A/B test, we will utilize the formula for sample size calculation for comparing two proportions:

$$ n = \left( \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1-p_1) + p_2(1-p_2))}{(p_1 - p_2)^2} \right) $$

Where:
- $Z_{\alpha/2}$: Z-value for the desired confidence level (e.g., 1.96 for 95% confidence).
- $Z_{\beta}$: Z-value for the desired power (e.g., 0.84 for 80% power).
- $p_1$: Conversion rate for control group.
- $p_2$: Expected conversion rate for treatment group.

Assuming $p_1 = 0.10$ (10% conversion rate) and we expect an increase to $p_2 = 0.15$ (15% conversion rate), we calculate:
- $Z_{\alpha/2} = 1.96$
- $Z_{\beta} = 0.84$

We can now compute the sample size.

Plugging the numbers into the formula:
1. Calculate $p_1(1-p_1) = 0.10(1-0.10) = 0.09$
2. Calculate $p_2(1-p_2) = 0.15(1-0.15) = 0.1275$
3. Compute the total variance: $(0.09 + 0.1275) = 0.2175$
4. The difference in proportions: $(0.15 - 0.10) = 0.05$
5. Now substitute values into the sample size formula:
$$ n = \left( \frac{(1.96 + 0.84)^2 \cdot 0.2175}{(0.05)^2} \right) \approx 1286.74 $$
6. Therefore, we need approximately 1287 users in each group, totaling 2574 users.

By conducting this A/B test with approximately 2574 users, we ensure that we have adequate power to detect a meaningful difference in conversion rates between the control and treatment groups. If we observe a significant increase in the conversion rate in the treatment group, we can conclude that the new pricing tiers have a positive impact on user behavior. Conversely, if no significant difference is found, we may need to reconsider the pricing structure or conduct further testing.

This A/B testing framework connects to several key statistical concepts:
1. **Central Limit Theorem**: As we increase sample size, the sampling distribution of the sample proportion will approach normality, allowing us to accurately apply Z-tests.
2. **Confidence Intervals**: Once results are obtained, we can calculate confidence intervals for the observed conversion rates to assess the precision of our estimates.
3. **Bayesian vs Frequentist Inference**: Depending on the company’s philosophy, we might consider a Bayesian approach to incorporate prior beliefs about conversion rates.
4. **Power Analysis**: The importance of determining the sample size beforehand reflects an understanding of statistical power, ensuring we can detect true effects in our experiment.

Design an A/B test for a search ranking change

Grafana Labs

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Grafana Labs Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example