Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

Start by formalizing the problem:

1. **Define the hypotheses**: H0 (null hypothesis) and H1 (alternative). Be precise about what you're testing.

2. **Choose the test**: t-test for comparing means, chi-squared for categorical data, ANOVA for multiple groups, Mann-Whitney for non-normal distributions.

3. **Set the significance level**: Typically alpha = 0.05. Justify if you need a different threshold (e.g., multiple comparisons).

**Compute the test statistic**: Apply the appropriate formula using the sample data.

**Find the p-value**: The probability of observing a result this extreme (or more extreme) under the null hypothesis.

**Make a decision**: If p-value < alpha, reject H0. But always consider: statistical significance does not imply practical significance. A 0.1% improvement may be statistically significant with a large enough sample but meaningless in practice.

**Multiple testing**: If you run 20 tests at alpha=0.05, you expect 1 false positive. Apply Bonferroni or FDR correction.

**Simpson's Paradox**: A trend that appears in aggregated data can reverse when segmented. Always check subgroups.

**Survivorship Bias**: You only see users who stayed. Those who churned might tell a different story.

**Confounding Variables**: Correlation does not imply causation. Use randomization (A/B tests) or causal inference methods to establish causal relationships.

Design an A/B test for a search ranking change

Grafana Labs

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Setting Up the Problem

Solution Approach

Submit Your Answer

Grafana Labs Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example