Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To evaluate the impact of a redesigned homepage on user engagement and conversion rates, we need to frame our analytical question: **How does the new homepage design affect the average number of transactions per user and user retention on Coinbase?** The key data we need includes:

1. User engagement metrics: Number of transactions, time spent on site, and click-through rates on key features. 
2. User retention data: Percentage of returning users over a defined period (e.g., 1 week, 1 month). 
3. User demographics: Age, location, and trading experience to analyze heterogeneous effects. 
4. Control and treatment group identifiers for A/B testing.

We will conduct a classic **A/B test** with a control group (current homepage) and a treatment group (redesigned homepage). To ensure statistical validity, we will calculate the sample size required to detect a meaningful difference in our primary metrics (transactions and retention). Using a two-tailed t-test, we can use the following formula for sample size estimation:

$$ n = \left( \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (\sigma_1^2 + \sigma_2^2)}{(\mu_1 - \mu_2)^2} \right) \cdot 2 $$

Where:  
- $Z_{\alpha/2}$ is the z-score for the desired confidence level (1.96 for 95%).  
- $Z_{\beta}$ is the z-score for power (0.84 for 80% power).  
- $\sigma_1^2$ and $\sigma_2^2$ are the variances of the control and treatment groups.  
- $\mu_1$ and $\mu_2$ are the expected means of transactions in both groups.

Assuming we expect the control group (current homepage) to have an average of 5 transactions per user with a standard deviation of 2, and we want to detect an increase to 6 transactions in the treatment group:

1. Set $\mu_1 = 5$, $\mu_2 = 6$  
2. Set $\sigma_1 = 2$ and $\sigma_2 = 2$  
3. Using the values:  
    - $Z_{\alpha/2} = 1.96$  
    - $Z_{\beta} = 0.84$

Plugging these into the sample size formula:

$$ n = \left( \frac{(1.96 + 0.84)^2 \cdot (2^2 + 2^2)}{(5 - 6)^2} \right) \cdot 2 $$  
$$ n = \left( \frac{(2.8)^2 \cdot 8}{1} \right) \cdot 2 \approx 157.76 $$  
Rounding up, we need **158 users in each group**.

Once the A/B test is conducted, we will analyze the results using statistical tests (t-tests for means). If the treatment group shows a statistically significant increase in the average number of transactions, we can infer that the new design positively impacts user engagement. Additionally, we will monitor guardrail metrics like bounce rates and session durations to ensure that the redesign does not negatively affect the user experience. If the treatment group shows increased retention, the redesign can be deemed successful, indicating that users are more likely to return after the change.

Common pitfalls in this type of analysis include:  
1. **Not accounting for novelty effects**: Users might initially engage more due to the novelty of the redesign, which could skew results. We should track metrics over time to assess long-term effects.  
2. **Ignoring segment differences**: Different user segments might respond variably to the redesign. Segment analysis can reveal insights that a general overview might miss.  
3. **Sample size miscalculations**: Underestimating the required sample size can lead to inconclusive results. Always verify assumptions about user behavior and variance in your calculations.

Design an A/B test for a search ranking change

Coinbase

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Coinbase Product Manager Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention