Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

The analytical question we want to answer is: **What is the impact of the redesigned homepage on user engagement and conversion rates?** To answer this, we need to collect data on user behaviors before and after the redesign. The data requirements include:
- **User metrics**: Click-through rate (CTR), time spent on the homepage, conversion rates (defined as a user completing a purchase or signing up), and bounce rates.
- **Demographic information**: User segments such as age, location, and previous engagement levels to identify heterogeneous effects.
- **Randomization details**: Ensure users are randomly assigned to either the control group (old homepage) or the treatment group (new homepage).

To determine the sample size for our A/B test, we will use the following formula for calculating sample size for comparing two proportions:

$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2} $$

Where:
- $ p_1 $ = baseline conversion rate (e.g., 0.10 for the old homepage)
- $ p_2 $ = expected conversion rate after redesign (e.g., 0.12)
- $ Z_{\alpha/2} $ = Z-score for significance level (1.96 for 95%)
- $ Z_{\beta} $ = Z-score for power (0.84 for 80%)

Assuming:
- $ p_1 = 0.10 $
- $ p_2 = 0.12 $

We calculate:
- $ Z_{\alpha/2} = 1.96 $
- $ Z_{\beta} = 0.84 $
- Sample size needed per group = 9,427 users. Therefore, we should aim for at least 19,000 total users.

Suppose we run the A/B test for two weeks and gather the following data:
- **Control Group (Old Homepage)**: 10,000 users, 1,000 conversions (CTR = 10%)
- **Treatment Group (New Homepage)**: 10,000 users, 1,200 conversions (CTR = 12%)

We’ll conduct a hypothesis test using a two-proportion Z-test:
1. **Null Hypothesis (H0)**: $ p_1 = p_2 $
2. **Alternative Hypothesis (H1)**: $ p_1 < p_2 $
3. Calculate the pooled proportion:
   $$ p_{pooled} = \frac{(1000 + 1200)}{(10000 + 10000)} = 0.11 $$
4. Calculate the standard error:
   $$ SE = \sqrt{p_{pooled}(1 - p_{pooled}) \left(\frac{1}{n_1} + \frac{1}{n_2}\right) } = \sqrt{0.11(0.89)\left(\frac{1}{10000} + \frac{1}{10000}\right)} = 0.0043 $$
5. Calculate the Z-score:
   $$ Z = \frac{(0.12 - 0.10)}{SE} = \frac{0.02}{0.0043} \approx 4.65 $$
6. Looking up Z = 4.65 gives a p-value < 0.0001, leading us to reject H0.

The result suggests that the redesigned homepage significantly increased the conversion rate from 10% to 12%, implying a 20% improvement in conversion rates. However, we must also consider **long-term effects**: will users maintain this engagement, or is there a novelty effect? Additionally, we should segment the data by demographics to check for heterogeneous treatment effects. For instance, did younger users respond better than older users? This could inform future design iterations.

Common mistakes in this analysis include:
- **Ignoring sample size**: Running the test with insufficient users may lead to Type I or Type II errors.
- **Not accounting for external factors**: Variations in traffic due to seasonal events or marketing campaigns can skew results.
- **Confounding variables**: Failing to control for user characteristics that might affect conversion rates.
- **Overlooking long-term vs. short-term metrics**: Focusing only on immediate results might miss potential long-term impacts. We must monitor engagement metrics over a longer period post-launch.

Design an A/B test for a search ranking change

xAI

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

xAI Data Scientist Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention