Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To evaluate the impact of a redesigned homepage on user engagement, we will conduct an A/B test. We'll define two groups:  
- **Group A (Control)**: Users see the current homepage.  
- **Group B (Treatment)**: Users see the redesigned homepage.

**Metrics**: The primary metric for evaluation will be the conversion rate, defined as the proportion of users who complete a desired action (e.g., making a purchase or signing up for a newsletter).

**Hypotheses**:  
- **Null Hypothesis (H0)**: The redesigned homepage has no effect on conversion rates compared to the current homepage.  
- **Alternative Hypothesis (H1)**: The redesigned homepage has a positive effect on conversion rates compared to the current homepage.

1. **Sample Size Calculation**: To ensure statistical significance, we need to determine the required sample size. We can use the formula for sample size in A/B testing:  
   
   $$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1-p_1) + p_2(1-p_2))}{(p_1 - p_2)^2} $$  
   
   Where:  
   - $ Z_{\alpha/2} $ is the critical value for the significance level (e.g., 1.96 for a 95% confidence level).  
   - $ Z_{\beta} $ is the critical value for the power (e.g., 0.84 for 80% power).  
   - $ p_1 $ is the conversion rate for the control group.  
   - $ p_2 $ is the expected conversion rate for the treatment group.  
   
2. **Analysis Plan**: We will conduct a two-proportion z-test to compare the conversion rates between the two groups and calculate p-values to determine if we can reject H0.

Assume:  
- Current conversion rate (p1) = 5% (0.05)  
- Expected conversion rate for redesigned homepage (p2) = 6% (0.06)  
- Significance level (α) = 0.05 (95% confidence)  
- Power = 0.80

1. **Calculate critical values**:  
   - $ Z_{0.025} \approx 1.96 $  
   - $ Z_{0.20} \approx 0.84 $  
2. **Calculate sample size (n)**:  
   $$ n = \frac{(1.96 + 0.84)^2 \cdot ((0.05)(0.95) + (0.06)(0.94))}{(0.05 - 0.06)^2} $$  
   $$ n \approx \frac{(2.8)^2 \cdot (0.0475 + 0.0564)}{(-0.01)^2} \approx \frac{7.84 \cdot 0.1039}{0.0001} \approx 81456 $$  
   Therefore, we need approximately 81,456 users in each group.

The calculated sample size of approximately 81,456 users in each group allows us to detect a statistically significant difference in conversion rates if the redesigned homepage truly leads to an increase. If after running the test, we find a p-value less than 0.05, we would reject H0, suggesting that the redesigned homepage positively impacts conversion rates. If the p-value is greater, we fail to reject H0, indicating no significant impact.

This A/B testing approach connects to related concepts such as:  
- **Bayesian vs Frequentist Inference**: In this example, we used a frequentist approach. A Bayesian approach could provide a probability distribution of the conversion rates, allowing us to incorporate prior beliefs about the treatment effect.  
- **Central Limit Theorem**: Given a large enough sample size, the sample proportion of conversions will be approximately normally distributed, justifying the use of the z-test.  
- **Power Analysis**: Post-experiment, we could perform a power analysis to evaluate if our sample size was adequate to detect a meaningful effect.

Design an A/B test for a search ranking change

Adobe

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Adobe Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example