Design an A/B test for a recommendation algorithm

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To design an A/B test for a redesigned homepage, we need to establish the hypotheses, define our population, and identify the primary metrics of interest.

**Hypotheses:**  
- Null Hypothesis (H0): The redesigned homepage has no effect on user engagement metrics (e.g., click-through rate, conversion rate).  
- Alternative Hypothesis (H1): The redesigned homepage improves user engagement metrics.

**Population:**  
Users visiting the Adobe website, ideally segmented by demographics to account for potential confounding factors.

**Metrics:**  
- Primary Metric: Click-through rate (CTR) on recommended items.  
- Secondary Metrics: Conversion rate, average session duration, and user retention rate.

To calculate the sample size required for our A/B test, we use the following steps:  
1. **Determine Effect Size:** Based on historical data, we assume a baseline CTR of 5% and we want to detect an increase to at least 6% (1% absolute increase).
2. **Set Alpha and Beta Levels:** Common choices are alpha = 0.05 (5% significance level) and beta = 0.2 (80% power).
3. **Use Sample Size Formula:**  
   
   $ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \times (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2} $  
   
   Where:  
   - $ p_1 = 0.05 $ (baseline CTR)  
   - $ p_2 = 0.06 $ (expected CTR)  
   - $ Z_{\alpha/2} = 1.96 $ (for 95% confidence level)  
   - $ Z_{\beta} = 0.84 $ (for 80% power)

4. **Calculate Sample Size:**  
   
   $$ n = \frac{(1.96 + 0.84)^2 \times (0.05(1 - 0.05) + 0.06(1 - 0.06))}{(0.05 - 0.06)^2} $$  
   
   After calculating, we find $ n \approx 10,000 $ users per group (A and B).

Let’s assume we are running our A/B test for two weeks.  
- Total users visiting the homepage during this period = 100,000.  
- Split evenly: 50,000 users in Group A (control) and 50,000 users in Group B (redesigned homepage).  
- Assume Group A has a CTR of 5% (2,500 clicks) and Group B has a CTR of 6% (3,000 clicks).  
- Calculate the p-value using a two-proportion z-test:  
   
   $$ z = \frac{(p_2 - p_1)}{\sqrt{p(1 - p)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}} $$  
   Where:  
   - $ p = \frac{(2,500 + 3,000)}{(50,000 + 50,000)} = 0.055 $  
   - After calculating, we find a p-value of approximately 0.04, indicating statistical significance.

The p-value of 0.04 suggests that we can reject the null hypothesis at the 5% significance level, indicating that the redesigned homepage significantly improves the CTR compared to the old one. However, we should assess practical significance: although the p-value is low, the actual increase in CTR (1%) might not be impactful enough for business objectives, especially if the cost of redesigning the homepage outweighs the benefits. Additionally, we must consider potential confounders like user demographics and website traffic sources that may affect user behavior.

This A/B test connects to several related concepts:  
- **Bayesian vs. Frequentist Inference:** Discussing whether a Bayesian approach could provide more nuanced insights into user behavior over time rather than a single point estimate of effectiveness.
- **Power Analysis:** Exploring how changing our expected effect size or power requirements might affect sample size.
- **Confounding Variables:** Identifying and controlling for variables such as seasonality, marketing campaigns, or user segmentation to ensure valid results.
- **Longitudinal Studies:** Considering follow-up tests over a longer horizon to validate the sustained impact of the redesigned homepage.

Design an A/B test for a recommendation algorithm

Adobe

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Adobe Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example