Design an A/B test for a new checkout flow

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

The analytical question for this A/B test is: What is the impact of the new pricing tiers on conversion rates during the checkout process at Cloudflare? To answer this question, we need the following data: 
1. **Historical Conversion Rates**: Data on the current conversion rates with existing pricing tiers over the same period in the past. 
2. **User Characteristics**: Demographic and behavioral data of users to assess segmentation. 
3. **Checkout Flow Metrics**: Time spent in the checkout flow, drop-off rates at each stage, and payment method preferences. 
4. **Sample Size Requirements**: We need to define the minimum sample size required to detect a meaningful difference in conversion rates with a specified level of statistical power (e.g., 80%) and significance level (e.g., 0.05).

The specific statistical approach for this problem involves conducting a randomized controlled trial (RCT). We will randomly assign users into two groups: Group A (current pricing tiers) and Group B (new pricing tiers). The primary metric for success will be the conversion rate, defined as the percentage of users who complete their purchase after entering the checkout flow. 
To calculate the sample size, we will use the formula for comparing two proportions:

$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1-p_1) + p_2(1-p_2))}{(p_1 - p_2)^2} $$

Where:
- $ p_1 $ = current conversion rate (e.g., 5%)
- $ p_2 $ = expected conversion rate with new pricing (e.g., 7%)
- $ Z_{\alpha/2} $ and $ Z_{\beta} $ are the Z-scores for the desired significance level and power, respectively.

Assuming the current conversion rate $ p_1 $ is 0.05 (5%) and we expect the new pricing tier to increase it to $ p_2 $ of 0.07 (7%). Using Z-scores for a significance level of 0.05 (1.96) and power of 0.80 (0.84):

$$ n = \frac{(1.96 + 0.84)^2 \cdot (0.05(1-0.05) + 0.07(1-0.07))}{(0.05 - 0.07)^2} $$

Calculating each component:
- $ (1.96 + 0.84)^2 = 7.84 $
- $ 0.05(0.95) + 0.07(0.93) \approx 0.0475 + 0.0651 = 0.1126 $
- $ (0.05 - 0.07)^2 = 0.0004 $

Plugging these values into the formula:

$$ n = \frac{7.84 \cdot 0.1126}{0.0004} \approx 221,840 $$

This means we need approximately 221,840 users in each group for the test.

After running the A/B test, we will analyze the conversion rates for both groups. If the conversion rate for Group B (new pricing) is statistically significantly higher than Group A (current pricing) based on a chi-squared test or a two-proportion Z-test, we can conclude that the new pricing tiers have a positive impact on conversion rates. However, we must also consider the long-term implications of changing the pricing structure, as there could be novelty effects where initial results may not sustain over time. Additionally, we will segment the data by user characteristics to identify any heterogeneous treatment effects, such as whether different customer segments respond differently to the new pricing.

Common pitfalls in this type of analysis include:
1. **Failure to Randomize**: If users are not properly randomized into control and treatment groups, it could lead to biased results. 
2. **Ignoring Segmentation**: Not analyzing results by user segments can mask important insights about how different groups respond to the new pricing. 
3. **Short-term Focus**: Evaluating the impact of the A/B test solely on short-term metrics like immediate conversion rates can overlook long-term customer value and retention. 
4. **Neglecting Guardrail Metrics**: It’s crucial to monitor guardrail metrics (e.g., average order value, churn rates) to ensure that the new pricing does not negatively impact overall business health.

Design an A/B test for a new checkout flow

Cloudflare

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Cloudflare Product Manager Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention