Design an A/B test for a pricing model

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

The analytical question here is: "What is the impact of the redesigned homepage on user engagement and conversion rates for Capital One?" To address this, we need to gather data on key metrics such as: 1. **User Engagement**: measured by click-through rates (CTR) on key features of the homepage, 2. **Conversion Rates**: percentage of users completing a desired action (e.g., signing up for a credit card or logging into their account), and 3. **Bounce Rate**: percentage of users who leave the homepage without further interaction. Additionally, we will segment the user data by demographics (age, location) and previous banking behavior (existing customers vs new customers).

We will conduct a randomized controlled A/B test where users are randomly assigned to either the control group (existing homepage) or the treatment group (redesigned homepage). A suitable sample size can be calculated using the formula for comparing two proportions. Assuming we want to detect a 5% increase in conversion rate from an initial rate of 20%, with a power of 0.8 and a significance level of 0.05, we first estimate the required sample size using the formula:

$$ n = \frac{{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1 - p_1) + p_2(1 - p_2))}}{{(p_2 - p_1)^2}} $$

where:
- $ p_1 = 0.20 $ (control conversion rate)
- $ p_2 = 0.25 $ (treatment conversion rate)
- $ Z_{\alpha/2} = 1.96 $ (for 95% confidence)
- $ Z_{\beta} = 0.84 $ (for 80% power)

Calculating gives us a required sample size of approximately 1,024 users in each group.

Assuming that we run the A/B test with a sample of 1,024 users in each group, we observe the following results after two weeks:
- **Control Group**: 200 conversions (conversion rate = 19.5%)
- **Treatment Group**: 260 conversions (conversion rate = 25.4%)

Next, we can perform a hypothesis test to determine if this difference is statistically significant. We will use a two-proportion Z-test:

$$ Z = \frac{{(p_1 - p_2)}}{{\sqrt{{p(1 - p) \cdot (\frac{1}{n_1} + \frac{1}{n_2})}}}} $$

where $ p = \frac{200 + 260}{1024 + 1024} = 0.227 $. Plugging in these values, we can calculate Z and compare it against the critical Z value of 1.96 for significance at the 0.05 level.

If our calculated Z-value exceeds 1.96, we can reject the null hypothesis and conclude that the redesigned homepage significantly increases conversion rates compared to the existing homepage. For instance, if our Z-value is 2.5, we would interpret this as strong evidence that the redesigned homepage leads to higher user engagement and conversion. Moreover, we should analyze the segmented data to see if the effect varies across different demographics, which could inform future design choices.

Common mistakes in this type of analysis include: 1. **Insufficient Sample Size**: Underestimating sample size can lead to Type II errors (false negatives), failing to detect a true effect. 2. **Ignoring External Factors**: Variables like seasonality or promotional events can skew results; thus, controlling for these factors in the analysis is crucial. 3. **Overlooking Segmentation**: Failing to segment the results by user demographics may mask important insights. For example, if younger users respond positively to the redesign but older users do not, this could inform tailored marketing strategies.

Design an A/B test for a pricing model

Capital One

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Capital One Data Scientist Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention