Design an A/B test for a pricing model

Question

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To design an A/B test for evaluating the impact of a redesigned homepage on user engagement, we first define our hypotheses:

- **Null Hypothesis (H0)**: The redesigned homepage does not lead to an increase in user engagement compared to the original homepage.
- **Alternative Hypothesis (H1)**: The redesigned homepage leads to a statistically significant increase in user engagement compared to the original homepage.

Next, we identify our primary metric for user engagement, which could be the average time spent on the homepage or the conversion rate (percentage of users who complete a desired action, such as making a purchase). We decide to focus on the conversion rate as our main metric for this experiment.

We will also need to determine the target sample size based on expected effect size, significance level, and power.

To calculate the required sample size, we use the formula for sample size in proportion testing:

$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1 (1 - p_1) + p_2 (1 - p_2))}{(p_2 - p_1)^2} $$

Where:
- $ p_1 $ = baseline conversion rate (let's assume it is 10% or 0.10)
- $ p_2 $ = expected conversion rate after redesign (let's assume we expect an increase to 12% or 0.12)
- $ Z_{\alpha/2} $ = Z-score for the desired significance level (for a 95% confidence level, $ Z_{0.025} = 1.96 $)
- $ Z_{\beta} $ = Z-score for the desired power (for 80% power, $ Z_{0.2} = 0.84 $)

We plug in these values to calculate $ n $.

Using the values defined:
- $ p_1 = 0.10 $
- $ p_2 = 0.12 $
- $ Z_{\alpha/2} = 1.96 $
- $ Z_{\beta} = 0.84 $

We calculate:
$$ n = \frac{(1.96 + 0.84)^2 \cdot (0.10(1 - 0.10) + 0.12(1 - 0.12))}{(0.12 - 0.10)^2} $$
$$ n = \frac{(2.8)^2 \cdot (0.10 \cdot 0.90 + 0.12 \cdot 0.88)}{(0.02)^2} $$
$$ n = \frac{7.84 \cdot (0.09 + 0.1056)}{0.0004} $$
$$ n = \frac{7.84 \cdot 0.1956}{0.0004} \approx 38308 $$

Thus, we need approximately 38,308 users in each group (A and B), totaling around 76,616 users for the test.

The result indicates that to detect a statistically significant increase in conversion rates from 10% to 12% with 95% confidence and 80% power, we need a large sample size of about 76,616 users. This is practical for a company like Databricks, which likely has a sufficient user base. If the experimental results show a significant difference in conversion rates post-test, we can conclude that the redesigned homepage effectively increases user engagement. However, we must also consider practical significance; even if the result is statistically significant, we should assess whether the increase in conversion rate justifies the redesign effort.

This A/B testing framework connects to several related concepts:
- **Causal Inference**: Establishing the causal impact of the redesign requires careful experimental design to avoid confounding variables, such as seasonality or marketing campaigns that might influence user behavior.
- **Confidence Intervals**: Beyond just p-values, we should calculate confidence intervals for our conversion rates to understand the range of likely outcomes and the robustness of our findings.
- **Bayesian Inference**: An alternative approach could involve Bayesian methods to update our beliefs about the redesign's effectiveness as we gather data. This could provide more nuanced insights over time and allow us to incorporate prior knowledge.
- **Randomization and Control**: Ensuring random assignment of users to the control and experimental groups helps mitigate bias and ensures that results are attributable solely to the homepage changes.

Design an A/B test for a pricing model

Databricks

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Databricks Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example