Design an A/B test for a pricing model

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To design an A/B test for Figma's new pricing tiers, we need to establish a clear framework. Let’s assume we are testing three pricing models: Tier A (current pricing), Tier B (new pricing tier 1), and Tier C (new pricing tier 2). Our primary goal is to assess the impact of these pricing tiers on conversion rates (the proportion of visitors who make a purchase). We will define our hypotheses as follows:

- **Null Hypothesis (H0)**: The conversion rates for all pricing tiers are equal (pA = pB = pC).
- **Alternative Hypothesis (H1)**: At least one pricing tier has a different conversion rate (pA ≠ pB ≠ pC).

We will conduct a multi-arm A/B test to compare these three pricing models.

1. **Sample Size Calculation**: To ensure adequate power for our test, we first calculate the required sample size. We want to detect a minimal effect size of 5% increase in conversion rate with a power of 0.8 and a significance level of 0.05. We will use the formula for sample size in a multi-group test:
   
   $$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1-p_1) + p_2(1-p_2) + p_3(1-p_3))}{(p_1 - p_2)^2} $$
   
   where $Z_{\alpha/2} $ is the critical value for a two-tailed test and $Z_{\beta} $ is the critical value for power.

2. **Metrics**: The primary metric will be the conversion rate, while secondary metrics could include average revenue per user (ARPU) and customer lifetime value (CLV).

3. **Analysis Plan**: We will use ANOVA for multiple comparisons to determine if there are significant differences in conversion rates among the tiers. If significant differences are found, we will apply the Bonferroni correction to control for Type I error across multiple comparisons.

Let's assume the baseline conversion rate (p1) for Tier A is 10%. We want to detect an increase to 15% in either Tier B or C. Using a Z-table:
- $Z_{\alpha/2} \approx 1.96$ for a 0.05 significance level (two-tailed).
- $Z_{\beta} \approx 0.84$ for 80% power.

Plugging in the numbers:
- $p1 = 0.10$, $p2 = 0.15$, hence:
- $$ n = \frac{(1.96 + 0.84)^2 \cdot (0.10(1-0.10) + 0.15(1-0.15) + 0.10(1-0.10))}{(0.15 - 0.10)^2} $$
- $$ n = \frac{(2.8)^2 \cdot (0.09 + 0.1275 + 0.09)}{(0.05)^2} $$
- $$ n \approx \frac{7.84 \cdot 0.3075}{0.0025} \approx 964 $$

Therefore, we would need approximately 964 participants per group, totaling around 2,892 participants for the three groups.

Once the data is collected and analyzed, we will report the conversion rates for each tier along with confidence intervals. If we find that Tier B has a significantly higher conversion rate than Tier A and C, this suggests that the new pricing model may be a viable option for increasing sales. However, if the p-values after applying the Bonferroni correction are still above our significance level, we conclude that there is no statistically significant difference, and we may need to reevaluate our pricing strategies or consider additional tiers.

This A/B test design connects to various advanced statistical concepts. For instance, we could explore **Bayesian methods** to update our beliefs about conversion rates as data accumulates. Additionally, we might consider **causal inference** to account for confounding factors that could affect conversion rates, such as seasonality or website traffic sources. Lastly, discussing **Simpson's Paradox** is essential; we must ensure that the increase in conversion rates is not misleading due to different demographics responding differently to the pricing tiers.

Design an A/B test for a pricing model

Figma

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Figma Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example