Design an A/B test for a pricing model

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To evaluate the impact of new pricing tiers on customer conversion and revenue, we need to frame our analytical question as follows: **How do the new pricing tiers affect the conversion rate and average revenue per user (ARPU) compared to the existing pricing model?**

**Data Required:**  
1. Historical conversion rates and revenue data from the current pricing model.  
2. Customer demographics and behavior data (e.g., previous purchase history, engagement metrics).  
3. A/B test data including which users were assigned to which pricing tier, their conversion status, and revenue generated during the test period.

We will employ a randomized controlled A/B test with two groups:  
- **Control Group:** Users exposed to the existing pricing tiers.  
- **Treatment Group:** Users exposed to the new pricing tiers.

**Sample Size Calculation:**  
Assuming we anticipate a 10% increase in conversion rate with new pricing tiers (from 20% to 22%), we can calculate the required sample size using the following formula for two proportions:

$$ n = \left( rac{(Z_{\alpha/2} + Z_{eta})^2 \cdot (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2} ight) $$  
Where:  
- $ p_1 = 0.20 $ (control group conversion rate)  
- $ p_2 = 0.22 $ (treatment group conversion rate)  
- $ Z_{\alpha/2} = 1.96 $ (for a 95% confidence level)  
- $ Z_{eta} = 0.84 $ (for a power of 80%)

Plugging in the values:  
$$ n = \left( rac{(1.96 + 0.84)^2 \cdot (0.20 \cdot 0.80 + 0.22 \cdot 0.78)}{(0.02)^2} ight) \approx 4,180 $$  
Thus, we need approximately 4,180 users in each group.

Let's assume we run the A/B test with 4,180 users in each group. After the test period, the results are:  
- **Control Group:** 20% conversion rate with 836 conversions (20% of 4,180) and ARPU of $50.  
- **Treatment Group:** 22% conversion rate with 920 conversions (22% of 4,180) and ARPU of $55.

**Calculating Revenue:**  
- Revenue from Control Group = 836 conversions * $50 = $41,800  
- Revenue from Treatment Group = 920 conversions * $55 = $50,600

**Conversion Rate Difference:**  
- Difference in conversion rates = 22% - 20% = 2%  
- Difference in ARPU = $55 - $50 = $5

The results indicate that the new pricing tiers led to a 2% increase in conversion rates and a $5 increase in ARPU. This suggests that the new pricing model is effective in enhancing customer engagement and profitability.  
However, we must consider external factors, such as seasonality or concurrent marketing campaigns, that might have influenced these metrics. Moreover, we should analyze if the increase in revenue is sustainable over time and across different customer segments.

Several common mistakes can arise in this analysis:  
1. **Ignoring Confounding Variables:** Failing to account for external factors such as promotions or market trends during the test period can skew results.  
2. **Insufficient Sample Size:** Underestimating the sample size required may lead to inconclusive results due to low statistical power.  
3. **Short Test Duration:** Running the A/B test for too brief a period may not capture long-term trends or behavioral changes.  
4. **Misinterpretation of Results:** Assuming causation from correlation without proper statistical validation can lead to incorrect conclusions about the pricing tiers' effectiveness.

Design an A/B test for a pricing model

Scale AI

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Scale AI Product Manager Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention