Design an A/B test for a pricing model

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

The analytical question is: "What is the impact of new pricing tiers on booking rates and revenue for Airbnb listings?" To answer this, we need data on current booking rates, average revenue per booking, customer demographics, and historical pricing data. We will also need to define the new pricing tiers and ensure we have enough listings across different regions to conduct a meaningful A/B test.

We will use a randomized controlled trial (RCT) design for this A/B test. The experimental group will be exposed to the new pricing tiers, while the control group will maintain the current pricing model. After defining our sample, we will perform a sample size calculation using the following parameters: expected lift in booking rates (e.g., 10%), current booking rate (e.g., 30%), and significance level (alpha = 0.05) with power (1 - beta = 0.8). Using a two-proportion z-test formula, we can determine the necessary sample size for both groups.

Assuming we expect a 10% increase in booking rate from 30% to 33%, we apply the sample size formula for proportions:

n = (Z_alpha/2 + Z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2

Where: 
- p1 = 0.30 (control booking rate) 
- p2 = 0.33 (expected booking rate for the new pricing tier) 
- Z_alpha/2 (for 0.05 significance level) ≈ 1.96 
- Z_beta (for 0.8 power) ≈ 0.84

Calculating gives us: 
- n ≈ (1.96 + 0.84)^2 * (0.30 * 0.70 + 0.33 * 0.67) / (0.03)^2 ≈ 1,000 listings per group. Therefore, we need 2,000 total listings to achieve sufficient power.

After conducting the A/B test, we would analyze the booking rates and revenue generated from both groups. If the experimental group shows a statistically significant increase in bookings and revenue compared to the control, we could conclude that the new pricing tiers have a positive impact. We would also examine guardrail metrics, such as customer satisfaction ratings and cancellation rates, to ensure that the new pricing model does not negatively impact user experience.

Common mistakes in this type of analysis include failing to account for network effects, where the pricing model impacts not just the listings being tested but also others nearby (e.g., clustering of listings could skew results). Another pitfall is not having sufficient segmentation; different markets may respond differently to pricing changes, so it’s crucial to analyze results across various demographics and geographic locations. Additionally, neglecting to monitor guardrail metrics could lead to overlooking adverse effects on customer behavior.

Design an A/B test for a pricing model

Airbnb

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Airbnb Data Scientist Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention