Design an A/B test for a new checkout flow

Last updated: July 1, 2025

Quick Overview

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Stripe

Statistics & Math

Data Scientist

Stripe

July 1, 2025

Data Scientist

Take-home Project

Statistics & Math

Medium

223

3,616 solved

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

This statistics question from Stripe's Take-home Project tests your ability to apply mathematical reasoning to practical problems. The interviewer expects precise definitions, correct methodology, and awareness of assumptions and limitations.

What the Interviewer Expects

Set up the problem formally with proper notation
Apply the correct statistical test with clear justification
Interpret results with appropriate caveats and confidence levels
Discuss practical significance vs statistical significance
Identify potential confounders and how to address them

Key Topics to Cover

Bayesian vs frequentist inference

Regression analysis

Causal inference basics

Power analysis and sample size calculation

Confidence intervals and significance levels

How to Approach This

Define your hypotheses (H0 and H1) clearly before performing any test.
Calculate required sample size BEFORE running an experiment, using power analysis.
Remember the Central Limit Theorem: sample means become approximately normal with large n.
Watch for Simpson's paradox. Always segment data by key dimensions.
Distinguish between statistical significance and practical significance.

Possible Follow-up Questions

How would you handle multiple comparisons?
What assumptions does this test make, and how would you validate them?
How would you design a follow-up experiment based on these results?
What if the sample size is very small?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Statistics Questions

Sample Answer

Problem Formulation

To evaluate the impact of new pricing tiers on user conversion rates during the checkout flow, we will set up an A/B test. Let $p_0$ represent the conversion rate under the current pricing tier (c...

Solution Approach

Sample Size Calculation: We will use the formula for sample size for comparing two proportions:

$n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 (p_0(1 - p_0) + p_1(1 - p_1))}{(p_1 - p_0)^2}$ ...

Submit Your Answer

Markdown supported

Stripe Data Scientist Interview Guide

Interview process, tips, and preparation timeline