Design an A/B test for a pricing model
Last updated: December 29, 2025
Quick Overview
Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.
Roblox
December 29, 202547
7
1,181 solved
Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.
Roblox values data-driven decision making. This Phone Screen question assesses whether you can design experiments, interpret results correctly, and avoid common statistical pitfalls like p-hacking or Simpson's paradox.
What the Interviewer Expects
- Derive results from first principles when needed
- Handle complex scenarios with multiple interacting variables
- Design experiments that account for real-world complications
- Discuss advanced topics: Bayesian methods, causal inference, resampling
- Connect statistical concepts to business decision-making
- Identify subtle errors in reasoning (Simpson's paradox, survivorship bias)
Key Topics to Cover
How to Approach This
- Define your hypotheses (H0 and H1) clearly before performing any test.
- Calculate required sample size BEFORE running an experiment, using power analysis.
- Remember the Central Limit Theorem: sample means become approximately normal with large n.
- Watch for Simpson's paradox. Always segment data by key dimensions.
- Distinguish between statistical significance and practical significance.
Possible Follow-up Questions
- What alternative statistical method could you use here?
- What assumptions does this test make, and how would you validate them?
- What if the sample size is very small?
- How would you explain this result to a non-technical audience?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Browse Statistics QuestionsSample Answer
Problem Formulation
To design an A/B test for a new pricing model for Roblox, we need to define the pricing tiers we want to test. Let's say we are considering three tiers: Basic (10/month), and Pre...
Solution Approach
To determine the sample size needed for this A/B test, we can use the following formula for comparing means between groups:
[ n = \left( \frac{Z_{\alpha/2} + Z_{\beta}}{\mu_1 - \mu_2} / \sigma \righ...
Submit Your Answer
