Design an A/B test for a pricing model
Last updated: August 29, 2025
Quick Overview
Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.
Grubhub
August 29, 20256
13
243 solved
Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.
Grubhub values data-driven decision making. This Onsite question assesses whether you can design experiments, interpret results correctly, and avoid common statistical pitfalls like p-hacking or Simpson's paradox.
What the Interviewer Expects
- Derive results from first principles when needed
- Handle complex scenarios with multiple interacting variables
- Design experiments that account for real-world complications
- Discuss advanced topics: Bayesian methods, causal inference, resampling
- Connect statistical concepts to business decision-making
- Identify subtle errors in reasoning (Simpson's paradox, survivorship bias)
Key Topics to Cover
How to Approach This
- Define your hypotheses (H0 and H1) clearly before performing any test.
- Calculate required sample size BEFORE running an experiment, using power analysis.
- Remember the Central Limit Theorem: sample means become approximately normal with large n.
- Watch for Simpson's paradox. Always segment data by key dimensions.
- Distinguish between statistical significance and practical significance.
Possible Follow-up Questions
- What assumptions does this test make, and how would you validate them?
- How would you handle multiple comparisons?
- What alternative statistical method could you use here?
- How would you explain this result to a non-technical audience?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Browse Statistics QuestionsSample Answer
Problem Formulation
To design an A/B test for Grubhub's redesigned homepage, we need to formulate our experiment in terms of hypotheses, metrics, and sample size.
Hypotheses:
- Null Hypothesis (H0): The redesigne...
Solution Approach
Let’s assume we expect a baseline CTR of 5% (0.05) and hope to detect an increase to 7% (0.07) with the new design.
- Calculate z-scores:
- For a 95% confidence level: ( Z_{\alpha/2} = 1....
Submit Your Answer
