Design an A/B test for a search ranking change

Last updated: August 14, 2025

Quick Overview

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

What the Interviewer Expects

Set up the problem formally with proper notation
Apply the correct statistical test with clear justification
Interpret results with appropriate caveats and confidence levels
Discuss practical significance vs statistical significance
Identify potential confounders and how to address them

Key Topics to Cover

Causal inference basics

Multiple testing correction (Bonferroni, FDR)

Confidence intervals and significance levels

Bayesian vs frequentist inference

Regression analysis

Power analysis and sample size calculation

How to Approach This

Define your hypotheses (H0 and H1) clearly before performing any test.
Calculate required sample size BEFORE running an experiment, using power analysis.
Remember the Central Limit Theorem: sample means become approximately normal with large n.
Watch for Simpson's paradox. Always segment data by key dimensions.
Distinguish between statistical significance and practical significance.

Possible Follow-up Questions

How would you handle multiple comparisons?
What if the sample size is very small?
What assumptions does this test make, and how would you validate them?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Statistics Questions

Sample Answer

Setting Up the Problem

Start by formalizing the problem: 1. **Define the hypotheses**: H0 (null hypothesis) and H1 (alternative). Be precise about what you're testing. 2. ...

Solution Approach

**Compute the test statistic**: Apply the appropriate formula using the sample data. **Find the p-value**: The probability of observing a result this...

Submit Your Answer

Markdown supported

Meta Data Scientist Interview Guide

Interview process, tips, and preparation timeline