Design an A/B test for a search ranking change

Last updated: May 14, 2026

Quick Overview

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Booking.com

Statistics & Math

Data Scientist

Booking.com

May 14, 2026

Data Scientist

Technical Screen

Statistics & Math

Medium

3,015 solved

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Booking.com values data-driven decision making. This Technical Screen question assesses whether you can design experiments, interpret results correctly, and avoid common statistical pitfalls like p-hacking or Simpson's paradox.

What the Interviewer Expects

Set up the problem formally with proper notation
Apply the correct statistical test with clear justification
Interpret results with appropriate caveats and confidence levels
Discuss practical significance vs statistical significance
Identify potential confounders and how to address them

Key Topics to Cover

Central Limit Theorem

Multiple testing correction (Bonferroni, FDR)

Causal inference basics

Confidence intervals and significance levels

Conditional probability and Bayes theorem

Bayesian vs frequentist inference

How to Approach This

Define your hypotheses (H0 and H1) clearly before performing any test.
Calculate required sample size BEFORE running an experiment, using power analysis.
Remember the Central Limit Theorem: sample means become approximately normal with large n.
Watch for Simpson's paradox. Always segment data by key dimensions.
Distinguish between statistical significance and practical significance.

Possible Follow-up Questions

What if the sample size is very small?
How would you handle multiple comparisons?
How would you explain this result to a non-technical audience?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Statistics Questions

Sample Answer

Problem Formulation

To evaluate the impact of new pricing tiers on user engagement, we will conduct an A/B test. Let $A$ represent the control group (current pricing tiers) and $B$ the treatment group (new pricin...

Solution Approach

Sample Size Calculation: We will use the formula for sample size estimation for two proportions: $n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 (p_1(1 - p_1) + p_2(1 - p_2))}{(p_2 - p_1)^2}$ ...

Submit Your Answer

Markdown supported

Booking.com Data Scientist Interview Guide

Interview process, tips, and preparation timeline