Design an A/B test for a new checkout flow

Last updated: August 10, 2025

Quick Overview

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Visa

Statistics & Math

Data Scientist

Visa

August 10, 2025

Data Scientist

Onsite

Statistics & Math

Medium

586 solved

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Visa values data-driven decision making. This Onsite question assesses whether you can design experiments, interpret results correctly, and avoid common statistical pitfalls like p-hacking or Simpson's paradox.

What the Interviewer Expects

Set up the problem formally with proper notation
Apply the correct statistical test with clear justification
Interpret results with appropriate caveats and confidence levels
Discuss practical significance vs statistical significance
Identify potential confounders and how to address them

Key Topics to Cover

Conditional probability and Bayes theorem

Power analysis and sample size calculation

Probability distributions

Multiple testing correction (Bonferroni, FDR)

Central Limit Theorem

How to Approach This

Define your hypotheses (H0 and H1) clearly before performing any test.
Calculate required sample size BEFORE running an experiment, using power analysis.
Remember the Central Limit Theorem: sample means become approximately normal with large n.
Watch for Simpson's paradox. Always segment data by key dimensions.
Distinguish between statistical significance and practical significance.

Possible Follow-up Questions

What if the sample size is very small?
How would you design a follow-up experiment based on these results?
What alternative statistical method could you use here?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Statistics Questions

Sample Answer

Problem Formulation

To test the impact of new pricing tiers on the checkout flow, we will define the following parameters:

Hypotheses:
- Null Hypothesis (H0): The new pricing tiers do not significantly affect th...

Solution Approach

Using the formula for sample size for comparing two proportions:

$n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p_1(1 - p_1) + p_2(1 - p_2))}{(p_1 - p_2)^2}$

Where:

$p_1 = 0.10$ (current con...

Submit Your Answer

Markdown supported

Visa Data Scientist Interview Guide

Interview process, tips, and preparation timeline