Design an A/B test for a new checkout flow

Last updated: January 19, 2026

Quick Overview

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Oracle

Statistics & Math

Data Scientist

Oracle

January 19, 2026

Data Scientist

Onsite

Statistics & Math

Easy

3,958 solved

Design an experiment to test the impact of a redesigned homepage. Include sample size calculation, metrics, and analysis plan.

Oracle values data-driven decision making. This Onsite question assesses whether you can design experiments, interpret results correctly, and avoid common statistical pitfalls like p-hacking or Simpson's paradox.

What the Interviewer Expects

State the correct formula or theorem with clear definitions
Apply the concept to the given scenario step by step
Interpret the result in plain language
Identify assumptions and when they might be violated

Key Topics to Cover

Conditional probability and Bayes theorem

Regression analysis

Bayesian vs frequentist inference

Central Limit Theorem

Confidence intervals and significance levels

How to Approach This

Define your hypotheses (H0 and H1) clearly before performing any test.
Calculate required sample size BEFORE running an experiment, using power analysis.
Remember the Central Limit Theorem: sample means become approximately normal with large n.
Watch for Simpson's paradox. Always segment data by key dimensions.
Distinguish between statistical significance and practical significance.

Possible Follow-up Questions

What alternative statistical method could you use here?
What if the sample size is very small?
How would you design a follow-up experiment based on these results?
What assumptions does this test make, and how would you validate them?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Statistics Questions

Sample Answer

Problem Formulation

To assess the impact of a redesigned homepage on the checkout flow, we will implement an A/B test where:

Group A (Control): Users experience the current homepage.
Group B (Treatment): Use...

Solution Approach

Sample Size Calculation:
- We need to determine the sample size required to detect a meaningful difference in conversion rates between the two groups.
- Assume the current CR is 2% (0.02)...

Submit Your Answer

Markdown supported

Oracle Data Scientist Interview Guide

Interview process, tips, and preparation timeline