Calculate probability for coin flips

Last updated: January 9, 2026

Quick Overview

Given the following scenario about revenue per session, calculate the the sample size needed.

Analytics & Experimentation

Product Manager

January 9, 2026

Product Manager

Take-home Project

Analytics & Experimentation

Medium

3,827 solved

Given the following scenario about revenue per session, calculate the the sample size needed.

Reddit asks this during the Take-home Project to assess your experimentation skills. They want to see how you define success metrics, design controlled experiments, and interpret results with appropriate statistical rigor.

What the Interviewer Expects

Design a rigorous experiment with proper randomization and sample size calculation
Define primary and guardrail metrics with clear rationale
Address novelty effects, network effects, and interference
Segment results appropriately and identify heterogeneous treatment effects
Propose follow-up analyses when results are ambiguous

Key Topics to Cover

Funnel analysis and cohort analysis

Novelty and primacy effects

Segmentation and heterogeneous effects

Guardrail metrics

How to Approach This

Define success metrics carefully. A good metric is measurable, actionable, and aligned with business goals.
Run experiments long enough to account for novelty effects and weekly seasonality.
Use funnel analysis to identify where users drop off for maximum optimization impact.
Segment results by key dimensions (platform, country, user cohort) to catch hidden patterns.
Consider network effects and interference between treatment and control groups.

Possible Follow-up Questions

How would you handle an experiment where the control and treatment groups are different sizes?
What would you do if a stakeholder wants to end the experiment early because initial results look good?
How would you handle interference between treatment and control?
What if the experiment shows a positive short-term effect but you suspect a negative long-term impact?

Sharpen Your Skills on Codemia

Practice similar problems with our interactive workspace, get AI feedback, and track your progress.

Browse Analytics Questions

Sample Answer

Problem Setup

To determine the probability of success in terms of revenue per session on Reddit, we need to frame the analytical question: "What is the required sample size to detect a significant difference in rev...

Methodology

We will use the formula for sample size calculation for comparing two means:

$n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (\sigma_1^2 + \sigma_2^2)}{(\mu_1 - \mu_2)^2}$

Where:

( Z_{\alpha/2...

Submit Your Answer

Markdown supported

Reddit Product Manager Interview Guide

Interview process, tips, and preparation timeline