Calculate conditional probability for A/B test results
Last updated: May 5, 2026
Quick Overview
Given the following scenario about user conversion, calculate the the p-value.
Salesforce
May 5, 20265
14
574 solved
Given the following scenario about user conversion, calculate the the p-value.
Salesforce asks this during the Take-home Project to assess your experimentation skills. They want to see how you define success metrics, design controlled experiments, and interpret results with appropriate statistical rigor.
What the Interviewer Expects
- Design a rigorous experiment with proper randomization and sample size calculation
- Define primary and guardrail metrics with clear rationale
- Address novelty effects, network effects, and interference
- Segment results appropriately and identify heterogeneous treatment effects
- Propose follow-up analyses when results are ambiguous
Key Topics to Cover
How to Approach This
- Define success metrics carefully. A good metric is measurable, actionable, and aligned with business goals.
- Run experiments long enough to account for novelty effects and weekly seasonality.
- Use funnel analysis to identify where users drop off for maximum optimization impact.
- Segment results by key dimensions (platform, country, user cohort) to catch hidden patterns.
- Consider network effects and interference between treatment and control groups.
Possible Follow-up Questions
- What would you do if a stakeholder wants to end the experiment early because initial results look good?
- How would you handle seasonality in your experiment?
- What if you discover a bug in the logging during the experiment?
- What if the experiment shows a positive short-term effect but you suspect a negative long-term impact?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Browse Analytics QuestionsSample Answer
Problem Setup
To analyze the A/B test results for Salesforce, our analytical question is: "What is the p-value for the difference in conversion rates between the control group (Group A) and the treatment group (Gro...
Methodology
For this A/B test, we will use a two-proportion z-test to compare the conversion rates of the two groups. The null hypothesis (H0) states that there is no difference in conversion rates between Group ...
Submit Your Answer
