Calculate expected value for A/B test results
Last updated: August 27, 2025
Quick Overview
Given the following scenario about user conversion, calculate the the p-value.
CrowdStrike
August 27, 2025121
6
478 solved
Given the following scenario about user conversion, calculate the the p-value.
Analytics questions at CrowdStrike evaluate your ability to define metrics, design experiments, and derive actionable insights from data. This Take-home Project question tests your end-to-end analytical thinking.
What the Interviewer Expects
- Design a rigorous experiment with proper randomization and sample size calculation
- Define primary and guardrail metrics with clear rationale
- Address novelty effects, network effects, and interference
- Segment results appropriately and identify heterogeneous treatment effects
- Propose follow-up analyses when results are ambiguous
Key Topics to Cover
How to Approach This
- Define success metrics carefully. A good metric is measurable, actionable, and aligned with business goals.
- Run experiments long enough to account for novelty effects and weekly seasonality.
- Use funnel analysis to identify where users drop off for maximum optimization impact.
- Segment results by key dimensions (platform, country, user cohort) to catch hidden patterns.
- Consider network effects and interference between treatment and control groups.
Possible Follow-up Questions
- How would you handle an experiment where the control and treatment groups are different sizes?
- How would you handle interference between treatment and control?
- What if the experiment shows a positive short-term effect but you suspect a negative long-term impact?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Browse Analytics QuestionsSample Answer
Problem Setup
For this A/B test at CrowdStrike, we want to determine the impact of a new feature on user conversion rates. Our analytical question is: "Does the new feature significantly increase the user conversio...
Methodology
To calculate the p-value for the A/B test, we will use a two-proportion z-test. This statistical test compares the conversion rates of the two groups. The null hypothesis (H0) states that there is no ...
Submit Your Answer
