Calculate conditional probability for A/B test results

Question

Given the following scenario about revenue per session, calculate the the p-value.

Codemia · Accepted Answer

The analytical question here is to determine whether the revenue per session differs significantly between two groups in an A/B test conducted by Postmates. We need to calculate the p-value to assess this difference. We will require data that includes:

1. **Revenue per session** for both the control group (Group A) and the treatment group (Group B).
2. **Sample sizes** for each group to ensure adequate statistical power.
3. **Session counts** or interactions per user to account for varying user engagement levels.
4. **Confounding variables** such as time of day, day of the week, and user demographics that might influence revenue.

To calculate the p-value for this A/B test, we will use a two-sample t-test. This test is appropriate here because we are comparing means from two independent samples (Group A vs. Group B). The null hypothesis (H0) is that there is no difference in revenue per session between the two groups, while the alternative hypothesis (H1) posits that there is a difference. The steps are:

1. Calculate the mean revenue per session for both groups.
2. Calculate the standard deviation and standard error of the means.
3. Use the t-statistic formula:
   
   $$ t = \frac{\bar{X}_A - \bar{X}_B}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}} $$
   
4. Determine the degrees of freedom and use it to find the p-value from the t-distribution.

Assume the following hypothetical data:
- Group A (Control): Mean revenue per session = $15, Sample size (nA) = 100, Standard deviation (sA) = $5.
- Group B (Treatment): Mean revenue per session = $18, Sample size (nB) = 100, Standard deviation (sB) = $6.

1. Calculate the t-statistic:
   
   $$ t = \frac{15 - 18}{\sqrt{\frac{5^2}{100} + \frac{6^2}{100}}} = \frac{-3}{\sqrt{\frac{25}{100} + \frac{36}{100}}} = \frac{-3}{\sqrt{0.25 + 0.36}} = \frac{-3}{\sqrt{0.61}} \approx -3.84 $$

2. Degrees of freedom (df) can be approximated using:
   
   $$ df = n_A + n_B - 2 = 100 + 100 - 2 = 198 $$

3. Using a t-table or calculator, we find the p-value associated with t = -3.84 and df = 198. The p-value is approximately 0.0001.

The p-value of 0.0001 indicates a very low probability that the observed difference in revenue per session occurred by random chance under the null hypothesis. Since this p-value is far below a common significance level of 0.05, we reject the null hypothesis. This suggests that the treatment (Group B) significantly increased revenue per session compared to the control group (Group A). However, we must also consider the context of the results, such as whether the increase in revenue justifies any changes made and if the results are consistent over a longer term.

Common mistakes in this type of analysis include:
1. **Ignoring confounding variables:** Not controlling for external factors like time of day or user demographics can skew results.
2. **Misinterpreting the p-value:** A low p-value does not imply a strong effect or practical significance. It's essential to evaluate the actual revenue increase in the business context.
3. **Sample size issues:** Small sample sizes can lead to unreliable results and larger variability in estimates. Ensure that both groups are adequately powered to detect meaningful differences.
4. **Assuming causation:** Even if a significant difference is found, one must not assume that the treatment caused the increase without further investigation into other influencing factors.

Calculate conditional probability for A/B test results

Postmates

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Postmates Data Scientist Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention