Calculate expected value for A/B test results

Question

Given the following scenario about user conversion, calculate the the p-value.

Codemia · Accepted Answer

In this scenario, we are tasked with calculating the p-value for an A/B test concerning user conversion rates. We start by defining our hypotheses:

- Null Hypothesis (H0): There is no difference in conversion rates between the two groups (A and B).
- Alternative Hypothesis (H1): There is a difference in conversion rates between the two groups (A and B).

Assuming we have conversion data from both groups, we denote:
- $ p_A $: conversion rate for group A (control group)
- $ p_B $: conversion rate for group B (treatment group)

We will use a two-proportion z-test to calculate the p-value, which is appropriate for comparing two proportions.

To perform the two-proportion z-test, we follow these steps:

1. **Calculate the sample proportions**:
   - $ \hat{p}_A = rac{x_A}{n_A} $ where $ x_A $ is the number of conversions in group A and $ n_A $ is the total number of users in group A.
   - $ \hat{p}_B = rac{x_B}{n_B} $ where $ x_B $ is the number of conversions in group B and $ n_B $ is the total number of users in group B.

2. **Calculate the pooled proportion**:
   - $ \hat{p} = rac{x_A + x_B}{n_A + n_B} $

3. **Calculate the standard error (SE)**:
   - $ SE = \sqrt{\hat{p}(1 - \hat{p})\left(rac{1}{n_A} + rac{1}{n_B}ight)} $

4. **Calculate the z-statistic**:
   - $ z = rac{\hat{p}_A - \hat{p}_B}{SE} $

5. **Determine the p-value**:
   - Use the z-statistic to find the p-value from the standard normal distribution.

Let's assume the following data from our A/B test:
- Group A (control): 80 conversions from 1000 users (i.e., $ x_A = 80, n_A = 1000 $)
- Group B (treatment): 100 conversions from 1000 users (i.e., $ x_B = 100, n_B = 1000 $)

Now, we calculate:
1. Sample proportions:
   - $ \hat{p}_A = rac{80}{1000} = 0.08 $
   - $ \hat{p}_B = rac{100}{1000} = 0.10 $

2. Pooled proportion:
   - $ \hat{p} = rac{80 + 100}{1000 + 1000} = rac{180}{2000} = 0.09 $

3. Standard error:
   - $ SE = \sqrt{0.09(1 - 0.09)\left(rac{1}{1000} + rac{1}{1000}ight)} = \sqrt{0.09 \cdot 0.91 \cdot 0.002} \approx 0.013 $

4. Z-statistic:
   - $ z = rac{0.08 - 0.10}{0.013} \approx -1.54 $

5. P-value:
   - Using a standard normal distribution table, a z-value of -1.54 corresponds to a p-value of approximately 0.061.
   - Since this is a two-tailed test, we multiply by 2: p-value ≈ 0.122.

The calculated p-value of 0.122 suggests that there is not enough evidence to reject the null hypothesis at a common significance level of 0.05. In practical terms, this means that the difference in conversion rates between the control and treatment groups is not statistically significant, and we do not have strong evidence to conclude that the new treatment (group B) provides a better conversion rate compared to the control (group A). The company may choose to keep the current strategy or consider further testing.

This problem connects to several related concepts:
- **Type I and Type II Errors**: Understanding the risk of incorrectly rejecting the null hypothesis (Type I error) versus failing to reject a false null hypothesis (Type II error).
- **Power of the Test**: Discussing the probability of correctly rejecting the null hypothesis when it is false, which is related to sample size and effect size.
- **Bayesian vs Frequentist Approaches**: Exploring how Bayesian methods could provide a different perspective on the conversion rates by incorporating prior beliefs about conversion rates into the analysis, contrasting with the frequentist interpretation used here.

Calculate expected value for A/B test results

Doordash

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Doordash Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example