Calculate probability for A/B test results

Question

Given the following scenario about user conversion, calculate the the p-value.

Codemia · Accepted Answer

The analytical question we need to answer is: "What is the probability that the observed differences in user conversion rates between the control group and the treatment group are statistically significant?" To do this, we will need the following data: 1. The number of users in the control group (n_control) and treatment group (n_treatment). 2. The number of conversions in the control group (conversions_control) and treatment group (conversions_treatment). 3. The significance level (commonly set at 0.05) to determine the threshold for the p-value.

We will use a two-proportion z-test to calculate the p-value. This test is appropriate as we are comparing the conversion rates of two independent groups. The formula for the z-statistic is:

$$ z = \frac{(p_1 - p_2)}{\sqrt{p(1 - p)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}} $$

Where:  
- p_1 is the conversion rate of the treatment group  
- p_2 is the conversion rate of the control group  
- p is the pooled conversion rate calculated as $$ p = \frac{conversions_{control} + conversions_{treatment}}{n_{control} + n_{treatment}} $$

Suppose we have the following data:  
- Control group: n_control = 1000, conversions_control = 100 (conversion rate p_2 = 0.1)  
- Treatment group: n_treatment = 1000, conversions_treatment = 150 (conversion rate p_1 = 0.15)

Step 1: Calculate the pooled conversion rate:  
$$ p = \frac{100 + 150}{1000 + 1000} = \frac{250}{2000} = 0.125 $$

Step 2: Calculate the z-statistic:  
$$ z = \frac{(0.15 - 0.1)}{\sqrt{0.125(1 - 0.125)\left(\frac{1}{1000} + \frac{1}{1000}\right)}}} = \frac{0.05}{\sqrt{0.125 \cdot 0.875 \cdot 0.002}} $$  
$$ = \frac{0.05}{\sqrt{0.000109375}} \approx \frac{0.05}{0.01043} \approx 4.79 $$

Step 3: Find the p-value using the z-table for z = 4.79, which gives a p-value < 0.0001. This indicates a statistically significant difference in conversion rates.

The p-value < 0.0001 suggests that there is a very low probability that the observed difference in conversion rates (from 10% to 15%) occurred by chance. Therefore, we can reject the null hypothesis that there is no difference in conversion rates between the control and treatment groups, and conclude that the treatment likely has a positive effect on user conversion.

Common mistakes in this type of analysis include: 1. Failing to ensure proper randomization in assigning users to the control and treatment groups, which can lead to biased results. 2. Not accounting for potential confounding variables that may affect user conversion rates. 3. Misinterpreting the p-value; a small p-value does not imply a large effect size or practical significance. 4. Ignoring the implications of network effects and interference, such that changes in one group may affect the other, leading to incorrect assumptions about the treatment's impact.

Calculate probability for A/B test results

Supabase

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Setup

Methodology

Submit Your Answer

Supabase Product Manager Interview Guide

Related Questions

Explain Bayesian vs frequentist with an example

Explain p-value with an example

Interpret no significant difference from an experiment

Design an A/B test for a recommendation algorithm

Calculate conditional probability for user retention