Calculate expected value for click-through rates

Question

Given the following scenario about user conversion, calculate the the sample size needed.

Codemia · Accepted Answer

To calculate the expected value for click-through rates (CTR) and determine the sample size needed for a user conversion test at CrowdStrike, we first define the key parameters:

1. **Click-Through Rate (CTR)**: This is the proportion of users who click on a link or advertisement. Denote the true CTR as $ p $.
2. **Sample Size (n)**: This is the number of users we will test to achieve a statistically significant result.
3. **Significance Level ($ \alpha $)**: This is the probability of incorrectly rejecting the null hypothesis (Type I error). Common values are 0.05 or 0.01.
4. **Power (1 - $ \beta $)**: This is the probability of correctly rejecting the null hypothesis when it is false, typically set at 0.8 or 0.9.
5. **Effect Size (d)**: This is the minimum difference in CTR we want to detect (e.g., an increase from 5% to 6%).

With these definitions, we can use the formula for sample size calculation for proportions:  
$$ n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot (p(1 - p))}{d^2} $$

To derive the sample size, we follow these steps:
1. **Determine Z-scores**: Use the Z-table to find the Z-scores corresponding to the desired significance level and power. For $ \alpha = 0.05 $ (two-tailed), $ Z_{\alpha/2} \approx 1.96 $ and for $ \beta = 0.2 $ (80% power), $ Z_{\beta} \approx 0.84 $.
2. **Estimate the CTR (p)**: Assume a baseline CTR of 0.05 (5%).
3. **Define the effect size (d)**: We want to detect a change of 0.01 (1%), so $ d = 0.01 $.
4. **Plug values into the formula**: 
   $$ n = \frac{(1.96 + 0.84)^2 \cdot (0.05(1 - 0.05))}{0.01^2} $$  
   This simplifies to:  
   $$ n = \frac{(2.8)^2 \cdot (0.05 \cdot 0.95)}{0.0001} $$  
   $$ n = \frac{7.84 \cdot 0.0475}{0.0001} $$  
   $$ n \approx 372,000 $$

Using the values we derived:
- $ p = 0.05 $
- $ d = 0.01 $

Calculating sample size:
$$ n = \frac{(1.96 + 0.84)^2 \cdot (0.05 \cdot 0.95)}{0.01^2} $$
$$ n = \frac{(2.8)^2 \cdot (0.0475)}{0.0001} $$
$$ n = \frac{7.84 \cdot 0.0475}{0.0001} $$
$$ n \approx 372,000 $$

Thus, to detect a 1% increase in CTR from a baseline of 5% with 80% power at a 5% significance level, we would require approximately 372,000 users in our test.

The result indicates that to achieve statistically significant results for a 1% increase in click-through rates from 5% to 6%, CrowdStrike would need to test around 372,000 users. This large sample size emphasizes the challenge of detecting small effect sizes in A/B testing and highlights the importance of planning experiments with sufficient power to avoid Type II errors (failing to detect a true effect).

This problem connects to several advanced statistical concepts:
1. **Bayesian vs. Frequentist Inference**: The sample size calculation here is based on frequentist methods. In a Bayesian framework, we could update our beliefs about the CTR as new data comes in, which might influence how we think about sample sizes.
2. **Multiple Testing Correction**: If CrowdStrike plans to test multiple variations of their advertisement (e.g., different headlines), the Bonferroni correction or False Discovery Rate (FDR) methods would be necessary to adjust for the increased risk of Type I errors.
3. **Causal Inference Basics**: Understanding the causal relationship between the advertisement and user behavior is critical. We must ensure that any observed changes in CTR are due to the advertisement and not confounding variables.
4. **Resampling Techniques**: Methods like bootstrapping could be employed to assess the variability of our estimates, providing a more robust understanding of the CTR.

Calculate expected value for click-through rates

CrowdStrike

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

CrowdStrike Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example