Calculate expected value for click-through rates

Question

Given the following scenario about revenue per session, calculate the the p-value.

Codemia · Accepted Answer

To calculate the expected value for click-through rates (CTR) and subsequently the p-value, we first need to establish our hypotheses and relevant parameters. Let:
- $ H_0 $: The null hypothesis, which posits that the mean click-through rate (CTR) for a new advertising strategy is equal to the current CTR.
- $ H_1 $: The alternative hypothesis, which asserts that the mean CTR for the new strategy is different from the current CTR.

Define the current CTR as $ \mu_0 $ and the new CTR from the experiment as $ \mu_1 $. We will collect data from a sample of user sessions and calculate the sample mean and standard deviation of the CTR, denoted as $ ar{x} $ and $ s $, respectively, from a sample size $ n $.

1. **Collect Data**: Gather data from the user sessions for both the current strategy and the new strategy.
2. **Calculate Sample Statistics**: Compute the sample mean $ ar{x} $ and the standard deviation $ s $ for the CTR.
3. **Conduct a Hypothesis Test**: Use a t-test or z-test to determine the p-value.
   - If the population variance is known, use a z-test:
     - $ z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}} $
   - If the population variance is unknown, use a t-test:
     - $ t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} $
4. **Calculate the p-value**: Determine the p-value associated with the calculated test statistic.

Suppose from our experiment we found:
- Current CTR ($ \mu_0 $): 0.05 (5%)
- Sample mean CTR from new strategy ($ ar{x} $): 0.07 (7%)
- Standard deviation ($ s $): 0.01
- Sample size ($ n $): 100

1. Calculate the t-statistic:
   - $ t = \frac{0.07 - 0.05}{0.01 / \sqrt{100}} = \frac{0.02}{0.001} = 20 $
2. Look up $ t $ in the t-distribution table with $ n-1 = 99 $ degrees of freedom. The two-tailed p-value for $ t = 20 $ is approximately 0.0001.

Thus, the p-value is 0.0001.

The p-value of 0.0001 indicates a very strong evidence against the null hypothesis $ H_0 $. This means that there is a statistically significant difference in click-through rates between the current strategy and the new strategy. Given Adobe's emphasis on data-driven decision-making, this low p-value suggests that the new advertising strategy is likely to perform better, warranting further investigation into implementing it on a larger scale. However, while statistically significant, it is crucial to also consider practical significance — whether the increase in CTR translates to meaningful revenue growth.

In an interview setting, the interviewer might ask about potential confounders that could bias the results, such as seasonal effects or user demographics. One way to mitigate these confounders is to conduct a stratified analysis or use regression techniques to control for these variables. Furthermore, discussing multiple testing corrections, such as Bonferroni or False Discovery Rate (FDR), is important if multiple CTR comparisons are made in the same study, to avoid inflated Type I error rates. This connects to the broader topic of experiment design and the importance of pre-registration to avoid p-hacking.

Calculate expected value for click-through rates

Adobe

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Adobe Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example