Design an A/B test for a search ranking change

Question

Design an experiment to test the impact of new pricing tiers. Include sample size calculation, metrics, and analysis plan.

Codemia · Accepted Answer

To design an A/B test for the impact of new pricing tiers, we need to define the following elements:

1. **Hypothesis**: The null hypothesis (H0) states that the new pricing tiers have no impact on the search ranking performance (e.g., click-through rate, conversion rate). The alternative hypothesis (H1) states that the new pricing tiers do impact performance.

2. **Metrics**: Choose metrics that reflect the desired outcomes of the pricing change, such as:
   - Conversion Rate (CR): The percentage of users that complete a desired action (e.g., purchase) after searching.
   - Average Revenue Per User (ARPU): Total revenue divided by the number of users.

3. **Sample Size Calculation**: Using the formula for sample size in A/B testing:
   $$ n = \frac{2 \times (Z_{\alpha/2} + Z_{\beta})^2 \times \sigma^2}{(p_1 - p_2)^2} $$
   where:
   - $Z_{\alpha/2}$ is the Z-score for the significance level (e.g., 1.96 for 95% confidence).
   - $Z_{\beta}$ is the Z-score for the power of the test (e.g., 0.84 for 80% power).
   - $\sigma$ is the estimated standard deviation.
   - $p_1$ and $p_2$ are the expected conversion rates in groups A and B.

1. **Define Parameters**: Let's assume:
   - Expected conversion rate for control group (p1) = 0.10
   - Expected conversion rate for treatment group (p2) = 0.12
   - Standard deviation (sigma) can be calculated using $\sigma = \sqrt{p(1-p)}$, where $p = (p1 + p2)/2$.

2. **Calculate sigma**:
   $$ p = \frac{0.10 + 0.12}{2} = 0.11 $$
   $$ \sigma = \sqrt{0.11 \times (1 - 0.11)} = \sqrt{0.11 \times 0.89} \approx 0.329 $$

3. **Calculate Sample Size**:
   Using the sample size formula:
   $$ n = \frac{2 \times (1.96 + 0.84)^2 \times (0.329)^2}{(0.12 - 0.10)^2} $$
   $$ n \approx \frac{2 \times (2.8)^2 \times 0.108}{0.0004} \approx \frac{16.032}{0.0004} \approx 40080 $$
   Thus, we need approximately 40,080 users in each group.

Let's say we conduct the A/B test with:
- 40,080 users in the control group and 40,080 users in the treatment group.

Assuming the results show:
- Control Group (A): 4,008 conversions (10% conversion rate)
- Treatment Group (B): 4,810 conversions (12% conversion rate)

We then calculate:
- **Conversion Rate for Group B**: $CR_B = \frac{4,810}{40,080} = 0.120$ or 12%
- **Statistical Significance Test**: Use a two-proportion Z-test to evaluate significance:
   $$ Z = \frac{CR_B - CR_A}{\sqrt{\sigma^2(\frac{1}{n_A} + \frac{1}{n_B})}} $$
   where $\sigma^2 = 0.11 \times 0.89$. After calculating, if $Z > 1.96$, we reject H0.

If the Z-test shows significance, we conclude that the new pricing tiers have a statistically significant positive impact on conversion rates. Practically, this means the new pricing strategy is likely effective, and Salesforce can consider implementing it across its user base. If not significant, it suggests the changes may not yield the expected benefits, warranting further analysis or adjustments.

This A/B testing framework connects to several related concepts:
1. **Bayesian vs. Frequentist Approaches**: Discussing how a Bayesian framework could offer advantages in terms of incorporating prior distributions about conversion rates.
2. **Multiple Testing Corrections**: If testing multiple pricing tiers, we would need to apply corrections like Bonferroni or False Discovery Rate (FDR) to maintain the overall significance level.
3. **Power Analysis**: Discussing how power analysis informs sample size calculations and impacts the likelihood of detecting true effects and avoiding Type II errors.

Design an A/B test for a search ranking change

Salesforce

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Formulation

Solution Approach

Submit Your Answer

Salesforce Data Scientist Interview Guide

Related Questions

Interpret mixed results across segments from an experiment

Explain instrumental variables with an example

Calculate conditional probability for coin flips

Explain difference-in-differences with an example

Explain p-value with an example