Diagnose a drop in NPS

Question

The activation rate dropped by 15% this week. Walk through your investigation process.

Codemia · Accepted Answer

To understand the 15% drop in activation rate, we need to first define what 'activation' means within the context of Snowflake. Activation could refer to users successfully loading their first dataset, running their first query, or integrating Snowflake with other tools. Given this, we should scope the investigation to focus on user onboarding processes and identify which step in the activation journey has the most significant decline. Additionally, collecting data on the specific cohort of users who experienced this drop, including their previous engagement levels and types of datasets they are working with, will be crucial.

The primary users of Snowflake are data engineers, analysts, and business intelligence professionals, often within medium to large enterprises. These users prioritize quick and reliable data access, seamless integration with existing tools, and efficient data processing capabilities. In the context of the NPS drop, it’s essential to understand if the onboarding experience is not meeting their expectations or if there are technical barriers preventing them from realizing value quickly. Conducting user interviews and surveys focused on the activation phase will help us gather qualitative insights into their experiences.

Based on the investigation, if we find that users are struggling with the onboarding process, we should consider implementing a guided onboarding feature that provides step-by-step instructions, tooltips, and sample datasets to help users navigate the platform more effectively. This could include a dedicated onboarding checklist that users can follow to ensure they complete critical actions that lead to activation. Additionally, creating a dedicated support channel during the first two weeks of use could help address immediate questions and concerns. This solution directly targets the pain points discovered during user research and aims to improve the activation rate.

To measure the success of the onboarding feature and its impact on activation rates, we should track the following metrics: 1. **Activation Rate**: The percentage of users completing the onboarding checklist. 2. **Time to First Query**: The average time it takes for a user to run their first query after signup. 3. **User Engagement Score**: A composite measure based on log-ins and feature usage during the first week. 4. **Support Ticket Volume**: The number of support tickets related to onboarding issues. 5. **NPS Score**: Collect feedback specifically post-onboarding to see if users feel more satisfied.

Implementing a guided onboarding process may require trade-offs, such as additional engineering resources and potential delays in other feature developments. There's also a risk that overly prescriptive onboarding could frustrate experienced users who prefer flexibility. Balancing the need for a tailored experience for newcomers while not alienating existing users will be key. Additionally, we must consider the cost of additional support resources during the onboarding phase against the potential long-term benefits of improved activation rates and customer satisfaction.

Diagnose a drop in NPS

Snowflake

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Framing

User & Market Context