Diagnose a drop in NPS

Question

The engagement dropped by 15% this week. Walk through your investigation process.

Codemia · Accepted Answer

To understand the drop in NPS by 15% this week at HRT, we need to analyze the underlying causes. First, we'll segment our user base to identify if this decline is widespread or concentrated among specific user personas. Key areas to investigate include recent product changes, user engagement patterns, and feedback trends. For this specific problem, we will focus on the last two weeks of user interactions, highlighting any new features or issues that could correlate with the timing of the NPS drop.

Our primary users are traders and financial analysts utilizing HRT's platform for high-frequency trading. Their needs revolve around data accuracy, speed of execution, and reliability. Given the recent drop in NPS, we need to engage with users who reported dissatisfaction to understand their pain points. This may include traders experiencing lag in transaction execution or analysts unable to access critical data. Understanding their workflows and immediate frustrations will allow us to prioritize fixes that align with their core needs, ensuring that we not only address the symptoms but also the root cause of dissatisfaction.

Based on initial investigations, a likely contributor to the drop in NPS is a recent update that affected real-time data feeds. My recommendation is to implement a rollback of this feature while simultaneously conducting a thorough review of the affected functionality. We can introduce a phased rollout strategy for the updated data feed with a controlled group of users to monitor performance and feedback before a full-scale release. This approach minimizes disruption and allows us to gather valuable user insights to refine the feature.

To gauge the effectiveness of the proposed solution, we will track the following metrics:
1. **NPS Score**: Monitor changes in the NPS post-rollout to evaluate user satisfaction.
2. **User Engagement Rate**: Measure the frequency of user sessions and interactions with the platform.
3. **Feature Usage**: Track how often the updated data feed is accessed by users to assess its adoption.
4. **Support Tickets**: Analyze the volume and nature of support inquiries related to data feed issues.
5. **Churn Rate**: Monitor the retention of users before and after the implementation to assess long-term impact.

The key trade-offs in this decision include:
- **Speed vs. Quality**: Rolling back the feature immediately may address user dissatisfaction quickly, but it could also delay the overall improvement of the product.
- **Short-term Satisfaction vs. Long-term Vision**: Focusing on fixing immediate NPS issues might divert resources from longer-term strategic initiatives, such as enhancing platform capabilities or building new features.
- **Resource Allocation**: The rollback and phased rollout will require significant engineering resources, potentially delaying other planned features or updates that could benefit users.

Diagnose a drop in NPS

HRT

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Problem Framing

User & Market Context