Design a Sliding-Hour Top-100 User Event Pipeline

Question

Design a system to identify the top 100 users with the highest event counts over the past hour from a stream of user events.

Codemia · Accepted Answer

To design a system that identifies the top 100 users based on event counts over the past hour, we need to process a stream of events formatted as (userId, eventType, timestamp). The system should be able to handle high throughput and provide near real-time insights.

The architecture can be divided into several components:
1. **Event Ingestion**: Use a message broker like Kafka to handle incoming event streams.
2. **Stream Processing**: Utilize a framework like Apache Flink or Spark Streaming to process the events in real-time.
3. **Sliding Window**: Implement a sliding time window of 1 hour to aggregate event counts for each user.
4. **Storage**: Store the results in a fast-access database (e.g., Redis) for quick retrieval of the top 100 users.

In this scenario, streaming processing is preferred due to the need for real-time analytics. While batch processing could be used, it would introduce latency that is not acceptable for this use case. Streaming frameworks like Flink allow for low-latency processing and support complex event processing, making them suitable for this task.

When designing this system, we need to consider:
- **Scalability**: Ensure that the system can handle spikes in event volume without degradation in performance.
- **Fault Tolerance**: Implement checkpointing in Flink to recover from failures without losing data.
- **Latency**: Minimize the time taken from event ingestion to the retrieval of the top 100 users.

In summary, by leveraging a streaming architecture with a sliding time window, we can efficiently compute the top 100 users based on event counts in real-time. The choice of technology, such as Flink, plays a crucial role in achieving the desired performance and scalability.

Design a Sliding-Hour Top-100 User Event Pipeline

Oracle

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Requirements

High-Level Design

Submit Your Answer

Oracle Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search