Design a event-driven Feature Flag System

Question

Design a event-driven feature flag system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Feature Management:** Ability to create, update, and delete feature flags in real-time.
2. **User Targeting:** Enable feature flags to be toggled based on user attributes (e.g., user ID, role).
3. **Event-Driven Updates:** Feature flags should be updated based on events (e.g., user actions, system events).
4. **Audit Logging:** Maintain logs of who changed what and when for compliance.
5. **SDK Support:** Provide SDKs for various programming languages to access feature flags.

### Non-Functional Requirements
1. **Scalability:** Must handle millions of requests per second with low latency.
2. **Availability:** 99.99% uptime to ensure features are consistently delivered.
3. **Consistency:** Eventual consistency model is acceptable; stale flags should be retrievable.
4. **Performance:** Response time under 50ms for flag retrieval under normal load.

Assuming Oracle's feature flag system needs to handle 10 million users and each user checks an average of 5 feature flags per second.

### Calculation:
- **Total Requests per Second:** 10 million users * 5 requests/user = 50 million requests/second.
- **Scaling Factor:** To account for bursts and scaling, a factor of 2 is used, resulting in 100 million requests/second peak capacity.

### Storage Estimation:
If each feature flag configuration is approximately 1KB, to store 1 million feature flags:
- **Storage Size:** 1 million flags * 1KB = ~1GB of data.
- **Database Throughput:** Considering read-heavy operations, a system capable of handling 10,000 reads/second per database shard is needed.

### Component Diagram
- **API Gateway:** Handles all incoming requests for feature flag operations.
- **Event Bus (Apache Kafka):** Facilitates event-driven updates and communication between services.
- **Feature Flag Service:** Microservice that manages feature flags, querying the database and publishing changes to the event bus.
- **Database (Cassandra):** A NoSQL database for storing feature flags, chosen for its high availability and scalability.
- **Caching Layer (Redis):** For fast access to frequently requested feature flags, reducing database load.
- **Monitoring (Prometheus & Grafana):** For tracking system performance and alerting on anomalies.

### Technology Choices:
- **Kafka** for event streaming.
- **Cassandra** for distributed data storage.
- **Redis** for in-memory caching.

### Schema Design
- **FeatureFlag Table:**
  - `feature_id` (UUID, Primary Key)
  - `name` (String)
  - `description` (String)
  - `is_enabled` (Boolean)
  - `user_targeting` (JSON)
  - `created_at` (Timestamp)
  - `updated_at` (Timestamp)

### Access Patterns:
- **Retrieval:** Fetching flags by `feature_id` for a user with caching in Redis.
- **Update:** Write operations for changes in flags, which trigger events to update all relevant users.

### Architectural Decisions
1. **Eventual Consistency vs. Strong Consistency:** Opted for eventual consistency to allow for rapid feature flag updates and scalability, accepting that users might see stale flags for a brief period.
2. **Caching Strategy:** Redis caching introduces a layer of complexity but significantly enhances performance; trade-off between fresh data and speed.
3. **Microservices vs. Monolith:** Chose microservices for scalability and independent deployability, albeit with increased operational overhead.
4. **Database choice:** Cassandra offers high availability and partition tolerance, but can introduce latency compared to relational databases during complex queries.

Design a event-driven Feature Flag System

Oracle

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Calculation:

Submit Your Answer

Oracle Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search