Design Feature Flag Infrastructure for global users

Question

Design a fault-tolerant feature flag system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. Create, read, update, and delete feature flags.  
  2. Support multivariate flags and gradual rollouts (percentage-based).  
  3. Provide a RESTful API for clients to fetch feature flag statuses.  
  4. Allow users to enable/disable flags in real-time with low latency.  
  5. Audit logging of changes to feature flags for compliance.  
  
- **Non-Functional Requirements:**  
  1. **Scalability:** Support millions of requests per day, with peak loads during product launches.  
  2. **Availability:** 99.99% uptime with fault tolerance across multiple regions.  
  3. **Consistency:** Ensure eventual consistency for feature flags across global deployments.  
  4. **Performance:** API responses should be under 100ms in normal conditions.  
  5. **Security:** Implement OAuth 2.0 for API access and ensure data privacy.

- **User Base:** Assuming Workday has approximately 10 million global users.  
- **Traffic Estimation:**  
  1. Assume each user checks feature flags 5 times a day.  
  2. Total daily requests = 10 million users * 5 requests = 50 million requests/day.  
  3. Peak traffic estimated at 10% of daily requests during launches = 5 million requests/hour (~1,400 requests/second).  
  
- **Storage Needs:**  
  1. Assume each feature flag entry requires 1 KB of data (metadata, status, rules).  
  2. For 100,000 flags, total storage = 100,000 flags * 1 KB = 100 MB, with additional space for historical versions and audit logs, estimating total storage needs to be around 1 GB.

- **Architecture Components:**  
  1. **API Gateway:** Using AWS API Gateway to handle incoming requests and enforce rate limiting.  
  2. **Load Balancer:** Distributing traffic across multiple application servers deployed in a multi-region setup (US, EU, APAC).  
  3. **Application Servers:** Stateless microservices built with Node.js for handling business logic and interacting with databases.  
  4. **Database:** Use Amazon DynamoDB for storing feature flags with a global table setup for multi-region replication.  
  5. **Caching Layer:** Implement Redis as an in-memory cache to reduce database load for frequently accessed feature flags.  
  6. **Message Queue:** Use Apache Kafka for processing flag updates asynchronously and to trigger notifications to clients for real-time updates.  
  7. **Monitoring:** Use Prometheus and Grafana for observability and monitoring of system performance metrics.

- **Feature Flag Schema:**  
  ```json  
  {  
      "flag_id": "string",  
      "name": "string",  
      "description": "string",  
      "status": "boolean",  
      "rules": [{  
          "condition": "string",  
          "percentage": "integer"  
      }],  
      "created_at": "timestamp",  
      "updated_at": "timestamp",  
      "version": "integer"  
  }  
  ```  
- **Access Patterns:**  
  1. Read: Fetching the status of a flag based on `flag_id`.  
  2. Write: Creating or updating flags, which should be batched to reduce write load on DynamoDB.

- **Consistency vs. Availability (CAP Theorem):**  
  - Opting for eventual consistency in favor of high availability, allowing for temporary stale reads while ensuring system resilience.  
  
- **Caching vs. Direct DB Access:**  
  - Utilizing Redis for caching frequently accessed flags to improve performance, with trade-offs in consistency (stale data) addressed by cache invalidation strategies.  
  
- **Synchronous vs. Asynchronous Processing:**  
  - Processing flag updates asynchronously via Kafka to improve responsiveness but risking potential delays in real-time flag visibility for clients until the processing is completed.  
  
- **Deployment Strategy:**  
  - Rolling deployments with canary releases for new feature flags, allowing gradual exposure to users and minimizing risk during launches.

Design Feature Flag Infrastructure for global users

Workday

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Workday Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search