Requirements

Functional Requirements:

• The service must efficiently match available feature flags with the need of a user.
• The service must report current inventory and health status of each.
• The service can request, reserve and release these servers based on the feature flags.
• The service can add and remove feature flags.
• The service must atomically reserve one or more matching servers for a user with lease terms.
• The service must reclaim servers when a lease or consumer heartbeat expires.

Non-Functional Requirements:

• Strict-consistency: Zero double-allocations under high concurrency.
• Low Latency: High-throughput matching across multi-attribute feature flags (O(1) to O(k) candidate lookup, sub-50ms allocation).
• Scale: hundreds of thousands of servers, up to 1000 sparse attributes/flags per server.
• Availability: consistency is mentioned above for allocation time but the system itself must have high availability.

Capacity Estimation

~300K-500K servers, avg. reservation ~2hrs.

~20-50 active flags/server (sparse over a ~1000 vocab)

Peak alloc/release ≈ 5–10K ops/sec, bursty (10K concurrent users)

Inventory/health reads = 10x allocation volume

~100K total users/tenants in the org

API Design

gRPC: allocate / release / heartbeat sit on the low-latency hot path and heartbeat renewal is naturally a stream - gRPC fits;

rpc RequestServers(RequestServersReq) returns (RequestServersResp); message RequestServersReq { repeated string flags = 1; int32 count = 2; } message RequestServersResp { repeated Server servers = 1; }

REST (admin/read): adding flags and browsing inventory is opes-facing dashboard traffic.

High-Level Design

Client -> allocate (gRPC) -> API Gateway -> Load Balancer -> Allocate APIs -> Index (roaring bitmaps)

Client -> Lease Mgr (TTL = reaper) -> Shared Store (etcd/FoundationDB - replicated Quorum) -> flag index (CAS reserve, return to pool)

Servers -> -> Health Cache (eventual) -> Index (excludes unhealthy)

Database Design

Partitioned KV w/ CAS (etcd / FoundationDB)In-memory inverted index (Redis + roaring bitmaps)Postgres (durable catalog)

Detailed Component Design

• Sharded pool + local CAS: partition servers so allocation is a compare-and-swap inside one shard's consensus group, never a global lock
• Cross-shard requests: fan out candidate lookups, rank, CAS-reserve on the winning shard, retry the next candidate on conflict
• Optimistic reserve-then-verify on popular flag combos instead of serializing every request through one lock (avoids thundering herd)
• Alloc API is stateless — N instances behind a gRPC-aware L7 load balancer, health-checked out of rotation, scaled horizontally on load