System requirements

Functional:

• Support incrementing and decrementing counters by a specific value across a distributed environment
• Support getting counter value
• Support resetting the counter
• Out of scope
  • Authentication

Non-Functional:

• Highly available (99.99%)
• Eventual consistency with low latency
• Latency for all APIs < 200ms
• Fault tolerant and highly scalable

Capacity estimation

Assumptions:

• Assume we support 100M counters, where each counter is a int64 (8 bytes)
• Read/write ratio: assume it is 10:1
• Assume each byte and its metadata is ~100 bytes
• Assume on average write per sec is 0.1/sec on average, with 100X at peak times for 100k of those counters

Storage

• 100M * 100 = 10GB, which is small but we need to shard it

Network

• Write QPS: 10M writes/sec
• Read QPS: 100M reads/sec

API design

// Create and delete counters

POST /counters

{

name,

}

DELETE /counters/

// Increment / decrement / reset counter

POST /counters/

{

action, // increment | decrement | reset

value,

}

// Read counter

GET /counters/ -> value

Database design

For individual counters, our max read concurrency is 0.1 * 10 * 100 = 100 /sec. This is can be handled by redis, provided we shard the cache properly.

We choose redis. Say each instance can handle 100k/sec. On average each counter has 1 write/sec. so we fit 100k counters into each instance, we results in 1000 shards. In reality we probably don't need that number of shards. We should monitor memory and CPU usage to determine the actual number of shards needed.

High-level design

APIGateway

• Handles rate limiting, authentication, SSL termination, etc.

Update service and queue

• Implements update API. Validates input. Queues the counter update request.

CounterService

• Write the update into Redis

ReadService

• Handles read request, reads from redis

Request flows

Update flow

• Client sends an update request to UpdateService via APIGateway
• UpdateService pushes the request into queue, and responds HTTP 200
• CounterService picks up the request from queue, updates redis, and acks the message

Read flow

• Clients sends a read request to ReadService via APIGateway
• ReadService reads from redis

Detailed component design

Redis

• Shard by counter name.
• The sharded redis should form a consistent hash ring, in order to evenly balance the load, and minimize rebalancing when servers are added/deleted
• Redis should enable high availability and replication.

Update queue

• Assume we use Kafka
• We have ~10M writes/sec. Assume each shard can handle 1M /sec. We need ~20 shards (brokers) to give it some extra buffer.
• Data size is small and should not be a problem.

Trade offs/Tech choices

We use redis as opposed to a DB for its performance, and because eventual consistency is acceptable.

Failure scenarios/bottlenecks

UpdateService, CounterService and ReadService are stateless and can be horizontally scaled.

Redis

• For extra reliability, we should enable high availability mode with replication

UpdateQueue

• We enable replication.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?