Loading...
Assuming DAU is 1M, each user make ~10 operations per minutes,
10M per minutes
~ 14400 M per day
Capacity
Assume record counter history, each operation creates 8 bytes data
So it's 10^5 M bytes = 100 G per day
RPS
10M per minutes 150 K RPS for write
Read: Write 10:1 1.5 M RPS for read
api/counter/increment
POST
{
id: uuid
count: 1
}
api/counter/decrement
POST
{
id: uuid
count: 1
}
api/counter/{id} GET
For the counter itself
It could be just K-V pairs, since we need higher read access, low latency and eventual consitency, we could use Dynamo DB
Key as the id of the counter, Value as the count
For the history tracking, it depends on granular level, say we need counter history for each hour, we could just snapshot the counter DB every hour
Increment/Decrement: User post count to counter service
Retriever counter: User fetch counter with the counter id
For write:
Take increment for example, when user post count to counter serivce, if the service could adjust the counter in DB directly, it will be a bottleneck to write if there are tons of writes.
So, we could firstly add a service level data layer, where it stores all increment/decrement operations, then we do accumulation and flush the operations to cache at intervals. We could reduce the operations with the accumulation. For example, if counter X +5, +1, -2, we could combine the operation as +4. And every 5 seconds, we send the operation to some key value cache layer (like Redic/Memcached), then we could use write through strategy to write to the DB and update cache correspondingly.
Decrement flow is pretty similar to increment flow.
For read:
When user try to read a specific counter, we could use read through strategy, basically firstly try to read from the cache layer first, if not found, we would read from DB, and update the cache.
For the counter service, it's intuitive to scale by sharding based on the counter id, so operations for the same counter id would always go to the same service. The service data layer is a in memory map that store operations as key as counter id, value as the increment/decrement value (e.g 1, 2, -1), to optimize, we could adjust the value part based on the operations, basically calculate the sum of increment & decrement. And we run a cron job to flush all counters to the cache layer.
At cache layer, we would write through, basically increment counter at DB level, then get the updated counter and store it in cache.
For read operation, we would always try to read from cache layer first to reduce latency.
The solution provided ensure user could write and read fast. We add a data layer at service level to accumulate the operations, which help to reduce write operations to DB, but it also means there is a delay to get latest counter value.
For read operation, we try to return cached counter value first, this reduce read latency, but user may get stale counter data.
Counter service data layer may be down, and we lost user operations stored. One way to resolve is we write logs of operations, this way when server recovers or backup server takes over, it could retrieve the missed operations.
Also, the cache service could go down, we could add replica. The downside is we may increase write counter time. Could mitigate with async replicate, with the possibility result in inconsistent counter.
For the DB, since we use NoSQL DB (Dynamo), it scales & replicates by nature. But we could still run into hot shard issue if we shard by counter ids.
We could mitigate by adding more DB servers to shard servers further. Using consisten hashing, we could minize the data movement when adding servers.
Now we would have a 5 seconds delay to update the counter in DB. To optimize, we could reduce the flushing operations interval, but this means we would have more writes to DB. We could also try to introduce some client cache so the counter is calculated based on user operations for few seconds, and sync to server at intervals or when user refresh.