Loading...
DAU = 10M. Assuming 20% of the user post online, we have 2M posts per day.
Writes = 2M/day = 20 posts/sec
Peak write = 60 posts/sec (3x traffic)
Reads = 100x writes = 2000 reads/sec
Peak reads = 6000 reads/sec
Storage = Assuming each posts contains 5-10 tags, taking an average of 8 tags per post, and assuming each tag containing 10 characters -> we have roughly 80 bytes of data, along with metadata, we can assume each posts' tag to be roughly 150bytes.
Total tags storage in a day = 2 * 10^6 * 150 bytes / day = 300MB
Monthly storage = 9GB
Yearly storage = 110 GB
5 Years storage = 550GB
This is a read heavy operation. Storage is not going to be the bottleneck here.
POST /v1/tags
Header: Idempotency-Key: uuid
Body: {
user_id: uuid,
tags: string,
}
Response: 202 Accepted
{
item_id: uuid,
success: True
}
Get v1/tags/search-posts?q=tagA&q=tagB, limit=10
{
post_ids: [post_id1, post_id2, ....] # Info related to each tag is in the dictionary
cursor: id
success: True,
}
GET v1/tags/auto-complete?q=trav, limit=10
{
tags: [tag_id_1, tag_id_2, ...],
cursor: id
success: True
}
GET v1/tags/tag_name?q=tag_id
{
tag_name: string,
success: True
}
Client -> Load Balancer -> API Gateway
Write Path:
Write is divided into 2 parts.
Read Path:
We follow a Cache Aside method. Reads are served from Redis, and in case of cache miss, it is searched in ES and returned to DB.
We also have Monitoring Service subscribed to Kafka topics, which helps us in tracking the stats and monitoring the performance of our services.
We chose Elastic Search for its better search and auto complete capabilities over PostgresDB. We are using Postgres DB for storing all the related data and persistence.
We will use S3 to store the objects. The tags services will ask S3 for an URL where the clients can directly upload the object, so the extra load doesn't go through our servers.
An aggregator service will poll the Kafka topics and aggregate the count and we will batch update the count of tags in Redis using Redis INCR counter for popular tag counts.
Postgres:
Post Table - (post_id, user_id, raw_tags, post_url) -> with post_id as the primary key
Tags table - (tag_id, normalised_tag)
Post Tag Table - (post_id, tag_id) -> indexed at both post_id and tag_id
Elastic Search
We insert the post info into Elastic Search as JSON object
{
post_id: uuid,
user_id: uuid
tags: ["a", "b", "c",.....]
created_at: timestamp,
count: int
}
Redis:
We will have 2 types of keys here:
popular_tags_{keyword}: [comma separated top tags starting with the keyword]
popular_posts_{keyword}: [comma separated post ids starting with the keyword]
We can hash by tag_id in Redis
We will deep dive into the following things:
Normalized Tags and Storage -> We will convert normalize each tag and then store it instead of the raw tags in order to improve our search and autocomplete functionality.
We will store the post_id, tags_id in Postgres and index it (as it is the Source of Truth).
But for better search speeds, we will pass the JSON object to ES and mark the tags field as keyword, so that we match them exactly.
Elastic Search also provide autocomplete functionality using edge n-gram and we can sort by usage counts to give us the top N tags matching the prefix query.
To achieve High Write Throughput and low latency, we have separated the Write path into 2 parts. One writes the raw data to Postgres and handles object uploading in S3 bucket. The user gets the 202 Accepted confirmation. The optimization for search and autocomplete functionality is intentionally decoupled so that both can scale independently. This way any slowness in ES or Kafka, will not stop the critical Tags adding part.
Monitoring Service reads data from Kafka and writes to Influx DB. This will help us track any issues in our system.
Failure Handling.
If the system is throttled due to heavy pressure, we can pause the non-critical parts such as Monitoring Services, add more consumers to the Kafka (bounded by the size constraints) to handle backpressure.
Kafka also helps us in durability in case any consumers go down.
If Redis node is down, the read paths can still read through the Elastic Search.
We have 4 layers of Caching -> CDN -> Redis -> Elastic Search -> Postgres