Loading...
Capacity Estimation:
The format in which the tags are to be stored is UTF-8. The character size is as follows:
English letters (a, b, c) | 1 byte |
Accented (é, ñ) | 2 bytes |
Some symbols (₹, €) | 3 bytes |
Emojis (😀, 🔥) | 4 bytes |
So if we consider an average tag size of 10 characters with at least one emoji and one symbol that would be 8bytes + 3 bytes + 4 bytes equal to 15bytes on average.
But we have to consider for maximum worst case so we will go with tag with 100 character with all of them being emojis so 4bytes * 100 = 400 bytes just for the tag if we have to consider the metadata for the tag we can add another 250 bytes making it around 650 bytes per entry.
Let us say that we are getting 1 million new tags everyday that would be 1 million // 86400 = ~12 writes/sec. that would be 650 bytes * 86400 * 12 writes / sec = ~642 MB/day storage would be required.
642 MB * 30 days= 19,260 MB = ~19 GB/month storage would be required
19 GB * 12 = 228 GB / year storage would be required.
POST /api/v1/create
Request Body:
Text: text to create a tag of
Response (201 Created):
Tag: The tag user has created.
tagID: Id of the tag created
CreatedAt: ISO 8601 timestamp for when the tag was created
Response (400 Bad Request) If a request fails
Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.
PUT /api/v1/update
Request Body:
Text: text to create a tag of
Response (200 OK):
Tag: The tag user has updated.
TagID: Id of the tag updated
UpdatedAt: ISO 8601 timestamp for when the tag was updated
Response (400 Bad Request) If a request fails
Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.
DELETE /api/v1/delete
Request Body:
TagID: Id of the tag updated
Response (204 No Content) when deleted successfully.
POST /api/v1/attach
Request Body:
TagID: Id to be attached
ItemID: Id of the item the tag be attached to
Response (200 OK):
TagID: Id of the tag attached
ItemID: Id of the item to which tag was attached.
AttachedAt: ISO 8601 timestamp for when the tag was attached.
GET /api/v1/items/{tagId}
Request Body:
TagID: ID of the tag
Pages: No of pages to get.
Response (200 Found)
Items: List of items that have the tag provided.
Response(404 NOT FOUND) If the tag is not present.
Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.
GET /api/v1/popular
Response (200 Found)
Tags: List of popular tags
Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.
GET /api/v1/suggestions
Response (200 Found)
Tags: List of suggested tags for user
Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.
The data model for a tag looks simple but is kind of complex. A tag is just a text the user gives as an input. Now the tag created can be attached a single item, 2 items or say 100 items meaning that it is many to many relationship. Also it can be attached to no item as well. We can create a table for the details as follows:
Tag Details:
TagID: Must be unique. We can go with generating the tag of length 6 using [0-9 a-z A-Z] making it 62 ^ 6 which is equivalent to 56 billion. I think this should work for us. We can have a check to see if the generated id is already present to avoid collisions.
Tag: Text the user gives to create the tag.
CreatedAt: ISO 8601 timestamp for when the tag was created.
ItemID: The item the tag is attached to. Can be null.
Since a tag can be attached to multiple items and an item can have multiple tags it is a many to many relationship.
The actual digital item will be in a object store and a prefix or an id of the item will be used to point to it.
Keeping all this in mind I think the database of choice would be Postgres which can scale for our use case of 12 writes/sec and can easily handle complex queries with joins.
We need to create indexes on TagID, ItemID and TagID+Tag to make the queries efficient.
We can use CDN for hosting our frontend which will also cache the reads making read operations more efficient.
We can use API gateway for routing our api hits and it can also rate limit the traffic.
Load balancer can be used to balance out the traffic to the servers.
We can use redis for caching the hits. If we have a miss then the request goes to the postgres. we will be using cache aside method to fill our cache and LRU to invalidate our cache and a cleanup service will be there to do this in both redis and in cdn
We will have an auto scalable servers where our services would run.
For key generation as mentioned earlier we would be using the [0-9 a-z A-Z] of length 6 making around 56 billions possibilities for us. Before using a generated key we would check if its already present and only use it if its not there already.
A separate search service would be used to search for tags. Whenever a user tries to search for a item with a specific tag the search service would first check the redis cluster to see if its already present in the cache and if it is then its a hit making the response subsecond. If the item is not present in the cache then it will be fetched from the postgres and at the same time will be pushed into the cache.
API Gateway: Helps in routing the api hits as they come smoothly and can also rate limit the requests making the app resilient against DDoS attachs and brute force attacks.
Tag Search Service: This is a separate service created for searching the items based on tags. So when ever a user searches for an item if it is cached in CDN the request is handled by CDN itself. If not it will come to Tag Search Service which first looks into Redis cluster to see if any entry of the searched item is present if so it returns it otherwise it would get it from db and return it also while making an entry in the redis with its ttl.
Redis: We use it to cache the tags. We will be using cache aside strategy to cache the data and would Evict the data using LRU. Making the cache fresh and stale free.
The postgres DB is used to store our data. It supports many to many relationship which is required in this usecase. We will be handling normal tag search of one tag per item normally. For a multi tag search we can use self join or since the tables are already indexed we can make the queries faster using those.
Tag management service: This service handles the majority of the tag related activities like tag creation, updation, deletion etc.,
Also tag that are getting created needs to be normalized by making them lowercase remove starting and trailing spaces. This makes them even and consistent throughout.