Requirements

Functional Requirements:

• Allow users to create new tags.
• Allow users to rename or edit existing tags.
• Allow users to delete tags they no longer need.
• Allow users to attach one or more tags to any digital item.
• Support searching for items using one or more tags, and return matching items.
• Support suggesting relevant tags when a user is tagging an item, based on item content analysis, existing tags, or user behavior patterns.
• Normalize tags to ensure consistency, lowercasing, trimming whitespace and mapping synonyms.

Non-Functional Requirements:

• Needs to have low latency (creating/renaming/deleting tags should be less than 10 ms, while searching and querying for items should be less than 10-100ms by using caching and indexing)
• Needs be reliable (searching for items by tags should always be accurate)
• Needs to have high availability (Needs to have at least 99.9% uptime)
• Needs to be scalable (Should be able to handle increasing number of users by millions and items by millions without losing performance)

API Design

1. Create Tags
   1. path: /createTag
   2. request: { userId: string, tag: string }
   3. response: { tagId: string }
2. Update Tags
   1. path: /updateTag
   2. request: { userId: string, tagId: string, tag: string }
   3. response: { tagId: string, tag: string }
3. Delete Tag
   1. path: /deleteTag
   2. request: { userId: string, tagId: string }
   3. response: None
4. Attach Tag
   1. path: /attachTag
   2. request: { userId: string, itemId: string, tagIds: List[string] }
   3. response: { itemId: string, tagIds: List[string] }
5. Search Items by Tags
   1. path: /searchItemsByTags
   2. request: { userId: string, tagIds: List[string] }
   3. response: { itemIds: List[string] }
6. Suggest Tags
   1. path: /suggestTags
   2. request: { userId: string, itemId: string }
   3. response: { tagIds: List[string] }
7. Get Popular Tags
   1. path: /tags/popular
   2. response: { tagIds: List[string] }

High-Level Design

1. The Client will first connect to a CDN to retrieve any static content to improve performance.
2. The Client will also connect to a Rate Limiter to prevent abuse by limiting the amount of tags that they can create to 5 per second. If they attempt to create more than that, they will receive an error.
3. The Rate Limiter will then connect to a Load Balancer so that we can distribute the requests evenly between servers.
4. The Load Balancer will then connect to the Server which will send the requests to the Tag Handling Service.
5. The Tag Handling Service will handle all of the APIs that we have listed above.
6. When a Tag is created, we can send it over to the Tag Normalization Service to handle converting it lowercase, trimming whitespace, map it to a synonym, and other forms of validations before sending it to the cache/database.
7. The Tag Handling Service will connect to the Cache and the Database for when we are creating items or getting items. We will get items from the Cache first and if there is a cache miss, we will fetch from the Database.
8. The Tag Handling Service will also connect to ElasticSearch when the user wants to search items by tags or get suggested tags.
9. In order to get popular tags, we can have the Tag Handling Service send a payload to our SQS to increase the usage for a tag by 1. We will process the messages in batches by having a buffer in our Tag Handling Service that stores incoming requests. Once the messages reach a certain threshold (ex: 1000 requests) or a certain amount of time has passed (5 minutes), we will handle all of the batch requests. Any failed SQS messages will go to the DLQ for manual review and redrive. We can also implement a rate limiter to limit the number of index updates we do.

Detailed Component Design

1. Tag Handling Service
   1. Create Tag will just take in a tag name and return the corresponding tag id. After creation, it will be sent to the Tag Normalization Service to fix the tag name. This information will also be stored in the cache for quick retrieval and then in the database.
   2. Update Tag will take in a tag id and a tag name for us to update the existing tag with the same tag id. This information will also be stored in the cache and the database.
   3. Delete Tag will take in a tag id and will remove the corresponding tag from the cache and the database.
   4. Attach Tag will take in an item id and a list of tag ids to attach to the item. This information will be stored in the cache and the database. Afterwards, we will send an SQS message to increase the tag usage by 1 so we know how popular it is.
   5. Search Items by Tag will take in a list of tag ids and will check the cache first and then query the database to find related items that have those tags if there is a cache miss.
   6. Suggest Tags will take in an item id and will use ElasticSearch to find tags that may be relevant to the item. This will also function as an autocomplete by checking what the user is typing and suggesting relevant tags. We can use things such as ElasticSearch's Full-Text Search to index item content and associated tags and perform search queries that match the content of that item with relevant tags.
   7. Get Popular Tags will check the cache/database to see which items have the highest number of usage.
2. Tag Normalization Service
   1. This service will take in a string and will clear out whitespace, convert it to lowercase, and will check a predefined mapping to see if there are tags that use the same synonyms already. We can also check if a tag with that name already exists by first checking the database. Furthermore, we can implement idempotency by adding an idempotency key in the request and ensuring that we don't update it in the database if that same idempotency key already exists in that item.

Tech Choice/Trade Offs

1. DynamoDB, Redis, ElasticSearch
2. We will go with a non relational database instead of a relational database due to the high number of read and write requests. We also want to scale horizontally when we get a larger number of tag creation/popular tags. Users may see stale data if they query immediately after a write operation.
3. We have went with an asynchronous approach for handling tag usage. This is because having large amounts of write requests directly to the database may degrade the performance so we decided to batch the request asynchronously. However, this comes at a cost of the popular tags not being immediately updated in real time.