Requirements

Functional Requirements:

• Create a short URL for a given long URL.
• Return the long URL associated with a given short URL.

Non-Functional Requirements:

• The service has to scale on demand
• the data base connections has to be preserved
• 100 ms of Latency
• It has to follow the OWASP Top 10 items
• The Id will have 7 digits of AlphaNumeric using Snowflake pattern and converted to Base62
• 100 millions of urls generated per day
• High availability
• Low redirect latency
• Horizontal scalability

API Design

For this api will be need two paths in the same domain,

POST shortener/url/

Response Status Code: 201

GET shortener/url/{Id}

Response Status Code: 302

High-Level Design

The Client will fill up the form with the url that has to be shortened and will submit It.

Before the api there is a layer of ensurance and DNS management.

On the Server-side there is a gateway will evalute de token sent by front end calling the OAuth server.

The url shortener service is an Fargate published in a LoadBalance.

The IdManager service is an Fargate published in a LoadBalance will store da Ids in a DynamoDb Table.

The main DataBase will be for the URLs dictionary and other Data will be a DybamoDb.

For Read Access the most accessed url is stored in a redis cache.

CDN

As this will became a worldwide domain, to keep a low latency for everyone and avoid to turn the service a bottleneck to the flow, It will have a CDN that will cache the response for the edges of each region through the world.

Gateway

The gateway will have a cache too and will help when rate increase, because It can be configured to return the cached responses and will help on redirecting responses.

Auto scale

When the services reaches 50% of CPU capacity the auto scaling service will add more machines horizontally.

WAF and SSH area insurance layer to protect data and the services against attacks and malicious activities.

Detailed Component Design

Api Gateway

The AWS Api Gateway has a RateLimit configuration where a traffic limitation can be set. API Gateway throttling would be configured using rate and burst limits to absorb traffic spikes and protect downstream services such as Redis and DynamoDB from overload. When limits are exceeded, clients would receive HTTP 429 responses and should retry using exponential backoff with jitter.

Service URL Shortener

The services are using .NET and and following the Clean Architecture, creating a possible expansion of It. The integration with other services will be made using Circuit and retry policy.

Every data inserted into the database will be available on cache for 5 minutes, after that time only the most accessed url will be there for a long term following some internal rules.

Before the service GET a data from Dynamo, It will query the cache and if needed the service will create a new cache record on redis.

When the GET route be requested It will return an status code known as 302 because it give to the requesting client browser temporarily the information of redirect, the 301 status code gives the information of permanent redirection, and this action increase the complexity of understanding the analytics data.

CDN

CDN is being use as cache for backend, but it also protects backend from sudden traffic spikes and abusive clients.

Cache Validation / invalidation

The CDN will have a cache with 1 minute of TTL, and de redis will have 5 minutes.

When the CDN cache expire, the redis will hold the requests against the database.

It will also provide the possibility of getting updates when the DataBase ShortCode be updated.

Background refresh - Cache TLL Handler Service

Here is a how to deal to prevent hotspots, even though the design already has load balance and cache services.

The service can have a algorithm to refresh the cache that handle the TTL deadline. But It can create a race condition to refresh that data.

The way to solve this is creating a new service worker to update the TTL proactively.

1- This service can get a dictionary of cached data created on redis by the shortener url and query database and confirm on database if it is necessary to have cache of this data, comparing the due date and last update count.

1.1 The Dictionary will be used to manage cache bloat and maintain performance, because not all the Id can be kept on Cache, so following pareto principle 80/20, maybe a dictionary with 20% of the most accessed data can solve the heavy traffic and even with sudden traffic, the system will be protected.

2 - The service will access Kinesis data and confirm and update data of accessed Shorten URLs. If there is data on Kinesis for a specific Id, this data will be updated on database with the count increment, and either have the Id added on the redis dictionary or removed if the Id is not present.

3 - This cache can runs every 10 seconds and update all TLLs found in the dictionary which are under 30 seconds to expire, increasing them to 300 seconds further.

4 - The invalidation will happens when the cache achieves its configured TTLL, or if the services identifies changes on update_at or status = pending (if so, it should be changes to updated).

This service will be a Lambda Function written in c# .NET.

Id Generator

Base62 is built with 0-9, a-z, A-Z. It is 62 characters ^ 2.

100 millions of Shortened URL mean that I will need 100 million of Ids a day.

100.000.000 * 365 = 36,500 billion a year, but a 62^7(characters) will provide 3,5 trillion, It is much more than needed for a whole year and provide Ids for over 10 years.

I Will write the Id in DataBase only after handle a conversion behavior to avoid discoverable url Ids.

So The architecture will use Redis Cluster incremental Id to generate Ids that is after 14 million (that is harder to be discovered) . The Redis cluster will generate unique Id even in race condition, because It creates a queue of incremental numbers. This will be used as a system to generate Id to all the services running in the LoadBalance.

DataBase Service Shortener URL Dynamo DB

Dynamo provides high availability, low latency and good performance on high throughput and massive reads and writes because It can scale horizontally.

100 millions / 24 / 60 / 60 = 1160 Write Per Second

if a url has 100 character or 100 bytes, it will has: 9.31 GB of data a day (100 bytes × 100.000.000)

I will need at least 3.6 Terabytes of disk to live through a year.

Usually for each 1 write there are 10 reads, so 1160 * 10 = 11.600 RPS