Requirements

Functional Requirements:

Query:

Get(redirect 301/302 redirect): short url -> redirect to long url

manage:

• POST - Create a short URL for a given long URL.
• DELETE - delete a short url / long url
• PATCH disabled a short url

Non-Functional Requirements:

Let's assume we have (scalop)

• Security: Basic abuse protection -- ddos & bot attack. WAF/bot migration + rate limit for redirect scan.
• Consistence: We should guarantee that each short url redirect to the consistent same long url. We don't deduplicate the long urls, each create creates a new short url for the long url.
• Availability: high availability. When a user enters a valid short url we should always redirect the user to a right place. Or if it is invalid url, we should return 404/410
• Latency: Read: low latency p95 100s, p99 250ms. Write can be higher 1s. I assume read query the short url can be 1M qps. But write query should be a lot lower 100 qps or 10qps in a geo.
• Observability: (LETS) latency- errors - traffic - saturation(how exhausted our facilities(server gpu, memory. ect)) + cache hit rate; DB/KV latency
• peak: Query - Read can be 1 M qps, write 100qps. Storage: 10M rows per day * 500b per row = ~ 5GB.

API Design

Query:

Get(redirect 301/302 redirect): short url -> redirect to long url

manage:

• POST - Create a short URL for a given long URL.
• DELETE - delete a short url / long url
• PATCH disabled a short url

High-Level Design

1. Client (user browser, mobile any device)
2. CDN
   1. cache the redirect request -- the short url redirect to the long url
   2. WAF/bot DDOS attack.
   3. Block bad lists short url per geo/ bad request.
   4. Block list of malicious requests
3. Api gateway
   1. Auth - JWT/apikey
   2. rate limits.
   3. Forward the right request to the right server.
   4. request id + validation
4. Load balancer
   1. Distribute the request load to different servers.
   2. Health check. failover.Rolling deploys
5. Redirect Service
   1. Handle cdn miss cache get request.
   2. look up from cache, if cache doesnt have it, load from db. If db has, load into cache.
   3. If no short url record, return 404.
   4. if short url record disabled return 410
6. Redirect service group cache
   1. KV cache store short (key) url - long url
   2. TTL 60s
7. Manager server
   1. Handle Post/patch/delete requests
   2. Generate Id and update records
   3. write the new short url / long url pair into db
8. DB
   1. KV, short url code (key):
   2. {
   3. long url :string ,
   4. create_at: timestamp,
   5. expired_date: timestamp,
   6. disabled: boolean
   7. }
9. Async pipeline:
   1. for analysis -- fetch the visit log from cnd
   2. update expired data to be 90days from today when a record is visited today.

Read Flow:

1. Client sends get request
2. CDN: it arrives at cdn first. And 99% visit should be stopped and redirect successfully at the cdn part. if the short url found, the return 301/302
3. redirect server: if cdn cache missed, cdn forward to redirect server, where it reads from cache see if the long url exist and then db. If both not exist, return 404/410.
4. This can handle 10M qps, thanks to the cdn will handle 99% requests.

Write Flow:

1. Client sends post/patch/delete requests
2. Api gateway forwards to manage service
3. Load balancer will handle the load
4. Manage service will handle create the short url code and generate the code id
5. Manager service will handle delete the record by update db
6. Manager service will handle disable the record by update db

The short url is combine: domain/{code}

In the manager server: we need to generate code from the long url path.

Detailed Component Design

Generate short url code:

worker listen to cdn log:

1. if a url has access log, update the data entry's expire_at in the db