Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Endpoint redirect-url:
Endpoint shorten-url:
See High level diagram.
The app follow the clean architecture principles (port/adapter, domain/apllication/infra) and CQRS.
The client reach the Browser, and then Browser reach the low Balancer.
The low Balancer check the availability of the app and scall it horizontaly (create new instances if needed) and assume that multiple instances behind the LB, DB primary/replica with failover.
To enhance the perf and clarity the App is split into a Write component that expose an interface (endpoint, shortenController) to allow the user to create a shorten url of a long one. A handler check it (format / reachable, not blacklisted) return an error if it's not valid or generate the Short URL.
The Short URL is push in Cache system and written in the DB through the UrlWriteRepository.
The read side is adressed with the SetLongerUrlHandler that is called by the RedirectController endpoint. It gets the url trough the UrlCacheReadRepository from the cache and use the BdUrlRepository if it miss it, if no url are retrieved an error is returned.
Metrics are exported to Prometheus and shown in Grafana.
Backups every 4 h: scheduled snapshots to object storage.
trade off:
If cache goes down, the system is slower but still continue to work, a systemd restart it. Base64 is very fast and not ressource hungry.
see ER diagram.
see Class digram.
Here's a short version you can paste:
Detailed Component Design
1. Write side (Command)
POST /api/urls): receives {longUrl}, builds a ShortenUrlCommand and returns 201 {shortUrl}, 400 for a bad format, or 422 if the URL is blacklisted or unreachable.LongUrl.isValidFormat(): a regex check on scheme and host.BlacklistChecker: looks the domain up in the BLACKLIST table.UrlChecker: sends a HEAD request with a ~150 ms timeout to stay within the 250 ms budget, and expects a 2xx.CodeGenerator (Base62, 7 characters, about 3.5 trillion combinations): retries if exists(code) returns true.UrlWriteRepository.save(): writes to the Postgres primary.2. Read side (Query)
GET /{code}): returns a 302 to the long URL, or 404. It uses 302 rather than 301 so every click reaches the server and can be counted.UrlReadRepository:code, the value is long_url, with LRU eviction. A hit takes about 1 ms.3. Analytics (async)
ClickEvent through ClickEventPublisher to a queue. It never waits, so the redirect stays ≤ 10 ms.CLICK table.4. Data
URL(short_code PK, long_url, created_at), CLICK(id, short_code FK, clicked_at, ip, user_agent, referrer), BLACKLIST(domain PK, added_at).A few things to fix in your HLD text so it matches the diagrams:
+ and /, which aren't URL-safe.SetLongerUrlHandler, UrlCacheReadRepository and BdUrlRepository should be GetLongUrlHandler, RedisUrlCache and PostgresUrlRepository.5. Cache policy (Redis)
GETEX), so popular URLs stay cached and unused ones expire. A ±10% random jitter on the TTL keeps many keys from expiring at the same moment.maxmemory-policy allkeys-lfu. When memory is full, Redis evicts the least frequently used keys. That fits a URL shortener better than LRU, because a small set of URLs gets most of the clicks.DEL code).SET NX), while the others wait briefly and then read the cache.6. Burst handling (rate limiting)
POST /api/urls: 1 token every 6 s (10/min), bucket size 20.GET /{code}: 100 tokens/s, bucket size 200.429 Too Many Requests with Retry-After set to the seconds until the next token. Clients retry with exponential backoff plus jitter (1 s, 2 s, 4 s…, capped at 30 s).