Loading...
Primary
To create short urls aliases for user provided urls
Secondary
Analytics, expiration of urls updates and deletes
Low latency for redirects
High availability for redirects
200 write rps
20000 redirect rps
100 bytes of long url
+ 8 bytes of short url + buffer for other fields ~= 256 bytes per url_mapping
5x10^6 bytes per second
3600 * 24 * 365 = 31 ~10^6 seconds in a year
Which translates as redirect operations need multiple servers but the write operations could be served with a simple server
shortenUrl
input: user_id, long_url
output: short_url
redirect
input: short_url
output: redirects to long_url
Url_Mappings
short_url, primary key (8bytes)
long_url (100 bytes)
user_id
... other fields
Users table (secondary)
CDN for redirects
Api Gateway for security and rate limiting
Load balancer for horizontal scaling
Separate Redirect and create url service for appropriate scaling
Datacenter In-memory Redis cache
DynamoDB NoSQL database
Write Request
create url service receives the request
generates a short url randomly checks for non duplication
and writes it to DynamoDB regenerating it if there's a collision
Read Request
When a request first arrives it goes through
CDN -> Service -> Redis cache -> Dynamo DB
The CDN and redis cache are read through caches which pull data on cache miss
The CDNs and the the datacenter cache use LFU eviction policy, keeping the most frequent data near users, and delegating less used data to the datacenter cache.
They also use read through caching.
The CDN is responsible for distributing the traffic for hot spots as most frequent content is present.
For the datacenter cache we will choose Redis, for its replication and availability features rather than simpler options like memcache.
Explicit updating and deletion of both the CDN and Redis cache will be issued from the server when this operations are requested
The services use lightweight servers with event driven frameworks, to scale horizontally efficiently and use an API gateway and load balancers to keep scale.
As their only function is very specific and the load is very different scaling of each sevice is decoupled. The event driven server optimizes usage for I/O operations.
DynamoDB is a managed database with great scalability fast writes, and reads which handles partitioning and rebalancing. It partitions based on the primary key which are the short url randomly generated hashes
Our database choice of Dynamo DB exchanges cost due to its managed nature for scale. We also sacrifice the relational query capabilities which could be useful for additional features.
The choice of using a CDN and Redis caching introduces extra complexity and cost, as memory is more costly than storage, while improving read performance, scalability and reducing read latency.
If volume follows even distributions caching the most frequently used could generate many cache misses, defeating the purpose of the cache and incurring in unforeseen costs in DynamoDB.
In the case of cache failures DynamoDB would autoscale and increase latency, this is expensive.
In case of service failures, horizontal scaling provides reliability
No SPOF is set
Analytics might be harder to implement due to the reliance on the CDN for redirects, in which case owning the CDN, or extracting analytics from their service with scheduled jobs, or API services would be required.
Dynamo DB is very limited in query
capabilities so no complex features could be easily supported by this system.