Loading...
around 100,000 unique short URLs created per day, resulting in about 900,000 read requests.
Each shortened url will be hashed in order to evenly distribute keys across DB shards. I'm thinking we can use sha-256 for this. Given that, each key will be 32 bytes, and let's say each original url stored along with it will be on average 30 bytes, so each row comes out to about 62 bytes. At 100k writes per day, this equates to needing a capacity of at least 62*100k bytes=6.2MB per day
GET /<url>: this will hash the url input, and perform a key lookup on the hash to find the original url associated with it. The API will return a 404 if the original url is not found or a 301 redirect if it is.
POST /api/v1/shorten: this will accept a request body containing the original url and, optionally, a custom alias. If the alias is not provided, the shortened url will be autogenerated.
As we are performing 9x the number of reads as writes, we want a database that optimizes for reads. This makes either a hash index or B-tree favorable to a db that utilizes LSM-tree+SSTables. Since we want durability though, and hash indexes are typically in-memory, that leaves B-trees as the natural choice. Given that, Postgres instances shared by a hash function, with each instance utilizing read replicas, would be a good choice here.
There are two main scenarios:
1) User requests a new link to be generated: in this case, the shortened url will be either randomly generated, or in the case of a custom alias, validated then provided as is. The proposed url will then be hashed and stored along with the original url. In the event of a hash collision, the app will try probing for an unused hash until one is accepted.
2) Link is viewed: for analytics, this will write a click event to an unstructured data store such as s3 or hdfs. Apache Spark streaming jobs will process clicks in micro-batches and write them to the analytics db (this could be the main db, or a separate analytics db depending on how much of a load this puts on the existing system). In parallel to analytics processing, the link read request will first attempt to read from the cache; in the event of a cache miss, the db will be read and the output will both be returned to the user and written to the cache.
See high-level design above
See high-level design above
see above.
Spark should offload the db from analytics processing, but could still be an issue if sharing the primary db. As mentioned above, a separate db just for analytics could be used. This would likely be more write heavy than read heavy in this case; in that scenario, Cassandra could be a good fit.
Should add a profanity filter for URLs.