Loading...
2KB/url * 10K/s = 20MB/s which is 20KB * 8 = 160Mbps download speed. The upload is trippling the download bandwidth which is 500Mbps. Considering the 3 times than normal burst traffic, a 10Gbps network is good enough for this service. All moden datacenter has the capacity for this requirement.
2KB/url * 10K/s =20MB/s, and for a day, 20MB * 86400 roughly equals to 2TB a day. The system keep the data for 90 days which means 2TB * 90 = 180 TB data all time. It is considered a quite large dataset. Triditional HDD disk is more eco friendly.
lets assume we kept 5 mins data in distributed KV system for example Redis to boost the performance. 2KB/url * 30K/s = 60MB/s, and 60MB * 300 = 18GB memory is needed for redis to cache the url.
Use RESTFUL API to describe the entire API
The format is like below
POST /urls
Authorization: Bearer your_token
The request body json format like below
{
"id" : 1234,
"url": "https://www.example.com/a/b/c?p1=v1&p2=v2"
}
The response body jason format like below
{
"id": 1234,
"shortUrl": "https://www.shorturl.com/urls/s1"
}
The http status code is 200 for successful, 403 for unauthorized access, and 500 for error occured in the service.
GET /urls/{shorturl}
There is no access authorization for this API.
The sample url is like "https://www.shorturl.com/urls/{shorturl}"
The return will be a json response like below
{
"url": "https://www.example.com/a/b/c?p1=v1&p2=v2"
}
DELETE /urls/{shorturl}
Authorization: bearer your_token
The http status code is 200 for successful, 403 for unauthorized access, and 500 for server error.
I pick postgres for the persistent storage. Considering 30K concurrent is a quite large read demand for SQL database. We'll use leader/follower mode, and use component patroni for automatically fail over. PGPOOL II can be used for read/write separation, and it can read status from patroni so that it can update master node accordingly. We need 3 database nodes in all, one for leader, and the rest works as
The schema is like below
create table if not exists short_url
(
id bigserial,
full_url varchar(2048) not null,
short_url_id bigint not null,
primary key (id, short_url_id)
) partition by hash(short_url_id);
create table if not exists short_url_p0 partition of short_url for values with (modulus 4, remainder 0);
create table if not exists short_url_p1 partition of short_url for values with (modulus 4, remainder 1);
create table if not exists short_url_p2 partition of short_url for values with (modulus 4, remainder 2);
create table if not exists short_url_p3 partition of short_url for values with (modulus 4, remainder 3);
create index if not exists short_url_p0_idx on short_url_p0 (short_url_id, full_url);
create index if not exists short_url_p1_idx on short_url_p1 (short_url_id, full_url);
create index if not exists short_url_p2_idx on short_url_p2 (short_url_id, full_url);
create index if not exists short_url_p3_idx on short_url_p3 (short_url_id, full_url);
Create a fully covered index to avoid go back table when searching the short url mapped full url. Considering the records of the database, I partition the table into 4.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Create URL request reaches the nginx server, and nginx works as a load balancer forwarding the creation request to the spring boot server. The full url is shortend there and persists into the postgresql database.
When get url request reach the nginx server, similarly it will be forwarded to spring boot server. The spring boot server will first try to load the full url from the redis cluster. When missing the full url in redis, the spring boot server will find it from the postgresql and then update the redis.
When delete url request reach the nginx server, the nginx will forward it to a spring boot server, first the spring boot server will delete the url from postgresql, and then remove the record from redis.
Spring boot server does the mapping full url to short url job. We roughly have 10K * 86400 records a day, and for 90 days, system need to keep 100B record. We use snow flake algorithm to generate global unique number and then use base64 to encode the unique id. After that, saving the record into database as the short url. After calculation, we need 14 characters for the generated short url.
Every short url keeping in redis should have an expiry time as 5 mins to avoid overflow the memory of redis server.
Postgresql is a well known and widely adopted database, and with this amount of data, more help from community and supporting company can give better solution.
When postgres master fails, the detection script can only realise that after certain amount of time. During this time of period, the whole services could degrade, and if the record is not in redis, the service have to wait for the pgpoolii upgrade its written rules to new pg master node.
Potentially, it's worth trying to use NOSQL databse for example scylladb, mongodb to replace the pg cluster because it needs less human intervention and things like partitioning.